Question 24

College Answer

a)

The SMR of ICU A is above the upper 99% CI indicating the SMR is significantly higher than similar hospitals. Your ICU has significantly more deaths than expected compared to similar hospitals.

The overall SMR for the group is less than 1 and the SMR for ICU A is less than 1

b)

SMR = O/E    O= observed number of deaths, E = expected number of deaths

E is derived from the average of the sample/ population.

Usually a risk adjustment model is used to calculate and account for severity of illness.

c)

Can be “apparent” or “real”.

Data quality

Incomplete or errors in data submission causing underestimated expected risk Widely different casemix of this ICU compared to others.

Statistical model (risk adjustment) may no longer well calibrated True increase in mortality which can be due to

i.    Factors internal to ICU: very high occupancy, poor processes,, inadequate staffing,

ii.    Factors external to ICU; problems in services that are high users of ICU e.g. surgery, system issues

Additional Examiners’ Comments:

Many candidates showed a significant knowledge gap relating to this commonly used quality indicator with insufficient details and structure in their answers.

Discussion

Again, the college did not include their images here. The SMR funnel plot above has been ripped off from the ANZICS own database for 2013-2014, via the propaganda materials from The Alfred. The red dot in the bottom right of the funnel plot is that abovementioned centre of excellence, soaring ever higher into "good outlier" territory.

Generally, the college loves SMRs. Specifically, they like to discuss which it might be abnormally high, and what are its limitations as a measure of the quality of care.

• Question 24 from the second paper of 2015 asked to explain causes of a raised SMR, as well as to interpret a funnel plot of SMR data and to explain how it is calculated.
• Question 22 from the first paper of 2007 asked why an SMR might be raised suddenly
• Question 22 from the second paper of 2007 was identical to Question 22 from the first paper of 2007, i.e. the same question used twice within the same year.
• Question 13 from the first paper of 2003 wanted you to discuss the limitations of the SMR
• Question 30 from the second paper of 2006 was identical to Question 13 from the first paper of 2003

a)  ICU "A" is clearly having some sort of crisis. However, the overall SMR is still less than 1, which means that mortality has not exceeded the average mortality predicted by APACHE data. Which is good, because some might say APACHE data is crap for predicting mortality: these day mortality is probably lower for any given APACHE score than the outdated system might predict.

b)

• SMR is the ratio of the observed  mortality vs. predicted mortality for a specified time period.
• One can use this to compare hospitals and ICUs
• One needs to first calculate the predicted hospital mortality using an illness severity scoring system.
• An SMR of 1 means the mortality is as expected.
• An SMR of < 1 is better than expected, and >1 is worse than expected.

The above definition is probably enough to satisfy the minimum requirements for the college. 
In short,

SMR = observed number of deaths / expected number of deaths

In order to calculate this, one requires three main variables:

• A time interval (you decide - three months, one year, etc.)
• A measurement of the observed number of deaths (ICU mortality numbers should be available widely)
• An estimate of the predicted mortality (this can be achieved using a scoring system).

c)

Reasons for a spuriously elevated SMR

• Unfair mortality measurements should not be a problem, as there is no serious situation in which mortality might get over-reported (i.e. some patients walking around after ICU discharge who are formally dead according to the official record). An under-reporting of mortality is much more likely, and might occur in resource-poor environments where public record-keeping is rudimentary. Such dodgy records might still list the person as alive, whereas they died shortly after ICU discharge and nobody bothered to file the paperwork.
• Unfair comparison group mortality / APACHE mortality estimate is usually the culprit.
  • Poor data entry, eg. constant overestimation of GCS or failure to tick the chronic illness boxes, which makes the patients appear healthier than they actually are.
  • Missing data is (in the author's experience) usually rich in APACHE points, eg. the missed urine output entry obscuring anuria, or the failure to document biochemistry results which conceals the potassium level of 8.0mmol/L.
  • "Lead time bias" - treatment received prior to ICU admission may result in artifically normalised acute physiology scores
  • "Healthy worker effect" - a change towards selective ICU admission practices may be favouring patients who score low on illness severity scales, eg. young elective surgical patients

Reasons for a truly elevated SMR

• Issues external to the ICU
  • A population with greater pre-ICU morbidity is suddenly available (eg. to borrow an example from LITFL, you have suddenly decided to become a destination for the state's ECMO retrieval service).
  • Pre-ICU care has changed its practice (for the worse)
  • Parameters which govern ICU admission have changed (eg. administrative pressure is being placed on the ICU to rapidly admit ED patients who have had little management or workup)
  • Discharge arrangements have changed (eg. a local palliative care ward had shut down, and you keep dying patients in the ICU because it would be insensitive to transfer them to the next nearby palliative care unit)
• Issues internal to the ICU (i.e. genuine under-performance)
  • This could be a long list...
  • A new staffing model is in place (inexperienced staff)
  • Understaffing has impaired patient care
  • Junior people are not following unfamiliar protocols, or the new protocols are crap
  • New equipment you bought is useless

Young, Paul, et al. "End points for phase II trials in intensive care: Recommendations from the Australian and New Zealand clinical trials group consensus panel meeting." Critical Care and Resuscitation 14.3 (2012): 2111.

Suter, P., et al. "Predicting outcome in ICU patients." Intensive Care Medicine20.5 (1994): 390-397.

Martinez, Elizabeth A., et al. "Identifying Meaningful Outcome Measures for the Intensive Care Unit." American Journal of Medical Quality (2013): 1062860613491823.

Tipping, Claire J., et al. "A systematic review of measurements of physical function in critically ill adults." Critical Care and Resuscitation 14.4 (2012): 302.

Gunning, Kevin, and Kathy Rowan. "Outcome data and scoring systems." Bmj319.7204 (1999): 241-244.

Woodman, Richard, et al. Measuring and reporting mortality in hospital patients. Australian Institute of Health and Welfare, 2009.

Vincent, J-L. "Is Mortality the Only Outcome Measure in ICU Patients?."Anaesthesia, Pain, Intensive Care and Emergency Medicine—APICE. Springer Milan, 1999. 113-117.

Rosenberg, Andrew L., et al. "Accepting critically ill transfer patients: adverse effect on a referral center's outcome and benchmark measures." Annals of internal medicine 138.11 (2003): 882-890.

Burack, Joshua H., et al. "Public reporting of surgical mortality: a survey of New York State cardiothoracic surgeons." The Annals of thoracic surgery 68.4 (1999): 1195-1200.

Hayes, J. A., et al. "Outcome measures for adult critical care: a systematic review." Health technology assessment (Winchester, England) 4.24 (1999): 1-111.

RUBENFELD, GORDON D., et al. "Outcomes research in critical care: results of the American Thoracic Society critical care assembly workshop on outcomes research." American journal of respiratory and critical care medicine 160.1 (1999): 358-367.

Turnbull, Alison E., et al. "Outcome Measurement in ICU Survivorship Research From 1970 to 2013: A Scoping Review of 425 Publications." Critical care medicine (2016).

Solomon, Patricia J., Jessica Kasza, and John L. Moran. "Identifying unusual performance in Australian and New Zealand intensive care units from 2000 to 2010." BMC medical research methodology 14.1 (2014): 1.

Liddell, F. D. "Simple exact analysis of the standardised mortality ratio." Journal of Epidemiology and Community Health 38.1 (1984): 85-88.

Ben-Tovim, David, et al. "Measuring and reporting mortality in hospital patients." Canberra: Australian Institute of Health and Welfare (2009).

McMichael, Anthony J. "Standardized Mortality Ratios and the'Healthy Worker Effect': Scratching Beneath the Surface." Journal of Occupational and Environmental Medicine 18.3 (1976): 165-168.

Wolfe, Robert A. "The standardized mortality ratio revisited: improvements, innovations, and limitations." American Journal of Kidney Diseases 24.2 (1994): 290-297.

Kramer, Andrew A., Thomas L. Higgins, and Jack E. Zimmerman. "Comparing observed and predicted mortality among ICUs using different prognostic systems: why do performance assessments differ?." Critical care medicine 43.2 (2015): 261-269.

Spiegelhalter, David J. "Funnel plots for comparing institutional performance." Statistics in medicine 24.8 (2005): 1185-1202.

Teres, Daniel. "The value and limits of severity adjusted mortality for ICU patients." Journal of critical care 19.4 (2004): 257-263.