As part of a nationwide quality improvement program, the standardised mortality ratio (SMR) of your Intensive Care Unit was compared to other similar Intensive Care Units using a funnel plot.
You are ICU “A”
a) What does the graph show about your ICU “A”? (20% marks)
b) Explain how the SMR is calculated. (20% marks)
c) Give the causes of an increased SMR. (60% marks)
a)
The SMR of ICU A is above the upper 99% CI indicating the SMR is significantly higher than similar hospitals. Your ICU has significantly more deaths than expected compared to similar hospitals.
The overall SMR for the group is less than 1 and the SMR for ICU A is less than 1
b)
SMR = O/E O= observed number of deaths, E = expected number of deaths
E is derived from the average of the sample/ population.
Usually a risk adjustment model is used to calculate and account for severity of illness.
c)
Can be “apparent” or “real”.
Data quality
Incomplete or errors in data submission causing underestimated expected risk Widely different casemix of this ICU compared to others.
Statistical model (risk adjustment) may no longer well calibrated True increase in mortality which can be due to
i. Factors internal to ICU: very high occupancy, poor processes,, inadequate staffing,
ii. Factors external to ICU; problems in services that are high users of ICU e.g. surgery, system issues
Additional Examiners’ Comments:
Many candidates showed a significant knowledge gap relating to this commonly used quality indicator with insufficient details and structure in their answers.
Again, the college did not include their images here. The SMR funnel plot above has been ripped off from the ANZICS own database for 2013-2014, via the propaganda materials from The Alfred. The red dot in the bottom right of the funnel plot is that abovementioned centre of excellence, soaring ever higher into "good outlier" territory.
Generally, the college loves SMRs. Specifically, they like to discuss which it might be abnormally high, and what are its limitations as a measure of the quality of care.
a) ICU "A" is clearly having some sort of crisis. However, the overall SMR is still less than 1, which means that mortality has not exceeded the average mortality predicted by APACHE data. Which is good, because some might say APACHE data is crap for predicting mortality: these day mortality is probably lower for any given APACHE score than the outdated system might predict.
b)
The above definition is probably enough to satisfy the minimum requirements for the college.
In short,
SMR = observed number of deaths / expected number of deaths
In order to calculate this, one requires three main variables:
c)
Reasons for a spuriously elevated SMR
Reasons for a truly elevated SMR
Young, Paul, et al. "End points for phase II trials in intensive care: Recommendations from the Australian and New Zealand clinical trials group consensus panel meeting." Critical Care and Resuscitation 14.3 (2012): 2111.
Suter, P., et al. "Predicting outcome in ICU patients." Intensive Care Medicine20.5 (1994): 390-397.
Martinez, Elizabeth A., et al. "Identifying Meaningful Outcome Measures for the Intensive Care Unit." American Journal of Medical Quality (2013): 1062860613491823.
Tipping, Claire J., et al. "A systematic review of measurements of physical function in critically ill adults." Critical Care and Resuscitation 14.4 (2012): 302.
Gunning, Kevin, and Kathy Rowan. "Outcome data and scoring systems." Bmj319.7204 (1999): 241-244.
Woodman, Richard, et al. Measuring and reporting mortality in hospital patients. Australian Institute of Health and Welfare, 2009.
Vincent, J-L. "Is Mortality the Only Outcome Measure in ICU Patients?."Anaesthesia, Pain, Intensive Care and Emergency Medicine—APICE. Springer Milan, 1999. 113-117.
Rosenberg, Andrew L., et al. "Accepting critically ill transfer patients: adverse effect on a referral center's outcome and benchmark measures." Annals of internal medicine 138.11 (2003): 882-890.
Burack, Joshua H., et al. "Public reporting of surgical mortality: a survey of New York State cardiothoracic surgeons." The Annals of thoracic surgery 68.4 (1999): 1195-1200.
Hayes, J. A., et al. "Outcome measures for adult critical care: a systematic review." Health technology assessment (Winchester, England) 4.24 (1999): 1-111.
RUBENFELD, GORDON D., et al. "Outcomes research in critical care: results of the American Thoracic Society critical care assembly workshop on outcomes research." American journal of respiratory and critical care medicine 160.1 (1999): 358-367.
Turnbull, Alison E., et al. "Outcome Measurement in ICU Survivorship Research From 1970 to 2013: A Scoping Review of 425 Publications." Critical care medicine (2016).
Solomon, Patricia J., Jessica Kasza, and John L. Moran. "Identifying unusual performance in Australian and New Zealand intensive care units from 2000 to 2010." BMC medical research methodology 14.1 (2014): 1.
Liddell, F. D. "Simple exact analysis of the standardised mortality ratio." Journal of Epidemiology and Community Health 38.1 (1984): 85-88.
Ben-Tovim, David, et al. "Measuring and reporting mortality in hospital patients." Canberra: Australian Institute of Health and Welfare (2009).
McMichael, Anthony J. "Standardized Mortality Ratios and the'Healthy Worker Effect': Scratching Beneath the Surface." Journal of Occupational and Environmental Medicine 18.3 (1976): 165-168.
Wolfe, Robert A. "The standardized mortality ratio revisited: improvements, innovations, and limitations." American Journal of Kidney Diseases 24.2 (1994): 290-297.
Kramer, Andrew A., Thomas L. Higgins, and Jack E. Zimmerman. "Comparing observed and predicted mortality among ICUs using different prognostic systems: why do performance assessments differ?." Critical care medicine 43.2 (2015): 261-269.
Spiegelhalter, David J. "Funnel plots for comparing institutional performance." Statistics in medicine 24.8 (2005): 1185-1202.
Teres, Daniel. "The value and limits of severity adjusted mortality for ICU patients." Journal of critical care 19.4 (2004): 257-263.