The Standardised Mortality Ratio (SMR)

At a basic level, the SMR is a measure of how good your unit is at preventing acute illnesses from killing patients. This seems like an important metric, and indeed much is made of the SMR in the critical care community, at least locally. Units with well-developed media machinery use favourable SMR data to advertise their supremacy as centres of excellence; units which perform poorly use damning SMR data as a kick in the arse required to stimulate policy change.

The college loves SMR. Specifically, they like to discuss which it might be abnormally high, and what are its limitations as a measure of the quality of care.

• Question 11 from the first paper of 2015 asked how one might calculate a SMR, and what might have caused it to increase
• Question 24 from the second paper of 2015 asked to explain causes of a raised SMR, as well as to interpret a funnel plot of SMR data and to explain how it is calculated.
• Question 22 from the first paper of 2007 asked why an SMR might be raised suddenly
• Question 22 from the second paper of 2007 was identical to Question 22 from the first paper of 2007, i.e. the same question used twice within the same year.
• Question 13 from the first paper of 2003 wanted you to discuss the limitations of the SMR
• Question 30 from the second paper of 2006 was identical to Question 13 from the first paper of 2003

More information can be derived from the following resources:

• LITFL - an excellent overview
• NM-IBIS resource on SMR calculation
• Wolfe, 1994 - more detail on the calculation of the SMR
• Ben-Tovim et al (2009) - the AIHW publication on epidemiological data collection standards, which may be viewed as the definitive resource for this topic. Fortunately for the exam candidate crazed with stress, it has that magic combination of being both absurdly detailed, unbearably dry and extremely long (well over 100 pages long, not including the extensive appendices).

Statistical epidemiology literature appears at first glance to be written in machine code. Some attempt has been made to decompile these materials into a human-readable format.

Definition and calculation of the SMR

The SMR, in a nushell:

• SMR is the ratio of the observed  mortality vs. predicted mortality for a specified time period.
• One can use this to compare hospitals and ICUs
• One needs to first calculate the predicted hospital mortality using an illness severity scoring system.
• An SMR of 1 means the mortality is as expected.
• An SMR of < 1 is better than expected, and >1 is worse than expected.

The above definition is probably enough to satisfy the minimum requirements for the college. 
In short,

SMR = observed number of deaths / expected number of deaths

In order to calculate this, one requires three main variables:

• A time interval (you decide - three months, one year, etc.)
• A measurement of the observed number of deaths (ICU mortality numbers should be available widely)
• An estimate of the predicted mortality (this can be achieved using a scoring system).

How does one know that the SMR is statistically significant? Confidence intervals can be calculated if one is that interested. NM-IBIS has a handy explanation of how to calculate the 95%CI using standard error. This sort of thing is probably sufficiently esoteric to not appear in the exam.

The SMRs of many ICUs can then be plotted on a funnel-like graph to compare their performance.

This graph was from Question 24 from the second paper of 2015. ICU "A" is clearly having some sort of crisis. The dashed lines represent the boundaries of the 95% CI for a given workload, i.e. the acceptable range of expected mortality. The width of the funnel represent the precision of the SMR estimate. Lines funnel out into wild hyperbolae at very small case numbers because of the imprecision of mortality measurements in such situations. If the ICU only admits 10 patients a year and all of them happen to die in spite of having a low-ish APACHE score, this would give an enormously high SMR -  but it my represent random noise, and the CI would be impossibly wide. As the case load becomes larger, the precision of the SMR improves. The anatomy of the institutional performance funnel plot (and the related "caterpillar plot") is better discussed by Spiegelhalter (2005).

Properties of the SMR as a tool

Limitations of the SMR

• SMR depends on the choice of "standard" population to compare mortality with, or - in the case of ICU - on the population which was used to validate the chosen illness severity scoring system. For the APACHE II system, this was a population of 5815 American ICU patients admitted between 1979 and 1982 (at least in the original study, though - to be fair- the APACHE II system has been validated in other settings since). The choice of scoring system matters: for instance, Kramer et al (2015) have demonstrated that different systems may rate the same ICU as having wildly different SMRs. In some cases, the APACHE IVa and NFQ methodology produced results of both significantly less than 1.0 and over 1.75 using exactly the same data; in fact they only agreed on the general direction of the SMR in 45% of cases.
• If you are not using an illness severity scoring sytem, the populations used to calculate the predicted hospital mortality are potentially non-representative (i.e. the population may also contains a number of dying critically ill patients, or it may contain an unusually large proportion of people in robust health).
• In-ICU mortality may not be the best data to be collecting (why not in-hospital, 3 month, 1 year?)
• Acceptable deviations from the SMR are not defined
• Suffers from inaccuracies associated with data collection
• Morbidity data is not available (conceivably, an ICU could have an excellent SMR and still be discharging a population 100% of whom are alive in a persistent vegetative state).
• SMR may be influenced by ICU admission and discharge practices (eg. discharging patients who are palliated, or admitting patients who are inevitably going to die).

Advantages of the SMR

• Mortality is a pretty unambiguous parameter which has some meaning in terms of what one might realistically want to avoid in a hospital. It's not some sort of abstract notion.
• Redundant records (hospital, state, insurance company etc) make the data more available
• These records should be cheap to access
• SMR is simple to calculate (relatively)
• The use of the same illness severity scoring system over a whole health service protects to some extent  against the disagreement of scoring systems which is mentioned above.
• Allows the charting of a trend over time

Limitations of comparing ICUs with the SMR:

• The SMR assumes all pre-ICU care is identical; this is probably homogenised in big cities but has implications for hospitals suffering from isolation, eg. a patient en route to Royal Darwin ICU may need to travel for 1000km.
• Inter-ICU transfers influence the validity and are unaccounted for (imagine a good ICU being dragged down by accepting numerous transfers annually from a crappy ICU)
• Ignores differences in case mix (patient type, severity, rates of ventilation, variations in cultural considerations, etc...). To borrow an example from Teres et al (2004), "One would not compare SMR of a medical ICU with a preponderance of chronic obstructive pulmonary disease (COPD) patients to the SMR of a neurosurgical ICU".
• Sample sizes need to be large enough to obey the laws of logistic regression; i.e. ICU size influences the random noise.
• Data is assumed to be flawless and complete. Unfortunately, physiology scores generated from monitors or records may “over-score” or "undr-score" patients
• The is no formal allowance for regression to the mean: ICUs which are extreme outliers are not corrected for "extremeness" to reflect the fact that extreme bad (or good) luck lies behind their outlier value
• Mortality is not a surrogate for quality of care, which it is frequently mistaken for (consider how an  SMR might contribute to the ranking of an ICU which accepts moribund patients for controlled palliation as a matter of policy, as compared to an ICU which demands they be extubated unceremoniously in the ED).
• Because of cultural and administrative differences, the SMR is not "portable" internationally.

Why is my SMR so high? Does my ICU suck?

Perhaps. A variety of factors influence the SMR, of which several are completely unrelated to the ICU's performance (leaving aside the problem of defining quality of care through mortality statistics). 

Reasons for a spuriously elevated SMR

Some devilry is causing your SMR to rise in spite of the fact that you are still clearly awesome. This can be for a number of reasons. To take it back to basics, either the mortality measurement is wrong, or the ICU is failing because of some internal or external factors

Unfair mortality measurements should not be a problem, as there is no serious situation in which mortality might get over-reported (i.e. some patients walking around after ICU discharge who are formally dead according to the official record). An under-reporting of mortality is much more likely, and might occur in resource-poor environments where public record-keeping is rudimentary. Such dodgy records might still list the person as alive, whereas they died shortly after ICU discharge and nobody bothered to file the paperwork.

Unfair comparison group mortality / APACHE mortality estimate is usually the culprit.

• Poor data entry, eg. constant overestimation of GCS or failure to tick the chronic illness boxes, which makes the patients appear healthier than they actually are.
• Missing data is (in the author's experience) usually rich in APACHE points, eg. the missed urine output entry obscuring anuria, or the failure to document biochemistry results which conceals the potassium level of 8.0mmol/L.
• "Lead time bias" - treatment received prior to ICU admission may result in artifically normalised acute physiology scores
• "Healthy worker effect" - a change towards selective ICU admission practices may be favouring patients who score low on illness severity scales, eg. young elective surgical patients

Reasons for a truly elevated SMR

Issues external to the ICU

• A population with greater pre-ICU morbidity is suddenly available (eg. to borrow an example from LITFL, you have suddenly decided to become a destination for the state's ECMO retrieval service).
• Pre-ICU care has changed its practice (for the worse)
• Parameters which govern ICU admission have changed (eg. administrative pressure is being placed on the ICU to rapidly admit ED patients who have had little management or workup)
• Discharge arrangements have changed (eg. a local palliative care ward had shut down, and you keep dying patients in the ICU because it would be insensitive to transfer them to the next nearby palliative care unit)

Issues internal to the ICU (i.e. genuine under-performance)

• This could be a long list...
• A new staffing model is in place (inexperienced staff)
• Understaffing has impaired patient care
• Junior people are not following unfamiliar protocols, or the new protocols are crap
• New equipment you bought is useless