Question 19

College Answer

For a good answer candidates were expected to list the following levels of evidence, eg Level I (evidence obtained from a systematic review of all (at least 2) relevant randomized controlled trials), Level II (evidence obtained from at least one properly designed randomized controlled trial, Level III (evidence obtained from other well-designed experimental or analytical studies (not RCCT’s), Level IV (evidence obtained from descriptive studies, reports of expert committees or from opinions of respected authorities based on clinical experience).

Candidates were expected to define a meta-analysis (process of combining the results of different (randomised) trials to derive a pooled estimate of effect) and a systematic review (process of obtaining and evaluating all relevant trials, their statistical analyses and interpretation of results). In relation to strengths, a good answer required mentioning increased power of pooled data, analysis and conclusions based on inclusion of high quality trials (weighting of trial quality), overcomes the uncertainty associated with single-centre trials, robust methodology; combines similar patient groups, interventions and end-points to inform the analysis and established methods to find all relevant trials. In relation to weaknesses, a good answer required mentioning publication bias, heterogeneity of included trials, pooled result may be biased toward the largest included trials, historical (outdated) data, and that positive results generally require confirmation by a large RCT. 

Syllabus: EBM 2a 

References: Myles & Gin Statistical methods for Anaesthesia and Intensive Care, pg114-118

Discussion

This question is virtually identical to Question 8 from the second paper of 2013. The discussion sections for both questions are therefore very similar.

a) Define levels of evidence with respect to Evidence Based Medicine (EBM).

There are in fact several systems. The NHMRC classification system is discussed in the document "NHMRC additional levels of evidence and grades for recommendations for developers of guidelines". Instead of wading through the entire 23-page morass, the time-poor candidate is invited to explore Table 3 on page 15. In brief:

• Level I: systematic review of RCTs
• Level II: RCT
• Level III-1: pseudorandomised trial of high quality
• Level III-2: cohort studies or case control studies - but with a control group
• Level III-3: cohort studies with historical controls, or no control group
• Level IV: case series

b) Discuss the strengths and weaknesses of meta-analysis.

From the college answer, we observe that the candidates were "expected to define a meta-analysis ... and a systematic review", but it is unclear how this expectation could be determined from actually reading the question. The question clearly asks to "discuss the strengths and weaknesses" and nothing more. No mention whatsoever is made of systematic review. However, if the college insist that some hidden meaning is encoded into their SAQ, so we must play along and answer it.

There are several possible definitions in addition to the canonical college answer:

• In Question 30 from the second Fellowship paper of 2007, meta-analysis is defined as "a form of systematic review that uses statistical methods to combine the results from different studies"
• Bartolucci et al (2010) defines it as "the process of combining the quantitative results of separate (but similar) studies by means of formal statistical methods" 
• The website of the Centre for Cognitive Ageing and Cognitive Epidemiology offers the following definitions: "a systematic review answers a defined research question by collecting and summarising all empirical evidence that fits pre-specified eligibility criteria; a meta-analysis is the use of statistical methods to summarise the results of these studies."

Now, to answer the actual question as it was asked:

Advantages of meta-analysis

• A more objective quantitative appraisal of evidence
• Reduces the probability of false negative results
• The combination of samples leads to an improvement of statistical power
• Increased sample size may "normalise" the sample distribution and render the results more generalisable, i.e. increase the external validity of the findings
• Increased sample size may increase the accuracy of the estimate
• May explain heterogeneity between the results of different studies
• Inconsistencies among trials may be quantified and analysed
• RCT heterogeneity may be resolved
• Publication bias may be revealed
• Future research directions may be identified
• Avoids Simpson’s paradox, in which a consistent effect in constituent trials is reversed when results are simply pooled.

Disadvantages of meta-analysis

• Frustrated by heterogeneity of population samples and methodologies
• Selection of studies may be biased
• Negative studies are rarely published, and thus may not be included
• The meta-analysis uses summary data rather than individual data
• Reliability may be sabotaged by inclusion of poor quality studies. If the studies are of an overall poor quality, then the conclusions of the overall review will also suffer in their quality. One cannot soar like an eagle if one is collectively analysing the studies of turkeys.
• The studies included in the review may not have the same outcome measures, which might make it difficult to collectively analyse the results.
• Positive meta-analysis findings do not by themselves constitute evidence of a sufficiently high quality to merit a change in practice, and still need to be confirmed by a large scale RCT

Sackett, David L., et al. "Evidence based medicine: what it is and what it isn't." (1996): 71-72.

Brown, Gary C., Melissa M. Brown, and Sanjay Sharma. "Value-based medicine: evidence-based medicine and beyond." Ocular immunology and inflammation 11.3 (2003): 157-170.

Bartolucci, Alfred A., and William B. Hillegass. "Overview, strengths, and limitations of systematic reviews and meta-analyses." Evidence-Based Practice: Toward Optimizing Clinical Outcomes. Springer Berlin Heidelberg, 2010. 17-33.