Question 8

College Answer

There is more than one classification system for level of evidence with respect to EBM, and any of those was acceptable. Most candidates struggled with the second part of the question. The strengths/weaknesses listed by candidates were often not related to the scientific or statistical principles/properties of a meta-analysis. A good answer would include a discussion about publication bias, duplicate publication, heterogeneity (different interventions), inclusions of outdated studies, inclusions of non-randomised trials, that pooled result may be biased towards the largest included trials as potential weaknesses and increased sample size and that more variables, sub-groups and outcomes can be examined can be considered as strengths.

Discussion

This question is virtually identical to Question 19 from the second paper of 2010, except that in 2010 for whatever reason the "levels of evidence" question was weighed higher, as 30% of the marks. Also, during the first appearance of it, the examiners gave more of an answer- actually listing the NHMRC levels of evidence, for example. Even a text reference was offered (Myles & Gin Statistical methods for Anaesthesia and Intensive Care, pg114-118). 

a) Define levels of evidence with respect to Evidence Based Medicine (EBM).

There are in fact several systems. The NHMRC classification system is discussed in the document "NHMRC additional levels of evidence and grades for recommendations for developers of guidelines". Instead of wading through the entire 23-page morass, the time-poor candidate is invited to explore Table 3 on page 15. In brief:

• Level I: systematic review of RCTs
• Level II: RCT
• Level III-1: pseudorandomised trial of high quality
• Level III-2: cohort studies or case control studies - but with a control group
• Level III-3: cohort studies with historical controls, or no control group
• Level IV: case series

b) Discuss the strengths and weaknesses of meta-analysis.

From the college answer, we observe that the candidates were "expected to define a meta-analysis ... and a systematic review", but it is unclear how this expectation could be determined from actually reading the question. The question clearly asks to "discuss the strengths and weaknesses" and nothing more. No mention whatsoever is made of systematic review. However, if the college insist that some hidden meaning is encoded into their SAQ, so we must play along and answer it.

There are several possible definitions in addition to the canonical college answer:

• In Question 30 from the second Fellowship paper of 2007, meta-analysis is defined as "a form of systematic review that uses statistical methods to combine the results from different studies"
• Bartolucci et al (2010) defines it as "the process of combining the quantitative results of separate (but similar) studies by means of formal statistical methods" 
• The website of the Centre for Cognitive Ageing and Cognitive Epidemiology offers the following definitions: "a systematic review answers a defined research question by collecting and summarising all empirical evidence that fits pre-specified eligibility criteria; a meta-analysis is the use of statistical methods to summarise the results of these studies."

Now, to answer the actual question as it was asked:

Advantages of meta-analysis

• A more objective quantitative appraisal of evidence
• Reduces the probability of false negative results
• The combination of samples leads to an improvement of statistical power
• Increased sample size may "normalise" the sample distribution and render the results more generalisable, i.e. increase the external validity of the findings
• Increased sample size may increase the accuracy of the estimate
• May explain heterogeneity between the results of different studies
• Inconsistencies among trials may be quantified and analysed
• RCT heterogeneity may be resolved
• Publication bias may be revealed
• Future research directions may be identified
• Avoids Simpson’s paradox, in which a consistent effect in constituent trials is reversed when results are simply pooled.

Disadvantages of meta-analysis

• Frustrated by heterogeneity of population samples and methodologies
• Selection of studies may be biased
• Negative studies are rarely published, and thus may not be included
• The meta-analysis uses summary data rather than individual data
• Reliability may be sabotaged by inclusion of poor quality studies. If the studies are of an overall poor quality, then the conclusions of the overall review will also suffer in their quality. One cannot soar like an eagle if one is collectively analysing the studies of turkeys.
• The studies included in the review may not have the same outcome measures, which might make it difficult to collectively analyse the results.
• Positive meta-analysis findings do not by themselves constitute evidence of a sufficiently high quality to merit a change in practice, and still need to be confirmed by a large scale RCT

Sackett, David L., et al. "Evidence based medicine: what it is and what it isn't." (1996): 71-72.

Brown, Gary C., Melissa M. Brown, and Sanjay Sharma. "Value-based medicine: evidence-based medicine and beyond." Ocular immunology and inflammation 11.3 (2003): 157-170.

Bartolucci, Alfred A., and William B. Hillegass. "Overview, strengths, and limitations of systematic reviews and meta-analyses." Evidence-Based Practice: Toward Optimizing Clinical Outcomes. Springer Berlin Heidelberg, 2010. 17-33.