The idea behind these is that there may be a benefit in summing up all the evidence from several similar trials, analysing all of it together. This way, as the sample numbers grow, more subtle treatment effects may surface (because smaller trials may have been underpowered and thus many type 2 errors may have been committed).
However, the statistical analysis of the evidence in a metaanalysis of trials can occasionally produce results which contradict the actual trials. One is left wondering: which methodology is flawed? Whose statistics are faulty?
This topic presents in both the Primary and the Fellowship Exams. Particularly these "critically evaluate" questions have cropped up more and more often these days, ever since the simple "calculate this or define that" questions have been largely banished to the CICM Part I. Historically, both the Part I and Part II questions have required the candidate to define metaanalysis, analyse forest and funnel plots, and to discuss the advantages and disadvantages of metaanalysis. For purposes of simplifying revision, the material common to Part I and Part II SAQs is duplicated in this chapter as well as in the Primary Exam required reading chapter on metaanalysis and systematic review. Systematic review questions have not appeared in the Fellowship exam thus far.
Past paper questions have included the following:
 Question 8 from the first paper of 2015
 Question 13 from the second paper of 2014
 Question 5 from the second paper of 2013
 Question 10 from the first paper of 2009
 Question 30 from the second paper of 2007
Critical evaluation of the metaanalysis as a tool of EBM
Rationale for metaanalysis
 Pools data from mutiple trials to increase sample size
 Increased sample = increased statistical power
 This improves the accuracy of effect size estimates
The idea behind these is that there may be a benefit in summing up all the evidence from several similar trials, analysing all of it together. This way, as the sample numbers grow, more subtle treatment effects may surface (because smaller trials may have been underpowered and thus many type 2 errors may have been committed).
However, the statistical analysis of the evidence in a metaanalysis of trials can occasionally produce results which contradict the actual trials. One is left wondering: which methodology is flawed? Whose statistics are faulty?
Method
 Literature search is performed, and unpublished trials are searched for
 Large trials weighted more heavily
 OR is used and represented graphically
 Findings expressed as NNT
Advantages
 A more objective quantitative appraisal of evidence
 Reduces the probability of false negative results
 The combination of samples leads to an improvement of statistical power
 Increased sample size may "normalise" the sample distribution and render the results more generalisable, i.e. increase the external validity of the findings
 Increased sample size may increase the accuracy of the estimate
 May explain heterogeneity between the results of different studies
 Inconsistencies among trials may be quantified and analysed
 RCT heterogeneity may be resolved
 Publication bias may be revealed
 Future research directions may be identified
 Avoids Simpson’s paradox, in which a consistent effect in constituent trials is reversed when results are simply pooled.
Disadvantages
 Frustrated by heterogeneity of population samples and methodologies
 Selection of studies may be biased
 Negative studies are rarely published, and thus may not be included
 The metaanalysis uses summary data rather than individual data
Analysis of the validity of a metaanalysis: list of desirable features
 Research questions clearly defined
 Transparent search strategy
 Thorough search protocol
 Authors contacted and unpublished data collected
 Definition of inclusion and exclusion criteria for studies
 Sensible exclusion and inclusion criteria
 Assessment of methodological quality of the included studies
 Transparent methodology of assessment
 Calculation of a pooled estimate
 Plot of the results (Forest Plot)
 Measurement of heterogeneity
 Assessment of publication bias (Funnel Plot)
 Reproduceable metaanalysis strategy (eg. multiple reviewers perform the same metaanalysis, according to the same methods)
A forest plot
Candidates were asked to identify this graph in Question 8 from the first paper of 2015 and Question 10 from the first paper of 2009. The standards for labelling and graphical representation are well summarised by this Cochrane document (however, it appears that careful adherence to standards is no defence against the absence of useful content). A more thorough discussion of the forest plot takes place in the relevant Required Reading chapter from the Primary Exam Collection.
In summary, to answer the abovementioned SAQs one needs to known only these features:
 The horizontal lines: the confidence interval of the individual study
 The position of the square: a point estimate of the odds ratio (OR)
 The size of the square: the weight of the study according to the weighing rules of the metaanalysis, likely representing the sample size and statistical power.
A funnel plot
A funnel plot is scatter plot of the intervention effect estimates from individual studies against some measure of each study’s size or precision. At least this is the definition given to it by the Cochrane Handbook. Question 13 from the second paper of 2014 presented this graphic device to the candidates, asking them to identify its lines and reasons for assymmetry within it.
Cardinal features:
 The effect is plotted on the horsiontal axis
 The study size is plotted on the vertical axis
 Small studies have very varied findings and scatter wildly (at the bottom)
 As the studies get bigger, they get more precise. The data concentrates into the "spout" of the funnel.
The lines? what do they mean? Said best by the laconic college:
 Outer dashed linestriangular region where 95% of studies are expected to lie. This triangle is centred on a fixed effect summary estimate, and extens 1.96 standard errors in each direction. If no bias is present, this triangle will include about 95% of studies, provided the true treatment effect is the same in each study (i.e. none were using some sort of dodgy homemade levosimendan, for instance).
 Solid vertical line no intervention effect. This corresponds to an OR of 1.00.
Causes of assymmetry are well summarised by Sterne et al (2011), whose Box 1 I have shamelessly stolen:
Reporting biases
Poor methodological quality

True heterogeneity
Artefactual
Chance

References
DerSimonian, Rebecca, and Nan Laird. "Metaanalysis in clinical trials."Controlled clinical trials 7.3 (1986): 177188.
Rockette, H. E., and C. K. Redmond. "Limitations and advantages of metaanalysis in clinical trials." Cancer Clinical Trials. Springer Berlin Heidelberg, 1988. 99104.
Walker, Esteban, Adrian V. Hernandez, and Michael W. Kattan. "Metaanalysis: Its strengths and limitations." Cleveland Clinic Journal of Medicine75.6 (2008): 431439.
Methodological Expectations of Cochrane Intervention Reviews
Sterne, Jonathan AC, et al. "Recommendations for examining and interpreting funnel plot asymmetry in metaanalyses of randomised controlled trials." Bmj 343 (2011): d4002.