The idea behind these is that there may be a benefit in summing up all the evidence from several similar trials, analysing all of it together. This way, as the sample numbers grow, more subtle treatment effects may surface (because smaller trials may have been underpowered and thus many type 2 errors may have been committed).
However, the statistical analysis of the evidence in a meta-analysis of trials can occasionally produce results which contradict the actual trials. One is left wondering: which methodology is flawed? Whose statistics are faulty?
This topic presents in both the Primary and the Fellowship Exams. Particularly these "critically evaluate" questions have cropped up more and more often these days, ever since the simple "calculate this or define that" questions have been largely banished to the CICM Part I. Historically, both the Part I and Part II questions have required the candidate to define meta-analysis, analyse forest and funnel plots, and to discuss the advantages and disadvantages of meta-analysis. For purposes of simplifying revision, the material common to Part I and Part II SAQs is duplicated in this chapter as well as in the Primary Exam required reading chapter on meta-analysis and systematic review. Systematic review questions have not appeared in the Fellowship exam thus far.
Past paper questions have included the following:
The idea behind these is that there may be a benefit in summing up all the evidence from several similar trials, analysing all of it together. This way, as the sample numbers grow, more subtle treatment effects may surface (because smaller trials may have been underpowered and thus many type 2 errors may have been committed).
However, the statistical analysis of the evidence in a meta-analysis of trials can occasionally produce results which contradict the actual trials. One is left wondering: which methodology is flawed? Whose statistics are faulty?
Candidates were asked to identify this graph in Question 8 from the first paper of 2015 and Question 10 from the first paper of 2009. The standards for labelling and graphical representation are well summarised by this Cochrane document (however, it appears that careful adherence to standards is no defence against the absence of useful content). A more thorough discussion of the forest plot takes place in the relevant Required Reading chapter from the Primary Exam Collection.
In summary, to answer the abovementioned SAQs one needs to known only these features:
A funnel plot is scatter plot of the intervention effect estimates from individual studies against some measure of each study’s size or precision. At least this is the definition given to it by the Cochrane Handbook. Question 13 from the second paper of 2014 presented this graphic device to the candidates, asking them to identify its lines and reasons for assymmetry within it.
Cardinal features:
The lines? what do they mean? Said best by the laconic college:
Causes of assymmetry are well summarised by Sterne et al (2011), whose Box 1 I have shamelessly stolen:
Reporting biases
Poor methodological quality
|
True heterogeneity
Artefactual
Chance
|
DerSimonian, Rebecca, and Nan Laird. "Meta-analysis in clinical trials."Controlled clinical trials 7.3 (1986): 177-188.
Rockette, H. E., and C. K. Redmond. "Limitations and advantages of meta-analysis in clinical trials." Cancer Clinical Trials. Springer Berlin Heidelberg, 1988. 99-104.
Walker, Esteban, Adrian V. Hernandez, and Michael W. Kattan. "Meta-analysis: Its strengths and limitations." Cleveland Clinic Journal of Medicine75.6 (2008): 431-439.
Methodological Expectations of Cochrane Intervention Reviews
Sterne, Jonathan AC, et al. "Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials." Bmj 343 (2011): d4002.