Critical appraisal of meta-analysis data

The idea behind these is that there may be a benefit in summing up all the evidence from several similar trials, analysing all of it together. This way, as the sample numbers grow, more subtle treatment effects may surface (because smaller trials may have been underpowered and thus many type 2 errors may have been committed).

However, the statistical analysis of the evidence in a meta-analysis of trials can occasionally produce results which contradict the actual trials. One is left wondering: which methodology is flawed? Whose statistics are faulty?

This topic presents in both the Primary and the Fellowship Exams. Particularly these "critically evaluate" questions have cropped up more and more often these days, ever since the simple "calculate this or define that" questions have been largely banished to the CICM Part I. Historically, both the Part I and Part II questions have required the candidate to define meta-analysis, analyse forest and funnel plots, and to discuss the advantages and disadvantages of meta-analysis. For purposes of simplifying revision, the material common to Part I and Part II SAQs is duplicated in this chapter as well as in the Primary Exam required reading chapter on meta-analysis and systematic review. Systematic review questions have not appeared in the Fellowship exam thus far.

Past paper questions have included the following:

• Question 8 from the first paper of 2015
• Question 13 from the second paper of 2014
• Question 5 from the second paper of 2013
• Question 10 from the first paper of 2009
• Question 30 from the second paper of 2007

Critical evaluation of the meta-analysis as a tool of EBM

Rationale for meta-analysis

• Pools data from mutiple trials to increase sample size
  • Increased sample = increased statistical power
• This improves the accuracy of effect size estimates

The idea behind these is that there may be a benefit in summing up all the evidence from several similar trials, analysing all of it together. This way, as the sample numbers grow, more subtle treatment effects may surface (because smaller trials may have been underpowered and thus many type 2 errors may have been committed).

However, the statistical analysis of the evidence in a meta-analysis of trials can occasionally produce results which contradict the actual trials. One is left wondering: which methodology is flawed? Whose statistics are faulty?

Method

• Literature search is performed, and unpublished trials are searched for
• Large trials weighted more heavily
• OR is used and represented graphically
• Findings expressed as NNT

Advantages

• A more objective quantitative appraisal of evidence
• Reduces the probability of false negative results
• The combination of samples leads to an improvement of statistical power
• Increased sample size may "normalise" the sample distribution and render the results more generalisable, i.e. increase the external validity of the findings
• Increased sample size may increase the accuracy of the estimate
• May explain heterogeneity between the results of different studies
• Inconsistencies among trials may be quantified and analysed
• RCT heterogeneity may be resolved
• Publication bias may be revealed
• Future research directions may be identified
• Avoids Simpson’s paradox, in which a consistent effect in constituent trials is reversed when results are simply pooled.

Disadvantages

• Frustrated by heterogeneity of population samples and methodologies
• Selection of studies may be biased
• Negative studies are rarely published, and thus may not be included
• The meta-analysis uses summary data rather than individual data

Analysis of the validity of a meta-analysis: list of desirable features

• Research questions clearly defined
• Transparent search strategy
• Thorough search protocol
• Authors contacted and unpublished data collected
• Definition of inclusion and exclusion criteria for studies
• Sensible exclusion and inclusion criteria
• Assessment of methodological quality of the included studies
• Transparent methodology of assessment
• Calculation of a pooled estimate
• Plot of the results (Forest Plot)
• Measurement of heterogeneity
• Assessment of publication bias (Funnel Plot)
• Reproduceable meta-analysis strategy (eg. multiple reviewers perform the same meta-analysis, according to the same methods)

A forest plot

Candidates were asked to identify this graph in Question 8 from the first paper of 2015 and Question 10 from the first paper of 2009. The standards for labelling and graphical representation are well summarised by this Cochrane document (however, it appears that careful adherence to standards is no defence against the absence of useful content). A more thorough discussion of the forest plot takes place in the relevant Required Reading chapter from the Primary Exam Collection.

In summary, to answer the abovementioned SAQs one needs to known only these features:

• The horizontal lines: the confidence interval of the individual study
• The position of the square: a point estimate of the odds ratio (OR)
• The size of the square: the weight of the study according to the weighing rules of the meta-analysis, likely representing the sample size and statistical power. 

A funnel plot

A funnel plot is scatter plot of the intervention effect estimates from individual studies against some measure of each study’s size or precision. At least this is the definition given to it by the Cochrane Handbook.  Question 13 from the second paper of 2014 presented this graphic device to the candidates, asking them to identify its lines and reasons for assymmetry within it.

Cardinal features:

• The effect is plotted on the horsiontal axis
• The study size is plotted on the vertical axis
• Small studies have very varied findings and scatter wildly (at the bottom)
• As the studies get bigger, they get more precise. The data concentrates into the "spout" of the funnel.

The lines? what do they mean? Said best by the laconic college:

• Outer dashed lines-triangular region where 95% of studies are expected to lie. This triangle is centred on a fixed effect summary estimate, and extens 1.96 standard errors in each direction. If no bias is present, this triangle will  include about 95% of studies, provided the true treatment effect is the same in each study (i.e. none were using some sort of dodgy home-made levosimendan, for instance).
• Solid vertical line- no intervention effect. This corresponds to an OR of 1.00.

Causes of assymmetry are well summarised by Sterne et al (2011), whose Box 1 I have shamelessly stolen:

Sources of Assymmetry in Funnel Plots

Reporting biases Delayed publication (also known as time lag or pipeline) bias Location biases (eg, language bias, citation bias, multiple publication bias) Selective outcome reporting Selective analysis reporting Poor methodological quality i.e. smalle studies inflated the effect size Poor methodological design Inadequate analysis Fraud

True heterogeneity Size of effect differs according to study size (eg, in smaller studies the intervention was less intense: eg. PROSEVA trial) Artefactual In some circumstances, sampling variation can lead to an association between the intervention effect and its standard error Chance Asymmetry may occur by chance, which motivates the use of asymmetry tests