Question 13

a) With respect to meta-analysis of randomised controlled trials, what is a funnel plot?

b) In the funnel plot above:

i. What do the outer dashed lines indicate?

ii. To what does the solid vertical line correspond?

c) List three factors that result in asymmetry in funnel plots.

[Click here to toggle visibility of the answers]

College Answer

a) A funnel plot is a scatter plot of the effect estimates from individual studies against some measure of each study’s size or precision. The standard error of the effect estimate is often chosen as the measure of study size and plotted on the vertical axis with a reversed scale that places the larger, most powerful studies towards the top. The effect estimates from smaller studies should scatter more widely at the bottom, with the spread narrowing among larger studies.

Outer dashed lines-triangular region where 95% of studies are expected to lie

Solid vertical line- no intervention effect

i) Heterogeneity

Size of effect differs according to study size
Clinical differences
Methodological differences

ii) Reporting bias

Publication bias- delayed publication, language, citation, multiple publication bias
Selective outcome reporting
Selective analysis/inadequate analysis reporting
Poor design
Fraud

iii) Chance

It was expected that candidates regularly attending journal club would have the knowledge to answer this question but overall it was not well answered and explanation of terms was poor

Discussion

The abovedepicted plot is not the gospel plot from the CICM paper, but one which I have confabulated myself. Hopefully, it bears some resemblance to the original.

a) is answered by the college in a manner which precisely reflects the wording of the Cochrane Handbook. That is indeed " a simple scatter plot of the intervention effect estimates from individual studies against some measure of each study’s size or precision ".

b)
The lines? what do they mean? Said best by the laconic college:

Outer dashed lines-triangular region where 95% of studies are expected to lie. This triangle is centred on a fixed effect summary estimate, and extens 1.96 standard errors in each direction. If no bias is present, this triangle will include about 95% of studies, provided the true treatment effect is the same in each study (i.e. none were using some sort of dodgy home-made levosimendan, for instance).
Solid vertical line- no intervention effect. This corresponds to an OR of 1.00.

Causes of assymmetry are well summarised by Sterne et al (2011), whose Box 1 I have shamelessly stolen:

Sources of Assymmetry in Funnel Plots
Reporting biases Delayed publication (also known as time lag or pipeline) bias Location biases (eg, language bias, citation bias, multiple publication bias) Selective outcome reporting Selective analysis reporting Poor methodological quality i.e. smalle studies inflated the effect size Poor methodological design Inadequate analysis Fraud	True heterogeneity Size of effect differs according to study size (eg, in smaller studies the intervention was less intense: eg. PROSEVA trial) Artefactual In some circumstances, sampling variation can lead to an association between the intervention effect and its standard error Chance Asymmetry may occur by chance, which motivates the use of asymmetry tests

References

DerSimonian, Rebecca, and Nan Laird. "Meta-analysis in clinical trials."Controlled clinical trials 7.3 (1986): 177-188.

Rockette, H. E., and C. K. Redmond. "Limitations and advantages of meta-analysis in clinical trials." Cancer Clinical Trials. Springer Berlin Heidelberg, 1988. 99-104.

Walker, Esteban, Adrian V. Hernandez, and Michael W. Kattan. "Meta-analysis: Its strengths and limitations." Cleveland Clinic Journal of Medicine75.6 (2008): 431-439.

Methodological Expectations of Cochrane Intervention Reviews

Sterne, Jonathan AC, et al. "Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials." Bmj 343 (2011): d4002.

[Submit a comment or correction]