With reference to the reporting of clinical trials in the literature:
a) What is a meta-analysis?
b) What are the advantages of a meta-analysis over the interpretation of an individual study?
c) List the features of a well-conducted meta-analysis.
d) What is “publication bias” and how can this impact on the validity of a meta-analysis?
 

[Click here to toggle visibility of the answers]

College Answer

a)

  • A form of systematic review that uses statistical methods to combine the results from different studies

b)

  • ↑ Statistical power by ↑ sample size.
  • Resolve uncertainty when studies disagree.
  • Improve estimates of effect size.
  • Inconsistency of results across studies can be quantified and analysed e.g. heterogeneity of studies, sampling error.
  • Presence of publication bias can be investigated.
  • Establish questions for future RCTs.
  • May provide information regarding generalisability of results.

 

c)

  • Clearly defined research question.
  • Thorough search strategy that makes it unlikely that significant studies have been missed.
  • Reproducible and clear criteria for inclusion in the meta-analysis.
  • Adequate and reproducible assessment of the methodological quality of the included studies.
  • Use of appropriate statistical methods to assess for heterogeneity between studies and pooling of the results of studies when appropriate.
  • Utilisation of methods to ensure that the results of the meta-analysis are reproducible; e.g. two reviewers perform aspects of the study (the search, the application of the inclusion/exclusion criteria, the assessment of validity, the data extraction).
  • Assessment for the presence of publication bias/small study bias with report of the results of these analyses.

 

d)

  • Publication bias is the publication or non-publication of studies depending on the direction and statistical significance of the results. A meta-analysis evaluating studies where there has been publication bias will be flawed, no matter how well conducted in other aspects.
  • Publication bias may also extend to bias of selection of studies for inclusion in a meta-analysis based on language, journal of publication, ease of access, field of research etc., (dissemination bias).

 

Salient points

  • Meta analysis = tool of quantitative systematic review
  • Advantages:
    • increased statistical power
    • resolves heterogeneity
    • avoids Simpson's paradox
  • A good meta-analysis has:
    • a well-structured question
    • broad search strategy
    • transparent methodology
    • attempt to exclude publication bias
    • Forest plot
    • measures of heterogeneity

Discussion

 

LITFL have an excellent resource for this.

a) What is a meta-analysis?

Meta-analysis is a tool of quantitative systematic review.

It is used to weigh the available evidence from RCTs and other studies based on the numbers of patients included, the effect size, and on statistical tests of agreement with other trials

 

b) What are the advantages of a meta-analysis over the interpretation of an individual study?

  • A more objective quantitative appraisal of evidence
  • Reduces the probability of false negative results
  • The combination of samples leads to an improvement of statistical power
  • Increased sample size may "normalise" the sample distribution and render the results more generalisable, i.e. increase the external validity of the findings
  • Increased sample size may increase the accuracy of the estimate
  • May explain heterogeneity between the results of different studies
  • Inconsistencies among trials may be quantified and analysed
  • RCT heterogeneity may be resolved
  • Publication bias may be revealed
  • Future research directions may be identified
  • Avoids Simpson’s paradox, in which a consistent effect in constituent trials is reversed when results are simply pooled.

 

c) List the features of a well-conducted meta-analysis.

  • Research questions clearly defined
  • Transparent search strategy
  • Thorough search protocol
  • Authors contacted and unpublished data collected
  • Definition of inclusion and exclusion criteria for studies
  • Sensible exclusion and inclusion criteria
  • Assessment of methodological quality of the included studies
  • Transparent methodology of assessment
  • Calculation of a pooled estimate
  • Plot of the results (Forest Plot)
  • Measurement of heterogeneity
  • Assessment of publication bias (Funnel Plot)
  • Reproduceable meta-analysis strategy (eg. multiple reviewers perform the same meta-analysis, according to the same methods)

 

d) What is “publication bias” and how can this impact on the validity of a meta-analysis?

  • Publication bias is the influence of study results on the likelihood of their publication
  • A funnel plot can be used to identify publication bias.
  • A meta-analysis can be invalidated if publication bias has influenced the included studies.
  • Publication bias leads to the selection of mostly positive (or mostly negative) studies, which in turn leads to positive meta-analysis results. Studies with the opposite effect may not have been selected for publication, and may not be available to the meta-analysis authors.
  • Meta-analysis authors may develop an inherent publication bias by only using English-language studies, only free-access articles, or only focusing their search within a narrow field of research.
  • Publication bias can be overcome by contacting relevant authors and requesting unpublished trial data, by searching for publications in all languages, and by searching broadly in multiple cross-specialty databases.

 

References

References

DerSimonian, Rebecca, and Nan Laird. "Meta-analysis in clinical trials."Controlled clinical trials 7.3 (1986): 177-188.

 

Rockette, H. E., and C. K. Redmond. "Limitations and advantages of meta-analysis in clinical trials." Cancer Clinical Trials. Springer Berlin Heidelberg, 1988. 99-104.

 

Walker, Esteban, Adrian V. Hernandez, and Michael W. Kattan. "Meta-analysis: Its strengths and limitations." Cleveland Clinic Journal of Medicine75.6 (2008): 431-439.

 

Methodological Expectations of Cochrane Intervention Reviews