Meta-analysis and systematic review

This chapter answers parts from Section A(a) of the old 2011 Primary Syllabus; "Describe the features of evidence-based medicine, including levels of evidence (eg. NH&MRC), meta-analysis and systematic review". It closely resembles the Fellowship exam revision chapter, "Types of studies, their advantages and disadvantages". The specific reference to meta-analysis and systematic review also resembles the sort of stuff that might be expected of a fellowship exam candidate, with the exception of the fact that in the Part II written paper we might be expected to perform a Critical Appraisal of Meta-Analysis Data. 

This topic has come up at least twice in the Primary (Question 19 from the second paper of 2010 and Question 8 from the second paper of 2013), and also about five times in the Fellowship Exam, with slightly different expectations from the college. Only slightly. Both Fellowship and Primary candidates have been expected to define meta-analysis and to offer some advantages and disadvantages. Both have at one stage been expected to analyse (or identify the major features) of forest plots and funnel plots. Only Primary candidates have been expected to discuss systematic reviews. Only Fellowship candidates have been expected to delve deeper into the critical appraisal of meta-analysis data to discuss what makes a well-conducted meta-analysis.  Thus though material common to Part I and Part II SAQs is duplicated in this chapter, there is a different emphasis when compared to the chapter on Critical Appraisal of Meta-Analysis Data from the Required Reading section for the Fellowship Exam. 

As far as definitve resources for this topic go, one cannot go past the excellent review by Bartolucci et al (2010). Unless otherwise stated, this is the primary reference for most of the material which follows. 

Definition of meta-analysis

Meta-analysis is a tool of quantitative systematic review. It is used to weigh the available evidence from RCTs and other studies based on the numbers of patients included, the effect size, and on statistical tests of agreement with other trials.

The definition offered by the CICM Primary examiners can probably be viewed as canonical, and is as follows:

Meta-analysis is process of combining the results of different (randomised) trials to derive a pooled estimate of effect.

The answer from a Part II question (Question 30 from the second paper of 2007) reads slightly differently, "A form of systematic review that uses statistical methods to combine the results from different studies". The Bartolucci paper defines it as "the process of combining the quantitative results of separate (but similar) studies by means of formal statistical methods"  Presumably one might get away with using either definition, as they all sound very similar and begin to blend into one another when one reads enough of them.

Advantages and disadvantages of meta-analysis

This SEO-destroying copypasta is from the chapter on Critical Appraisal of Meta-Analysis Data:

Advantages

• A more objective quantitative appraisal of evidence
• Reduces the probability of false negative results
• The combination of samples leads to an improvement of statistical power
• Increased sample size may "normalise" the sample distribution and render the results more generalisable, i.e. increase the external validity of the findings
• Increased sample size may increase the accuracy of the estimate
• May explain heterogeneity between the results of different studies
• Inconsistencies among trials may be quantified and analysed
• RCT heterogeneity may be resolved
• Publication bias may be revealed
• Future research directions may be identified
• Avoids Simpson’s paradox, in which a consistent effect in constituent trials is reversed when results are simply pooled.

Disadvantages

• Frustrated by heterogeneity of population samples and methodologies
• Selection of studies may be biased
• Negative studies are rarely published, and thus may not be included
• The meta-analysis uses summary data rather than individual data
• Positive meta-analysis findings do not by themselves constitute evidence of a sufficiently high quality to merit a change in practice, and still need to be confirmed by a large scale RCT

Definition of systematic review

The definition offered by the CICM examiners is as follows:

A systematic review is a process of obtaining and evaluating all relevant trials, their statistical analyses and interpretation of results.

How is that different from meta-analysis? Well. An excellent clarification can be found at the website of the Centre for Cognitive Ageing and Cognitive Epidemiology.

Quoting them:

A systematic review answers a defined research question by collecting and summarising all empirical evidence that fits pre-specified eligibility criteria.

A meta-analysis is the use of statistical methods to summarise the results of these studies.

Thus, it is possible to have a systematic review which does not include a meta-analysis of trials, but all meta-analysis articles are by this definition systematic reviews. A meta-analysis arises from a systematic review. 

Advantages of a systematic review

• Cheap: it is less expensive for a team to review existing studies than to design and conduct a new study which answers the same question
• Comprehensive: the review should employ a strategy whereby all available evidence is scraped together from all available literature searches. The benchmark of a good systematic review is the retrieval of all available data including unpublished studies, ongoing research and personal correspondence.
• Reproduceable: the review's search strategy should be clearly defined. As the result of such transparency, a future team can repeat the review, including evidence which has been published in the interim.
• Generalisable: because it encompasses multiple studies, the systematic review "irons out" the irregularities in the data which might arise as the result of using different populations.
• Reliable: the findings should be more robust because of the raw increase in numbers. By including a total of several thousand patients, one should theoretically decrease the fragility index of he findings (i.e. one should never ed up in a situation where the change of one patient's result will change the whole conclusion of your systematic review).  

 Disadvantages of systematic review

• Because it is not a meta-analysis (i.e. the data are not analysed in a systematic and formalised manner) the quality of the findings is lower, i.e. it occupies a lower tier on the famous "pyramid of evidence"
• It is time-consuming. The better the quality, the more time-consuming it is. One might estimate that a team working at normal capacity may be able to complete a single systematic review over the period of 12-24 months.
• Reliability may be sabotaged by inclusion of poor quality studies. If the studies are of an overall poor quality, then the conclusions of the overall review will also suffer in their quality. One cannot soar like an eagle if one is collectively analysing the studies of turkeys.
• The studies included in the review may not have the same outcome measures, which might make it difficult to collectively analyse the results.