Question 9

College Answer

Syllabus topic/section:

2.5.1    Research and Evidence Based practice in Intensive Care.

Aim:

To assess the candidate familiarity with research publications.

Discussion:

Candidates scored well if they gave a structured, detailed answer touching upon:

•    Methods section including research protocol, criteria for including and excluding studies in the systematic review, analytic framework and key questions, databases and other information sources used to identify relevant studies, search strategy, study selection process, data extraction process, methods for handling missing information, information to be extracted from included studies, methods to appraise the quality of individual studies, summary measures of effect size (e.g., risk ratio, difference in means), rationale for pooling (or not pooling) results of included studies, methods of synthesizing the evidence (qualitative and meta-analysis).
•    Result section including Study selection process, list of excluded studies and reasons for their exclusion, appraisal of individual studies’ quality, qualitative synthesis, meta-analysis of results. Defined Meta- analysis and differentiated it from SR.
•    Discussion section with summary of evidence, strengths, and weaknesses of the conclusion for each key question, gaps in evidence, future research.

Candidates scored poorly due to limited answers lacking detail and demonstrated confusion by describing an RCT rather than a MA/SR. Lack of knowledge was the main reason for scoring poor marks.

Strategies to improve:

•    Structure the answer.
•    Must gain knowledge of how a systematic review is conducted – in a structured, reproducible, and unbiased way.
 

Discussion

Meta-analysis was not included in the stem, and so one cannot flaw the candidates for not describing the methodological qualities of an "MA/SR", or for not defining the terms. What follows is specific for the evaluation of a systematic review, and comes from the excellent article on systematic literature reviews and meta-analyses from a series by Deutsches Ärzteblatt International, the JBI checklist for systematic reviews, and the Systematic review checklist from the CASP (Critical Appraisal Skills Program).

Definitions:

• Systematic review: a process of obtaining and evaluating all relevant trials, their statistical analyses and interpretation of results.
• A meta-analysis is the use of statistical methods to summarise the results of these studies.

Methods

• Specific and relevant questions
• Inclusion and exclusion criteria specified a priori
• Literature search is thorough and includes studies in all languages and unpublished studies (by contacting authors)
• Search strategy is described and databases are specified
• Pooling strategy is described (for pooled reanalysis)
• Pooled effect estimate calculation methodology is valid and appropriate (there's really only two, it will be either the fixed effect model or the random effect model)
• Data extraction strategy is described where pooled data is not used (transparent search strategy)
• Methods to minimize errors in data extraction are listed, eg. multiple reviewers perform the same meta-analysis, according to the same methods

Results

• Important characteristics of the individual studies should be presented in overview tables
• List of excluded studies should be available
• Large trials weighted more heavily
• Plot of the results (Forest Plot)
• Measurement of heterogeneity is presented
• Assessment of publication bias (Funnel Plot) is available

Discussion

• Summary statement available
• Recommendations are supported by the findings
• Gaps in evidence identified, future directions for research
• Strengths and weaknesses of the conclusions for each key question are identified