Question 8

College Answer

Points to consider in the answer would be:

• Does the population studied correspond with the population the candidate expects to treat?
• Were the inclusion/exclusion criteria appropriate?
• Was the trial methodology appropriate – was there adequate blinding and randomisation?
• Was the primary outcome a clinically relevant or a surrogate endpoint?
• Was the length of follow up adequate?
• Was the trial sufficiently powered to detect a clinically relevant effect?
• Were the groups studied equivalent at baseline?
• Is the statistical analysis appropriate – was there an intention to treat analysis, have differences between groups at baseline been adjusted for? Are there multiple sub group analyses, and if so were they specified a priori?
• Is this a single centre study or multi centre?
• Were the results clinically significant rather than just statistically significant?
• Is the primary hypothesis biologically plausible with pre existing supporting evidence?
• Are the findings supported by other evidence – have these results been replicated?
• Would there be logistical and/or financial implications in practice change?
• Are there important adverse effects of the treatment?

Discussion

This question really asks, "how do you assess an RCT for validity?"

This is addressed in greater detail elsewhere.

In brief:

Is the premise sound?

• Is the primary hypothesis biologically plausible?
• Is the research ethical?
• If the results are valid, are there any disastrous logistical ethical or financial emplications to a change in practice?

Is the methodology of high quality?

• Were the inclusion/exclusion criteria appropriate?
• Was the assignment of patients to treatments randomised? If yes, then was it truly random?
• Were the study groups homogenous?
• Were the groups treated equally?
• Are there any missing patients? Is every enrolled patient accounted for? 
• Was follow-up complete? Is the drop-out rate explained? Do we know what happened to the dropouts?

Is the reporting of an appropriate quality?

• Methods describtion should be complete: the trial should be reproduceable
• Do the results have confidence intervals?
• Results should present relative and absolute effect sizes
• Is a CONSORT-style flow diagram of patient selection available?
• Discussion should contain limitation, bias and imprecision
• Funding sources and the full trial protocol should be disclosed

Are the results of the study valid?

• Was there blinding? Was blinding even possible? Was it double-blind? If not, at least were the data interpreters and statisticians blinded?
• Was there allocation concealment?
• Was there intention-to-treat analysis?
• If there were sub-groups, were they identified a priori?

What were the results?

• How large was the treatment effect?
• How precisely was the effect estimated? (i.e. what was the 95% confidence interval)

Is this study helpful for me?

• Is this applicable to my patient? i.e. would my patient have been enrolled in this study?
• Does the population studied correspond with the population to which my patient belongs?
• Were all the clinically meaningful outcomes considered?
• Does the benefit outweigh the cost and risk?

Oh's Intensive Care manual: Chapter 10 (p83), Clinical trials in critical care by Simon Finfer and Anthony Delaney.

The JAMA collection via the John Hopkins Medical School

CASP (Critical Appraisal Skils Program) has checklists for the appraisal of many different sorts of studies; these actually come with tickboxes. One imagines reviewers wandering around a trial headquarters, ticking these boxes on their little clipboards.

CEBM (Centre for Evidence Based Medicine) also has checklists, which (in my opinion) are more informative.

Here is a link to their checklist for the critical appraisal of an RCT.