College Answer

Points to consider in the answer would be:

• Does the population studied correspond with the population the candidate expects to treat?
• Were the inclusion/exclusion criteria appropriate?
• Was the trial methodology appropriate – was there adequate blinding and randomisation?
• Was the primary outcome a clinically relevant or a surrogate endpoint?
• Was the length of follow up adequate?
• Was the trial sufficiently powered to detect a clinically relevant effect?
• Were the groups studied equivalent at baseline?
• Is the statistical analysis appropriate – was there an intention to treat analysis, have differences between groups at baseline been adjusted for? Are there multiple sub group analyses, and if so were they specified a priori?
• Is this a single centre study or multi centre?
• Were the results clinically significant rather than just statistically significant?
• Is the primary hypothesis biologically plausible with pre existing supporting evidence?
• Are the findings supported by other evidence – have these results been replicated?
• Would there be logistical and/or financial implications in practice change?
• Are there important adverse effects of the treatment?

Discussion

This question really asks, "how do you assess an RCT for validity?"

This is addressed in greater detail elsewhere.

In brief:

Is the premise sound?

• Is the primary hypothesis biologically plausible?
• Is the research ethical?
• If the results are valid, are there any disastrous logistical ethical or financial emplications to a change in practice?

Is the methodology of high quality?

• Were the inclusion/exclusion criteria appropriate?
• Was the assignment of patients to treatments randomised? If yes, then was it truly random?
• Were the study groups homogenous?
• Were the groups treated equally?
• Are there any missing patients? Is every enrolled patient accounted for? 
• Was follow-up complete? Is the drop-out rate explained? Do we know what happened to the dropouts?

Is the reporting of an appropriate quality?

• Methods describtion should be complete: the trial should be reproduceable
• Do the results have confidence intervals?
• Results should present relative and absolute effect sizes
• Is a CONSORT-style flow diagram of patient selection available?
• Discussion should contain limitation, bias and imprecision
• Funding sources and the full trial protocol should be disclosed

Are the results of the study valid?

• Was there blinding? Was blinding even possible? Was it double-blind? If not, at least were the data interpreters and statisticians blinded?
• Was there allocation concealment?
• Was there intention-to-treat analysis?
• If there were sub-groups, were they identified a priori?

What were the results?

• How large was the treatment effect?
• How precisely was the effect estimated? (i.e. what was the 95% confidence interval)

Is this study helpful for me?

• Is this applicable to my patient? i.e. would my patient have been enrolled in this study?
• Does the population studied correspond with the population to which my patient belongs?
• Were all the clinically meaningful outcomes considered?
• Does the benefit outweigh the cost and risk?