Question 6

College Answer

Various checklists are available for assessing methodological quality. One such list is that proposed by  David  Sackett.  It  includes  3  main  questions:  was  assignment  randomised  and  was  the randomisation list concealed (minimise potential for bias)?; was follow up of patients sufficiently long and complete (ensure endpoints accurately assessed)?; were patients analysed in the groups to which they were randomised (maintain benefits of randomisation)? It also includes 3 finer points to address: were patients and clinicians (and outcome assessors) kept blind to treatment (minimise bias)?; were groups treated equally apart from the experimental treatment (ensure intervention effect is only thing being assessed)?; were the groups similar at the start of the trial (were there any potentially confounding effects that randomisation did not eliminate)? In addition to these, the study should have enrolled enough patients to be sufficiently powered to detect the perceived clinically important benefit in the primary outcome variable! Standardised criteria have also been published (CONSORT) that were recommended to facilitate consistency and clarity in studies submitted for publication, allowing the reader to more readily assess the internal and external validity of a study.

(Sackett DL et al (eds.). Evidence-based medicine. Churchill Livingstone, London. 2000

Begg C et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996 Aug 28;276(8):637-9).

Discussion

Though not word-for-word identical, this question closely resembles Question 8 from the second paper of 2012. It discusses the assessment of the validity of a randomised controlled trial, which is discussed in greater detail elsewhere.

In brief:

Is the premise sound?

• Is the primary hypothesis biologically plausible?
• Is the research ethical?
• If the results are valid, are there any disastrous logistical ethical or financial emplications to a change in practice?

Is the methodology of high quality?

• Were the inclusion/exclusion criteria appropriate?
• Was the assignment of patients to treatments randomised? If yes, then was it truly random?
• Were the study groups homogenous?
• Were the groups treated equally?
• Are there any missing patients? Is every enrolled patient accounted for? 
• Was follow-up complete? Is the drop-out rate explained? Do we know what happened to the dropouts?

Is the reporting of an appropriate quality?

• Methods describtion should be complete: the trial should be reproduceable
• Do the results have confidence intervals?
• Results should present relative and absolute effect sizes
• Is a CONSORT-style flow diagram of patient selection available?
• Discussion should contain limitation, bias and imprecision
• Funding sources and the full trial protocol should be disclosed

Are the results of the study valid?

• Was there blinding? Was blinding even possible? Was it double-blind? If not, at least were the data interpreters and statisticians blinded?
• Was there allocation concealment?
• Was there intention-to-treat analysis?
• If there were sub-groups, were they identified a priori?

What were the results?

• How large was the treatment effect?
• How precisely was the effect estimated? (i.e. what was the 95% confidence interval)

Is this study helpful for me?

• Is this applicable to my patient? i.e. would my patient have been enrolled in this study?
• Does the population studied correspond with the population to which my patient belongs?
• Were all the clinically meaningful outcomes considered?
• Does the benefit outweigh the cost and risk?