Question 2c

An article appears reporting the positive effects of a new agent in a trial of 50 patients with septic shock.

(c) What criteria will you use to assess the validity of this article to your ICU?

[Click here to toggle visibility of the answers]

College Answer

The criteria for assessment of such an article include:

•    Is the trials design valid and powered to achieve a result? It seems doubtful in this case but a large effect in a specific group may be detected.

•    Was the hypothesis based on valid evidence?

•    Were all the entered patients accounted for?

•    Were the groups equivalent after randomisation?

•    Was there proper blinding of study personnel?

•    Apart from the experimental intervention were the groups treated equivalently?

•    Was the statistical analysis appropriate?

•    How large was the treatment affect?

•    Can the results be applied to my patients?

Discussion

Though not owrd-for-word identical, this question closely resembles Question 8 from the second paper of 2012, as well as Question 8 from the first paper of 2004. It discusses the assessment of the validity of a randomised controlled trial, which is discussed in greater detail elsewhere.

The answer is reproduced below, to simplify revision and damage SEO:

Is the premise sound?

  • Is the primary hypothesis biologically plausible?
  • Is the research ethical?
  • If the results are valid, are there any disastrous logistical ethical or financial emplications to a change in practice?

Is the methodology of high quality?

  • Were the inclusion/exclusion criteria appropriate?
  • Was the assignment of patients to treatments randomised? If yes, then was it truly random?
  • Were the study groups homogenous?
  • Were the groups treated equally?
  • Are there any missing patients? Is every enrolled patient accounted for? 
  • Was follow-up complete? Is the drop-out rate explained? Do we know what happened to the dropouts?

Is the reporting of an appropriate quality?

  • Methods describtion should be complete: the trial should be reproduceable
  • Do the results have confidence intervals?
  • Results should present relative and absolute effect sizes
  • Is a CONSORT-style flow diagram of patient selection available?
  • Discussion should contain limitation, bias and imprecision
  • Funding sources and the full trial protocol should be disclosed

Are the results of the study valid?

  • Was there blinding? Was blinding even possible? Was it double-blind? If not, at least were the data interpreters and statisticians blinded?
  • Was there allocation concealment?
  • Was there intention-to-treat analysis?
  • If there were sub-groups, were they identified a priori?

What were the results?

  • How large was the treatment effect?
  • How precisely was the effect estimated? (i.e. what was the 95% confidence interval)

Is this study helpful for me?

  • Is this applicable to my patient? i.e. would my patient have been enrolled in this study?
  • Does the population studied correspond with the population to which my patient belongs?
  • Were all the clinically meaningful outcomes considered?
  • Does the benefit outweigh the cost and risk?