Question 14

College Answer

For a good answerthe following areas should have been addressed. Diagnostic tests may be 
correct or incorrect. The accuracy of a test is assessed by its sensitivity (true positive rate) 
and its specificity (true negative rate). The ROC provides a graphical representation of the 
trade-off between sensitivity on the y axis and specificity or 1-specificity (false positive rate) 
on the x axis. Any increase in sensitivity will be accompanied by a decrease in specificity. It 
accounts for an arbitrary cut off level made for a test or comparing two or more diagnostic 
tests. A gradient of 1 (area under the curve of 0.5) suggests that the test has no predictive 
ability. A steeper gradient has increased area under the curve (ideally > 0.75) and improved 
predictive ability. The best point on the curve is dependent on the consequences of a false 
positive compared with a false negative of the test and is usually the L elbow of the curve. 
The ROC is not affected by changes in prevalence as sensitivity and specificity are not 
dependant on prevalence. An illustration of the ROC, with correctly labelled axis and 
features was essential to answer this question. Few candidates scored well in this question, 
but of those that did they generally achieved a good score. This area is well covered in the 
recommended text Statistical Methods for Anaesthesia and Intensive Care by P Myles and T 
Gin pages 98 to 99.

Discussion

As far as college answers go, this is one of the good ones. 

In brief:

• The ROC curve is a plot of sensitivity vs. false positive rate (1-specificity)
• Sensitivity is on the y-axis, from 0% to 100%
• The ROC is for tests which produce results on a numerical scale, rather than binary (positive vs. negative results)
• The ROC curve can be used to determine the cut off point at which the sensitivity and specificity are optimal.
• All possible combinations of sensitivity and specificity that can be achieved by changing the test's cutoff value can be summarised using a single parameter, the area under the ROC curve (AUC).
  • The higher the AUC, the more accurate the test
  • An AUC of 1.0 means the test is 100% accurate (i.e. the curve is square)
  • An AUC of 0.5 (50%) means the ROC curve is a a straight diagonal line, which represents the "ideal bad test", one which is only ever accurate by pure chance.
• When comparing two tests, the more accurate test is the one with an ROC curve further to the top left corner of the graph, with a higher AUC.
• The best cutoff point for a test (which separates positive from negative values) is the point on the ROC curve which is closest to the top left corner of the graph.
• The cutoff values can be selected according to whether one wants more sensitivity or more specificity.

Information gained from the ROC curve:

• A simple graphical comparison of accuracy between diagnostic tests
• Optimal cut-off value can be derived from the shape of the curve - simplest method is to choose the point which is closest to 1,0 coordinate (i.e. the top left corner)
• The tanget at any point is the likelihood ratio for the single test value at that point
• Slope between the origin and any point along the curve is the positive likelihood ratio where the selected point is the criterion for positivity
• The slope between any  two points is the likelihood ratio for a test with a defined level, bounded by the two points.

Hanley, James A., and Barbara J. McNeil. "The meaning and use of the area under a receiver operating characteristic (ROC) curve." Radiology 143.1 (1982): 29-36.

Zweig, Mark H., and Gregory Campbell. "Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine." Clinical chemistry 39.4 (1993): 561-577.

Cook, Nancy R. "Use and misuse of the receiver operating characteristic curve in risk prediction." Circulation 115.7 (2007): 928-935.

Jones, Catherine M., and Thanos Athanasiou. "Summary receiver operating characteristic curve analysis techniques in the evaluation of diagnostic tests."The Annals of thoracic surgery 79.1 (2005): 16-20.

Metz, Charles E. "ROC methodology in radiologic imaging." Investigative radiology 21.9 (1986): 720-733.

Søreide, Kjetil. "Receiver-operating characteristic curve analysis in diagnostic, prognostic and predictive biomarker research." Journal of clinical pathology 62.1 (2009): 1-5.

Lusted, Lee B. "ROC recollected." Medical Decision Making (1984): 131-135.