Elsevier

Academic Radiology

Volume 14, Issue 7, July 2007, Pages 871-876
Academic Radiology

Perspective
“Binary” and “Non-Binary” Detection Tasks: Are Current Performance Measures Optimal?

https://doi.org/10.1016/j.acra.2007.03.014Get rights and content

Rationale and Objectives

We have observed that a very large fraction of responses for several detection tasks during the performance of observer studies are in the extreme ranges of lower than 11% or higher than 89% regardless of the actual presence or absence of the abnormality in question or its subjectively rated “subtleness.” This observation raises questions regarding the validity and appropriateness of using multicategory rating scales for such detection tasks. Monte Carlo simulation of binary and multicategory ratings for these tasks demonstrate that the use of the former (binary) often results in a less biased and more precise summary index and hence may lead to a higher statistical power for determining differences between modalities.

References (20)

There are more references available in the full text version of this article.

Cited by (16)

  • Reply

    2011, Academic Radiology
  • The Importance of ROC Data

    2011, Academic Radiology
  • Is an ROC-type Response Truly Always Better Than a Binary Response in Observer Performance Studies?

    2010, Academic Radiology
    Citation Excerpt :

    The multicategory rating ROC approach allows for adjustments for the differences in latent thresholds during analysis and has been widely investigated and accepted as the preferred tool for assessing differences (if any) in performance summary indices between modes of operation. In one of our previous papers (20) we demonstrated the possible disadvantages that could result from forcing an “unnatural” multicategory rating scale onto a truly binary diagnostic task. However, there are tasks for which multicategory scales allow one to elicit information that can be useful for some purposes that could be only indirectly related to the question of interest.

View all citing articles on Scopus

Supported in part by Grants EB001694, EB002106, and EB003503 (to the University of Pittsburgh) from the National Institute for Biomedical Imaging and Bioengineering (NIBIB), National Institute of Health.

View full text