Review ArticleContinuing Education

The Value of Observer Performance Studies in Dose Optimization: A Focus on Free-Response Receiver Operating Characteristic Methods

John D. Thompson, David J. Manning and Peter Hogg
Journal of Nuclear Medicine Technology June 2013, 41 (2) 57-64; DOI: https://doi.org/10.2967/jnmt.112.116566

Article Figures & Data

Figures

  • FIGURE 1.
    FIGURE 1.

    Typical ROC curve appearances: perfect test (A), chance diagonal (B), and good test (C). TPF = true-positive fraction; FPF = false-positive fraction.

  • FIGURE 2.
    FIGURE 2.

    Potential for error in visual search task in conventional ROC methods. (A) One observer finds lesion (true) and scores confidence 4. (B) Another observer finds lesion mimic (false) and scores confidence 4. (C) Third observer finds lesion and scores confidence 1. Observers A and B are treated as equals by ROC analysis; observer C is penalized for having lower confidence. LROC and FROC account for this error type, and observer B would be penalized with false-positive (nonlesion localization).

  • FIGURE 3.
    FIGURE 3.

    Size of acceptance radius can influence classification of mark-rating pairs. If acceptance radius of low-conspicuity malignant lesion overlaps high-conspicuity benign lesion, perceptual hit could occur (25), being classified incorrectly as lesion localization.

  • FIGURE 4.
    FIGURE 4.

    Comparison of 18F (A and B) and 99mTc-methylene diphosphonate (C and D). Free-response study would have greater power than conventional ROC analysis in defining difference in diagnostic accuracy because of requirement to accurately localize lesions. (Reprinted from (45).)

Tables

  • TABLE 1

    Terminology Associated with ROC and FROC

    TermMeaning
    ConspicuityTarget visibility within structured surrounding
    CaseImage
    ModalityVariation (technique, image acquisition, imaging modality, dose)
    Lesion localizationCorrect marking of lesion
    Nonlesion localizationIncorrect marking of lesion
    TruthTrue disease status of image
    Mark-rating pairLocalization decision and confidence (rating) score
    Multireader multicaseStudy design using multiple readers to account for reader variability; results can be generalized to populations of readers and cases
  • TABLE 2

    Examples of Terminology Used in Discrete Rating Scales to Help Observers Classify Their Confidence and Use of Quasi Continuous Rating Scales

    ObserverROC confidence scale
    Discrete
     Chesters (12)0–5 (definitely not signal to definitely signal)
     Langoltz (43)0–5 (definitely benign to definitely malignant)
     Obuchowski (44)0–4 (normal to malignant)
    Quasi continuous
     Chakraborty (3)0–100 (101-point scale)
     Rockette et al. (17)0–100 (101-point scale)
     Wagner et al. (18)0–100 (101-point scale)
     Hadjiiski et al. (19)0–100 (101-point scale, rebinned to 0–5, 0–10, 0–20)
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools
Bookmark this article

Jump to section

Keywords