Article Figures & Data
Figures
- FIGURE 1.
Typical ROC curve appearances: perfect test (A), chance diagonal (B), and good test (C). TPF = true-positive fraction; FPF = false-positive fraction.
- FIGURE 2.
Potential for error in visual search task in conventional ROC methods. (A) One observer finds lesion (true) and scores confidence 4. (B) Another observer finds lesion mimic (false) and scores confidence 4. (C) Third observer finds lesion and scores confidence 1. Observers A and B are treated as equals by ROC analysis; observer C is penalized for having lower confidence. LROC and FROC account for this error type, and observer B would be penalized with false-positive (nonlesion localization).
- FIGURE 3.
Size of acceptance radius can influence classification of mark-rating pairs. If acceptance radius of low-conspicuity malignant lesion overlaps high-conspicuity benign lesion, perceptual hit could occur (25), being classified incorrectly as lesion localization.
- FIGURE 4.
Comparison of 18F (A and B) and 99mTc-methylene diphosphonate (C and D). Free-response study would have greater power than conventional ROC analysis in defining difference in diagnostic accuracy because of requirement to accurately localize lesions. (Reprinted from (45).)
Tables
Term Meaning Conspicuity Target visibility within structured surrounding Case Image Modality Variation (technique, image acquisition, imaging modality, dose) Lesion localization Correct marking of lesion Nonlesion localization Incorrect marking of lesion Truth True disease status of image Mark-rating pair Localization decision and confidence (rating) score Multireader multicase Study design using multiple readers to account for reader variability; results can be generalized to populations of readers and cases - TABLE 2
Examples of Terminology Used in Discrete Rating Scales to Help Observers Classify Their Confidence and Use of Quasi Continuous Rating Scales
Observer ROC confidence scale Discrete Chesters (12) 0–5 (definitely not signal to definitely signal) Langoltz (43) 0–5 (definitely benign to definitely malignant) Obuchowski (44) 0–4 (normal to malignant) Quasi continuous Chakraborty (3) 0–100 (101-point scale) Rockette et al. (17) 0–100 (101-point scale) Wagner et al. (18) 0–100 (101-point scale) Hadjiiski et al. (19) 0–100 (101-point scale, rebinned to 0–5, 0–10, 0–20)
Jump to section
- Article
- Abstract
- OBSERVER PERFORMANCE FOR ASSESSMENT OF SYSTEM PERFORMANCE
- ROC ANALYSIS
- ROC CURVES AND AUC
- ROC: CONFIDENCE SCALES
- THE VISUAL SEARCH: DEVELOPMENT OF LOCATION-BASED ANALYSIS
- LROC ANALYSIS
- FROC ANALYSIS
- ACCEPTANCE RADII
- ANALYZING FROC DATA
- STATISTICAL POWER FOR FROC STUDIES
- FROC STUDY EXAMPLE: DOSE REDUCTION IN HYBRID IMAGING
- FROC STUDY EXAMPLE: MODALITY COMPARISON
- DRAWBACKS OF ROC AND FROC
- ALTERNATIVE MEASURES OF OBSERVER PERFORMANCE
- CONCLUSION
- DISCLOSURE
- Footnotes
- REFERENCES
- Figures & Data
- Info & Metrics