A Brief History of Free-Response Receiver Operating Characteristic Paradigm Data Analysis

doi:10.1016/j.acra.2013.03.001

Academic Radiology

Volume 20, Issue 7, July 2013, Pages 915-919

https://doi.org/10.1016/j.acra.2013.03.001 Get rights and content

In the receiver operating characteristic paradigm the observer assigns a single rating to each image and the location of the perceived abnormality, if any, is ignored. In the free-response receiver operating characteristic paradigm the observer is free to mark and rate as many suspicious regions as are considered clinically reportable. Credit for a correct localization is given only if a mark is sufficiently close to an actual lesion; otherwise, the observer's mark is scored as a location-level false positive. Until fairly recently there existed no accepted method for analyzing the resulting relatively unstructured data containing random numbers of mark-rating pairs per image. This report reviews the history of work in this field, which has now spanned more than five decades. It introduces terminology used to describe the paradigm, proposed measures of performance (figures of merit), ways of visualizing the data (operating characteristics), and software for analyzing free-response receiver operating characteristic studies.

Section snippets

FROC data: Mark-Rating pairs

The mark is the location of the suspicious region and the rating is the confidence level that the region contains a lesion. The data analyst decides whether a mark is close enough to a real lesion to qualify as lesion localization (LL)—a location-level “true positive”—and otherwise the mark is classified as non-lesion localization (NL)—a location-level “false positive.” The quotes are intended to emphasize the confusion that can arise if one uses terminology developed for image-level ROC

Operating Characteristics and Figures-of-Merit

Data analysis starts with the selection of a figure-of-merit (FOM) and a procedure for estimating it from the observed collection of NLs and LLs, each with an associated rating (the rating does not have to be a discrete integer). A valid FOM rewards the observer for correct decisions and penalizes for incorrect decisions. Finding a suitable FOM usually starts with a way of visualizing the data. For example, the ROC curve suggests the area under the curve as a suitable FOM for ROC data. Bunch

FOMs

The AFROC plot (LLF versus FPF) is amenable to defining a nonparametric FOM. One compares all pairing of LLs and highest rated NLs on normal images. If the LL rating is greater, one cumulates unity; if they are equal, one cumulates 0.5, and at the end of the process, one divides by the total number of comparisons. Except for some nuances, described in a document available at www.devchakraborty.com, this is the FOM used in jackknife alternative FROC (JAFROC) analysis (23), currently the most

Discussion

This report has summarized the history of research in free-response data analysis. The history is essentially that of finding a good FOM and a method for testing the significance of the difference between two FOMs. The significance testing methodology has benefited immensely from work by DBM (25) and subsequent refinements by Hillis et al 26, 31, 32, 33, 34, 35.

Not discussed in this report are the modeling advances that have taken place in connection with FROC research 13, 14, 15, 36, 37, 38.

Acknowledgments

The author is grateful to Ms Kun-Wan Chen for proofing the manuscript. This work was supported by grants from the Department of Health and Human Services, National Institutes of Health (R01-EB005243 and R01-EB008688).

References (43)

D.P. Chakraborty et al.
Spatial localization accuracy of radiologists in free-response studies: inferring perceptual FROC curves from mark-rating data
Acad Radiol
(2007)
D.P. Chakraborty
New developments in observer performance methodology in medical imaging
Semin Nucl Med
(2011)
S.L. Hillis et al.
Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis
Acad Radiol
(2008)
S.V. Beiden et al.
Components-of variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis
Acad Radiol
(2000)
D.P. Chakraborty
Validation and statistical power comparison of methods for analyzing free-response observer performance studies
Acad Radiol
(2008)
S.L. Hillis et al.
Power estimation for the Dorfman-Berbaum-Metz method
Acad Radiol
(2004)
S.L. Hillis et al.
Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification
Acad Radiol
(2005)
D.P. Chakraborty
Recent developments in imaging system assessment methodology, FROC analysis and the search model
Nucl Instr and Meth A: Accelerators, Spectrometers, Detectors and Associated Equipment
(2011)
N.A. Obuchowski et al.
data analysis for detection and localization of multiple abnormalities with application to mammography
Acad Radiol
(2000)
C.M. Rutter
Bootstrap estimation of diagnostic accuracy with patient-clustered data
Acad Radiol
(2000)

J.P. Egan et al.

Operating characteristics, signal detectability and the method of free response

J Acoust Soc Am

(1961)

C.J. D'Orsi et al.

Breast Imaging Reporting and Data System: ACR BI-RADS – Breast Imaging Atlas

(2003)

H. Miller

The FROC curve: a representation of the observer's performance for the method of free response

J Acoust Soc Am

(1969)

P.C. Bunch et al.

A free-response approach to the measurement and characterization of radiographic-observer performance

J Appl Photogr Eng

(1978)

D.P. Chakraborty et al.

Digital and conventional chest imaging: a modified ROC study of observer performance using simulated nodules

Radiology

(1986)

L.T. Niklason et al.

Simulated pulmonary nodules: detection with dual-energy digital versus conventional radiography

Radiology

(1986)

M. Kallergi et al.

Evaluating the performance of detection algorithms in digital mammography

Med Phys

(1999)

T.M. Haygood et al.

On the choice of acceptance radius in free-response observer performance studies

Br J Radiol

(2013)

D.P. Chakraborty et al.

Free-response methodology: alternate analysis and a new observer-performance experiment

Radiology

(1990)

D.P. Chakraborty

Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data

Med Phys

(1989)

R.G. Swensson

Unified measurement of observer performance in detecting and localizing target objects on images

Med Phys

(1996)

Cited by (56)

A deep-learning model for intracranial aneurysm detection on CT angiography images in China: a stepwise, multicentre, early-stage clinical validation study
2024, The Lancet Digital Health
Artificial intelligence (AI) models in real-world implementation are scarce. Our study aimed to develop a CT angiography (CTA)-based AI model for intracranial aneurysm detection, assess how it helps clinicians improve diagnostic performance, and validate its application in real-world clinical implementation.
We developed a deep-learning model using 16 546 head and neck CTA examination images from 14 517 patients at eight Chinese hospitals. Using an adapted, stepwise implementation and evaluation, 120 certified clinicians from 15 geographically different hospitals were recruited. Initially, the AI model was externally validated with images of 900 digital subtraction angiography-verified CTA cases (examinations) and compared with the performance of 24 clinicians who each viewed 300 of these cases (stage 1). Next, as a further external validation a multi-reader multi-case study enrolled 48 clinicians to individually review 298 digital subtraction angiography-verified CTA cases (stage 2). The clinicians reviewed each CTA examination twice (ie, with and without the AI model), separated by a 4-week washout period. Then, a randomised open-label comparison study enrolled 48 clinicians to assess the acceptance and performance of this AI model (stage 3). Finally, the model was prospectively deployed and validated in 1562 real-world clinical CTA cases.
The AI model in the internal dataset achieved a patient-level diagnostic sensitivity of 0·957 (95% CI 0·939–0·971) and a higher patient-level diagnostic sensitivity than clinicians (0·943 [0·921–0·961] vs 0·658 [0·644–0·672]; p<0·0001) in the external dataset. In the multi-reader multi-case study, the AI-assisted strategy improved clinicians' diagnostic performance both on a per-patient basis (the area under the receiver operating characteristic curves [AUCs]; 0·795 [0·761–0·830] without AI vs 0·878 [0·850–0·906] with AI; p<0·0001) and a per-aneurysm basis (the area under the weighted alternative free-response receiver operating characteristic curves; 0·765 [0·732–0·799] vs 0·865 [0·839–0·891]; p<0·0001). Reading time decreased with the aid of the AI model (87·5 s vs 82·7 s, p<0·0001). In the randomised open-label comparison study, clinicians in the AI-assisted group had a high acceptance of the AI model (92·6% adoption rate), and a higher AUC when compared with the control group (0·858 [95% CI 0·850–0·866] vs 0·789 [0·780–0·799]; p<0·0001). In the prospective study, the AI model had a 0·51% (8/1570) error rate due to poor-quality CTA images and recognition failure. The model had a high negative predictive value of 0·998 (0·994–1·000) and significantly improved the diagnostic performance of clinicians; AUC improved from 0·787 (95% CI 0·766–0·808) to 0·909 (0·894–0·923; p<0·0001) and patient-level sensitivity improved from 0·590 (0·511–0·666) to 0·825 (0·759–0·880; p<0·0001).
This AI model demonstrated strong clinical potential for intracranial aneurysm detection with improved clinician diagnostic performance, high acceptance, and practical implementation in real-world clinical cases.
National Natural Science Foundation of China.
For the Chinese translation of the abstract see Supplementary Materials section.
Periapical Radiography versus Cone Beam Computed Tomography in Endodontic Disease Detection: A Free-response, Factorial Study
2023, Journal of Endodontics
To assess and compare reader performance in interpreting digital periapical (PA) radiography and cone beam computed tomography (CBCT) in endodontic disease detection, using a free-response, factorial model.
A reader performance study of 2 image test sets was undertaken using a factorial, free-response design, accounting for the independent variables: case type, case severity, reader type, and imaging modality. Twenty-two readers interpreted 60 PA and 60 CBCT images divided into 5 categories: diseased–subtle, diseased–moderate, diseased–obvious, nondiseased–subtle, and nondiseased–obvious. Lesion localization fraction, specificity, false positive (FP) marks, and the weighted alternative free-response receiver operating characteristic figure of merit were calculated.
CBCT had greater specificity than PA in the obvious nondiseased cases (P = .01) and no significant difference in the subtle nondiseased category. Weighted alternative free-response receiver operating characteristic values were higher for PA than CBCT in the subtle diseased (P = .02) and moderate diseased (P = .01) groups with no significant difference between in the obvious diseased groups. CBCT had higher mean FPs than PA (P < .05) in subtle diseased cases. Mean lesion localization fraction in the moderate diseased group was higher in PA than CBCT (P = .003). No relationships were found between clinical experience and all diagnostic performance measures, except for in the obvious diseased CBCT group, where increasing experience was associated mean FP marks (P = .04).
Reader performance in the detection of endodontic disease is better with PA radiography than CBCT. Clinical experience does not impact upon the accuracy of interpretation of both PA radiography and CBCT.
A Magnified Adaptive Feature Pyramid Network for automatic microaneurysms detection
2021, Computers in Biology and Medicine
Citation Excerpt :
The simplified CNN structure alleviated the local feature loss during the convolution and pooling processes. Their method outperformed most of the other methods in metrics of the free response operating characteristics (FROC) [27,28]. This method has achieved certain success and uses CNN to calculate the probability map for region proposal.
Diabetic retinopathy (DR), as an important complication of diabetes, is the primary cause of blindness in adults. Automatic DR detection poses a challenge which is crucial for early DR screening. Currently, the vast majority of DR is diagnosed through fundus images, where the microaneurysm (MA) has been widely used as the most distinguishable marker. Research works on automatic DR detection have traditionally utilized manually designed operators, while a few recent researchers have explored deep learning techniques for this topic. But due to issues such as the extremely small size of microaneurysms, low resolution of fundus pictures, and insufficient imaging depth, the DR detection problem is quite challenging and remains unsolved. To address these issues, this research proposes a new deep learning model (Magnified Adaptive Feature Pyramid Network, MAFP-Net) for DR detection, which conducts super-resolution on low quality fundus images and integrates an improved feature pyramid structure while utilizing a standard two-stage detection network as the backbone. Our proposed detection model needs no pre-segmented patches to train the CNN network. When tested on the E-ophtha-MA dataset, the sensitivity value of our method reached as high as 83.5% at false positives per image (FPI) of 8 and the F1 value achieved 0.676, exceeding all those of the state-of-the-art algorithms as well as the human performance of experienced physicians. Similar results were achieved on another public dataset of IDRiD.
Evaluation of an automatic method for detection of defects in linear and curvilinear ultrasound transducers
2021, Physica Medica
The high incidence of defective ultrasound transducers in clinical practice has been shown in several studies. Recently, a novel method using only stored images for automatic detection of defective transducers was presented. The method makes it possible to remotely monitor many transducers at the same time and send a notification when a defective transducer is found. The purpose of the present study was to evaluate the novel method and assess how well it performs when compared to an established method as reference.
To evaluate the novel method, in-air images were collected from 81 transducers in radiologic departments in nine hospitals. Two observers assessed the in-air images and marked the defects. Receiver operating characteristic (ROC)- and alternative free response receiver operating characteristic (AFROC)-curves and their figures of merit (FOM) were calculated for the novel method, using marked defects in the in-air images as reference truth.
The area under the ROC curve was 0.88 (SD 0.06), and the AFROC FOM was 0.71 (SE 0.07).
The result shows that the novel method has a good agreement with the in-air method for detecting defects in ultrasound systems. This indicates that the novel method could be a complement to the normal quality control for early, and automatic detection of defects.
Pulmonary nodule detection on chest radiographs using balanced convolutional neural network and classic candidate detection
2020, Artificial Intelligence in Medicine
Citation Excerpt :
First, we evaluate and analyze the overall performance of the CADe scheme for Databases A–C. A free-response receiver operating characteristic (FROC) curve is a tool that simultaneously characterizes the performance of a free-response system at all decision thresholds [38]. The CADe system does not simply detect whether medical images contain an abnormal organization, it detects the numbers and locations of the sites.
Computer-aided detection (CADe) systems play a crucial role in pulmonary nodule detection via chest radiographs (CXRs). A two-stage CADe scheme usually includes nodule candidate detection and false positive reduction. A pure deep learning model, such as faster region convolutional neural network (faster R-CNN), has been successfully applied for nodule candidate detection via computed tomography (CT). The model is yet to achieve a satisfactory performance in CXR, because the size of the CXR is relatively large and the nodule in CXR has been obscured by structures such as ribs. In contrast, the CNN has proved effective for false positive reduction compared to the shallow method. In this paper, we developed a CADe scheme using the balanced CNN with classic candidate detection. First, the scheme applied a multi-segment active shape model to accurately segment pulmonary parenchyma. The grayscale morphological enhancement technique was then used to improve the conspicuity of the nodule structure. Based on the nodule enhancement image, 200 nodule candidates were selected and a region of interest (ROI) was cropped for each. Nodules in CXR exhibit a large variation in density, and rib crossing and vessel tissue usually present similar features to the nodule. Compared to the original ROI image, the nodule enhancement ROI image has potential discriminative features from false positive reduction. In this study, the nodule enhancement ROI image, corresponding segmentation result, and original ROI image were encoded into a red–green–blue (RGB) color image instead of the duplicated original ROI image as input of the CNN (GoogLeNet) for false positive reduction. With the Japanese Society of Radiological Technology database, the CADe scheme achieved high performance of the published literatures (a sensitivity of 91.4 % and 97.1 %, with 2.0 false positives per image (FPs/image) and 5.0 FPs/image, respectively) for nodule cases.
Unsupervised domain adaptation with adversarial learning for mass detection in mammogram
2020, Neurocomputing
Many medical image datasets have been collected without proper annotations for deep learning training. In this paper, we propose a novel unsupervised domain adaptation framework with adversarial learning to minimize the annotation efforts. Our framework employs a task specific network, i.e., fully convolutional network (FCN), for spatial density prediction. Moreover, we employ a domain discriminator, in which adversarial learning is adopted to align the less-annotated target domain features with the well-annotated source domain features in the feature space. We further propose a novel training strategy for the adversarial learning by coupling data from source and target domains and alternating the subnet updates. We employ the public CBIS-DDSM dataset as the source domain, and perform two sets of experiments on two target domains (i.e., the public INbreast dataset and a self-collected dataset), respectively. Experimental results suggest consistent and comparable performance improvement over the state-of-the-art methods. Our proposed training strategy is also proved to converge much faster.

View all citing articles on Scopus

View full text

CommunicationA Brief History of Free-Response Receiver Operating Characteristic Paradigm Data Analysis

Section snippets

FROC data: Mark-Rating pairs

Operating Characteristics and Figures-of-Merit

FOMs

Discussion

Acknowledgments

Acad Radiol

Semin Nucl Med

Acad Radiol

Acad Radiol

Acad Radiol

Acad Radiol

Acad Radiol

Nucl Instr and Meth A: Accelerators, Spectrometers, Detectors and Associated Equipment

Acad Radiol

Acad Radiol

Operating characteristics, signal detectability and the method of free response

J Acoust Soc Am

Breast Imaging Reporting and Data System: ACR BI-RADS – Breast Imaging Atlas

The FROC curve: a representation of the observer's performance for the method of free response

J Acoust Soc Am

A free-response approach to the measurement and characterization of radiographic-observer performance

J Appl Photogr Eng

Digital and conventional chest imaging: a modified ROC study of observer performance using simulated nodules

Radiology

Simulated pulmonary nodules: detection with dual-energy digital versus conventional radiography

Radiology

Evaluating the performance of detection algorithms in digital mammography

Med Phys

On the choice of acceptance radius in free-response observer performance studies

Br J Radiol

Free-response methodology: alternate analysis and a new observer-performance experiment

Radiology

Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data

Med Phys

Unified measurement of observer performance in detecting and localizing target objects on images

Med Phys

Communication
A Brief History of Free-Response Receiver Operating Characteristic Paradigm Data Analysis