Abstract
This paper proposes a simple, but general, model for describing an observer's ability to find and report target objects in grayscale image backgrounds, which provides a basis for measuring the observer's combined detection- and-localization performance in various image-interpretation tasks. The model assumes that: 1) the observer's image detection response and first-choice of target location both depend on the “maximally suspicious” finding on the image, 2) a correct (first-choice) localization of the actual target occurs if and only if its location was regarded as the most suspicious and 3) a target's presence does not alter the suspicion engendered by any other (normal) image findings. Formalization of these assumptions relates the ROC curve, which measures the observer's ability to discriminate between target and nontarget images, to the “Localization Response” (LROC) curve, which measures the conjoint ability to detect and correctly localize the actual targets in the images. A two-parameter “binormal” version of this model provides the basis for an iterative, maximum-likelihood procedure that concurrently fits both the ROC and LROC curves, using an observer's image-ratings and target-localizations for a set of image interpretations. That same model can be extended to multiple-report (“free-response”) interpretations and to multiple-target images, provided that the observer's detection capability and criterion for reporting possible targets both remain invariant across images and across successive reports on a given image. That extension of the model leads to formulations for the so-called “Free-Response” (FROC) curve, and for a recently proposed “Alternative FROC” (AFROC) curve.
Preview
Unable to display preview. Download preview PDF.
References
K.S. Berbaum, E.A. Franken, Jr, D.D. Dorfman, T.J. Barloon: Influence of clinical history upon detection of nodules and other lesions. Investigative Radiology 23, 48–55 (1988).
K.S. Berbaum, E.A. Franken, Jr, D.D. Dorfman, S.A. Rooholamini, T.J. Barloon, F.M. Behlke, Y. Sato, C.H. Lu, G.Y.E. El-Khoury, F.D.W. Flickinger, W.J. Montgomery: Satisfaction of search in diagnostic radiology. Investigative Radiology 25, 133–140 (1990).
K.S. Berbaum, E.A. Franken, Jr, D.D. Dorfman, S.A. Rooholamini, C.E. Coffman, S.H. Cornell, A.H. Cragg, F.R. Galvin, H. Honda, S.C.S. Kao, D.A. Kimball, T.J. Ryals, W.J. Sickels, T.P. Smith: Time course of satisfaction of search. Investigative Radiology 26, 640–648 (1991).
P.C. Bunch, J.F. Hamilton, G.K. Sanderson, A.H. Simmons: A free response approach to the measurement and characterization of radiographic observer performance. Proceedings, Society of Photo-Optical and Instrumentation Engineers 154, 124–135 (1977).
A.E. Burgess: High level visual decision efficiencies. In C. Blakemore (ed.) Vision: coding and efficiency New York: Cambridge Univ. Press 1990, pp. 431–440.
A.E. Burgess, H. Ghandeharian: Visual signal detection. I: Ability to use phase information. Journal of the Optical Society of America A 1, 900–905 (1984).
A.E. Burgess, R.F. Wagner, R.J. Jennings, H.B. Barlow: Efficiency of human visual signal discrimination. Science 214, 93–94 (1981).
D.P. Chakraborty: Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Medical Physics 16, 561–568 (1989).
D.P. Chakraborty, E.A. Breatnach, M.V. Yester, B. Soto, G.T. Barnes, R.F. Fraser: Digital and conventional chest imaging: A modified ROC study of observer performance using simulated nodules. Radiology 158, 35–39 (1986).
D.P. Chakraboarty, L.H.L. Winter: Free-response methodology: Alternative analysis and a new observer-performance experiment. Radiology 174, 873–881 (1990).
H-P. Chan, K. Doi, C.J. Vyborny, R.A. Schmidt, C.E. Metz, K.L. Lam, T. Ogura, Y. Wu, H. MacMahon: Improvement in radiologists' detection of clustered microcalcifications on mammograms: The potential of computeraided diagnosis. Investigative Radiology 25, 1102–1110 (1990).
D.D. Dorfman, E. Alf: Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals — rating-method data. Journal of Mathematical Psychology 6, 487–496 (1969).
W. Feller: An introduction to probability theory and its applications: Vol. 1 New York: Wiley, 1950.
J.E. Goin, J.D. Haberman, M.K. Kinder, P.A. Lambird: Analysis of mammography: A blind interpretation of BCDDP radiographs. Radiology 148, 393–396 (1983).
D.M. Green, J.A. Swets: Signal detection theory and psychophysics New York: Wiley 1966 (reprinted Los Altos, CA: Peninsula Pub. 1988).
D.R. Grey, B.J.T. Morgan: Some aspects of ROC curve-fitting: Normal and logistic models. Journal of Mathematical Psychology 9, 128–139 (1972).
K.B. Herath, P.F. Sharp: Effects of “matched filter” smoothing as measured by receiver operating characteristic curve. Physics in Medicine and Biology 21, 442–446 (1976).
P.G. Herman, J. Drummey, R.G. Swensson, S.J. Hessel, J.P. Balikian: 350 kV chest radiography has no diagnostic advantage: a comparison with 140 kV technique. American Journal of Roentgenology 138, 485–489 (1982).
M.G. Kendall, A. Stuart: The advanced theory of statistics: Vol. 2. Inference and relationship. (4th ed.) London: Griffin, 1979.
M.F. Kijewski, R.G. Swensson, P.F. Judy: Analysis of rating data from multiple-alternative tasks. Journal of Mathematical Psychology 33, 428–451 (1989).
K.L. Kundel: Predictive value and threshold detectability of lung tumors. Radiology 139, 25–29 (1981).
N.A. Macmillan, C.D. Creelman: Detection theory: a user's guide. New York: Cambridge Univ. Press 1991.
T.F. Meaney, U. Raudkivi, W.J. McIntyre, J.H. Gallagher, J.R. Haaga, T.R. Havrilla, N.E. Reich: Detection of low-contrast lesions in computed body tomography: An experimental study of simulted lesions. Radiology 134, 149–154 (1980).
C.E. Metz: Some practical issues of experimental design and data analysis in radiological ROC studies. Investigative Radiology 24, 234–245 (1989).
C.E. Metz, P.L. Wang, H.B. Kronman: A new approach for testing the significance of differences between ROC curves measured from correlated data. In: F. Deconink (ed.), Information processing in medical imaging The Hague: Nijhoff, 1984, pp. 432–435.
C.E. Metz, S.J. Starr, L.B. Lusted, K. Rossmann: Progress in evaluation of human observer visual detection perfromance using the ROC curve approach. In: C Raynaud, A.E. Todd-Pokropek (eds.) Information Processing in scintigraphy Orsay, France: CEA, 1975, pp. 420–436.
C.R. Mitchel, J.A. Sorenson: Application of the Bunch transform in clinical FROC nodule detection studies. In W.W. Peppler, A. Alter (eds.) Proceedings of chest imaging conference '87 Madison, WI: Medical Physics Pub Co 1988, pp. 356–370.
B. Moulden, F. Kingdom: Effect of pixel width, display width, and number of alternative signal locations on the detection of a simple vertical-line signal in visual noise. Perception and Psychophysics 43, 592–598 (1988).
L.T. Niklason, N.M. Hickey, D.P. Chakraborty, E. A. Sabbagh, M.Y. Yester, R.G. Fraser, G.T. Barnes: Simulated pulmonary nodules: Detection with dual-energy digital versus conventional radiology. Radiology 160, 589–594 (1986).
J.C. Ogilvie, C.D. Creelman: Maximum-likelihood estimation of receiver operating characteristic curve parameters. Journal of Mathematical Psychology 5, 377–391 (1968).
S.E. Seltzer, R.G. Swensson, R.D. Nawfel, J.F. Lentini, I. Kazda, P.F. Judy: Visualization and detection-localization on computed tomographic images. Investigative Radiology 26, 285–291 (1991).
S.J. Starr, C.E. Metz, L.B. Lusted, D.J. Goodenough: Visual detection and localization of radiographic images. Radiology 116, 533–538 (1975).
S.J. Starr, C.E. Metz, L.B. Lusted: Letter to the editor: Comments on the generalization of receiver operating characteristic analysis to detection and localization tasks. Physics in Medicine and Biology 22, 376–379 (1977).
R.G. Swensson: Observer performance tests in chest imaging. In W.W. Peppler, A. Alter (eds.) Proceedings of chest imaging conference '87 Madison, Wisc.: Medical Physics Pub Co., 1988, pp. 337–355.
R.G. Swensson, K.H. Chan: Detection and localization of simulated pulmonary nodules on chest films. Radiology 181(P), 278 (1991).
R.G. Swensson, P.F. Judy: Detction of noisy visual targets: models for the effects of spatial uncertainty and signal-to-noise ratio. Perception and Psychophysics 29, 521–534 (1981).
R.G. Swensson, S.J. Hessel, P.G. Herman: Radiographic interpretation with and without search: visual search aids the recognition of chest pathology. Investigative Radiology 17, 145–151 (1982).
R.G. Swensson, S.J. Hessel, P.G. Herman: The value of searching films without specific preconceptions. Investigative Radiology 20, 100–107 (1985).
R.G. Swensson, S.J. Hessel, P.G. Herman: Different methods of analyzing radiographic interpretations do not alter the value of searching films (reply to editorial comments). Investigative Radiology 20, 111–114 (1985).
R.G. Swensson, G.H. Theodore: Search and nonsearch protocols for radiographic consultation. Radiology 177, 851–856 (1990).
J.A. Swets: On the supposed value of searching films (editorial comment). Investigative Radiology 20, 108–109 (1985).
J.A. Swets: On searching films and begging the question (letter to the editor). Investigative Radiology 20, 115–116 (1985).
J.A. Swets, R.M. Pickett: Evaluation of diagnostic systems: Methods from signal detection theory New York: Academic Press, 1982.
W.J. Tuddenham: Visual search, image organization, and reader error in roentgen diagnosis: Studies of the psychophysiology of roentgen image perception. Radiology 78, 694–704 (1962).
R.F. Wagner: Decision theory and the detail signal-to-noise ratio of Otto Shade. Photographic Science and Engineering 22, 41–46 (1978).
R.F. Wagner, D.G. Brown: Unified SNR analysis of medical imaging systems. Physics in Medicine and Biology 30, 489–518 (1985).
T.D. Wickins: Maximum-likelihood estimation of a multivariate gaussian rating model with excluded data. Journal of Mathematical Psychology 36, 213–234 (1992).
A. Yoshimura, M.L. Giger, K. Doi, H. MacMahon, S.M. Montner: Computerized Scheme for the detection of pulmonary nodules: a nonlinear filtering technique. Investigative Radiology 27, 124–129 (1992).
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1993 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Swensson, R.G. (1993). Measuring detection and localization performance. In: Barrett, H.H., Gmitro, A.F. (eds) Information Processing in Medical Imaging. IPMI 1993. Lecture Notes in Computer Science, vol 687. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0013809
Download citation
DOI: https://doi.org/10.1007/BFb0013809
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-56800-1
Online ISBN: 978-3-540-47742-6
eBook Packages: Springer Book Archive