Abstract

The localization receiver operating characteristic (LROC) curve is a standard method to quantify performance for the task of detecting and locating a signal. This curve is generalized to arbitrary detection/estimation tasks to give the estimation ROC (EROC) curve. For a two-alternative forced-choice study, where the observer must decide which of a pair of images has the signal and then estimate parameters pertaining to the signal, it is shown that the average value of the utility on those image pairs where the observer chooses the correct image is an estimate of the area under the EROC curve (AEROC). The ideal LROC observer is generalized to the ideal EROC observer, whose EROC curve lies above those of all other observers for the given detection/estimation task. When the utility function is nonnegative, the ideal EROC observer is shown to share many mathematical properties with the ideal observer for the pure detection task. When the utility function is concave, the ideal EROC observer makes use of the posterior mean estimator. Other estimators that arise as special cases include maximum a posteriori estimators and maximum-likelihood estimators.

© 2007 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |

  1. H. C. Gifford, R. G. Wells, and M. A. King, "A comparison of human observer LROC and numerical observer ROC tumor detection in SPECT images," IEEE Trans. Nucl. Med. 46, 1032-1037, (1999).
  2. R. G. Swensson, "Using localization data from image interpretations to improve estimates of performance accuracy," Med. Decis Making 20, 170-185 (2000).
    [CrossRef] [PubMed]
  3. P. Khurd and G. Gindi, "Decision strategies maximizing the area under the LROC curve," Proc. SPIE 5749, 150-161 (2005).
    [CrossRef]
  4. P. Khurd and G. Gindi, "Decision strategies that maximize the area under the LROC curve," IEEE Trans. Med. Imaging 24, 1626-1636, (2005).
    [CrossRef]
  5. H. H. Barrett, C. K. Abbey, and E. Clarkson, "Objective assessment of image quality III: ROC metrics, ideal observers and likelihood-generating functions," J. Opt. Soc. Am. A 15, 1520-1535 (1998).
    [CrossRef]
  6. E. Clarkson and H. H. Barrett, "Statistical decision theory and tumor detection," in Image Processing Techniques for Tumor Detection, R.Strickland, ed. (Dekker, 2002), Chap. 4.
  7. E. Clarkson and H. H. Barrett, "Approximations to ideal-observer performance on signal detection tasks," Appl. Opt. 39, 1783-1793 (2000).
    [CrossRef]
  8. E. Clarkson, "Bounds on the area under the receiver operating characteristic curve for the ideal observer," J. Opt. Soc. Am. A 19, 1963-1968, (2002).
    [CrossRef]
  9. F. Shen and E. Clarkson, "Using Fisher information to compute ideal observer performance on detection tasks," Proc. SPIE 5372, 22-30 (2004).
    [CrossRef]
  10. F. Shen and E. Clarkson, "Using Fisher information to approximate ideal observer performance on detection tasks for lumpy backgrounds," J. Opt. Soc. Am. A 23, 2406-2414 (2006).
    [CrossRef]
  11. E. Clarkson, "Estimation ROC curves and their corresponding ideal observers," Proc. SPIE , 6515,651504-1-651504-7 (2007).
  12. H.L.Van Trees, ed., Detection, Estimation, and Modulation Theory (Part I) (Academic, 1968).
  13. M. A. Kupinski, J. Hoppin, E. Clarkson, and H. H. Barrett, "Ideal observer computation using Markov-chain Monte Carlo," J. Opt. Soc. Am. A 20, 430-438 (2003).
    [CrossRef]
  14. E. Clarkson, M. A. Kupinski, and J. Hoppin, "Assessing the accuracy of estimates of the likelihood ratio," Proc. SPIE 5034, 135-143 (2003).
    [CrossRef]

2007 (1)

E. Clarkson, "Estimation ROC curves and their corresponding ideal observers," Proc. SPIE , 6515,651504-1-651504-7 (2007).

2006 (1)

2005 (2)

P. Khurd and G. Gindi, "Decision strategies maximizing the area under the LROC curve," Proc. SPIE 5749, 150-161 (2005).
[CrossRef]

P. Khurd and G. Gindi, "Decision strategies that maximize the area under the LROC curve," IEEE Trans. Med. Imaging 24, 1626-1636, (2005).
[CrossRef]

2004 (1)

F. Shen and E. Clarkson, "Using Fisher information to compute ideal observer performance on detection tasks," Proc. SPIE 5372, 22-30 (2004).
[CrossRef]

2003 (2)

E. Clarkson, M. A. Kupinski, and J. Hoppin, "Assessing the accuracy of estimates of the likelihood ratio," Proc. SPIE 5034, 135-143 (2003).
[CrossRef]

M. A. Kupinski, J. Hoppin, E. Clarkson, and H. H. Barrett, "Ideal observer computation using Markov-chain Monte Carlo," J. Opt. Soc. Am. A 20, 430-438 (2003).
[CrossRef]

2002 (1)

2000 (2)

R. G. Swensson, "Using localization data from image interpretations to improve estimates of performance accuracy," Med. Decis Making 20, 170-185 (2000).
[CrossRef] [PubMed]

E. Clarkson and H. H. Barrett, "Approximations to ideal-observer performance on signal detection tasks," Appl. Opt. 39, 1783-1793 (2000).
[CrossRef]

1998 (1)

Appl. Opt. (1)

IEEE Trans. Med. Imaging (1)

P. Khurd and G. Gindi, "Decision strategies that maximize the area under the LROC curve," IEEE Trans. Med. Imaging 24, 1626-1636, (2005).
[CrossRef]

J. Opt. Soc. Am. A (4)

Med. Decis Making (1)

R. G. Swensson, "Using localization data from image interpretations to improve estimates of performance accuracy," Med. Decis Making 20, 170-185 (2000).
[CrossRef] [PubMed]

Proc. SPIE (4)

P. Khurd and G. Gindi, "Decision strategies maximizing the area under the LROC curve," Proc. SPIE 5749, 150-161 (2005).
[CrossRef]

F. Shen and E. Clarkson, "Using Fisher information to compute ideal observer performance on detection tasks," Proc. SPIE 5372, 22-30 (2004).
[CrossRef]

E. Clarkson, "Estimation ROC curves and their corresponding ideal observers," Proc. SPIE , 6515,651504-1-651504-7 (2007).

E. Clarkson, M. A. Kupinski, and J. Hoppin, "Assessing the accuracy of estimates of the likelihood ratio," Proc. SPIE 5034, 135-143 (2003).
[CrossRef]

Other (3)

H. C. Gifford, R. G. Wells, and M. A. King, "A comparison of human observer LROC and numerical observer ROC tumor detection in SPECT images," IEEE Trans. Nucl. Med. 46, 1032-1037, (1999).

H.L.Van Trees, ed., Detection, Estimation, and Modulation Theory (Part I) (Academic, 1968).

E. Clarkson and H. H. Barrett, "Statistical decision theory and tumor detection," in Image Processing Techniques for Tumor Detection, R.Strickland, ed. (Dekker, 2002), Chap. 4.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Equations (74)

Equations on this page are rendered with MathJax. Learn more.

P FP ( T 0 ) = p ( g H 0 ) step [ T ( g ) T 0 ] d g ,
P FP ( T 0 ) = step [ T ( g ) T 0 ] g H 0 .
U TP ( T 0 ) = pr ( θ ) pr ( g θ , H 1 ) u [ θ ̂ ( g ) , θ ] step [ T ( g ) T 0 ] d g d θ ,
U TP ( T 0 ) = u [ θ ̂ ( g ) , θ ] step [ T ( g ) T 0 ] g , θ H 1 .
AEROC = U TP ( T 0 ) d P FP ( T 0 ) .
AEROC = U TP [ T ( g ) ] p ( g H 0 ) d g = U TP [ T ( g ) ] g H 0 .
AEROC = u [ θ ̂ ( g ) , θ ] step [ T ( g ) T ( g ) ] g , θ H 1 g H 0 ,
Λ ( g θ ) = pr ( g θ , H 1 ) pr ( g H 0 ) .
T I ( g ) = max θ { pr ( θ ) Λ ( g θ ) u ( θ , θ ) d θ } .
θ ̂ I ( g ) = arg max θ { pr ( θ ) Λ ( g θ ) u ( θ , θ ) d θ } .
T I ( g ) = pr ( θ ) Λ ( g θ ) u ( θ ̂ I ( g ) , θ ) d θ ,
pr ( θ ) pr ( g θ , H 1 ) u [ θ ̂ ( g ) , θ ] step [ T ( g ) T 0 ] d θ d g λ { p ( g H 0 ) step [ T ( g ) T 0 ] d g P } .
{ pr ( θ ) Λ ( g θ ) u [ θ ̂ ( g ) , θ ] d θ λ } step [ T ( g ) T 0 ] g H 0 λ P .
step [ T ( g ) T 0 ] > 0 pr ( θ ) Λ ( g θ ) u [ θ ̂ ( g ) , θ ] d θ λ > 0 .
AEROC I = T I ( g ) step [ T I ( g ) T I ( g ) ] g H 0 g H 0 .
AEROC I = T I step ( T I T I ) T I H 0 T I H 0
AUC I = Λ step ( Λ Λ ) Λ H 0 Λ H 0 ,
AEROC I = T I T I pr ( T I H 0 ) pr ( T I H 0 ) d T I d T I .
T I T I H 0 AEROC I = T I T I pr ( T I H 0 ) pr ( T I H 0 ) d T I d T I .
AEROC I = T I T I pr ( T I H 0 ) pr ( T I H 0 ) d T I d T I .
AEROC I = 1 2 max ( T I , T I ) pr ( T I H 0 ) pr ( T I H 0 ) d T I d T I .
T I T I H 0 AEROC I = 1 2 min ( T I , T I ) pr ( T I H 0 ) pr ( T I H 0 ) d T I d T I .
1 2 T I T I H 0 AEROC I .
T I ( g ) = max θ { pr ( g , θ H 1 ) pr ( g H 0 ) } ,
θ ̂ I ( g ) = arg max θ { pr ( θ ) pr ( g θ , H 1 ) } .
T I ( g ) = max θ { Λ ( g θ ) } ,
θ ̂ I ( g ) = arg max θ { pr ( g θ , H 1 ) } .
θ ̂ I ( g ) = θ pr ( θ ) pr ( g θ ) d θ
T I ( g ) = pr ( θ ) Λ ( g θ ) u ( θ ̂ I ( g ) , θ ) d θ .
pr ( g H 0 ) = 1 ( 2 π ) M det ( K ) exp [ 1 2 ( g b ) K 1 ( g b ) ] ,
pr ( g θ , H 1 ) = pr ( g s θ H 0 ) .
t I ( g ) = max θ { s θ K 1 ( g b ) 1 2 s θ K 1 s θ } .
θ ̂ I ( g ) = arg max θ { s θ K 1 ( g b ) 1 2 s θ K 1 s θ } .
AEROC I = T I T I H 0 1 4 π T I 1 2 i α T I H 0 2 d α α 2 + 1 4 .
pr 0 ( g ) = pr ( g H 0 ) ,
pr 1 ( g ) = T I ( g ) pr ( g H 0 ) T I ( g ) g H 0 .
Λ ( g ) = pr 1 ( g ) pr 0 ( g ) = T I ( g ) T I T I H 0 .
AEROC I = T I T I pr ( T I H 0 ) pr ( T I H 0 ) d T I d T I = T I T I H 0 Λ Λ pr ( Λ H 0 ) pr ( Λ H 0 ) d Λ d Λ = T I T I H 0 AUC Λ .
AUC Λ = 1 1 2 0 FPF 2 ( Λ ) d Λ ,
FPF ( Λ ) = Λ pr ( Λ H 0 ) d Λ = P FP ( T I T I H 0 Λ ) .
AEROC I = T I T I H 0 1 2 0 P FP 2 ( T ) d T .
T I T I H 0 1 2 T I 1 2 T I H 0 2 AEROC I T I T I H 0 .
exp [ μ ( β ) ] = T I β T I H 0 = T I T I H 0 β Λ β Λ H 0 .
2 μ ( 1 2 ) ln [ 2 ( T I T I H 0 AEROC I ) ] 2 μ ( 1 2 ) + [ 1 2 μ ( 1 2 ) ] 1 2 .
exp [ μ ( 1 ) ] AEROC I 1 2 + 1 2 erf { 1 4 [ μ ( 1 ) 2 μ ( 1 2 ) ] 1 2 } .
pr 0 ( g ) = pr ( g H 0 ) pr ( g 0 ) ,
pr 1 ( g ) = T I ( g α ) pr ( g H 0 ) T I ( g α ) g H 0 pr ( g α ) .
AUC Λ ( α ) 1 2 + 1 2 erf ( 1 2 F 0 1 2 α ) ,
F 0 = { d d α ln [ pr ( g α ) ] } α = 0 2 g H 0 .
Λ ( g α , θ ) = pr ( g α , θ , H 1 ) pr ( g H 0 ) ,
T I ( g α ) = pr ( θ ) Λ ( g α , θ ) u ( θ ̂ I ( g α ) , θ ) d θ .
d d α T I ( g α ) = pr ( θ ) [ d d α Λ ( g α , θ ) ] u ( θ ̂ I ( g α ) , θ ) d θ + [ d θ ̂ I ( g α ) d α ] [ θ pr ( θ ) Λ ( g α , θ ) u ( θ , θ ) d θ ] θ = θ ̂ I ( g α ) .
[ d d α T I ( g α ) ] α = 0 = pr ( θ ) [ ( d d α ) pr ( g α , θ , H 1 ) ] α = 0 u ( θ 0 , θ ) d θ pr ( g H 0 ) ,
[ d d α T I ( g α ) g H 0 ] α = 0 = [ d d α T I ( g α ) g H 0 ] α = 0 = 0 .
[ T I ( g α ) ] α = 0 = pr ( θ ) u ( θ 0 , θ ) d θ = u ¯ 0 ,
s ( g θ ) = { d d α ln [ pr ( g α , θ , H 1 ) ] } α = 0 .
[ d d α T I ( g α ) ] α = 0 = pr ( θ ) s ( g θ ) u ( θ 0 , θ ) d θ .
s ( g θ ) g H 0 = 0 .
F 0 ( θ , θ ) = s ( g θ ) s ( g θ ) g H 0 .
F 0 = u ( θ 0 , θ ) F 0 ( θ , θ ) u ( θ 0 , θ ) θ , θ .
AEROC I u ¯ 0 [ 1 2 + 1 2 erf ( 1 2 F 0 1 2 α ) ] ,
AEROC I = T I ( g ) step [ T I ( g ) T I ( g ) ] g H 0 g H 0 .
T I ( g ) = pr ( θ ) Λ ( g θ ) u ( θ ̂ I ( g ) , θ ) d θ .
T I ( g ) step [ T I ( g ) T I ( g ) ] g H 0 = u [ θ ̂ ( g ) , θ ] step [ T ( g ) T ( g ) ] g , θ H 1 .
t = ln ( T I ) .
AEROC I = exp ( t ) step ( t t ) t H 0 t H 0 .
ψ t ( ω ) = exp ( 2 π i ω t ) t H 0
AEROC I = ψ t ( ω ) [ ψ t ( ω 1 2 π i ) ] * [ 1 2 δ ( ω ) + P 1 2 π i ω ] d ω .
AEROC I = 1 2 [ ψ t ( 1 2 π i ) ] * + 1 2 π i P ψ t ( ω ) [ ψ t ( ω 1 2 π i ) ] * d ω ω .
AEROC I = 1 2 T I T I H 0 + 1 2 π i P T I 2 π i ω T I H 0 T I 2 π i ω + 1 T I H 0 * d ω ω .
AEROC I = 1 2 T I T I H 0 1 2 π i P T I i β T I H 0 T I 1 i β T I H 0 d β β .
T I x + i y T I H 0 T I x T I H 0 ,
T I x T I H 0 1 + T I T I H 0 ,
AEROC I = T I T H 0 1 4 π T I 1 2 + i α T I H 0 T I 1 2 i α T I H 0 d α α 2 + 1 4 .

Metrics