Abstract

The Bayesian ideal observer is optimal among all observers and sets an absolute upper bound for the performance of any observer in classification tasks [ Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, 1968). ]. Therefore, the ideal observer should be used for objective image quality assessment whenever possible. However, computation of ideal-observer performance is difficult in practice because this observer requires the full description of unknown, statistical properties of high-dimensional, complex data arising in real life problems. Previously, Markov-chain Monte Carlo (MCMC) methods were developed by Kupinski et al. [J. Opt. Soc. Am. A 20, 430(2003) ] and by Park et al. [J. Opt. Soc. Am. A 24, B136 (2007) and IEEE Trans. Med. Imaging 28, 657 (2009) ] to estimate the performance of the ideal observer and the channelized ideal observer (CIO), respectively, in classification tasks involving non-Gaussian random backgrounds. However, both algorithms had the disadvantage of long computation times. We propose a fast MCMC for real-time estimation of the likelihood ratio for the CIO. Our simulation results show that our method has the potential to speed up ideal-observer performance in tasks involving complex data when efficient channels are used for the CIO.

© 2009 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, 1968).
  2. M. A. Kupinski, J. W. Hoppin, E. Clarkson, and H. H. Barrett, “Ideal observer computation using Markov-chain Monte Carlo,” J. Opt. Soc. Am. A 20, 430-438 (2003).
    [CrossRef]
  3. X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535-1543 (2008).
    [CrossRef] [PubMed]
  4. C. K. Abbey and J. M. Boone, “An ideal observer for a model of x-ray imaging in breast parenchymal tissue,” in Digital Mammography, E.A.Krupinski, ed., Vol. 5116 of Lecture Notes in Computer Science, (Springer-Verlag, 2008), pp. 393-400.
    [CrossRef]
  5. S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre-Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136-B150 (2007).
    [CrossRef]
  6. B. D. Gallas and H. H. Barrett, “Validating the use of channels to estimate the ideal linear observer,” J. Opt. Soc. Am. A 20, 1725-1738 (2003).
    [CrossRef]
  7. S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657-667 (2009).
    [CrossRef] [PubMed]
  8. J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009).
    [CrossRef]
  9. S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008).
    [CrossRef]
  10. C. P. Robert and G. Casella, Monte Carlo Statistical Methods, 2nd ed. (Springer,2004).
  11. H. H. Barrett, C. K. Abbey, and E. Clarkson, “Objective assessment of image quality III: ROC metrics, ideal observers, and likelihood-generating functions,” J. Opt. Soc. Am. A 15, 1520-1535 (1998).
    [CrossRef]
  12. E. Clarkson, “Bounds on the area under the receiver operating characteristic curve for the ideal observer,” J. Opt. Soc. Am. A 19, 1963-1968 (2001).
    [CrossRef]
  13. J. P. Rolland and H. H. Barrett, “Effect of random background inhomogeneity on observer detection performance,” J. Opt. Soc. Am. A 9, 649-658 (1992).
    [CrossRef] [PubMed]
  14. E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic development of the MRMC method,” Acad. Radiol. 13(11), 1410-1421 (2006).
    [CrossRef] [PubMed]
  15. B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353-362 (2006).
    [CrossRef] [PubMed]

2009

S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657-667 (2009).
[CrossRef] [PubMed]

J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009).
[CrossRef]

2008

S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008).
[CrossRef]

X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535-1543 (2008).
[CrossRef] [PubMed]

2007

2006

E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic development of the MRMC method,” Acad. Radiol. 13(11), 1410-1421 (2006).
[CrossRef] [PubMed]

B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353-362 (2006).
[CrossRef] [PubMed]

2003

2001

1998

1992

Abbey, C. K.

H. H. Barrett, C. K. Abbey, and E. Clarkson, “Objective assessment of image quality III: ROC metrics, ideal observers, and likelihood-generating functions,” J. Opt. Soc. Am. A 15, 1520-1535 (1998).
[CrossRef]

C. K. Abbey and J. M. Boone, “An ideal observer for a model of x-ray imaging in breast parenchymal tissue,” in Digital Mammography, E.A.Krupinski, ed., Vol. 5116 of Lecture Notes in Computer Science, (Springer-Verlag, 2008), pp. 393-400.
[CrossRef]

Barrett, H. H.

Boone, J. M.

C. K. Abbey and J. M. Boone, “An ideal observer for a model of x-ray imaging in breast parenchymal tissue,” in Digital Mammography, E.A.Krupinski, ed., Vol. 5116 of Lecture Notes in Computer Science, (Springer-Verlag, 2008), pp. 393-400.
[CrossRef]

Caffo, B. S.

X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535-1543 (2008).
[CrossRef] [PubMed]

Casella, G.

C. P. Robert and G. Casella, Monte Carlo Statistical Methods, 2nd ed. (Springer,2004).

Clarkson, E.

Frey, E.

X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535-1543 (2008).
[CrossRef] [PubMed]

Gallas, B. D.

He, X.

X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535-1543 (2008).
[CrossRef] [PubMed]

Hoppin, J. W.

Kupinski, M. A.

Myers, K. J.

S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657-667 (2009).
[CrossRef] [PubMed]

J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009).
[CrossRef]

S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre-Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136-B150 (2007).
[CrossRef]

Park, S.

S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657-667 (2009).
[CrossRef] [PubMed]

J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009).
[CrossRef]

S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008).
[CrossRef]

S. Park, H. H. Barrett, E. Clarkson, M. A. Kupinski, and K. J. Myers, “A channelized-ideal observer using Laguerre-Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal,” J. Opt. Soc. Am. A 24, B136-B150 (2007).
[CrossRef]

Robert, C. P.

C. P. Robert and G. Casella, Monte Carlo Statistical Methods, 2nd ed. (Springer,2004).

Rolland, J. P.

Van Trees, H. L.

H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, 1968).

Witten, J.

J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009).
[CrossRef]

Witten, J. M.

S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657-667 (2009).
[CrossRef] [PubMed]

Acad. Radiol.

E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic development of the MRMC method,” Acad. Radiol. 13(11), 1410-1421 (2006).
[CrossRef] [PubMed]

B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353-362 (2006).
[CrossRef] [PubMed]

IEEE Trans. Med. Imaging

X. He, B. S. Caffo, and E. Frey, “Toward realistic and practical ideal observer estimation for the optimization of medical imaging systems,” IEEE Trans. Med. Imaging 27, 1535-1543 (2008).
[CrossRef] [PubMed]

S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657-667 (2009).
[CrossRef] [PubMed]

J. Opt. Soc. Am. A

Proc. SPIE

J. Witten, S. Park, and K. J. Myers, “Using partial least square to compute efficient channels for the Bayesian ideal observer,” Proc. SPIE 7263, 72630Q (2009).
[CrossRef]

S. Park and E. Clarkson, “Markov-chain Monte Carlo for the performance of a channelized-ideal observer in detection tasks with non-Gaussian lumpy backgrounds,” Proc. SPIE 6917, 69170T (2008).
[CrossRef]

Other

C. P. Robert and G. Casella, Monte Carlo Statistical Methods, 2nd ed. (Springer,2004).

H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, 1968).

C. K. Abbey and J. M. Boone, “An ideal observer for a model of x-ray imaging in breast parenchymal tissue,” in Digital Mammography, E.A.Krupinski, ed., Vol. 5116 of Lecture Notes in Computer Science, (Springer-Verlag, 2008), pp. 393-400.
[CrossRef]

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (17)

Fig. 1
Fig. 1

Top, overall histogram of the fifth element, v 5 , of the channelized background generated using 5000 random lumpy backgrounds of N ¯ = 5 . Bottom, histogram of v 5 of a given v b , generated using 30,808 sample backgrounds obtained via the Park extended MCMC. In this case, the true v 5 of the given channelized background, v b , for generating the channelized data, v, lies in the leftmost histogram.

Fig. 2
Fig. 2

Top, Gaussian signal object and one realization of the signal-absent lumpy object. Bottom, images (scaled by using the minimum and maximum of all image pixel values in both images) of the lumpy object with additive Gaussian noise (left) and Poisson noise (right). For the lumpy object, the magnitude and the spatial width of the Gaussian lump are 1 and 7. For the Gaussian signal, the magnitude and the spatial width of the signal are 0.2 and 3. For Gaussian noise, the amplitude width σ n is 20.

Fig. 3
Fig. 3

Histograms of 5000 lumpy-background images (top) using the same parameters as in the simulation and (bottom) using the same parameters except a Poisson distribution with each image pixel being the mean was used to realize the noise . The difference in the lengths of the tails from the mean of each histogram reveals how much non-Gaussian behavior each data set has.

Fig. 4
Fig. 4

Images in rows of the first 100 singular vectors of the linear imaging system, reproduced from [7], that are most strongly associated with the signal-only image. Originally published in S. Park, J. M. Witten, and K. J. Myers, “Singular vectors of a linear imaging system as efficient channels for the Bayesian ideal observer,” IEEE Trans. Med. Imaging 28, 657–667 (2009). (©2009 IEEE) Reprinted with permission.

Fig. 5
Fig. 5

Sample standard deviations of each of all the pixels in the channelized background v b .

Fig. 6
Fig. 6

Performance (AUC) of the analytical CIO with the Gaussian posterior model of width σ v ( = C v STD of v b ) as a function of C v and the number of channels N c . STD represents the sample standard deviation. In the detection task, 1000 pairs of signal-present and signal-absent were used. In this plot, N c = 5 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 . Note that the target performance for the CIO is the performance of the ideal observer, 0.91 ± 00.02, as shown in Table 3.

Fig. 7
Fig. 7

The performance (AUC) of the analytical CIO with the Gaussian “prior” model of width σ v ( = C v STD of v b ) as a function of C v and the number of channels N c . STD represents sample standard deviation. In the detection task, 1000 pairs of signal-present and signal-absent were used. In the plot, N c = 5 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 . Note that the target performance for the CIO is the performance of the ideal observer, 0.91 ±00.02, as shown in Table 3.

Fig. 8
Fig. 8

For the posterior CIO-MCMC, four empirical ROC curves obtained by using the likelihood ratio estimates for the ideal observer by the Kupinski MCMC and the CIO by the extended MCMC, CIO-MCMC, and analytical CIO. For all the likelihood ratios obtained by the MCMC algorithms, the same set of 200 pairs of images were used with five different Markov chains. For the likelihood ratios obtained by the analytical CIO method, another 1000 pairs of images were used. For the CIO, N c = 80 was used.

Fig. 9
Fig. 9

For the “prior” CIO-MCMC, four empirical ROC curves obtained by using the likelihood ratio estimates for the ideal observer by the Kupinski MCMC and the CIO by the extended MCMC, CIO-MCMC, and the analytical CIO. For all the likelihood ratios by the MCMC algorithms, the same set of 200 pairs of images were used with five different Markov chains. For the likelihood ratios by the analytical CIO method, another 1000 pairs of images were used. For the CIO, N c = 80 was used. Note that the ROC curve for CIO: MCMC-CIO appears above the ideal-observer ROC because of statistical fluctuation.

Fig. 10
Fig. 10

For the posterior CIO-MCMC, the plots show log ( Λ CBKE ) versus states of a Markov chain generated by the CIO-MCMC for the case of N c = 80 . Iteration number represents each state. The bottom plot shows the first 10,000 states in the top plot.

Fig. 11
Fig. 11

For the posterior CIO-MCMC, the plot shows Λ ̂ ( v ) as a function of the number, J, of Λ CBKE ( v | v b ) samples for the case of N c = 80 . In this plot, J=200k.

Fig. 12
Fig. 12

For the posterior CIO-MCMC, the plot shows the moment-generating function, M 0 ( β ) , using 10,000 estimates of Λ ̂ ( v | H 0 ) , calculated by the CIO-MCMC and the analytical method for the case of N c = 80 .

Fig. 13
Fig. 13

For the posterior CIO-MCMC, the plot shows Λ | H 1 var ( Λ | H 0 ) as a function of the number of samples, J, using 10,000 pairs of Λ ̂ ( v | H 0 ) and Λ ̂ ( v | H 1 ) estimated by the CIO-MCMC using the Gaussian posterior model for the case of N c = 80 .

Fig. 14
Fig. 14

For the “prior CIO-MCMC,” the plots show log ( Λ CBKE ) versus states of a Markov chain generated by the CIO-MCMC for the case of N c = 80 in Table 3. Iteration number represents each state. The bottom plot shows the first 1000 states in the top plot.

Fig. 15
Fig. 15

For the “prior” CIO-MCMC, the plot shows Λ ̂ ( v ) as a function of the number, J, of Λ CBKE ( v | v b ) samples for the case of N c = 80 in Table 3. In this plot, J = 200 k.

Fig. 16
Fig. 16

For the “prior” CIO-MCMC, the plot shows the moment-generating function, M 0 ( β ) , using 10,000 estimates of Λ ̂ ( v | H 0 ) , calculated by the CIO-MCMC and the analytical method for the case of Λ | H 1 var ( Λ | H 0 ) .

Fig. 17
Fig. 17

For the “prior” CIO-MCMC, the plot shows Λ | H 1 var ( Λ | H 0 ) as a function of the number of samples, J, using 1000 pairs of Λ ̂ ( v | H 0 ) and Λ ̂ ( v | H 1 ) estimated by the CIO-MCMC using the Gaussian prior model for the case of N c = 80 .

Tables (3)

Tables Icon

Table 1 Parameters for the Proposal, Posterior, and Prior Models

Tables Icon

Table 2 Analytical Expressions of the CIO Likelihood Ratio with Gaussian Noise

Tables Icon

Table 3 Comparison of Observer Performance a

Equations (41)

Equations on this page are rendered with MathJax. Learn more.

H 1 : g = b + s + n ,
H 0 : g = b + n ,
v = T g .
t ( g ) = pr ( g | H 1 ) pr ( g | H 0 ) ,
Λ ( v ) = pr ( v | H 1 ) pr ( v | H 0 ) .
Λ ( v ) = d θ Λ BKE ( v | b ( θ ) ) pr ( θ | v , H 0 ) ,
Λ BKE ( v | b ( θ ) ) = pr ( v | b ( θ ) , H 1 ) pr ( v | b ( θ ) , H 0 ) ,
pr ( θ | v , H 0 ) = pr ( v | b ( θ ) , H 0 ) pr ( θ ) d θ pr ( v | b ( θ ) , H 0 ) pr ( θ ) .
Λ ̂ ( v ) = 1 J J 0 i = J 0 + 1 J Λ BKE ( v | b ( θ ( i ) ) ) ,
Λ ( v ) = d v b Λ CBKE ( v | v b ) pr ( v b | v , H 0 ) ,
Λ CBKE ( v | v b ) = pr ( v | v b , H 1 ) pr ( v | v b , H 0 ) ,
pr ( v b | v , H 0 ) = pr ( v | v b , H 0 ) pr ( v b ) d v b * pr ( v | v b * , H 0 ) pr ( v b * ) .
Λ ̂ ( v ) = 1 J J 0 i = J 0 + 1 J Λ CBKE ( v | v b ( i ) ) ,
α v = min { 1 , pr ( v b ( * ) | v , H 0 ) q ( v b ( * ) | v b ( i ) ) pr ( v b ( i ) | v , H 0 ) q ( v b ( i ) | v b ( * ) ) } .
α v , post = min { 1 , pr ( v b ( * ) | v , H 0 ) pr ( v b ( i ) | v , H 0 ) } .
α v , prior = min { 1 , pr ( v | v b ( * ) , H 0 ) pr ( v b ( * ) ) pr ( v | v b ( i ) , H 0 ) pr ( v b ( i ) ) } .
M 0 ( β ) = 0 d Λ Λ β pr ( Λ | H 0 ) = Λ β 0 .
G ( β ) = ln [ M 0 ( β + 1 2 ) ] ( β 2 1 4 ) ,
1 1 2 exp [ 1 2 G ( 0 ) ] AUC Λ 1 1 2 exp [ 1 2 G ( 0 ) G ( 0 ) 1 8 G ( 0 ) ] .
pr ( v | v b , H j ) = exp ( 1 2 σ n 2 [ ( v v b j v s ) ] t [ T T t ] 1 [ ( v v b j v s ) ] ) ( 2 π ) N c det ( σ n 2 T T t ) , j = 0 , 1 .
Λ CBKE ( v | v b ) = exp ( 1 σ n 2 [ v v b 1 2 v s ) ] t [ T T t ] 1 [ v s ] ) ,
Λ ( v ) = π N c k = 1 N c σ v , k exp ( k = 1 N c [ v b , k ( v k v b , k ) σ v , k 2 ] + k = 1 N c [ v s , k ( v k v b , k 1 2 v s , k ) σ n 2 ] + k = 1 N c [ ( v s , k σ v , k ) 2 4 σ n 4 ] ) ,
pr ( v | H j ) = d v b pr ( v | v b , H j ) pr ( v b ) ,
Λ ( v ) = k = 1 N c exp ( v s , k ( v k v b , k 1 2 v s , k ) ( σ n 2 + σ v , k 2 ) ) .
pr ( v b | v , H 0 ) = d θ δ N c ( v b T b ( θ ) ) pr ( θ | v , H 0 ) ,
pr ( θ | v , H 0 ) = d v b δ N θ ( v b T b ( θ ) ) pr ( v b | v , H 0 ) ,
Λ ( v ) = d v b Λ BKE ( v | v b ) pr ( v b | v , H 0 ) ,
= d v b d θ Λ BKE ( v | v b ) δ N c ( v b T b ( θ ) ) pr ( θ | v , H 0 )
= d θ Λ BKE ( v | T b ( θ ) ) pr ( θ | v , H 0 )
= d θ Λ BKE ( v | θ ) pr ( θ | v , H 0 ) .
pr ( v b * | v ) = Gauss ( v b , B ) .
pr ( v | v b * ) = Gauss ( v b * , A ) ,
pr ( v b * ) = Gauss ( v b , B ) .
pr ( v b * | v ) = C pr ( v | v b * ) pr ( v b * ) = C exp ( 1 2 Q ) ,
Q = ( v v b * ) A 1 ( v v b * ) + ( v b * v b ) B 1 ( v b * v b ) .
v b * ( A 1 + B 1 ) v b * v b * ( A 1 v + B 1 v b ) [ v b * ( A 1 v + B 1 v b ) ] + v A 1 v + v b 1 B 1 v b .
Q = ( v b * x ) ( A 1 + B 1 ) ( v b * x ) ,
v b * ( A 1 + B 1 ) v b * v b * ( A 1 + B 1 ) x [ v b * ( A 1 + B 1 ) x ] + x ( A 1 + B 1 ) x .
pr ( v b * | v ) = Gauss ( x , K ) ,
Λ ( v ) = π N c k = 1 N c σ v , k exp ( k = 1 N c [ v b , k ( v k v b , k ) σ v , k 2 ] + k = 1 N c [ v s , k ( v k v b , k 1 2 v s , k ) σ n 2 ] + k = 1 N c [ ( v s , k σ v , k ) 2 4 σ n 4 ] )
Λ ( v ) = k = 1 N c exp ( v s , k ( v k v b , k 1 2 v s , k ) ( σ n 2 + σ v , k 2 ) )

Metrics