Abstract

The ideal observer sets an upper limit on the performance of an observer on a detection or classification task. The performance of the ideal observer can be used to optimize hardware components of imaging systems and also to determine another observer’s relative performance in comparison with the best possible observer. The ideal observer employs complete knowledge of the statistics of the imaging system, including the noise and object variability. Thus computing the ideal observer for images (large-dimensional vectors) is burdensome without severely restricting the randomness in the imaging system, e.g., assuming a flat object. We present a method for computing the ideal-observer test statistic and performance by using Markov-chain Monte Carlo techniques when we have a well-characterized imaging system, knowledge of the noise statistics, and a stochastic object model. We demonstrate the method by comparing three different parallel-hole collimator imaging systems in simulation.

© 2003 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. H. H. Barrett, “Objective assessment of image quality: effects of quantum noise and object variability,” J. Opt. Soc. Am. A 7, 1266–1278 (1990).
    [Crossref] [PubMed]
  2. H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, New York, 1968).
  3. K. Fukunaga, Statistical Pattern Recognition (Academic, San Diego, Calif., 1990).
  4. E. Clarkson, H. H. Barrett, “Approximation to ideal-observer performance on signal-detection tasks,” Appl. Opt. 39, 1783–1794 (2000).
    [Crossref]
  5. C. K. Abbey, H. H. Barrett, “Human- and model-observer performance in ramp-spectrum noise: effects of regularization and object variability,” J. Opt. Soc. Am. A 18, 473–488 (2001).
    [Crossref]
  6. A. E. Burgess, F. L. Jacobson, P. F. Judy, “Human observer detection experiments with mammograms and power-law noise,” Med. Phys. 28, 419–439 (2001).
    [Crossref] [PubMed]
  7. F. O. Bochud, C. K. Abbey, M. P. Eckstein, “Visual signal detection in structured backgrounds III. Calculation of figures of merit for model observers in statistically nonstationary backgrounds,” J. Opt. Soc. Am. A 17, 193–206 (2000).
    [Crossref]
  8. C. P. Robert, G. Casella, Monte Carlo Statistical Methods (Springer-Verlag, New York, 1999).
  9. W. R. Gilks, S. Richardson, D. J. Spiegelhalter, eds., Markov Chain Monte Carlo in Practice (Chapman & Hall, Boca Raton, Fla., 1996).
  10. H. H. Barrett, C. K. Abbey, E. Clarkson, “Objective assessment of image quality III. ROC metrics, ideal observers, and likelihood-generating functions,” J. Opt. Soc. Am. A 15, 1520–1535 (1998).
    [Crossref]
  11. J. Egan, Signal Detection Theory and ROC Analysis (Academic, New York, 1975).
  12. C. E. Metz, “Basic principles of ROC Analysis,” Semin. Nucl. Med. VIII, 283–298 (1978).
    [Crossref]
  13. J. P. Rolland, H. H. Barrett, “Effect of random background inhomogeneity on observer detection performance,” J. Opt. Soc. Am. A 9, 649–658 (1992).
    [Crossref] [PubMed]
  14. S. C. Zhu, Y. Wu, D. Mumford, “Filters, random fields and maximum entropy (FRAME),” Int. J. Comput. Vision 27, 1–20 (1998).
    [Crossref]
  15. E. P. Simoncelli, B. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1217 (2001).
    [Crossref] [PubMed]
  16. H. Zhang, B. Gallas, University of Arizona, Tucson, Ariz. 85721 (personal communication, 2001).
  17. C. E. Metz, B. A. Herman, J.-H. Shen, “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Stat. Med. 17, 1033–1053 (1998).
    [Crossref] [PubMed]
  18. J. Dugundji, Topology (Allyn and Bacon, Boston, Mass., 1965).

2001 (3)

C. K. Abbey, H. H. Barrett, “Human- and model-observer performance in ramp-spectrum noise: effects of regularization and object variability,” J. Opt. Soc. Am. A 18, 473–488 (2001).
[Crossref]

A. E. Burgess, F. L. Jacobson, P. F. Judy, “Human observer detection experiments with mammograms and power-law noise,” Med. Phys. 28, 419–439 (2001).
[Crossref] [PubMed]

E. P. Simoncelli, B. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1217 (2001).
[Crossref] [PubMed]

2000 (2)

1998 (3)

S. C. Zhu, Y. Wu, D. Mumford, “Filters, random fields and maximum entropy (FRAME),” Int. J. Comput. Vision 27, 1–20 (1998).
[Crossref]

H. H. Barrett, C. K. Abbey, E. Clarkson, “Objective assessment of image quality III. ROC metrics, ideal observers, and likelihood-generating functions,” J. Opt. Soc. Am. A 15, 1520–1535 (1998).
[Crossref]

C. E. Metz, B. A. Herman, J.-H. Shen, “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Stat. Med. 17, 1033–1053 (1998).
[Crossref] [PubMed]

1992 (1)

1990 (1)

1978 (1)

C. E. Metz, “Basic principles of ROC Analysis,” Semin. Nucl. Med. VIII, 283–298 (1978).
[Crossref]

Abbey, C. K.

Barrett, H. H.

Bochud, F. O.

Burgess, A. E.

A. E. Burgess, F. L. Jacobson, P. F. Judy, “Human observer detection experiments with mammograms and power-law noise,” Med. Phys. 28, 419–439 (2001).
[Crossref] [PubMed]

Casella, G.

C. P. Robert, G. Casella, Monte Carlo Statistical Methods (Springer-Verlag, New York, 1999).

Clarkson, E.

Dugundji, J.

J. Dugundji, Topology (Allyn and Bacon, Boston, Mass., 1965).

Eckstein, M. P.

Egan, J.

J. Egan, Signal Detection Theory and ROC Analysis (Academic, New York, 1975).

Fukunaga, K.

K. Fukunaga, Statistical Pattern Recognition (Academic, San Diego, Calif., 1990).

Gallas, B.

H. Zhang, B. Gallas, University of Arizona, Tucson, Ariz. 85721 (personal communication, 2001).

Herman, B. A.

C. E. Metz, B. A. Herman, J.-H. Shen, “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Stat. Med. 17, 1033–1053 (1998).
[Crossref] [PubMed]

Jacobson, F. L.

A. E. Burgess, F. L. Jacobson, P. F. Judy, “Human observer detection experiments with mammograms and power-law noise,” Med. Phys. 28, 419–439 (2001).
[Crossref] [PubMed]

Judy, P. F.

A. E. Burgess, F. L. Jacobson, P. F. Judy, “Human observer detection experiments with mammograms and power-law noise,” Med. Phys. 28, 419–439 (2001).
[Crossref] [PubMed]

Metz, C. E.

C. E. Metz, B. A. Herman, J.-H. Shen, “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Stat. Med. 17, 1033–1053 (1998).
[Crossref] [PubMed]

C. E. Metz, “Basic principles of ROC Analysis,” Semin. Nucl. Med. VIII, 283–298 (1978).
[Crossref]

Mumford, D.

S. C. Zhu, Y. Wu, D. Mumford, “Filters, random fields and maximum entropy (FRAME),” Int. J. Comput. Vision 27, 1–20 (1998).
[Crossref]

Olshausen, B.

E. P. Simoncelli, B. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1217 (2001).
[Crossref] [PubMed]

Robert, C. P.

C. P. Robert, G. Casella, Monte Carlo Statistical Methods (Springer-Verlag, New York, 1999).

Rolland, J. P.

Shen, J.-H.

C. E. Metz, B. A. Herman, J.-H. Shen, “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Stat. Med. 17, 1033–1053 (1998).
[Crossref] [PubMed]

Simoncelli, E. P.

E. P. Simoncelli, B. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1217 (2001).
[Crossref] [PubMed]

Van Trees, H. L.

H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, New York, 1968).

Wu, Y.

S. C. Zhu, Y. Wu, D. Mumford, “Filters, random fields and maximum entropy (FRAME),” Int. J. Comput. Vision 27, 1–20 (1998).
[Crossref]

Zhang, H.

H. Zhang, B. Gallas, University of Arizona, Tucson, Ariz. 85721 (personal communication, 2001).

Zhu, S. C.

S. C. Zhu, Y. Wu, D. Mumford, “Filters, random fields and maximum entropy (FRAME),” Int. J. Comput. Vision 27, 1–20 (1998).
[Crossref]

Annu. Rev. Neurosci. (1)

E. P. Simoncelli, B. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1217 (2001).
[Crossref] [PubMed]

Appl. Opt. (1)

Int. J. Comput. Vision (1)

S. C. Zhu, Y. Wu, D. Mumford, “Filters, random fields and maximum entropy (FRAME),” Int. J. Comput. Vision 27, 1–20 (1998).
[Crossref]

J. Opt. Soc. Am. A (5)

Med. Phys. (1)

A. E. Burgess, F. L. Jacobson, P. F. Judy, “Human observer detection experiments with mammograms and power-law noise,” Med. Phys. 28, 419–439 (2001).
[Crossref] [PubMed]

Semin. Nucl. Med. (1)

C. E. Metz, “Basic principles of ROC Analysis,” Semin. Nucl. Med. VIII, 283–298 (1978).
[Crossref]

Stat. Med. (1)

C. E. Metz, B. A. Herman, J.-H. Shen, “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Stat. Med. 17, 1033–1053 (1998).
[Crossref] [PubMed]

Other (7)

J. Dugundji, Topology (Allyn and Bacon, Boston, Mass., 1965).

H. Zhang, B. Gallas, University of Arizona, Tucson, Ariz. 85721 (personal communication, 2001).

H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I (Academic, New York, 1968).

K. Fukunaga, Statistical Pattern Recognition (Academic, San Diego, Calif., 1990).

J. Egan, Signal Detection Theory and ROC Analysis (Academic, New York, 1975).

C. P. Robert, G. Casella, Monte Carlo Statistical Methods (Springer-Verlag, New York, 1999).

W. R. Gilks, S. Richardson, D. J. Spiegelhalter, eds., Markov Chain Monte Carlo in Practice (Chapman & Hall, Boca Raton, Fla., 1996).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1
Fig. 1

Examples of two-dimensional lumpy objects. (a) Parameters N=39, a=1, and s=10; (b) parameters N=348, a=1, and s=4. The field of view (FOV) for both objects is 128×128.

Fig. 2
Fig. 2

Image realizations of the same lumpy object for (a) imaging system A, (b) imaging system B, and (c) imaging system C. There is no signal in these images.

Fig. 3
Fig. 3

ΛBKE versus iteration number. The iteration numbers here are the states of the Markov chain described in Subsection 3.A.

Fig. 4
Fig. 4

Log-likelihood histograms for (a) imaging system A, (b) imaging system B, and (c) imaging system C.

Fig. 5
Fig. 5

ROC curve comparison of the ideal-observer performance for the three imaging systems. The AUC values for imaging systems A, B, and C are 0.92, 0.88, and 0.67, respectively.

Tables (1)

Tables Icon

Table 1 Characteristics of the Three Imaging Systems

Equations (35)

Equations on this page are rendered with MathJax. Learn more.

g=Hf+n,
gm=Sdrhm(r)f(r)+nm,
H0 : g=Hfb+n,
H1 : g=H(fb+fs)+n.
b=Hfb,
s=Hfs.
Λ(g)=pr(g|H1)pr(g|H0),
fb=fb(r)=n=1NL(r-cn|a, s),
bm=n=1NSdrhm(r)L(r-cn|a, s).
Λ(g)=dbpr(b)pr(g|b, H1)dbpr(b)pr(g|b, H0).
Λ(g)=dbpr(b)pr(g|b, H1)dbpr(b)pr(g|b, H0),
Λ(g)=dbpr(g|b, H1)pr(g|b, H0)pr(b)pr(g|b, H0)dbpr(b)pr(g|b, H0).
Λ(g)=dbΛBKE(g|b)pr(b|g, H0),
ΛBKE(g|b)=pr(g|b, H1)pr(g|b, H0)
pr(b|g, H0)=pr(g|b, H0)pr(b)dbpr(g|b, H0)pr(b).
pr(g|b, H1)=m=1Mexp[-(bm+sm)](bm+sm)gmgm!,
pr(g|b, H0)=m=1Mexp(-bm)(bm)gmgm!.
ΛBKE(g|b)=m=1M1+smbmgmexp(-sm).
ΛBKE(g|b)=exp[(g-b-s/2)Kn-1s],
Λ(g)=dθΛBKE(g|b(θ))pr(θ|g, H0),
pr(θ|g, H0)=pr(g|b(θ), H0)pr(θ)dθpr(g|b(θ),H0)pr(θ).
Λˆ(g)=1Jj=1JΛBKE(g|b(θ(j)))
θ˜α(θ˜, θ(i))=min1,pr(θ˜|g, H0)q(θ˜|θ(i))pr(θ(i)|g, H0)q(θ(i)|θ˜).
θ(0),θ(1) ,, θ(K),
pr(g|b(θ), H0)pr(θ)=pr(g|b(θ), H0)pr(N)pr({cn})=m=1Mexp[-bm(θ)]bm(θ)gmgm!×exp(-N¯)N¯NN!N!FOVN
Φ=α1c1α2c2αNcN,
θ(Φ)={cn : αn=1}.
pr(b|g, H0)=dθδM(b-b(θ))pr(θ|g, H0),
Λ(g)=dbΛBKE(g|b)pr(b|g,H0)
=dbdθΛBKE(g|b)×δM(b-b(θ))pr(θ|g, H0)
=dθΛBKE(g|b(θ))pr(θ|g, H0),
Npr(N|g, H0)SNdc1dc2  dcN×ΛBKE(g|b({cn}))pr({cn}|g,H0).
q(Φ˜|Φ)=ηf(1-η)N-fj:α˜j=αj=11Cδ(c-j-c˜-j)×G(c˜j-cj),
f=i=1N|αi-α˜i|
C=αα˜.

Metrics