Multireader multicase variance analysis for binary data

Brandon D. Gallas; Gene A. Pennello; Kyle J. Myers

doi:10.1364/JOSAA.24.000B70

Multireader multicase variance analysis for binary data

Brandon D. Gallas, Gene A. Pennello, and Kyle J. Myers

Journal of the Optical Society of America A
Vol. 24,
Issue 12,
pp. B70-B80
(2007)
•https://doi.org/10.1364/JOSAA.24.000B70

Get Citation
Copy Citation Text
Brandon D. Gallas, Gene A. Pennello, and Kyle J. Myers, "Multireader multicase variance analysis for binary data," J. Opt. Soc. Am. A 24, B70-B80 (2007)

Export Citation
- BibTex
- Endnote (RIS)
- HTML
- Plain Text
Get PDF (701 KB)
Set citation alerts
Save article

Related Topics
Optics & Photonics Topics
?

The topics in this list come from the OSA Optics and Photonics Topics applied to this article.
Previously assigned OCIS codes
- Probability theory, stochastic processes, and statistics (000.5490)
- Image quality assessment (110.3000)
- Psychophysics (330.5510)

About this Article
History
- Original Manuscript: April 13, 2007
- Revised Manuscript: July 7, 2007
- Manuscript Accepted: August 2, 2007
- Published: September 28, 2007
Virtual Issues
Virtual Journal for Biomedical Optics Vol. 3, Iss. 1

Accessible

Open Access

Abstract

Multireader multicase (MRMC) variance analysis has become widely utilized to analyze observer studies for which the summary measure is the area under the receiver operating characteristic (ROC) curve. We extend MRMC variance analysis to binary data and also to generic study designs in which every reader may not interpret every case. A subset of the fundamental moments central to MRMC variance analysis of the area under the ROC curve (AUC) is found to be required. Through multiple simulation configurations, we compare our unbiased variance estimates to naïve estimates across a range of study designs, average percent correct, and numbers of readers and cases.

Full Article | PDF Article

OSA Recommended Articles

Estimation receiver operating characteristic curve and ideal observers for combined detection/estimation tasks

Eric Clarkson
J. Opt. Soc. Am. A 24(12) B91-B98 (2007)

Efficiency of the human observer for detecting a Gaussian signal at a known location in non-Gaussian distributed lumpy backgrounds

Subok Park, Brandon D. Gallas, Aldo Badano, Nicholas A. Petrick, and Kyle J. Myers
J. Opt. Soc. Am. A 24(4) 911-921 (2007)

Human linear template with mammographic backgrounds estimated with a genetic algorithm

Cyril Castella, Craig K. Abbey, Miguel P. Eckstein, Francis R. Verdun, Karen Kinkel, and François O. Bochud
J. Opt. Soc. Am. A 24(12) B1-B12 (2007)

References

View by:
Article Order
|
Year
|
Author
|
Publication

D. D. Dorfman, K. S. Berbaum, and C. E. Metz, “Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method,” Invest. Radiol. 27, 723–731 (1992).
[Crossref] [PubMed]
S. V. Beiden, R. F. Wagner, and G. Campbell, “Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis,” Acad. Radiol. 7, 341–349 (2000).
[Crossref] [PubMed]
N. A. Obuchowski, S. V. Beiden, K. S. Berbaum, S. L. Hillis, H. Ishwaran, H. H. Song, and R. F. Wagner, “Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods,” Acad. Radiol. 11, 980–995 (2004).
[Crossref] [PubMed]
B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353–362 (2006).
[Crossref] [PubMed]
B. D. Gallas and D. G. Brown, “Reader studies for validation of CAD systems,” submitted to Neural Networks.
C. A. Roe and C. E. Metz, “Dorfman–Berbaum–Metz method for statistical analysis of multireader, multimodality receiver operating characteristic (ROC) data: validation with computer simulation,” Acad. Radiol. 4, 298–303 (1997).
[Crossref] [PubMed]
M. Schiffman and M. E. Adrianza, “ASCUS-LSIL triage study: design, methods and characteristics of trial participants,” Acta Cytol. 44, 726–742 (2000).
[Crossref] [PubMed]
J. Jeronimo, L. S. Massad, and M. Schiffman, “Visual appearance of the uterine cervix: correlation with human papillomavirus detection and type,” Am. J. Obstet. Gynecol. 97, 47.e1–47.e8 (2007).
[Crossref]
S. L. Hillis and K. S. Berbaum, “Monte Carlo validation of the Dorfman–Berbaum–Metz method using normalized pseudovalues and less data-based model simplification,” Acad. Radiol. 12, 1534–1541 (2005).
[Crossref] [PubMed]
S. L. Hillis, N. A. Obuchowski, K. M. Schartz, and K. S. Berbaum, “A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data,” Stat. Med. 24, 1579–1607 (2005).
[Crossref] [PubMed]
X. Song and X.-H. Zhou, “A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data,” Biostatistics 6, 303–312 (2005).
[Crossref] [PubMed]
W. A. Yousef, R. F. Wagner, and M. H. Loew, “Assessing classifiers from two independent data sets using ROC analysis: a nonparametric approach,” IEEE Trans. Pattern Anal. Mach. Intell. 28, 1809–1817 (2006).
[Crossref] [PubMed]
M. S. Pepe, The Statistical Evaluation of Medical Tests for Classification and Prediction (Oxford U. Press, 2003).
C. A. Roe and C. E. Metz, “Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates,” Acad. Radiol. 4, 587–600 (1997).
[Crossref] [PubMed]
H. H. Barrett, M. A. Kupinski, and E. Clarkson, “Probabilistic Foundations of the MRMC Method,” Proc. SPIE 5749, 21–31 (2005).
[Crossref]
E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic model for the MRMC method. Part 1. theoretical development,” Acad. Radiol. 13, 1410–1421 (2006).
[Crossref] [PubMed]

2007 (1)

J. Jeronimo, L. S. Massad, and M. Schiffman, “Visual appearance of the uterine cervix: correlation with human papillomavirus detection and type,” Am. J. Obstet. Gynecol. 97, 47.e1–47.e8 (2007).
[Crossref]

2006 (3)

B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353–362 (2006).
[Crossref] [PubMed]

W. A. Yousef, R. F. Wagner, and M. H. Loew, “Assessing classifiers from two independent data sets using ROC analysis: a nonparametric approach,” IEEE Trans. Pattern Anal. Mach. Intell. 28, 1809–1817 (2006).
[Crossref] [PubMed]

E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic model for the MRMC method. Part 1. theoretical development,” Acad. Radiol. 13, 1410–1421 (2006).
[Crossref] [PubMed]

2005 (4)

H. H. Barrett, M. A. Kupinski, and E. Clarkson, “Probabilistic Foundations of the MRMC Method,” Proc. SPIE 5749, 21–31 (2005).
[Crossref]

S. L. Hillis and K. S. Berbaum, “Monte Carlo validation of the Dorfman–Berbaum–Metz method using normalized pseudovalues and less data-based model simplification,” Acad. Radiol. 12, 1534–1541 (2005).
[Crossref] [PubMed]

S. L. Hillis, N. A. Obuchowski, K. M. Schartz, and K. S. Berbaum, “A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data,” Stat. Med. 24, 1579–1607 (2005).
[Crossref] [PubMed]

X. Song and X.-H. Zhou, “A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data,” Biostatistics 6, 303–312 (2005).
[Crossref] [PubMed]

2004 (1)

N. A. Obuchowski, S. V. Beiden, K. S. Berbaum, S. L. Hillis, H. Ishwaran, H. H. Song, and R. F. Wagner, “Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods,” Acad. Radiol. 11, 980–995 (2004).
[Crossref] [PubMed]

2000 (2)

M. Schiffman and M. E. Adrianza, “ASCUS-LSIL triage study: design, methods and characteristics of trial participants,” Acta Cytol. 44, 726–742 (2000).
[Crossref] [PubMed]

S. V. Beiden, R. F. Wagner, and G. Campbell, “Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis,” Acad. Radiol. 7, 341–349 (2000).
[Crossref] [PubMed]

1997 (2)

C. A. Roe and C. E. Metz, “Dorfman–Berbaum–Metz method for statistical analysis of multireader, multimodality receiver operating characteristic (ROC) data: validation with computer simulation,” Acad. Radiol. 4, 298–303 (1997).
[Crossref] [PubMed]

C. A. Roe and C. E. Metz, “Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates,” Acad. Radiol. 4, 587–600 (1997).
[Crossref] [PubMed]

1992 (1)

D. D. Dorfman, K. S. Berbaum, and C. E. Metz, “Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method,” Invest. Radiol. 27, 723–731 (1992).
[Crossref] [PubMed]

Adrianza, M. E.

M. Schiffman and M. E. Adrianza, “ASCUS-LSIL triage study: design, methods and characteristics of trial participants,” Acta Cytol. 44, 726–742 (2000).
[Crossref] [PubMed]

Barrett, H. H.

E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic model for the MRMC method. Part 1. theoretical development,” Acad. Radiol. 13, 1410–1421 (2006).
[Crossref] [PubMed]

H. H. Barrett, M. A. Kupinski, and E. Clarkson, “Probabilistic Foundations of the MRMC Method,” Proc. SPIE 5749, 21–31 (2005).
[Crossref]

Beiden, S. V.

Berbaum, K. S.

Brown, D. G.

B. D. Gallas and D. G. Brown, “Reader studies for validation of CAD systems,” submitted to Neural Networks.

Campbell, G.

Clarkson, E.

E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic model for the MRMC method. Part 1. theoretical development,” Acad. Radiol. 13, 1410–1421 (2006).
[Crossref] [PubMed]

H. H. Barrett, M. A. Kupinski, and E. Clarkson, “Probabilistic Foundations of the MRMC Method,” Proc. SPIE 5749, 21–31 (2005).
[Crossref]

Dorfman, D. D.

Gallas, B. D.

B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353–362 (2006).
[Crossref] [PubMed]

B. D. Gallas and D. G. Brown, “Reader studies for validation of CAD systems,” submitted to Neural Networks.

Hillis, S. L.

Ishwaran, H.

Jeronimo, J.

Kupinski, M. A.

E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic model for the MRMC method. Part 1. theoretical development,” Acad. Radiol. 13, 1410–1421 (2006).
[Crossref] [PubMed]

H. H. Barrett, M. A. Kupinski, and E. Clarkson, “Probabilistic Foundations of the MRMC Method,” Proc. SPIE 5749, 21–31 (2005).
[Crossref]

Loew, M. H.

Massad, L. S.

Metz, C. E.

C. A. Roe and C. E. Metz, “Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates,” Acad. Radiol. 4, 587–600 (1997).
[Crossref] [PubMed]

Obuchowski, N. A.

Pepe, M. S.

M. S. Pepe, The Statistical Evaluation of Medical Tests for Classification and Prediction (Oxford U. Press, 2003).

Roe, C. A.

C. A. Roe and C. E. Metz, “Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates,” Acad. Radiol. 4, 587–600 (1997).
[Crossref] [PubMed]

Schartz, K. M.

Schiffman, M.

M. Schiffman and M. E. Adrianza, “ASCUS-LSIL triage study: design, methods and characteristics of trial participants,” Acta Cytol. 44, 726–742 (2000).
[Crossref] [PubMed]

Song, H. H.

Song, X.

X. Song and X.-H. Zhou, “A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data,” Biostatistics 6, 303–312 (2005).
[Crossref] [PubMed]

Wagner, R. F.

Yousef, W. A.

Zhou, X.-H.

X. Song and X.-H. Zhou, “A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data,” Biostatistics 6, 303–312 (2005).
[Crossref] [PubMed]

Acad. Radiol. (7)

B. D. Gallas, “One-shot estimate of MRMC variance: AUC,” Acad. Radiol. 13, 353–362 (2006).
[Crossref] [PubMed]

C. A. Roe and C. E. Metz, “Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates,” Acad. Radiol. 4, 587–600 (1997).
[Crossref] [PubMed]

E. Clarkson, M. A. Kupinski, and H. H. Barrett, “A probabilistic model for the MRMC method. Part 1. theoretical development,” Acad. Radiol. 13, 1410–1421 (2006).
[Crossref] [PubMed]

Acta Cytol. (1)

M. Schiffman and M. E. Adrianza, “ASCUS-LSIL triage study: design, methods and characteristics of trial participants,” Acta Cytol. 44, 726–742 (2000).
[Crossref] [PubMed]

Am. J. Obstet. Gynecol. (1)

Biostatistics (1)

X. Song and X.-H. Zhou, “A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data,” Biostatistics 6, 303–312 (2005).
[Crossref] [PubMed]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

Invest. Radiol. (1)

Proc. SPIE (1)

H. H. Barrett, M. A. Kupinski, and E. Clarkson, “Probabilistic Foundations of the MRMC Method,” Proc. SPIE 5749, 21–31 (2005).
[Crossref]

Stat. Med. (1)

Other (2)

B. D. Gallas and D. G. Brown, “Reader studies for validation of CAD systems,” submitted to Neural Networks.

M. S. Pepe, The Statistical Evaluation of Medical Tests for Classification and Prediction (Oxford U. Press, 2003).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.

Click here to see a list of articles that cite this paper

Fig. 1 Graphic on the left shows the (transpose) layout of data from a binary-outcome experiment with multiple readers. Compared to a fully crossed data set, which would fill the entire matrix, much data are missing. The table on the right shows a simple example. The PC in the last row weighs each reading equally. One estimate of the standard error of that average considers all the readings to be iid. The result is 3.7. One could instead obtain an average PC by averaging the reader-specific PCs, resulting in 60. Continuing, one might estimate the standard error of this average, yielding 20.6. While both of the averages are valid, both variance estimates are wrong.

Download Full Size | PPT Slide | PDF

Fig. 2 Population variances calculated using the integral expressions for the moments compared to those estimated from MC. A separate point is given for each of the 324 simulation configurations, 6 types of study designs, and both ways to weigh individual reader PCs.

Download Full Size | PPT Slide | PDF

Fig. 3 Population variances for all the high-PC (0.96) simulation configurations compared to the expected values (from MC averaging) of the naïve variance estimates. The expected values of our moment estimators are unbiased and thus equal the population variances. The naïve estimates bracket the true MRMC variances:

{\hat{V}}_{naive ̱ γ}

is biased high (dotted curve) and

{\hat{V}}_{naive ̱ g}

is biased low (dashed curve). In all but panel D,

V_{g}

lies on top of

V_{γ}

(notice some dots peaking out from behind the solid curve).

Download Full Size | PPT Slide | PDF

Fig. 4 RRMSE for the fully crossed study design: A, high PC (0.96); B, low PC (0.70).

Download Full Size | PPT Slide | PDF

Fig. 5 RRMSE for the broad uniform study design: A, high PC (0.96); B, low PC (0.70).

Download Full Size | PPT Slide | PDF

Tables (3)

Table 1 List of the Coefficients Needed to Appropriately Weight the Success Moments to Determine the Variance of $\hat{P}$ for a Fixed Study Design^a

View Table | View all tables in this article

Table 2 Parameters Investigated in the Simulations According to a Factorial Design, Yielding a Total of $3 \times 3 \times 3 \times 3 \times 2 \times 2 = 324$ Simulation Configurations

View Table | View all tables in this article

Table 3 One Example of the Six Distributions of Cases for Five Readers Reading 102 Cases on Average

View Table | View all tables in this article

Equations (52)

Equations on this page are rendered with MathJax. Learn more.

{\hat{p}}_{r} = \frac{1}{N_{g ∣ r}} \sum_{i = 1}^{N_{g}} d_{i r} s_{i r} .

\hat{P} = \sum_{r = 1}^{N_{γ}} w_{r} {\hat{p}}_{r} .

⟨ \hat{P} ∣ D ⟩ = ⟨ s (g, γ) ∣ D ⟩ = ⟨ s (g, γ) ⟩ .

V_{∣ D} = var (\hat{P} ∣ D) = c_{1} ⟨ s {(g, γ)}^{2} ⟩ + c_{4} ⟨ {⟨ s (g, γ) ∣ γ ⟩}^{2} ⟩ + c_{5} ⟨ {⟨ s (g, γ) ∣ g ⟩}^{2} ⟩ + c_{8} {⟨ s (g, γ) ⟩}^{2},

V_{∣ D} = \sum_{r = 1}^{N_{γ}} w_{r}^{2} var ({\hat{p}}_{r} ∣ D) + \sum_{r = 1}^{N_{γ}} \sum_{r^{'} \neq r}^{N_{γ}} w_{r} w_{r^{'}} cov ({\hat{p}}_{r}, {\hat{p}}_{r^{'}} ∣ D) .

var ({\hat{p}}_{r} {∣ N_{g}}_{∣ r}) = ⟨ {\hat{p}}_{r}^{2} {∣ N_{g}}_{∣ r} ⟩ - {⟨ {\hat{p}}_{r} ⟩}^{2}

= (\frac{1}{N_{g ∣ r}} M_{1} + \frac{N_{g ∣ r} - 1}{N_{g ∣ r}} M_{4}) - M_{8} .

cov ({\hat{p}}_{r}, {\hat{p}}_{r^{'}}) = \frac{1}{N_{g}} M_{5} - \frac{1}{N_{g}} M_{8},

{\hat{V}}_{∣ D} = {\underset{̱}{c}}^{t} M_{̱}^{̂},

{\hat{M}}_{1} = \sum_{r = 1}^{N_{γ}} w_{r} \sum_{i = 1}^{N_{g}} \frac{d_{i r}^{2} s_{i r}^{2}}{\sum_{i^{*} = 1}^{N_{g}} d_{i^{*} r}^{2}},

{\hat{M}}_{4} = \sum_{r = 1}^{N_{γ}} w_{r} \sum_{i = 1}^{N_{g}} \frac{d_{i r} s_{i r}}{\sum_{i^{*} = 1}^{N_{g}} d_{i^{*} r}} \sum_{i^{'} \neq i}^{N_{g}} \frac{d_{i^{'} r} s_{i^{'} r}}{\sum_{i^{' *} \neq i}^{N_{g}} d_{i^{' *} r}},

{\hat{M}}_{5} = \sum_{r = 1}^{N_{γ}} w_{r} \sum_{r^{'} \neq r}^{N_{γ}} \frac{w_{r^{'}}}{1 - w_{r}} \sum_{i = 1}^{N_{g}} \frac{d_{i r} d_{i r^{'}} s_{i r} s_{i r^{'}}}{\sum_{i^{*} = 1}^{N_{g}} d_{i^{*} r} d_{i^{*} r^{'}}},

{\hat{M}}_{8} = \sum_{r = 1}^{N_{γ}} w_{r} \sum_{i = 1}^{N_{g}} \frac{d_{i r} s_{i r}}{\sum_{i^{*} = 1}^{N_{g}} d_{i^{*} r}} \sum_{r^{'} \neq r}^{N_{γ}} \frac{w_{r^{'}}}{1 - w_{r}} \sum_{i^{'} \neq i}^{N_{g}} \frac{d_{i^{'} r^{'}} s_{i^{'} r^{'}}}{\sum_{i^{' *} \neq i}^{N_{g}} d_{i^{*'} r^{'}}} .

{\hat{V}}_{naive ̱ g} = \frac{1}{N_{total} (N_{total} - 1)} \sum_{r = 1}^{N_{γ}} \sum_{i = 1}^{N_{g}} d_{i r} {(s_{i r} - {\hat{P}}_{γ})}^{2},

{\hat{V}}_{naive ̱ γ} = \frac{1}{N_{γ} (N_{γ} - 1)} \sum_{r = 1}^{N_{γ}} {({\hat{p}}_{r} - {\hat{P}}_{γ})}^{2} .

s_{i r} = s (t_{1 i r} - t_{0 i r}) = {\begin{cases} 1 & if t_{1 i r} - t_{0 i r} > 0 \\ 0 & if t_{1 i r} - t_{0 i r} < 0 \end{cases} .

t_{0 i r} = 0 + {[R]}_{0 r} + {[C]}_{0 i} + {[R C]}_{0 i r},

t_{1 i r} = μ_{t} + {[R]}_{1 r} + {[C]}_{1 i} + {[R C]}_{1 i r} .

σ_{R}^{2} + σ_{C}^{2} + σ_{R C}^{2} = 1 .

p_{r} = ⟨ {\hat{p}}_{r} ∣ r ⟩ = ⟨ s (t_{1 i r} - t_{0 i r}) ∣ r ⟩,

p_{r} = Φ (\frac{μ_{t} + {[R]}_{1 r} - {[R]}_{0 r}}{\sqrt{2 σ_{C}^{2} + 2 σ_{R C}^{2}}}),

\Pr (p_{r} ⩽ τ) = \int_{- \infty}^{τ} \frac{d x \exp (- x^{2} ∕ 4 σ_{R}^{2})}{\sqrt{4 π σ_{R}^{2}}} Φ (\frac{μ_{t} + x}{\sqrt{2 σ_{C}^{2} + 2 σ_{R C}^{2}}}) .

M_{0} = ⟨ p_{r} ⟩ = ⟨ s (t_{1 i r} - t_{0 i r}) ⟩ = Φ (μ_{t} ∕ \sqrt{2}),

M_{4} = \int_{- \infty}^{\infty} \frac{d x \exp (- x^{2} ∕ 4 σ_{R}^{2})}{\sqrt{4 π σ_{R}^{2}}} {[Φ (\frac{μ_{t} + x}{\sqrt{2 σ_{C}^{2} + 2 σ_{R C}^{2}}})]}^{2},

M_{5} = \int_{- \infty}^{\infty} \frac{d x \exp (- x^{2} ∕ 4 σ_{C}^{2})}{\sqrt{4 π σ_{C}^{2}}} {[Φ (\frac{μ_{t} + x}{\sqrt{2 σ_{R}^{2} + 2 σ_{R C}^{2}}})]}^{2} .

RRMSE = \frac{1}{V} {({⟨ \hat{V} - ⟨ \hat{V} ⟩ ⟩}^{2} + var (\hat{V}))}^{1 ∕ 2},

⟨ {\hat{P}}^{2} ∣ D ⟩ = ⟨ {(\sum_{r = 1}^{N_{γ}} \frac{w_{r}}{N_{g ∣ r}} \sum_{i = 1}^{N_{g}} d_{i r} s_{i r})}^{2} ⟩ .

⟨ {\hat{P}}^{2} ∣ D ⟩ = \sum_{r = 1}^{N_{γ}} \frac{w_{r}^{2}}{N_{g ∣ r}^{2}} \sum_{i = 1}^{N_{g}} d_{i r}^{2} ⟨ s_{i r}^{2} ⟩ + \sum_{r = 1}^{N_{γ}} \frac{w_{r}^{2}}{N_{g ∣ r}^{2}} \sum_{i = 1}^{N_{g}} \sum_{i^{'} \neq i}^{N_{g}} d_{i r} d_{i^{'} r} ⟨ s_{i r} s_{i^{'} r} ⟩ + \sum_{r = 1}^{N_{γ}} \sum_{r^{'} \neq r}^{N_{γ}} \frac{w_{r} w_{r^{'}}}{N_{g ∣ r} N_{g ∣ r^{'}}} \sum_{i = 1}^{N_{g}} d_{i r} d_{i r^{'}} ⟨ s_{i r} s_{i r^{'}} ⟩ + \sum_{r = 1}^{N_{γ}} \sum_{r^{'} \neq r}^{N_{γ}} \frac{w_{r} w_{r^{'}}}{N_{g ∣ r} N_{g ∣ r^{'}}} \sum_{i = 1}^{N_{g}} \sum_{i^{'} \neq i}^{N_{g}} d_{i r} d_{i^{'} r^{'}} ⟨ s_{i r} s_{i^{'} r^{'}} ⟩ .

M_{1} = ⟨ s_{i r}^{2} ⟩ = ⟨ s {(g, γ)}^{2} ⟩,

M_{4} = ⟨ s_{i r} s_{i^{'} r} ⟩ = ⟨ {⟨ s (g, γ) ∣ γ ⟩}^{2} ⟩,

M_{5} = ⟨ s_{i r} s_{i r^{'}} ⟩ = ⟨ {⟨ s (g, γ) ∣ g ⟩}^{2} ⟩,

M_{8} = ⟨ s_{i r} s_{i^{'} r^{'}} ⟩ = {⟨ s (g, γ) ⟩}^{2} .

c_{1} = \sum_{r = 1}^{N_{γ}} \frac{w_{r}^{2}}{N_{g ∣ r}},

c_{4} = \sum_{r = 1}^{N_{γ}} w_{r}^{2} - \sum_{r = 1}^{N_{γ}} \frac{w_{r}^{2}}{N_{g ∣ r}},

c_{5} = \sum_{r = 1}^{N_{γ}} \sum_{r^{'} \neq r}^{N_{γ}} \frac{w_{r} w_{r^{'}}}{N_{g ∣ r} N_{g ∣ r^{'}}} \sum_{i = 1}^{N_{g}} d_{i r} d_{i r^{'}},

c_{8} = \sum_{r = 1}^{N_{γ}} \sum_{r^{'} \neq r}^{N_{γ}} w_{r} w_{r^{'}} - \sum_{r = 1}^{N_{γ}} \sum_{r^{'} \neq r}^{N_{γ}} \frac{w_{r} w_{r^{'}}}{N_{g ∣ r} N_{g ∣ r^{'}}} \sum_{i = 1}^{N_{g}} d_{i r} d_{i r^{'}} .

{\hat{p}}_{r} {\hat{p}}_{r^{'}} = \sum_{i = 1}^{N_{g}} \frac{d_{i r} s_{i r}}{N_{g ∣ r}} \frac{d_{i r^{'}} s_{i r^{'}}}{N_{g ∣ r^{'}}} + \sum_{i = 1}^{N_{g}} \sum_{i^{'} \neq i}^{N_{g}} \frac{d_{i r} s_{i r}}{N_{g ∣ r}} \frac{d_{i^{'} r^{'}} s_{i^{'} r^{'}}}{N_{g ∣ r^{'}}} .

⟨ {\hat{p}}_{r}^{2} ⟩ = \frac{1}{N_{g ∣ r}} M_{1} + \frac{(N_{g ∣ r} - 1)}{N_{g ∣ r}} M_{4},

⟨ {\hat{p}}_{r} {\hat{p}}_{r^{'}} ∣ FC ⟩ = \frac{1}{N_{g}} M_{5} + \frac{(N_{g} - 1)}{N_{g}} M_{8},

⟨ {\hat{p}}_{r} {\hat{p}}_{r^{'}} ∣ Dr - Pt ⟩ = M_{8} .

p_{γ} = ⟨ s (g, γ) ∣ γ ⟩ .

μ_{γ} = ⟨ p_{γ} ⟩ = ⟪ s (g, γ) ∣ γ ⟫ = M_{0},

σ_{γ}^{2} = ⟨ {⟨ s (g, γ) ∣ γ ⟩}^{2} ⟩ - {⟨ s (g, γ) ⟩}^{2} = M_{4} - M_{8} .

var ({\hat{p}}_{γ} {∣ N_{g}}_{∣ γ}) = var (⟨ {\hat{p}}_{γ} {∣ γ, N_{g}}_{∣ γ} ⟩) + ⟨ var ({\hat{p}}_{γ} {∣ γ, N_{g}}_{∣ γ}) ⟩

= var (p_{γ}) + ⟨ \frac{p_{γ} (1 - p_{γ})}{N_{g ∣ γ}} ⟩

= σ_{γ}^{2} + \frac{μ_{γ} (1 - μ_{γ}) - σ_{γ}^{2}}{N_{g ∣ r}} .

p_{g} = ⟨ s (g, γ) ∣ g ⟩ .

μ_{g} = ⟨ p_{g} ⟩ = ⟪ s (g, γ) ∣ g ⟫ = M_{0},

σ_{g}^{2} = ⟨ {⟨ s (g, γ) ∣ g ⟩}^{2} ⟩ - {⟨ s (g, γ) ⟩}^{2} = M_{5} - M_{8} .

{\hat{p}}_{G γ} = \bar{β} + β_{γ} + β_{G} + β_{G γ},

var (\hat{P} ∣ FC) = \frac{M_{4} - M_{8}}{N_{γ}} + \frac{M_{5} - M_{8}}{N_{g}} + \frac{M_{1} - M_{4} - M_{5} + M_{8}}{N_{γ} N_{g}} .

var (\hat{P} ∣ flat Dr - Pt) = \frac{M_{4} - M_{8}}{N_{γ}} + \frac{M_{1} - M_{4}}{N_{γ} N_{g ∣ r}} .

$\underset{̱}{c}$	Fully Crossed	Doctor–Patient
$\underset{̱}{c}$	Fully Crossed	$w_{r} = 1 ∕ N_{γ}$	$w_{r} = N_{g}_{∣ r} ∕ N_{g}$	General
$c_{1} =$	$1 ∕ N_{γ} N_{g}$	$1 ∕ N_{g}$	$1 ∕ N_{g}$	$\sum_{r = 1}^{R} w_{r}^{2} ∕ N_{g}_{∣ r}$
$c_{4} =$	$(N_{g} - 1) ∕ N_{γ} N_{g}$	$(N_{g} - N_{γ}) ∕ N_{γ} N_{g}$	$\sum_{r = 1}^{N_{γ}} (N_{g}^{2}_{∣ r} - N_{g}_{∣ r}) ∕ N_{g}^{2}$	$\sum_{r = 1}^{R} [w_{r}^{2} (N_{g}_{∣ r} - 1)] ∕ N_{g}_{∣ r}$
$c_{5} =$	$(N_{γ} - 1) ∕ N_{γ} N_{g}$	0	0	0
$c_{8} =$	$[(N_{g} - 1) (N_{γ} - 1) - N_{γ} N_{g}] ∕ N_{γ} N_{g}$	$- 1 ∕ N_{γ}$	$- \sum_{r = 1}^{N_{γ}} {(N_{g}_{∣ r} ∕ N_{g})}^{2}$	$\sum_{r = 1}^{R} \sum_{r^{'} \neq r}^{R} w_{r} w_{r^{'}}$

Experimental Design		Relative Components of Variance on the Scores
Performance:	$P C = [0.96, 0.86, 0.70]$	Reader:	$σ_{R}^{2} = [0.05, 0.10, 0.50]$
No. of readers:	$N_{γ} = [3, 5, 10]$	Case:	$σ_{C}^{2} = [0.05, 0.10, 0.50]$
Mean no. of trials/cases:	${\bar{N}}_{g}_{∣ r} = [12, 51, 102]$	Interaction:	$σ_{R C}^{2} = [0.05, 0.10, 0.50]$

Study Design	No. of Independent Cases per Reader
Study Design	R1	R2	R3	R4	R5
Flat	102	102	102	102	102
Poisson	93	100	89	106	98
Moderate uniform [52, 152]	85	142	120	81	123
Broad uniform [12, 192]	141	25	171	118	74
Fully crossed	Readers read same 102 cases
Hybrid	Please refer to the text

$\underset{̱}{c}$	Fully Crossed	Doctor–Patient
$\underset{̱}{c}$	Fully Crossed	$w_{r} = 1 ∕ N_{γ}$	$w_{r} = N_{g}_{∣ r} ∕ N_{g}$	General
$c_{1} =$	$1 ∕ N_{γ} N_{g}$	$1 ∕ N_{g}$	$1 ∕ N_{g}$	$\sum_{r = 1}^{R} w_{r}^{2} ∕ N_{g}_{∣ r}$
$c_{4} =$	$(N_{g} - 1) ∕ N_{γ} N_{g}$	$(N_{g} - N_{γ}) ∕ N_{γ} N_{g}$	$\sum_{r = 1}^{N_{γ}} (N_{g}^{2}_{∣ r} - N_{g}_{∣ r}) ∕ N_{g}^{2}$	$\sum_{r = 1}^{R} [w_{r}^{2} (N_{g}_{∣ r} - 1)] ∕ N_{g}_{∣ r}$
$c_{5} =$	$(N_{γ} - 1) ∕ N_{γ} N_{g}$	0	0	0
$c_{8} =$	$[(N_{g} - 1) (N_{γ} - 1) - N_{γ} N_{g}] ∕ N_{γ} N_{g}$	$- 1 ∕ N_{γ}$	$- \sum_{r = 1}^{N_{γ}} {(N_{g}_{∣ r} ∕ N_{g})}^{2}$	$\sum_{r = 1}^{R} \sum_{r^{'} \neq r}^{R} w_{r} w_{r^{'}}$

Experimental Design		Relative Components of Variance on the Scores
Performance:	$P C = [0.96, 0.86, 0.70]$	Reader:	$σ_{R}^{2} = [0.05, 0.10, 0.50]$
No. of readers:	$N_{γ} = [3, 5, 10]$	Case:	$σ_{C}^{2} = [0.05, 0.10, 0.50]$
Mean no. of trials/cases:	${\bar{N}}_{g}_{∣ r} = [12, 51, 102]$	Interaction:	$σ_{R C}^{2} = [0.05, 0.10, 0.50]$

Abstract

References

2007 (1)

2006 (3)

2005 (4)

2004 (1)

2000 (2)

1997 (2)

1992 (1)

Adrianza, M. E.

Barrett, H. H.

Beiden, S. V.

Berbaum, K. S.

Brown, D. G.

Campbell, G.

Clarkson, E.

Dorfman, D. D.

Gallas, B. D.

Hillis, S. L.

Ishwaran, H.

Jeronimo, J.

Kupinski, M. A.

Loew, M. H.

Massad, L. S.

Metz, C. E.

Obuchowski, N. A.

Pepe, M. S.

Roe, C. A.

Schartz, K. M.

Schiffman, M.

Song, H. H.

Song, X.

Wagner, R. F.

Yousef, W. A.

Zhou, X.-H.

Acad. Radiol. (7)

Acta Cytol. (1)

Am. J. Obstet. Gynecol. (1)

Biostatistics (1)

IEEE Trans. Pattern Anal. Mach. Intell. (1)

Invest. Radiol. (1)

Proc. SPIE (1)

Stat. Med. (1)

Other (2)

Cited By

Figures (5)

Tables (3)

Equations (52)

Metrics

Login or Create Account