Abstract

Unbiased Cramér–Rao lower bound (CRB) theory can be used to calculate lower bounds to the variances of unbiased estimates of a set of parameters given only the probability density function of a random vector conditioned on the true parameter values. However, when the estimated parameter values are required to satisfy inequality constraints such as positivity, the resulting estimator is typically biased. To calculate CRBs for biased estimates of the parameter values, an expression for the bias gradient matrix must also be known. Unfortunately, this expression often does not exist. Because expressions for biased CRBs are preferable to sample variance calculations, alternative methods for deriving biased CRB expressions associated with inequality constraints are needed. We present an alternative approach that is based upon creating the probability density function associated with a given biased estimate of these parameters using the available knowledge of the estimator properties. We apply this approach to the calculation of biased CRBs for estimators that use a positivity constraint with and without a support constraint for a specific measurement model and discuss the benefits and limitations of this approach.

© 2006 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. B. Porat, Digital Processing of Random Signals (Prentice Hall, 1994).
  2. A. O. Hero III, J. A. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the uniform Cramér-Rao bound," IEEE Trans. Signal Process. 44, 2026-2041 (1996).
    [CrossRef]
  3. D. C. Youla and H. Webb, "Image restoration by the method of convex projections: Part 1--theory," IEEE Trans. Med. Imaging MI-1, 81-94 (1982).
    [CrossRef]
  4. J. D. Gorman and A. O. Hero, "Lower bounds for parametric estimation with constraints," IEEE Trans. Inf. Theory 26, 1285-1301 (1990).
    [CrossRef]
  5. D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black object," J. R. Stat. Soc. Ser. B. Methodol. 54, 41-81 (1992).
  6. P. Stoica and T. L. Marzetta, "Parameter estimation problems with singular information matrices," IEEE Trans. Signal Process. 49, 87-90 (2001).
    [CrossRef]
  7. A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd ed. (McGraw-Hill, 1991).
  8. J. Goodman, Statistical Optics (Wiley, 1985).
  9. C. L. Matson, C. C. Beckner, and K. J. Schulze, "Fundamental limits to noise reduction in images using support--benefits from deconvolution," in Image Reconstruction from Incomplete Data III, P. J. Bones, M. A. Fiddy, and P. Millane, eds., Proc. SPIE 5562, 161-168 (2004).
    [CrossRef]

2004

C. L. Matson, C. C. Beckner, and K. J. Schulze, "Fundamental limits to noise reduction in images using support--benefits from deconvolution," in Image Reconstruction from Incomplete Data III, P. J. Bones, M. A. Fiddy, and P. Millane, eds., Proc. SPIE 5562, 161-168 (2004).
[CrossRef]

2001

P. Stoica and T. L. Marzetta, "Parameter estimation problems with singular information matrices," IEEE Trans. Signal Process. 49, 87-90 (2001).
[CrossRef]

1996

A. O. Hero III, J. A. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the uniform Cramér-Rao bound," IEEE Trans. Signal Process. 44, 2026-2041 (1996).
[CrossRef]

1992

D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black object," J. R. Stat. Soc. Ser. B. Methodol. 54, 41-81 (1992).

1990

J. D. Gorman and A. O. Hero, "Lower bounds for parametric estimation with constraints," IEEE Trans. Inf. Theory 26, 1285-1301 (1990).
[CrossRef]

1982

D. C. Youla and H. Webb, "Image restoration by the method of convex projections: Part 1--theory," IEEE Trans. Med. Imaging MI-1, 81-94 (1982).
[CrossRef]

Beckner, C. C.

C. L. Matson, C. C. Beckner, and K. J. Schulze, "Fundamental limits to noise reduction in images using support--benefits from deconvolution," in Image Reconstruction from Incomplete Data III, P. J. Bones, M. A. Fiddy, and P. Millane, eds., Proc. SPIE 5562, 161-168 (2004).
[CrossRef]

Donoho, D. L.

D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black object," J. R. Stat. Soc. Ser. B. Methodol. 54, 41-81 (1992).

Fessler, J. A.

A. O. Hero III, J. A. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the uniform Cramér-Rao bound," IEEE Trans. Signal Process. 44, 2026-2041 (1996).
[CrossRef]

Goodman, J.

J. Goodman, Statistical Optics (Wiley, 1985).

Gorman, J. D.

J. D. Gorman and A. O. Hero, "Lower bounds for parametric estimation with constraints," IEEE Trans. Inf. Theory 26, 1285-1301 (1990).
[CrossRef]

Hero, A. O.

A. O. Hero III, J. A. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the uniform Cramér-Rao bound," IEEE Trans. Signal Process. 44, 2026-2041 (1996).
[CrossRef]

J. D. Gorman and A. O. Hero, "Lower bounds for parametric estimation with constraints," IEEE Trans. Inf. Theory 26, 1285-1301 (1990).
[CrossRef]

Hoch, J. C.

D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black object," J. R. Stat. Soc. Ser. B. Methodol. 54, 41-81 (1992).

Johnstone, I. M.

D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black object," J. R. Stat. Soc. Ser. B. Methodol. 54, 41-81 (1992).

Marzetta, T. L.

P. Stoica and T. L. Marzetta, "Parameter estimation problems with singular information matrices," IEEE Trans. Signal Process. 49, 87-90 (2001).
[CrossRef]

Matson, C. L.

C. L. Matson, C. C. Beckner, and K. J. Schulze, "Fundamental limits to noise reduction in images using support--benefits from deconvolution," in Image Reconstruction from Incomplete Data III, P. J. Bones, M. A. Fiddy, and P. Millane, eds., Proc. SPIE 5562, 161-168 (2004).
[CrossRef]

Papoulis, A.

A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd ed. (McGraw-Hill, 1991).

Porat, B.

B. Porat, Digital Processing of Random Signals (Prentice Hall, 1994).

Schulze, K. J.

C. L. Matson, C. C. Beckner, and K. J. Schulze, "Fundamental limits to noise reduction in images using support--benefits from deconvolution," in Image Reconstruction from Incomplete Data III, P. J. Bones, M. A. Fiddy, and P. Millane, eds., Proc. SPIE 5562, 161-168 (2004).
[CrossRef]

Stern, A. S.

D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black object," J. R. Stat. Soc. Ser. B. Methodol. 54, 41-81 (1992).

Stoica, P.

P. Stoica and T. L. Marzetta, "Parameter estimation problems with singular information matrices," IEEE Trans. Signal Process. 49, 87-90 (2001).
[CrossRef]

Usman, M.

A. O. Hero III, J. A. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the uniform Cramér-Rao bound," IEEE Trans. Signal Process. 44, 2026-2041 (1996).
[CrossRef]

Webb, H.

D. C. Youla and H. Webb, "Image restoration by the method of convex projections: Part 1--theory," IEEE Trans. Med. Imaging MI-1, 81-94 (1982).
[CrossRef]

Youla, D. C.

D. C. Youla and H. Webb, "Image restoration by the method of convex projections: Part 1--theory," IEEE Trans. Med. Imaging MI-1, 81-94 (1982).
[CrossRef]

IEEE Trans. Inf. Theory

J. D. Gorman and A. O. Hero, "Lower bounds for parametric estimation with constraints," IEEE Trans. Inf. Theory 26, 1285-1301 (1990).
[CrossRef]

IEEE Trans. Med. Imaging

D. C. Youla and H. Webb, "Image restoration by the method of convex projections: Part 1--theory," IEEE Trans. Med. Imaging MI-1, 81-94 (1982).
[CrossRef]

IEEE Trans. Signal Process.

P. Stoica and T. L. Marzetta, "Parameter estimation problems with singular information matrices," IEEE Trans. Signal Process. 49, 87-90 (2001).
[CrossRef]

A. O. Hero III, J. A. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the uniform Cramér-Rao bound," IEEE Trans. Signal Process. 44, 2026-2041 (1996).
[CrossRef]

J. R. Stat. Soc. Ser. B. Methodol.

D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black object," J. R. Stat. Soc. Ser. B. Methodol. 54, 41-81 (1992).

Proc. SPIE

C. L. Matson, C. C. Beckner, and K. J. Schulze, "Fundamental limits to noise reduction in images using support--benefits from deconvolution," in Image Reconstruction from Incomplete Data III, P. J. Bones, M. A. Fiddy, and P. Millane, eds., Proc. SPIE 5562, 161-168 (2004).
[CrossRef]

Other

A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd ed. (McGraw-Hill, 1991).

J. Goodman, Statistical Optics (Wiley, 1985).

B. Porat, Digital Processing of Random Signals (Prentice Hall, 1994).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1
Fig. 1

Plots of positivity-constrained biased CRBs in the object support for two-element measurements with one element inside and one element outside the object support as a function of the ratio σ 1 2 σ 2 2 for (a) a single-element SNR of 5, (b) a single-element SNR of 1, and (c) a single-element SNR of 0.1. The asterisks, squares, and triangles are sample variances corresponding to cases (a), (b), and (c), respectively.

Fig. 2
Fig. 2

Plots of positivity-constrained biased CRBs in the object support for two-element measurements with both elements inside the object support as a function of the ratio σ 1 2 σ 2 2 for (a) both elements having an SNR of 5, (b) both elements having an SNR of 1, and (c) both elements having an SNR of 0.1. The asterisks, squares, and triangles are sample variances corresponding to cases (a), (b), and (c), respectively.

Fig. 3
Fig. 3

Plots of constrained CRBs in the object support for two-element measurements with one element inside the object support as a function of the ratio σ 1 2 σ 2 2 for a single-element SNR of 1 for (a) a positivity constraint, (b) a support constraint, and (c) both a positivity and a support constraint. The values in plots (a) and (c) are biased CRBs, while the values in plot (b) are unbiased CRBs. The squares and asterisks are sample variances corresponding to cases (b) and (c), respectively. The sample variances corresponding to case (a) are omitted here for clarity but are shown in Fig. 1.

Fig. 4
Fig. 4

Plots of positivity-constrained biased CRBs in the object support for two-element measurements with one element inside and one element outside the object support as a function of the single-element SNR for (a) σ 1 2 σ 2 2 = 1 , (b) σ 1 2 σ 2 2 = 100 , and (c) σ 1 2 σ 2 2 = 0.01 . The triangles, squares, and asterisks are sample variances corresponding to cases (a), (b), and (c), respectively.

Fig. 5
Fig. 5

Plots of positivity-constrained biased CRBs in the object support for two-element measurements with both elements inside the object support as a function of the single-element SNR for (a) σ 1 2 σ 2 2 = 1 , (b) σ 1 2 σ 2 2 = 100 , and (c) σ 1 2 σ 2 2 = 0.01 . The triangles, asterisks, and squares are sample variances corresponding to cases (a), (b), and (c), respectively.

Equations (25)

Equations on this page are rendered with MathJax. Learn more.

F pq ( θ ) = E [ ln f ( y ; θ ) θ p ln f ( y ; θ ) θ q ] = E [ 2 ln f ( y ; θ ) θ p θ q ] ,
CRB biased ( θ ) = diag { [ I + θ b ( θ ) ] F 1 ( θ ) [ I + θ b ( θ ) ] T } ,
E [ g ( y ) ] = g ( y ) f ( y ; θ ) d y = z f ( z ; θ ) d z ,
θ b g ( θ ) = θ [ g ( y ) f ( y ; θ ) d y θ ] = [ θ g ( y ) ] f ( y ; θ ) d y + g ( y ) [ θ f ( y ; θ ) ] d y I ,
θ b θ ̂ ( θ ) = θ [ θ ̂ f ( θ ̂ ; θ ) d θ ̂ θ ] = θ ̂ [ θ f ( θ ̂ ; θ ) ] d θ ̂ I .
CRB biased ( θ ) = diag { [ I + θ b θ ̂ ( θ ) ] F θ ̂ 1 ( θ ) [ I + θ b θ ̂ ( θ ) ] T } ,
f p ( θ ̂ p 1 = 0 , θ ̂ p 2 , , θ ̂ pN ; θ ) = f p ( θ ̂ p 2 , , θ ̂ pN θ ̂ p 1 = 0 ; θ ) f p ( θ ̂ p 1 = 0 , θ ̂ p 2 , , θ ̂ pN ; θ ) d θ ̂ p 2 d θ ̂ pN .
f p ( θ ̂ p 2 , , θ ̂ pN θ ̂ p 1 = 0 ; θ ) = f u ( θ ̂ u 2 , , θ ̂ uN θ ̂ u 1 = 0 ; θ ) = f u ( θ ̂ u 1 = 0 , θ ̂ u 2 , , θ ̂ uN ; θ ) [ f u ( θ ̂ u 1 = 0 , θ ̂ u 2 , , θ ̂ uN ; θ ) d θ ̂ u 2 d θ ̂ uN ] 1 .
f p ( θ ̂ p 1 = 0 , θ ̂ p 2 , , θ ̂ pN ; θ ) d θ ̂ p 2 d θ ̂ pN = 0 [ f u ( θ ̂ u 1 , , θ ̂ uN ; θ ) d θ ̂ u 2 d θ ̂ uN ] d θ ̂ u 1 ;
f p ( θ ̂ p 1 = 0 , θ ̂ p 2 , , θ ̂ pN ; θ ) = f u ( θ ̂ u 1 = 0 , θ ̂ u 2 , , θ ̂ uN ; θ ) [ f u ( θ ̂ u 1 = 0 , θ ̂ u 2 , , θ ̂ uN ; θ ) d θ ̂ u 2 d θ ̂ uN ] 1 × 0 [ f u ( θ ̂ u 1 , θ ̂ u 2 , , θ ̂ uN ; θ ) d θ ̂ u 2 d θ ̂ uN ] d θ ̂ u 1 δ ( θ ̂ p 1 ) u + ( θ ̂ p 2 ) u + ( θ ̂ pN ) ,
f p ( θ ̂ p 1 = 0 , , θ ̂ pM = 0 , θ ̂ p ( M + 1 ) , , θ ̂ pN ; θ ) = f u ( θ ̂ u 1 = 0 , , θ ̂ uM = 0 , θ ̂ u ( M + 1 ) , , θ ̂ uN ; θ ) [ f u ( θ ̂ u 1 = 0 , , θ ̂ uM = 0 , θ ̂ u ( M + 1 ) , , θ ̂ uN ; θ ) d θ ̂ u ( M + 1 ) d θ ̂ uN ] 1 × 0 0 [ f u ( θ ̂ u 1 , , θ ̂ uN ; θ ) d θ ̂ u ( M + 1 ) d θ ̂ uN ] d θ ̂ u 1 d θ ̂ uM × δ ( θ ̂ p 1 ) δ ( θ ̂ pM ) u + ( θ ̂ p ( M + 1 ) ) u + ( θ ̂ pN ) .
f p ( θ ̂ p 1 , , θ ̂ pN ; θ ) = f u ( θ ̂ u 1 , , θ ̂ uN ; θ ) u + ( θ ̂ p 1 ) u + ( θ ̂ pN ) ,
f p ( θ ̂ p 1 = 0 , , θ ̂ pN = 0 ; θ ) = { 1 0 0 [ sum of all the other terms in f p ( θ ̂ p ; θ ) ] d θ ̂ p 1 d θ ̂ pN } δ ( θ ̂ p 1 ) δ ( θ ̂ pN ) .
f s ( θ ̂ s 1 = 0 , , θ ̂ sM = 0 , θ ̂ s ( M + 1 ) , , θ ̂ sN ; θ ) = f s ( θ ̂ s ( M + 1 ) , , θ ̂ sN θ ̂ s 1 = 0 , , θ ̂ sM = 0 ; θ ) × f s ( θ ̂ s 1 = 0 , , θ ̂ sM = 0 , θ ̂ s ( M + 1 ) , , θ ̂ sN ; θ ) d θ ̂ s ( M + 1 ) d θ ̂ sN .
f u ( θ ̂ u ( M + 1 ) , , θ ̂ uN θ ̂ u 1 = 0 , , θ ̂ uM = 0 ; θ ) = f u ( θ ̂ u 1 = 0 , , θ ̂ uM = 0 , θ ̂ u ( M + 1 ) , , θ ̂ uN ; θ ) × [ f u ( θ ̂ u 1 = 0 , , θ ̂ uM = 0 , θ ̂ u ( M + 1 ) , , θ ̂ uN ; θ ) d θ ̂ u ( M + 1 ) d θ ̂ uN ] 1 .
f s ( θ ̂ s 1 , , θ ̂ sM = 0 , θ ̂ s ( M + 1 ) , , θ ̂ sN ; θ ) d θ ̂ s ( M + 1 ) d θ ̂ sN
= [ f u ( θ ̂ u 1 , , θ ̂ uN ; θ ) d θ ̂ u ( M + 1 ) d θ ̂ uN ] d θ ̂ u 1 d θ ̂ uM = 1 .
f s ( θ ̂ s ; θ ) = f u ( θ ̂ u 1 = 0 , , θ ̂ uM = 0 , θ ̂ u ( M + 1 ) , , θ ̂ uN ; θ ) × [ f u ( θ ̂ u 1 = 0 , , θ ̂ uM = 0 , θ ̂ u ( M + 1 ) , , θ ̂ uN ; θ ) d θ ̂ u ( M + 1 ) d θ ̂ uN ] 1 δ ( θ ̂ s 1 ) δ ( θ ̂ sM ) .
f ps ( θ ̂ ps 1 = 0 , , θ ̂ ps ( M + K ) = 0 , θ ̂ ps ( M + K + 1 ) , , θ ̂ pN ; θ ) = f u ( θ ̂ u 1 = 0 , , θ ̂ u ( M + K ) = 0 , θ ̂ u ( M + K + 1 ) , , θ ̂ uN ; θ ) [ f u ( θ ̂ u 1 = 0 , , θ ̂ u ( M + K ) = 0 , θ ̂ u ( M + K + 1 ) , , θ ̂ uN ; θ ) d θ ̂ n ( M + K + 1 ) d θ ̂ uN ] 1 × 0 0 [ f u ( θ ̂ u 1 , , θ ̂ uN ; θ ) d θ ̂ u ( M + 1 ) d θ ̂ uN ] d θ ̂ u 1 d θ ̂ uM × δ ( θ ̂ ps 1 ) δ ( θ ̂ ps ( M + K ) ) u + ( θ ̂ p ( M + K + 1 ) ) u + ( θ ̂ pN ) ,
y = θ + η ,
C = [ σ 1 2 + σ 2 2 σ 1 2 σ 2 2 σ 1 2 σ 2 2 σ 1 2 + σ 2 2 ] .
f p ( θ ̂ p 1 , θ ̂ p 2 ; θ ) = f Y ( y 1 = θ ̂ 1 , y 2 = θ ̂ 2 ; θ ) u + ( θ ̂ 1 ) u + ( θ ̂ 2 ) + f Y ( y 1 = θ ̂ 1 , y 2 = 0 ; θ ) f Y ( y 1 , y 2 = 0 ; θ ) d y 1 0 f Y ( y 1 , y 2 ; θ ) d y 1 d y 2 u + ( θ ̂ 1 ) δ ( θ ̂ 2 ) + f Y ( y 1 = 0 , y 2 = θ ̂ 2 ; θ ) f Y ( y 1 = 0 , y 2 ; θ ) d y 2 0 f Y ( y 1 , y 2 ; θ ) d y 1 d y 2 δ ( θ ̂ 1 ) u + ( θ ̂ 2 ) + [ 1 0 0 f Y ( y 1 = θ ̂ 1 , y 2 = θ ̂ 2 ; θ ) d θ ̂ 1 d θ ̂ 2 ] 0 f Y ( y 1 = θ ̂ 1 , y 2 = 0 ; θ ) d θ ̂ 1 f Y ( y 1 , y 2 = 0 ; θ ) d y 1 0 f Y ( y 1 , y 2 ; θ ) d y 1 d y 2 [ 0 f Y ( y 1 = 0 , y 2 = θ ̂ 2 ; θ ) d θ ̂ 2 f Y ( y 1 = 0 , y 2 ; θ ) d y 2 0 f Y ( y 1 , y 2 ; θ ) d y 1 d y 2 ] δ ( θ ̂ 1 ) δ ( θ ̂ 2 ) .
f s ( θ ̂ s 1 , θ ̂ s 2 ; θ ) = f Y ( y 1 = θ ̂ s 1 , y 2 = 0 ; θ ) f Y ( y 1 , y 2 = 0 ; θ ) d y 1 δ ( θ ̂ 2 ) .
f ps ( θ ̂ ps 1 , θ ̂ ps 2 ; θ ) = f Y ( y 1 = θ ̂ ps 1 , y 2 = 0 ; θ ) f Y ( y 1 , y 2 = 0 ; θ ) d y 1 u + ( θ ̂ ps 1 ) δ ( θ ̂ ps 2 ) + [ 1 0 f Y ( y 1 = θ ̂ ps 1 , y 2 = 0 ; θ ) d θ ̂ ps 1 f Y ( y 1 , y 2 = 0 ; θ ) d y 1 ] δ ( θ ̂ ps 1 ) δ ( θ ̂ ps 2 ) .
J ( α ) = ( y α 2 ) T C 1 ( y α 2 ) ,

Metrics