Abstract

A mathematical framework that permits the factorization of a joint probability distribution into its localized components for a two-dimensional array of pixels is presented. The factorization was used to estimate the contribution to mutual information of two- (I2) and three-pixel (I3) luminance correlations for a large ensemble of natural images analyzed at various spatial scales and pixel depths b. It is shown that both I2 and I3 saturate at b6 bits per pixel. Three-pixel correlations are shown to produce only a marginal increase of information redundancy (4%) over two-pixel correlations (50%). Implications for neural representation in visual cortex are discussed.

© 2003 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J. 27, 379–423 (1948).
    [CrossRef]
  2. E. R. Kretzmer, “Statistics of television signals,” Bell Syst. Tech. J. 31, 751–763 (1952).
    [CrossRef]
  3. W. F. Schreiber, “The measurement of third-order probability distributions of television signals,” IRE Trans. Inf. Theory IT-2, 94–105 (1956).
    [CrossRef]
  4. J. R. Parks, “Prediction and entropy of half-tone pictures,” Behav. Sci. 10, 436–445 (1965).
    [CrossRef] [PubMed]
  5. N. S. Tzannes, R. V. Spencer, A. J. Kaplan, “On estimating the entropy of random fields,” Inf. Control. 16, 1–6 (1970).
    [CrossRef]
  6. F. Attneave, “Some information aspects of visual perception,” Psychol. Rev. 61, 183–193 (1954).
    [CrossRef] [PubMed]
  7. H. B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. A. Rosenblith, ed. (MIT, Cambridge, Mass., 1961).
  8. J. J. Atick, “Could information theory provide an ecological theory of sensory processing?” Network 3, 213–251 (1992).
    [CrossRef]
  9. H. B. Barlow, P. Foldiak, The Computing Neuron (Addison-Wesley, Reading, Mass., 1989).
  10. S. Watanabe, “Pattern recognition as a quest for minimum entropy,” Pattern Recognition 13, 381–387 (1981).
    [CrossRef]
  11. S. Watanabe, Pattern Recognition: Human and Mechanical (Wiley, New York, 1985).
  12. S. B. Laughlin, “A simple coding procedure enhances a neuron’s information capacity,” Z. Naturforsch. 36, 910–912 (1981).
  13. J. J. Atick, A. N. Redlich, “Towards a theory of early visual processing,” Neural Comput. 2, 308–320 (1990).
    [CrossRef]
  14. J. J. Atick, Z. Li, A. N. Redlich, “Understanding retinal color coding from first principles,” Neural Comput. 4, 559–572 (1992).
    [CrossRef]
  15. J. J. Atick, A. N. Redlich, “What does the retina know about the natural scenes?” Neural Comput. 4, 196–210 (1992).
    [CrossRef]
  16. Zh. Li, J. J. Atick, “Towards a theory of the striate cortex,” Neural Comput. 6, 127–146 (1994).
    [CrossRef]
  17. A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
    [CrossRef]
  18. D. J. Field, “Relations between the statistics of natural images and the response properties of cortical cells,” J. Opt. Soc. Am. A 4, 2379–2394 (1987).
    [CrossRef] [PubMed]
  19. J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
    [CrossRef] [PubMed]
  20. D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
    [CrossRef]
  21. C. Jutten, J. Herault, “Blind separation of sources, part i: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
    [CrossRef]
  22. A. Hyvärinen, E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks 13, 411–430 (2000).
    [CrossRef]
  23. J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
    [CrossRef]
  24. A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces,” Neural Comput. 12, 1705–1720 (2000).
    [CrossRef]
  25. B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature 381, 607–609 (1996).
    [CrossRef] [PubMed]
  26. B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
    [CrossRef]
  27. M. G. A. Thomson, “Higher-order structure in natural scenes,” J. Opt. Soc. Am. A 16, 1549–1553 (1999).
    [CrossRef]
  28. M. G. A. Thomson, “Beats, kurtosis and visual coding,” Network Comput. Neural Syst. 12, 271–287 (2001).
    [CrossRef]
  29. A. B. Lee, K. S. Pedersen, D. Mumford, “The complex statistics of high-contrast patches in natural images,” presented at the IEEE Workshop on Statistical and Computational Theories of Vision, Vancouver, Canada, 2001; http://www.cis.ohio-state.edu/szhu/SCTV2001.html
  30. N. G. Deriugin, “The power spectrum and the correlation function of the television signal,” Telecommunications 1, 1–12 (1957).
  31. D. J. Tolhurst, Y. Tadmor, T. Chao, “Amplitude spectra of natural images,” Ophthalmic Physiol. Opt. 12, 229–232 (1992).
    [CrossRef] [PubMed]
  32. D. L. Ruderman, “The statistics of natural images,” Network 5, 517–548 (1994).
    [CrossRef]
  33. D. Kersten, “Predictability and redundancy in natural images,” J. Opt. Soc. Am. A 4, 2395–2400 (1987).
    [CrossRef] [PubMed]

2001 (1)

M. G. A. Thomson, “Beats, kurtosis and visual coding,” Network Comput. Neural Syst. 12, 271–287 (2001).
[CrossRef]

2000 (2)

A. Hyvärinen, E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks 13, 411–430 (2000).
[CrossRef]

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

1999 (1)

1998 (1)

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

1997 (2)

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

1996 (1)

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature 381, 607–609 (1996).
[CrossRef] [PubMed]

1994 (3)

D. L. Ruderman, “The statistics of natural images,” Network 5, 517–548 (1994).
[CrossRef]

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

Zh. Li, J. J. Atick, “Towards a theory of the striate cortex,” Neural Comput. 6, 127–146 (1994).
[CrossRef]

1992 (4)

J. J. Atick, Z. Li, A. N. Redlich, “Understanding retinal color coding from first principles,” Neural Comput. 4, 559–572 (1992).
[CrossRef]

J. J. Atick, A. N. Redlich, “What does the retina know about the natural scenes?” Neural Comput. 4, 196–210 (1992).
[CrossRef]

J. J. Atick, “Could information theory provide an ecological theory of sensory processing?” Network 3, 213–251 (1992).
[CrossRef]

D. J. Tolhurst, Y. Tadmor, T. Chao, “Amplitude spectra of natural images,” Ophthalmic Physiol. Opt. 12, 229–232 (1992).
[CrossRef] [PubMed]

1991 (1)

C. Jutten, J. Herault, “Blind separation of sources, part i: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

1990 (1)

J. J. Atick, A. N. Redlich, “Towards a theory of early visual processing,” Neural Comput. 2, 308–320 (1990).
[CrossRef]

1989 (1)

J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
[CrossRef] [PubMed]

1987 (2)

1981 (2)

S. Watanabe, “Pattern recognition as a quest for minimum entropy,” Pattern Recognition 13, 381–387 (1981).
[CrossRef]

S. B. Laughlin, “A simple coding procedure enhances a neuron’s information capacity,” Z. Naturforsch. 36, 910–912 (1981).

1970 (1)

N. S. Tzannes, R. V. Spencer, A. J. Kaplan, “On estimating the entropy of random fields,” Inf. Control. 16, 1–6 (1970).
[CrossRef]

1965 (1)

J. R. Parks, “Prediction and entropy of half-tone pictures,” Behav. Sci. 10, 436–445 (1965).
[CrossRef] [PubMed]

1957 (1)

N. G. Deriugin, “The power spectrum and the correlation function of the television signal,” Telecommunications 1, 1–12 (1957).

1956 (1)

W. F. Schreiber, “The measurement of third-order probability distributions of television signals,” IRE Trans. Inf. Theory IT-2, 94–105 (1956).
[CrossRef]

1954 (1)

F. Attneave, “Some information aspects of visual perception,” Psychol. Rev. 61, 183–193 (1954).
[CrossRef] [PubMed]

1952 (1)

E. R. Kretzmer, “Statistics of television signals,” Bell Syst. Tech. J. 31, 751–763 (1952).
[CrossRef]

1948 (1)

C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J. 27, 379–423 (1948).
[CrossRef]

Atick, J. J.

Zh. Li, J. J. Atick, “Towards a theory of the striate cortex,” Neural Comput. 6, 127–146 (1994).
[CrossRef]

J. J. Atick, Z. Li, A. N. Redlich, “Understanding retinal color coding from first principles,” Neural Comput. 4, 559–572 (1992).
[CrossRef]

J. J. Atick, A. N. Redlich, “What does the retina know about the natural scenes?” Neural Comput. 4, 196–210 (1992).
[CrossRef]

J. J. Atick, “Could information theory provide an ecological theory of sensory processing?” Network 3, 213–251 (1992).
[CrossRef]

J. J. Atick, A. N. Redlich, “Towards a theory of early visual processing,” Neural Comput. 2, 308–320 (1990).
[CrossRef]

Attneave, F.

F. Attneave, “Some information aspects of visual perception,” Psychol. Rev. 61, 183–193 (1954).
[CrossRef] [PubMed]

Barlow, H. B.

H. B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. A. Rosenblith, ed. (MIT, Cambridge, Mass., 1961).

H. B. Barlow, P. Foldiak, The Computing Neuron (Addison-Wesley, Reading, Mass., 1989).

Bell, A. J.

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

Chao, T.

D. J. Tolhurst, Y. Tadmor, T. Chao, “Amplitude spectra of natural images,” Ophthalmic Physiol. Opt. 12, 229–232 (1992).
[CrossRef] [PubMed]

Daugman, J. G.

J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
[CrossRef] [PubMed]

Deriugin, N. G.

N. G. Deriugin, “The power spectrum and the correlation function of the television signal,” Telecommunications 1, 1–12 (1957).

Field, D. J.

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature 381, 607–609 (1996).
[CrossRef] [PubMed]

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

D. J. Field, “Relations between the statistics of natural images and the response properties of cortical cells,” J. Opt. Soc. Am. A 4, 2379–2394 (1987).
[CrossRef] [PubMed]

Foldiak, P.

H. B. Barlow, P. Foldiak, The Computing Neuron (Addison-Wesley, Reading, Mass., 1989).

Herault, J.

C. Jutten, J. Herault, “Blind separation of sources, part i: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Hoyer, P. O.

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

Hyvärinen, A.

A. Hyvärinen, E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks 13, 411–430 (2000).
[CrossRef]

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

Jutten, C.

C. Jutten, J. Herault, “Blind separation of sources, part i: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Kaplan, A. J.

N. S. Tzannes, R. V. Spencer, A. J. Kaplan, “On estimating the entropy of random fields,” Inf. Control. 16, 1–6 (1970).
[CrossRef]

Kersten, D.

Kretzmer, E. R.

E. R. Kretzmer, “Statistics of television signals,” Bell Syst. Tech. J. 31, 751–763 (1952).
[CrossRef]

Laughlin, S. B.

S. B. Laughlin, “A simple coding procedure enhances a neuron’s information capacity,” Z. Naturforsch. 36, 910–912 (1981).

Lee, A. B.

A. B. Lee, K. S. Pedersen, D. Mumford, “The complex statistics of high-contrast patches in natural images,” presented at the IEEE Workshop on Statistical and Computational Theories of Vision, Vancouver, Canada, 2001; http://www.cis.ohio-state.edu/szhu/SCTV2001.html

Li, Z.

J. J. Atick, Z. Li, A. N. Redlich, “Understanding retinal color coding from first principles,” Neural Comput. 4, 559–572 (1992).
[CrossRef]

Li, Zh.

Zh. Li, J. J. Atick, “Towards a theory of the striate cortex,” Neural Comput. 6, 127–146 (1994).
[CrossRef]

Mumford, D.

A. B. Lee, K. S. Pedersen, D. Mumford, “The complex statistics of high-contrast patches in natural images,” presented at the IEEE Workshop on Statistical and Computational Theories of Vision, Vancouver, Canada, 2001; http://www.cis.ohio-state.edu/szhu/SCTV2001.html

Oja, E.

A. Hyvärinen, E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks 13, 411–430 (2000).
[CrossRef]

Olshausen, B. A.

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature 381, 607–609 (1996).
[CrossRef] [PubMed]

Parks, J. R.

J. R. Parks, “Prediction and entropy of half-tone pictures,” Behav. Sci. 10, 436–445 (1965).
[CrossRef] [PubMed]

Pedersen, K. S.

A. B. Lee, K. S. Pedersen, D. Mumford, “The complex statistics of high-contrast patches in natural images,” presented at the IEEE Workshop on Statistical and Computational Theories of Vision, Vancouver, Canada, 2001; http://www.cis.ohio-state.edu/szhu/SCTV2001.html

Redlich, A. N.

J. J. Atick, Z. Li, A. N. Redlich, “Understanding retinal color coding from first principles,” Neural Comput. 4, 559–572 (1992).
[CrossRef]

J. J. Atick, A. N. Redlich, “What does the retina know about the natural scenes?” Neural Comput. 4, 196–210 (1992).
[CrossRef]

J. J. Atick, A. N. Redlich, “Towards a theory of early visual processing,” Neural Comput. 2, 308–320 (1990).
[CrossRef]

Ruderman, D. L.

D. L. Ruderman, “The statistics of natural images,” Network 5, 517–548 (1994).
[CrossRef]

Schreiber, W. F.

W. F. Schreiber, “The measurement of third-order probability distributions of television signals,” IRE Trans. Inf. Theory IT-2, 94–105 (1956).
[CrossRef]

Sejnowski, T. J.

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

Shannon, C. E.

C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J. 27, 379–423 (1948).
[CrossRef]

Spencer, R. V.

N. S. Tzannes, R. V. Spencer, A. J. Kaplan, “On estimating the entropy of random fields,” Inf. Control. 16, 1–6 (1970).
[CrossRef]

Tadmor, Y.

D. J. Tolhurst, Y. Tadmor, T. Chao, “Amplitude spectra of natural images,” Ophthalmic Physiol. Opt. 12, 229–232 (1992).
[CrossRef] [PubMed]

Thomson, M. G. A.

M. G. A. Thomson, “Beats, kurtosis and visual coding,” Network Comput. Neural Syst. 12, 271–287 (2001).
[CrossRef]

M. G. A. Thomson, “Higher-order structure in natural scenes,” J. Opt. Soc. Am. A 16, 1549–1553 (1999).
[CrossRef]

Tolhurst, D. J.

D. J. Tolhurst, Y. Tadmor, T. Chao, “Amplitude spectra of natural images,” Ophthalmic Physiol. Opt. 12, 229–232 (1992).
[CrossRef] [PubMed]

Tzannes, N. S.

N. S. Tzannes, R. V. Spencer, A. J. Kaplan, “On estimating the entropy of random fields,” Inf. Control. 16, 1–6 (1970).
[CrossRef]

van der Schaaf, A.

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

van Hateren, J. H.

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

Watanabe, S.

S. Watanabe, “Pattern recognition as a quest for minimum entropy,” Pattern Recognition 13, 381–387 (1981).
[CrossRef]

S. Watanabe, Pattern Recognition: Human and Mechanical (Wiley, New York, 1985).

Behav. Sci. (1)

J. R. Parks, “Prediction and entropy of half-tone pictures,” Behav. Sci. 10, 436–445 (1965).
[CrossRef] [PubMed]

Bell Syst. Tech. J. (2)

C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J. 27, 379–423 (1948).
[CrossRef]

E. R. Kretzmer, “Statistics of television signals,” Bell Syst. Tech. J. 31, 751–763 (1952).
[CrossRef]

IEEE Trans. Biomed. Eng. (1)

J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
[CrossRef] [PubMed]

Inf. Control. (1)

N. S. Tzannes, R. V. Spencer, A. J. Kaplan, “On estimating the entropy of random fields,” Inf. Control. 16, 1–6 (1970).
[CrossRef]

IRE Trans. Inf. Theory (1)

W. F. Schreiber, “The measurement of third-order probability distributions of television signals,” IRE Trans. Inf. Theory IT-2, 94–105 (1956).
[CrossRef]

J. Opt. Soc. Am. A (3)

Nature (1)

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature 381, 607–609 (1996).
[CrossRef] [PubMed]

Network (2)

D. L. Ruderman, “The statistics of natural images,” Network 5, 517–548 (1994).
[CrossRef]

J. J. Atick, “Could information theory provide an ecological theory of sensory processing?” Network 3, 213–251 (1992).
[CrossRef]

Network Comput. Neural Syst. (1)

M. G. A. Thomson, “Beats, kurtosis and visual coding,” Network Comput. Neural Syst. 12, 271–287 (2001).
[CrossRef]

Neural Comput. (6)

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

J. J. Atick, A. N. Redlich, “Towards a theory of early visual processing,” Neural Comput. 2, 308–320 (1990).
[CrossRef]

J. J. Atick, Z. Li, A. N. Redlich, “Understanding retinal color coding from first principles,” Neural Comput. 4, 559–572 (1992).
[CrossRef]

J. J. Atick, A. N. Redlich, “What does the retina know about the natural scenes?” Neural Comput. 4, 196–210 (1992).
[CrossRef]

Zh. Li, J. J. Atick, “Towards a theory of the striate cortex,” Neural Comput. 6, 127–146 (1994).
[CrossRef]

Neural Networks (1)

A. Hyvärinen, E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks 13, 411–430 (2000).
[CrossRef]

Ophthalmic Physiol. Opt. (1)

D. J. Tolhurst, Y. Tadmor, T. Chao, “Amplitude spectra of natural images,” Ophthalmic Physiol. Opt. 12, 229–232 (1992).
[CrossRef] [PubMed]

Pattern Recognition (1)

S. Watanabe, “Pattern recognition as a quest for minimum entropy,” Pattern Recognition 13, 381–387 (1981).
[CrossRef]

Proc. R. Soc. London Ser. B (1)

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

Psychol. Rev. (1)

F. Attneave, “Some information aspects of visual perception,” Psychol. Rev. 61, 183–193 (1954).
[CrossRef] [PubMed]

Signal Process. (1)

C. Jutten, J. Herault, “Blind separation of sources, part i: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Telecommunications (1)

N. G. Deriugin, “The power spectrum and the correlation function of the television signal,” Telecommunications 1, 1–12 (1957).

Vision Res. (2)

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

Z. Naturforsch. (1)

S. B. Laughlin, “A simple coding procedure enhances a neuron’s information capacity,” Z. Naturforsch. 36, 910–912 (1981).

Other (4)

S. Watanabe, Pattern Recognition: Human and Mechanical (Wiley, New York, 1985).

H. B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. A. Rosenblith, ed. (MIT, Cambridge, Mass., 1961).

H. B. Barlow, P. Foldiak, The Computing Neuron (Addison-Wesley, Reading, Mass., 1989).

A. B. Lee, K. S. Pedersen, D. Mumford, “The complex statistics of high-contrast patches in natural images,” presented at the IEEE Workshop on Statistical and Computational Theories of Vision, Vancouver, Canada, 2001; http://www.cis.ohio-state.edu/szhu/SCTV2001.html

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (4)

Fig. 1
Fig. 1

Entropy as a function of kurtosis for distribution P(x)exp[-(|x|/σ)m] (see insets) of the same variance or second-order statistics (by adjusting σ with m). Arrows link points on H(κ) to corresponding P(x)’s.

Fig. 2
Fig. 2

(a) Joint probability factorization for a square grid. Dashed curves indicate the maximum correlation radius used for three-pixel factors. Open circles mark nodes in the factorization beyond the correlation radius. The set of diagrams illustrates factors in Eq. (9) (shown in reversed order here); gray lines connect the end pixels for each factor. (b) Correlation factors B and C [see Fig. 2(a)] transformed using Eqs. (7) and (8) to the effect of factoring out two- and three-pixel correlations. The second line in C shows transformation of the last factor from the previous line. (c) One-, two-, and three-pixel correlation factors resulting from the local probability factorization.

Fig. 3
Fig. 3

Information distribution matrices for b=7. Lower left and upper right corners in each matrix correspond to (0, 0) and (127, 127) pixel pair intensity, respectively, while positive and negative elements are shown by light and dark shades. (a) Two-pixel matrices with corresponding pixel geometry shown in the insets. (b) Vertical row three-pixel matrix and (c) one of the four corner, three-pixel matrices, visualized by means of four planar cross sections. Corresponding three-dimensional cuts are shown below each cross section; thick line marks the main diagonal.

Fig. 4
Fig. 4

Local information as a function of pixel depth and spatial scale. Small, medium, and large squares represent scales 1×, 2×, and 3×, respectively. (a) Single-pixel entropy. (b) Two-pixel mutual information as given by relation (14). (c) Three-pixel contribution to mutual information as given by relation (15). Redundancy factors are shown as insets: (a) intrapixel redundancy factor; (b), (c) two- and three-pixel contributions, respectively, to the interpixel correlation redundancy factor.

Tables (2)

Tables Icon

Table 1 Individual Contributions from Two-Pixel and Three-Pixel Configurations to Mutual Information at Scale 1× and 12 Pixel Depths

Tables Icon

Table 2 One-, Two-, and Three-Pixel Contributions to Information Content of Natural Images for 3 Spatial Scales and 12 Pixel Depthsa

Equations (39)

Equations on this page are rendered with MathJax. Learn more.

P(1, 2, 3, )=P(1)P(2|1)P(3|2, 1)P(4|3, 2, 1) =P(1)×P(2) P(2|1)P(2)P(3) P(3|2)P(3) P(3|2, 1)P(3|2)×P(4) P(4|3)P(4) P(4|3, 2)P(4|3) P(4|3, 2, 1)P(4|3, 2) .
P(, 1, 2, 3, , i, )=i=-P˜(i|i-1, i-2, ),
P˜(i|i-1, i-2, )=P(i) P(i|i-1)P(i)×P(i|i-1, i-2)P(i|i-1) P(i|i-1, i-2, i-3)P(i|i-1, i-2)  
H=-E log2[P(, 1, 2, 3, , i, )]=-i=-E log2P(i) P(i|i-1)P(i) P(i|i-1, i-2)P(i|i-1)  =-i=-E log2[P(i)]+E log2P(i|i-1)P(i)+E log2P(i|i-1, i-2)P(i|i-1)  ,
H=N[H1-I2-I3-I4- ],
H1=-E log2[P(i)],I2=E log2P(i|i-1)P(i),I3=E log2P(i|i-1, i-2)P(i|i-1), .
P(a|b, c|d)P(a|b, c, d)P(a|b, c)P(a, b, c, d)P(b, c)P(a, b, c)P(b, c, d)=P(d|b, c|a).
P(a|b, c|d)=P(b|a, c|d) P(a|c|d)P(b|c|d),
P˜(7|6, 5, )=P(7)P(7|6)P(7|6|5)P(7|6, 5|4)P(7|6, 5, 4|3)×P(7|6, 5, 4, 3|2)P(7|6, 5, 4, 3, 2|1)=P(7)P(7|6)P(7|6|5)P(7|6, 5|4)P(7|6, 4|3)×P(7|4, 3|2)P(7|4, 3, 2|1),
P(7|4, 3|2)=P(4|7, 3|2) P(7|3|2)P(4|3|2),
P(7|6, 5|4)=P(7|6, 4|5) P(7|6|4)P(7|6|5).
P(7|6, 4|3)=P(7|6, 3|4) P(7|6|3)P(7|6|4),P(7|6|3)=P(6|7|3) P(7|3)P(6|3).
P˜(7|6, 5, )P(7) P(76)P(73)P(63) P(7|3|2)P(6|7|3)P(4|3|2),
I2=E{log2[P(76)]+log2[P(73)]-log2[P(63)]},
I3=E{log2[P(7|3|2)+log2(P(6|7|3)]-log2[P(4|3|2)]}.
Hnoisy-H=i=1nPi log2 Pi-i=1n(Pi+νi)log2(Pi+νi)=-i=1nνi log2 Pi-i=1nPi1+νiPilog21+νiPi.
i=1mPi1+νiPilog21+νiPi=1log 2 i=1mPi1+νiPik=1 -1k -νiPik.
i=m+1n(Pi+νi)log21+νiPiR>0
Hnoisy-H<-i=1nνi log2 Pi-1log 2 i=1mνi-12 log 2 i=1mνi2Pi-13 νi3Pi2.
EHnoisy<H-1N2 log 2 i=1m1-13Ni,
I1,2noisy-I1,2(H1noisy-H1)+(H2noisy-H2)-(H1,2noisy-H1,2).
EI1,2noisy-I1,21N2 log 2 i,j=1m1-13Nij-2im1-13Ni+R1,2-R1-R2.
Pn(i/2)=P2n(i)+P2n(i+1).
P1=P+δP=P2n(i),P2=P-δP=P2n(i+1),
Hi1(2n)-Hi1(n)=(P1+P2)log2(P1+P2)-P1 log2 P1-P2 log2 P2=2P-P log21-δPP2+δP×log21-δPP-log21+δPP.
Hi1(2n)-Hi1(n)=2P-1log 2 k=1 1k(2k-1) (δP)2kP2k-1.
H1(2n)-H1(n)=i=1n2Pi-1log 2 i=1nk=1 1k(2k-1) (δPi)2kPi2k-1=1-1log 2 i=1nk=1 1k(2k-1) (δPi)2kPi2k-1,
Pn(i/2, j/2)=P2n(i, j)+P2n(i, j+1)+P2n(i+1, j)+P2n(i+1, j+1).
P1=P+δP1=P2n(i, j),P2=P+δP2=P2n(i, j+1),P3=P+δP3=P2n(i+1, j),P4=P+δP4=P2n(i+1, j+1),
Hij1,2(2n)-Hij1,2(n)=(P1+P2+P3+P4)log2(P1+P2+P3+P4)-P1 log2 P1-P2 log2 P2-P3 log2 P3-P4 log2 P48P-(δP1)2+(δP2)2+(δP3)2+(δP4)22P log 2.
H1,2(2n)-H1,2(n)=i,j=1n8Pij-12 log 2 i,j=1n 1Pij a=14(δPija)2=2-12 log 2 i,j=1n 1Pij a=14(δPija)2.
I2(2n)-I2(n)=12 log 2 i,j=1n 1Pij a=14(δPija)2-4i=1n (δPi)2Pi.
y={y1, , yn}={f1(x1), f2(x2), , fn(xn)}
Hn(y)=Hn(x)+E log2|J(x)|,
Hn(y)=Hn(x)+E log2 i=1n dfidxi=Hn(x)+i=1nE log2 dfidxi,
I3(y1, y2, y3)=H(y1, y2, y3)-H(y1, y2)-H(y2, y3)+H(y2)=I3(x1, x2, x3)+i=13E log2 dfidxi-i=12E log2 dfidxi-i=23E log2 dfidxi+E log2 df2dx2=I3(x1, x2, x3).
Gn(x)exp(-12xTR-1x)(2π)n/2[det(R)]1/2,
H(Gn)=log2{(2πe)n/2[det(R)]1/2}
I(G2)=2H(G1)-H(G2)=-log21-x1x22x1221/2.

Metrics