Abstract

A wide variety of recent studies have argued that the human visual system provides an efficient means of processing the information in the natural environment. However, the amount of information (entropy) in the signal can be estimated in a number of ways, and it is has been unclear how much of the information is carried by the different sources of redundancy. The primary difficulty is that there has been no rational way to estimate the entropy of such complex scenes. In this paper, we provide a technique that uses a recent approach to estimating the entropy and dimensionality of natural scenes [D. M. Chandler and D. J. Field, J. Opt. Soc. Am. A 24, 922–941 (2007)] to estimate the amount of information attributable to the power and phase spectra in natural-scene patches. By comparing the entropies of patches that have swapped phase spectra and fixed phase spectra, we demonstrate how to estimate both the amount of information in each type of spectrum and the amount of information that is shared by these spectra (mutual information). We applied this technique to small patches (4×4 and 8×8). From our estimates, we show that the power spectrum of 8×8 patches carries approximately 54% of the total information, the phase spectrum carries 56%, and 10% is mutual information (54%+56%10%=100%). This technique is currently limited to relatively small image patches, due to the number of patches currently in our collection (on the order of 106). However, the technique can, in theory, be extended to larger images. Even with these relatively small patches, we discuss how these results can provide important insights into both compression techniques and efficient coding techniques that work with relatively small image patches (e.g., JPEG, sparse coding, independent components analysis).

© 2012 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. D. J. Field, “Relations between the statistics of natural images and the response properties of cortical cells,” J. Opt. Soc. Am. A 4, 2379–2394 (1987).
    [CrossRef] [PubMed]
  2. D. J. Field, “Scale-invariance and self-similar ‘wavelet’ transforms: an analysis of natural scenes and mammalian visual systems,” in Wavelets, Fractals and Fourier Transforms: New Developments and New Applications, M.Farge, J.C. R.Hunt, and J.C.Vassilicos, eds. (Oxford University, 1993), pp. 151–193.
  3. D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
    [CrossRef]
  4. B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vis. Res. 37, 3311–3325 (1997).
    [CrossRef]
  5. E. P. Simoncelli and B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216(2001).
    [CrossRef] [PubMed]
  6. W. E. Vinje and J. L. Gallant, “Sparse coding and decorrelation in primary visual cortex during natural vision,” Science 287, 1273–1276 (2000).
    [CrossRef] [PubMed]
  7. A. Bell and T. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vis. Res. 37, 3327–3338 (1997).
    [CrossRef]
  8. P. O. Hoyer and A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vis. Res. 42, 1593–1605 (2002).
    [CrossRef] [PubMed]
  9. L. Wiskott and T. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770(2002).
    [CrossRef] [PubMed]
  10. W. E. Vinje and J. L. Gallant, “Natural stimulation of the nonclassical receptive field increases information transmission efficiency in V1,” J. Neurosci. 22, 2904–2915 (2002).
    [PubMed]
  11. P. Garrigues and B. A. Olshausen, “Learning horizontal connections in a sparse coding model of natural images,” in Advances in Neural Information Processing Systems 20, J.C.Platt, D.Koller, Y.Singer, and S.Roweis, eds. (MIT Press, 2008), pp. 505–512.
  12. A. B. Lee, K. S. Pedersen, and D. Mumford, “The nonlinear statistics of high-contrast patches in natural images,” Int. J. Comput. Vis. 54, 83–103 (2003).
    [CrossRef]
  13. W. B. Pennebaker and J. L. Mitchell, The JPEG Still Image Data Compression Standard (Van Nostrand Reinhold, 1993).
  14. International Organization for Standardization, “Information technology—JPEG 2000 image coding system: core coding system,” Tech. Rep. ISO/IEC FDIS15444-1:2000 (International Organization for Standardization, 2000).
  15. D. M. Chandler and D. J. Field, “Estimates of the information content and dimensionality of natural scenes from proximity distributions,” J. Opt. Soc. Am. A 24, 922–941 (2007).
    [CrossRef]
  16. R. Hosseini, F. Sinz, and M. Bethge, “Lower bounds on the redundancy of natural images,” Vis. Res. 50, 2213–2222 (2010).
    [CrossRef] [PubMed]
  17. G. J. Burton and I. R. Moorhead, “Color and spatial structure in natural scenes,” Appl. Opt. 26, 157–170 (1987).
    [CrossRef] [PubMed]
  18. D. J. Tolhurst, Y. Tadmor, and T. Chao, “Amplitude spectra of natural images,” Ophthalmol. Physiol. Opt. 12, 229–232 (1992).
    [CrossRef]
  19. A. V. Oppenheim and J. S. Lim, “The importance of phase in signals,” Proc. IEEE 69, 529–541 (1981).
    [CrossRef]
  20. L. N. Piotrowski and F. W. Campbell, “A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase,” Perception 11, 337–346 (1982).
    [CrossRef] [PubMed]
  21. R. P. Millane and W. H. Hsiao, “The basis of phase dominance,” Opt. Lett. 34, 2607–2609 (2009).
    [CrossRef] [PubMed]
  22. M. J. Morgan, J. Ross, and A. Hayes, “The relative importance of local phase and local amplitude in patchwise image reconstruction,” Biol. Cybern. 65, 113–119 (1991).
    [CrossRef]
  23. D. L. Ruderman, “Origins of scaling in natural images,” Vis. Res. 37, 3385–3398 (1997).
    [CrossRef]
  24. W. H. Hsiao and R. P. Millane, “Effects of occlusion, edges, and scaling on the power spectra of natural images,” J. Opt. Soc. Am. A 22, 1789–1797 (2005).
    [CrossRef]
  25. D. J. Graham and D. J. Field, “Statistical regularities of art images and natural scenes: spectra, sparseness and nonlinearities,” Spatial Vis. 21, 149–64 (2007). PMID: 18073056.
    [CrossRef]
  26. A. Torralba and A. Oliva, “Statistics of natural image categories,” Network: Comput. Neural Syst. 14, 391–412(2003).
    [CrossRef]
  27. Y. Tadmor and D. Tolhurst, “Both the phase and the amplitude spectrum may determine the appearance of natural images,” Vis. Res. 33, 141–145 (1993).
    [CrossRef] [PubMed]
  28. L. F. Kozachenko and N. N. Leonenko, “A statistical estimate for the entropy of a random vector,” Probl. Inf. Transm. 23, 9–16(1987).
  29. J. D. Victor, “Binless strategies for estimation of information from neural data,” Phys. Rev. E 66051903 (2002).
    [CrossRef]
  30. A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual information,” Phys. Rev. E 69 (2004).
    [CrossRef]
  31. J. Kybic, “High-dimensional mutual information estimation for image registration,” in Proceedings of International Conference on Image Processing, 2004, Vol. 3 (IEEE, 2005), pp. 1779–1782.
  32. J. H. van Hateren and A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. Lond. Ser. B 265, 359–366(1998).
    [CrossRef]
  33. R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE Trans. Inf. Theory 44, 2325–2384 (1998).
    [CrossRef]
  34. The values of T=65,536 and T=16,384 and ten trials of the search procedure (using ten randomizations of Group T) were sufficient to yield a consistent sample mean.
  35. M. R. Celis, J. E. Dennis, and R. A. Tapia, “A trust region strategy for nonlinear equality constrained optimization,” in Numerical Optimization 1984, P.T.Boggs, R.H.Byrd, and R.B.Schnabel, eds. (SIAM, 1985), pp. 71–82.

2010 (1)

R. Hosseini, F. Sinz, and M. Bethge, “Lower bounds on the redundancy of natural images,” Vis. Res. 50, 2213–2222 (2010).
[CrossRef] [PubMed]

2009 (1)

2007 (2)

D. M. Chandler and D. J. Field, “Estimates of the information content and dimensionality of natural scenes from proximity distributions,” J. Opt. Soc. Am. A 24, 922–941 (2007).
[CrossRef]

D. J. Graham and D. J. Field, “Statistical regularities of art images and natural scenes: spectra, sparseness and nonlinearities,” Spatial Vis. 21, 149–64 (2007). PMID: 18073056.
[CrossRef]

2005 (1)

2004 (1)

A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual information,” Phys. Rev. E 69 (2004).
[CrossRef]

2003 (2)

A. Torralba and A. Oliva, “Statistics of natural image categories,” Network: Comput. Neural Syst. 14, 391–412(2003).
[CrossRef]

A. B. Lee, K. S. Pedersen, and D. Mumford, “The nonlinear statistics of high-contrast patches in natural images,” Int. J. Comput. Vis. 54, 83–103 (2003).
[CrossRef]

2002 (4)

P. O. Hoyer and A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vis. Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

L. Wiskott and T. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770(2002).
[CrossRef] [PubMed]

W. E. Vinje and J. L. Gallant, “Natural stimulation of the nonclassical receptive field increases information transmission efficiency in V1,” J. Neurosci. 22, 2904–2915 (2002).
[PubMed]

J. D. Victor, “Binless strategies for estimation of information from neural data,” Phys. Rev. E 66051903 (2002).
[CrossRef]

2001 (1)

E. P. Simoncelli and B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216(2001).
[CrossRef] [PubMed]

2000 (1)

W. E. Vinje and J. L. Gallant, “Sparse coding and decorrelation in primary visual cortex during natural vision,” Science 287, 1273–1276 (2000).
[CrossRef] [PubMed]

1998 (2)

J. H. van Hateren and A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. Lond. Ser. B 265, 359–366(1998).
[CrossRef]

R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE Trans. Inf. Theory 44, 2325–2384 (1998).
[CrossRef]

1997 (3)

A. Bell and T. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vis. Res. 37, 3327–3338 (1997).
[CrossRef]

B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vis. Res. 37, 3311–3325 (1997).
[CrossRef]

D. L. Ruderman, “Origins of scaling in natural images,” Vis. Res. 37, 3385–3398 (1997).
[CrossRef]

1994 (1)

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

1993 (1)

Y. Tadmor and D. Tolhurst, “Both the phase and the amplitude spectrum may determine the appearance of natural images,” Vis. Res. 33, 141–145 (1993).
[CrossRef] [PubMed]

1992 (1)

D. J. Tolhurst, Y. Tadmor, and T. Chao, “Amplitude spectra of natural images,” Ophthalmol. Physiol. Opt. 12, 229–232 (1992).
[CrossRef]

1991 (1)

M. J. Morgan, J. Ross, and A. Hayes, “The relative importance of local phase and local amplitude in patchwise image reconstruction,” Biol. Cybern. 65, 113–119 (1991).
[CrossRef]

1987 (3)

1982 (1)

L. N. Piotrowski and F. W. Campbell, “A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase,” Perception 11, 337–346 (1982).
[CrossRef] [PubMed]

1981 (1)

A. V. Oppenheim and J. S. Lim, “The importance of phase in signals,” Proc. IEEE 69, 529–541 (1981).
[CrossRef]

Bell, A.

A. Bell and T. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vis. Res. 37, 3327–3338 (1997).
[CrossRef]

Bethge, M.

R. Hosseini, F. Sinz, and M. Bethge, “Lower bounds on the redundancy of natural images,” Vis. Res. 50, 2213–2222 (2010).
[CrossRef] [PubMed]

Burton, G. J.

Campbell, F. W.

L. N. Piotrowski and F. W. Campbell, “A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase,” Perception 11, 337–346 (1982).
[CrossRef] [PubMed]

Celis, M. R.

M. R. Celis, J. E. Dennis, and R. A. Tapia, “A trust region strategy for nonlinear equality constrained optimization,” in Numerical Optimization 1984, P.T.Boggs, R.H.Byrd, and R.B.Schnabel, eds. (SIAM, 1985), pp. 71–82.

Chandler, D. M.

Chao, T.

D. J. Tolhurst, Y. Tadmor, and T. Chao, “Amplitude spectra of natural images,” Ophthalmol. Physiol. Opt. 12, 229–232 (1992).
[CrossRef]

Dennis, J. E.

M. R. Celis, J. E. Dennis, and R. A. Tapia, “A trust region strategy for nonlinear equality constrained optimization,” in Numerical Optimization 1984, P.T.Boggs, R.H.Byrd, and R.B.Schnabel, eds. (SIAM, 1985), pp. 71–82.

Field, D. J.

D. J. Graham and D. J. Field, “Statistical regularities of art images and natural scenes: spectra, sparseness and nonlinearities,” Spatial Vis. 21, 149–64 (2007). PMID: 18073056.
[CrossRef]

D. M. Chandler and D. J. Field, “Estimates of the information content and dimensionality of natural scenes from proximity distributions,” J. Opt. Soc. Am. A 24, 922–941 (2007).
[CrossRef]

B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vis. Res. 37, 3311–3325 (1997).
[CrossRef]

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

D. J. Field, “Relations between the statistics of natural images and the response properties of cortical cells,” J. Opt. Soc. Am. A 4, 2379–2394 (1987).
[CrossRef] [PubMed]

D. J. Field, “Scale-invariance and self-similar ‘wavelet’ transforms: an analysis of natural scenes and mammalian visual systems,” in Wavelets, Fractals and Fourier Transforms: New Developments and New Applications, M.Farge, J.C. R.Hunt, and J.C.Vassilicos, eds. (Oxford University, 1993), pp. 151–193.

Gallant, J. L.

W. E. Vinje and J. L. Gallant, “Natural stimulation of the nonclassical receptive field increases information transmission efficiency in V1,” J. Neurosci. 22, 2904–2915 (2002).
[PubMed]

W. E. Vinje and J. L. Gallant, “Sparse coding and decorrelation in primary visual cortex during natural vision,” Science 287, 1273–1276 (2000).
[CrossRef] [PubMed]

Garrigues, P.

P. Garrigues and B. A. Olshausen, “Learning horizontal connections in a sparse coding model of natural images,” in Advances in Neural Information Processing Systems 20, J.C.Platt, D.Koller, Y.Singer, and S.Roweis, eds. (MIT Press, 2008), pp. 505–512.

Graham, D. J.

D. J. Graham and D. J. Field, “Statistical regularities of art images and natural scenes: spectra, sparseness and nonlinearities,” Spatial Vis. 21, 149–64 (2007). PMID: 18073056.
[CrossRef]

Grassberger, P.

A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual information,” Phys. Rev. E 69 (2004).
[CrossRef]

Gray, R. M.

R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE Trans. Inf. Theory 44, 2325–2384 (1998).
[CrossRef]

Hayes, A.

M. J. Morgan, J. Ross, and A. Hayes, “The relative importance of local phase and local amplitude in patchwise image reconstruction,” Biol. Cybern. 65, 113–119 (1991).
[CrossRef]

Hosseini, R.

R. Hosseini, F. Sinz, and M. Bethge, “Lower bounds on the redundancy of natural images,” Vis. Res. 50, 2213–2222 (2010).
[CrossRef] [PubMed]

Hoyer, P. O.

P. O. Hoyer and A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vis. Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

Hsiao, W. H.

Hyvärinen, A.

P. O. Hoyer and A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vis. Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

Kozachenko, L. F.

L. F. Kozachenko and N. N. Leonenko, “A statistical estimate for the entropy of a random vector,” Probl. Inf. Transm. 23, 9–16(1987).

Kraskov, A.

A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual information,” Phys. Rev. E 69 (2004).
[CrossRef]

Kybic, J.

J. Kybic, “High-dimensional mutual information estimation for image registration,” in Proceedings of International Conference on Image Processing, 2004, Vol. 3 (IEEE, 2005), pp. 1779–1782.

Lee, A. B.

A. B. Lee, K. S. Pedersen, and D. Mumford, “The nonlinear statistics of high-contrast patches in natural images,” Int. J. Comput. Vis. 54, 83–103 (2003).
[CrossRef]

Leonenko, N. N.

L. F. Kozachenko and N. N. Leonenko, “A statistical estimate for the entropy of a random vector,” Probl. Inf. Transm. 23, 9–16(1987).

Lim, J. S.

A. V. Oppenheim and J. S. Lim, “The importance of phase in signals,” Proc. IEEE 69, 529–541 (1981).
[CrossRef]

Millane, R. P.

Mitchell, J. L.

W. B. Pennebaker and J. L. Mitchell, The JPEG Still Image Data Compression Standard (Van Nostrand Reinhold, 1993).

Moorhead, I. R.

Morgan, M. J.

M. J. Morgan, J. Ross, and A. Hayes, “The relative importance of local phase and local amplitude in patchwise image reconstruction,” Biol. Cybern. 65, 113–119 (1991).
[CrossRef]

Mumford, D.

A. B. Lee, K. S. Pedersen, and D. Mumford, “The nonlinear statistics of high-contrast patches in natural images,” Int. J. Comput. Vis. 54, 83–103 (2003).
[CrossRef]

Neuhoff, D. L.

R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE Trans. Inf. Theory 44, 2325–2384 (1998).
[CrossRef]

Oliva, A.

A. Torralba and A. Oliva, “Statistics of natural image categories,” Network: Comput. Neural Syst. 14, 391–412(2003).
[CrossRef]

Olshausen, B. A.

E. P. Simoncelli and B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216(2001).
[CrossRef] [PubMed]

B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vis. Res. 37, 3311–3325 (1997).
[CrossRef]

P. Garrigues and B. A. Olshausen, “Learning horizontal connections in a sparse coding model of natural images,” in Advances in Neural Information Processing Systems 20, J.C.Platt, D.Koller, Y.Singer, and S.Roweis, eds. (MIT Press, 2008), pp. 505–512.

Oppenheim, A. V.

A. V. Oppenheim and J. S. Lim, “The importance of phase in signals,” Proc. IEEE 69, 529–541 (1981).
[CrossRef]

Pedersen, K. S.

A. B. Lee, K. S. Pedersen, and D. Mumford, “The nonlinear statistics of high-contrast patches in natural images,” Int. J. Comput. Vis. 54, 83–103 (2003).
[CrossRef]

Pennebaker, W. B.

W. B. Pennebaker and J. L. Mitchell, The JPEG Still Image Data Compression Standard (Van Nostrand Reinhold, 1993).

Piotrowski, L. N.

L. N. Piotrowski and F. W. Campbell, “A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase,” Perception 11, 337–346 (1982).
[CrossRef] [PubMed]

Ross, J.

M. J. Morgan, J. Ross, and A. Hayes, “The relative importance of local phase and local amplitude in patchwise image reconstruction,” Biol. Cybern. 65, 113–119 (1991).
[CrossRef]

Ruderman, D. L.

D. L. Ruderman, “Origins of scaling in natural images,” Vis. Res. 37, 3385–3398 (1997).
[CrossRef]

Sejnowski, T.

L. Wiskott and T. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770(2002).
[CrossRef] [PubMed]

A. Bell and T. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vis. Res. 37, 3327–3338 (1997).
[CrossRef]

Simoncelli, E. P.

E. P. Simoncelli and B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216(2001).
[CrossRef] [PubMed]

Sinz, F.

R. Hosseini, F. Sinz, and M. Bethge, “Lower bounds on the redundancy of natural images,” Vis. Res. 50, 2213–2222 (2010).
[CrossRef] [PubMed]

Standardization, International Organization for

International Organization for Standardization, “Information technology—JPEG 2000 image coding system: core coding system,” Tech. Rep. ISO/IEC FDIS15444-1:2000 (International Organization for Standardization, 2000).

Stögbauer, H.

A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual information,” Phys. Rev. E 69 (2004).
[CrossRef]

Tadmor, Y.

Y. Tadmor and D. Tolhurst, “Both the phase and the amplitude spectrum may determine the appearance of natural images,” Vis. Res. 33, 141–145 (1993).
[CrossRef] [PubMed]

D. J. Tolhurst, Y. Tadmor, and T. Chao, “Amplitude spectra of natural images,” Ophthalmol. Physiol. Opt. 12, 229–232 (1992).
[CrossRef]

Tapia, R. A.

M. R. Celis, J. E. Dennis, and R. A. Tapia, “A trust region strategy for nonlinear equality constrained optimization,” in Numerical Optimization 1984, P.T.Boggs, R.H.Byrd, and R.B.Schnabel, eds. (SIAM, 1985), pp. 71–82.

Tolhurst, D.

Y. Tadmor and D. Tolhurst, “Both the phase and the amplitude spectrum may determine the appearance of natural images,” Vis. Res. 33, 141–145 (1993).
[CrossRef] [PubMed]

Tolhurst, D. J.

D. J. Tolhurst, Y. Tadmor, and T. Chao, “Amplitude spectra of natural images,” Ophthalmol. Physiol. Opt. 12, 229–232 (1992).
[CrossRef]

Torralba, A.

A. Torralba and A. Oliva, “Statistics of natural image categories,” Network: Comput. Neural Syst. 14, 391–412(2003).
[CrossRef]

van der Schaaf, A.

J. H. van Hateren and A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. Lond. Ser. B 265, 359–366(1998).
[CrossRef]

van Hateren, J. H.

J. H. van Hateren and A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. Lond. Ser. B 265, 359–366(1998).
[CrossRef]

Victor, J. D.

J. D. Victor, “Binless strategies for estimation of information from neural data,” Phys. Rev. E 66051903 (2002).
[CrossRef]

Vinje, W. E.

W. E. Vinje and J. L. Gallant, “Natural stimulation of the nonclassical receptive field increases information transmission efficiency in V1,” J. Neurosci. 22, 2904–2915 (2002).
[PubMed]

W. E. Vinje and J. L. Gallant, “Sparse coding and decorrelation in primary visual cortex during natural vision,” Science 287, 1273–1276 (2000).
[CrossRef] [PubMed]

Wiskott, L.

L. Wiskott and T. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770(2002).
[CrossRef] [PubMed]

Annu. Rev. Neurosci. (1)

E. P. Simoncelli and B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216(2001).
[CrossRef] [PubMed]

Appl. Opt. (1)

Biol. Cybern. (1)

M. J. Morgan, J. Ross, and A. Hayes, “The relative importance of local phase and local amplitude in patchwise image reconstruction,” Biol. Cybern. 65, 113–119 (1991).
[CrossRef]

IEEE Trans. Inf. Theory (1)

R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE Trans. Inf. Theory 44, 2325–2384 (1998).
[CrossRef]

Int. J. Comput. Vis. (1)

A. B. Lee, K. S. Pedersen, and D. Mumford, “The nonlinear statistics of high-contrast patches in natural images,” Int. J. Comput. Vis. 54, 83–103 (2003).
[CrossRef]

J. Neurosci. (1)

W. E. Vinje and J. L. Gallant, “Natural stimulation of the nonclassical receptive field increases information transmission efficiency in V1,” J. Neurosci. 22, 2904–2915 (2002).
[PubMed]

J. Opt. Soc. Am. A (3)

Network: Comput. Neural Syst. (1)

A. Torralba and A. Oliva, “Statistics of natural image categories,” Network: Comput. Neural Syst. 14, 391–412(2003).
[CrossRef]

Neural Comput. (2)

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

L. Wiskott and T. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770(2002).
[CrossRef] [PubMed]

Ophthalmol. Physiol. Opt. (1)

D. J. Tolhurst, Y. Tadmor, and T. Chao, “Amplitude spectra of natural images,” Ophthalmol. Physiol. Opt. 12, 229–232 (1992).
[CrossRef]

Opt. Lett. (1)

Perception (1)

L. N. Piotrowski and F. W. Campbell, “A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase,” Perception 11, 337–346 (1982).
[CrossRef] [PubMed]

Phys. Rev. E (2)

J. D. Victor, “Binless strategies for estimation of information from neural data,” Phys. Rev. E 66051903 (2002).
[CrossRef]

A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual information,” Phys. Rev. E 69 (2004).
[CrossRef]

Probl. Inf. Transm. (1)

L. F. Kozachenko and N. N. Leonenko, “A statistical estimate for the entropy of a random vector,” Probl. Inf. Transm. 23, 9–16(1987).

Proc. IEEE (1)

A. V. Oppenheim and J. S. Lim, “The importance of phase in signals,” Proc. IEEE 69, 529–541 (1981).
[CrossRef]

Proc. R. Soc. Lond. Ser. B (1)

J. H. van Hateren and A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. Lond. Ser. B 265, 359–366(1998).
[CrossRef]

Science (1)

W. E. Vinje and J. L. Gallant, “Sparse coding and decorrelation in primary visual cortex during natural vision,” Science 287, 1273–1276 (2000).
[CrossRef] [PubMed]

Spatial Vis. (1)

D. J. Graham and D. J. Field, “Statistical regularities of art images and natural scenes: spectra, sparseness and nonlinearities,” Spatial Vis. 21, 149–64 (2007). PMID: 18073056.
[CrossRef]

Vis. Res. (6)

Y. Tadmor and D. Tolhurst, “Both the phase and the amplitude spectrum may determine the appearance of natural images,” Vis. Res. 33, 141–145 (1993).
[CrossRef] [PubMed]

D. L. Ruderman, “Origins of scaling in natural images,” Vis. Res. 37, 3385–3398 (1997).
[CrossRef]

A. Bell and T. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vis. Res. 37, 3327–3338 (1997).
[CrossRef]

P. O. Hoyer and A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vis. Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vis. Res. 37, 3311–3325 (1997).
[CrossRef]

R. Hosseini, F. Sinz, and M. Bethge, “Lower bounds on the redundancy of natural images,” Vis. Res. 50, 2213–2222 (2010).
[CrossRef] [PubMed]

Other (7)

P. Garrigues and B. A. Olshausen, “Learning horizontal connections in a sparse coding model of natural images,” in Advances in Neural Information Processing Systems 20, J.C.Platt, D.Koller, Y.Singer, and S.Roweis, eds. (MIT Press, 2008), pp. 505–512.

W. B. Pennebaker and J. L. Mitchell, The JPEG Still Image Data Compression Standard (Van Nostrand Reinhold, 1993).

International Organization for Standardization, “Information technology—JPEG 2000 image coding system: core coding system,” Tech. Rep. ISO/IEC FDIS15444-1:2000 (International Organization for Standardization, 2000).

D. J. Field, “Scale-invariance and self-similar ‘wavelet’ transforms: an analysis of natural scenes and mammalian visual systems,” in Wavelets, Fractals and Fourier Transforms: New Developments and New Applications, M.Farge, J.C. R.Hunt, and J.C.Vassilicos, eds. (Oxford University, 1993), pp. 151–193.

J. Kybic, “High-dimensional mutual information estimation for image registration,” in Proceedings of International Conference on Image Processing, 2004, Vol. 3 (IEEE, 2005), pp. 1779–1782.

The values of T=65,536 and T=16,384 and ten trials of the search procedure (using ten randomizations of Group T) were sufficient to yield a consistent sample mean.

M. R. Celis, J. E. Dennis, and R. A. Tapia, “A trust region strategy for nonlinear equality constrained optimization,” in Numerical Optimization 1984, P.T.Boggs, R.H.Byrd, and R.B.Schnabel, eds. (SIAM, 1985), pp. 71–82.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1

Proximity distribution functions for 8 × 8 patches of Gaussian white noise, noise with an amplitude spectrum of 1 / f , and natural scenes. The horizontal axis denotes N (the number of patches in Group N ); the vertical axis denotes the corresponding E { log 2 D N * } estimated via a sample mean over all target patches. Black circles: Gaussian white noise; light-gray circles: 1 / f noise; gray circles: natural scenes.

Fig. 2
Fig. 2

Graphical illustration of the contributions of the information in the amplitude spectrum H ( A ) and the information in the phase spectrum H ( P ) to the total entropy H ( A , P ) . (a) If the amplitude and phase spectra are assumed to be independent of each other, then H ( A , P ) = H ( A ) + H ( P ) . (b) Illustration of the contributions of H ( A ) and H ( P ) to the total entropy when the amplitude and phase are statistically dependent and therefore possess some amount of mutual information (overlap) I ( A ; P ) .

Fig. 3
Fig. 3

Graphical illustration of the strategy used in the current study to compute H ( A ) , H ( P ) , and I ( A ; P ) . (a) The configuration shows the assumed entropy of natural scenes H ( A , P ) in which H ( A ) and H ( P ) possess some amount of mutual information (overlap) I ( A ; P ) . (b) The configuration is achieved by setting the phase spectrum of all patches to a fixed-phase spectrum; this operation removes all variations due to phase, which forces H ( P ) = 0 , and therefore H ( A ) is given by the entropy of the fixed-phase patches [i.e., H ( A ) = H fixed phase ( A , P ) ]. (c) The configuration is achieved by creating hybrid images in which the phase spectrum of each patch is replaced with the phase spectrum of another natural-scene patch selected at random; this operation removes any statistical dependence between amplitude and phase while maintaining the individual quantities H ( A ) and H ( P ) . The entropy of the hybrid patches is therefore H hybrid = H ( A ) + H ( P ) , which allows us to compute both H ( P ) and I ( A ; P ) via H ( P ) = H hybrid ( A , P ) H fixed-phase ( A , P ) and I ( A ; P ) = H hybrid ( A , P ) H ( A , P ) .

Fig. 4
Fig. 4

Stimuli used in the experiment consisted of 4 × 4 and 8 × 8 natural-scene patches, fixed-phase patches, and hybrid patches (only 8 × 8 patches are shown). Each image depicts a random subset of sixty-four 8 × 8 patches. Left: Normal 8 × 8 natural-scene patches. Middle: Fixed-phase 8 × 8 patches generated by assigning the same fixed-phase spectrum to all patches; the amplitude spectrum was not adjusted. Right: Hybrid 8 × 8 patches generated by replacing the phase spectrum of each patch with the phase spectrum of another, randomly selected 8 × 8 natural-scene patch; the amplitude spectrum was not adjusted.

Fig. 5
Fig. 5

The fixed-phase spectrum used for all patches in the fixed-phase condition was generated by computing the discrete Fourier transform of a 4 × 4 or 8 × 8 patch consisting of only the central pixel set to a value of 255. The phase spectra Z shown on the right were computed via Z = tan 1 ( { Z } { Z } ) , where Z denotes the discrete Fourier transform of the central-pixel-on patches shown on the left. (The specific phase spectrum employed here is not important; rather, the key is to assign this same phase spectrum to all patches.)

Fig. 6
Fig. 6

Proximity distribution curves for 4 × 4 (top) and 8 × 8 (bottom) patches. In each graph, the horizontal axis denotes N (the number of patches in Group N ) and the vertical axis denotes the corresponding E { log 2 D N * } estimated via a sample mean over all target patches (average of 10 randomizations of Groups T and N ). Error bars denote ± 1 standard deviation. The data for Gaussian noise, provided for reference, were computed analytically via Eq. (10) from [15].

Fig. 7
Fig. 7

Relative dimensionality curves for 4 × 4 (top) and 8 × 8 (bottom) patches. In each graph, the horizontal axis denotes N (the number of patches in Group N ) and the vertical axis denotes the corresponding relative dimensionality computed via d log 2 ( N ) / d E { log 2 D N * } at each N. Error bars denote ± 1 standard deviation. The solid gray line denotes the intrinsic dimensionality of k = 16 ( 4 × 4 ) or k = 64 ( 8 × 8 ). The data for Gaussian noise were computed analytically via Eq. (10) from [15].

Fig. 8
Fig. 8

Extrapolated relative dimensionality curves for 4 × 4 (top) and 8 × 8 (bottom) patches using the XEntropy C estimator from [15]. In each graph, the horizontal axis denotes N (the number of patches in Group N ) and the vertical axis denotes the correspond ing relative dimensionality computed via RD ( N ) = ( log N + b 0 ) 2 / ( a 2 [ log N ] 2 + 2 a 2 b 0 log N + a 1 b 0 a 0 ) , where a 2 = 1 / k and a 0 = 65.05 , and where the parameters a 1 and b 0 were adjusted to fit the measured data. The solid gray line denotes the intrinsic dimensionality of k = 16 ( 4 × 4 ) or k = 64 ( 8 × 8 ).

Fig. 9
Fig. 9

Entropy estimates for 4 × 4 (top) and 8 × 8 (bottom) patches computed for each value of N using the XEntropy C estimator from [15]. The final entropy estimate listed for each curve was computed by using Eq. (A3) with k = 16 ( 4 × 4 ) or k = 64 ( 8 × 8 ) and N chosen such that the relative dimensionality was within 1% of the intrinsic dimensionality.

Fig. 10
Fig. 10

Top: Distribution of rotationally averaged slopes for 2000 natural-scene patches of 8 × 8 , 32 × 32 , and 256 × 256 pixels. Bottom: Standard deviation of average slopes as a function of patch size. Notice that there is much more variation in slope for smaller patches. (The slope here refers to the parameter α obtained by fitting the rotationally averaged amplitude spectrum with a function of the form 1 / f α , where f corresponds to radial frequency; only values of f > 0 were used in the fitting.)

Fig. 11
Fig. 11

Random collection of 8 × 8 (left) and 64 × 64 (right) patches with a standard deviation of pixel values greater than 35. For small patch sizes such as 8 × 8 , most of the patches are dominated by a single edge; in this case, the orientation of the edge is well described by the power spectrum with most of the power at that orientation, and thus the power spectrum provides a fairly good account of the perception of the edge. However, for larger patches, features tend to occur at multiple orientations and scales, and thus the power spectrum is closer to 1 / f ; in this latter case, the visual appearance of each patch is dictated more by its phase spectrum.

Tables (1)

Tables Icon

Table 1 Entropy Estimates for 4 × 4 and 8 × 8 Patches a

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

D n , t = X ( T t ) X ( N n ) L 2 = ( i = 1 k ( X i ( T t ) X i ( N n ) ) 2 ) 1 / 2 ,
D N , t * = min n [ 1 , N ] { D n , t } .
E { log 2 D N * } 1 T t = 1 T log 2 D N , t * .
h k E { log 2 D N * } + log 2 ( A k N k ) + γ ln 2 ,
H ( A , P ) = H ( A ) + H ( P ) .
H ( A , P ) = H ( A ) + H ( P ) I ( A ; P ) ,
H ( P ) = H hybrid ( A , P ) H fixed-phase ( A , P ) ,
= [ H ( A ) + H ( P ) ] H ( A ) .
I ( A ; P ) = H ( A ) + H ( P ) H ( A , P ) .
| X | = { X } 2 + { X } 2 ,
Y = | X | e j Z ,
RD ( N ) = ( log N + b 0 ) 2 / ( a 2 [ log N ] 2 + 2 a 2 b 0 log N + a 1 b 0 a 0 ) ,
h ( X ) x A f X ( x ) log 2 f X ( x ) d x = x A f X ( x ) i X ( x ) d x = E { i X ( x ) } 1 M m = 1 M i ^ X ( x m ) ,
i ^ X ( x ) = k E { log 2 D N * } + log 2 ( A k N k ) + γ ln 2 ,
h ( X ) k M m = 1 M log 2 D N , m * + log 2 ( A k N k ) + γ ln 2 ,

Metrics