Abstract

We apply a Bayesian method for inferring an optimal basis to the problem of finding efficient image codes for natural scenes. The basis functions learned by the algorithm are oriented and localized in both space and frequency, bearing a resemblance to two-dimensional Gabor functions, and increasing the number of basis functions results in a greater sampling density in position, orientation, and scale. These properties also resemble the spatial receptive fields of neurons in the primary visual cortex of mammals, suggesting that the receptive-field structure of these neurons can be accounted for by a general efficient coding principle. The probabilistic framework provides a method for comparing the coding efficiency of different bases objectively by calculating their probability given the observed data or by measuring the entropy of the basis function coefficients. The learned bases are shown to have better coding efficiency than traditional Fourier and wavelet bases. This framework also provides a Bayesian solution to the problems of image denoising and filling in of missing pixels. We demonstrate that the results obtained by applying the learned bases to these problems are improved over those obtained with traditional techniques.

© 1999 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. J. G. Daugman, “Uncertainty relation for resolution in space, spatial-frequency, and orientation optimized by two-dimensional visual cortical filters,” J. Opt. Soc. Am. A 2, 1160–1169 (1985).
    [CrossRef] [PubMed]
  2. J. G. Daugman, “Complete discrete 2-D Gabor transforms by neural networks for image-analysis and compression,” IEEE Trans. Acoust., Speech, Signal Process. 36, 1169–1179 (1988).
    [CrossRef]
  3. J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive-fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
    [CrossRef] [PubMed]
  4. D. J. Field, “What is the goal of sensory coding,” Neural Comput. 6, 559–601 (1994).
    [CrossRef]
  5. T. S. Lee, “Image representation using 2D Gabor wavelets,” IEEE Trans. Pattern. Anal. Mach. Intell. 18, 959–971 (1996).
    [CrossRef]
  6. M. S. Lewicki, T. J. Sejnowski, “Learning overcomplete representations,” Neural Comput. (to be published).
  7. C. Jutten, J. Herault, “Blind separation of sources. 1. An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
    [CrossRef]
  8. P. Comon, “Independent component analysis, a new concept,” Signal Process. 36, 287–314 (1994).
    [CrossRef]
  9. A. J. Bell, T. J. Sejnowski, “An information maximization approach to blind separation and blind deconvolution,” Neural Comput. 7, 1129–1159 (1995).
    [CrossRef] [PubMed]
  10. B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
    [CrossRef]
  11. E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992).
    [CrossRef]
  12. S. Chen, D. L. Donoho, M. A. Saunders, “Atomic decomposition by basis pursuit,” tech. rep. (Stanford University, Stanford, Calif., 1996).
  13. R. R. Coifman, M. V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Inf. Theory 38, 713–718 (1992).
    [CrossRef]
  14. S. G. Mallat, Z. F. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process. 41, 3397–3415 (1993).
    [CrossRef]
  15. S. C. Zhu, Y. N. Wu, D. Mumford, “Minimax entropy principle and its application to texture modeling,” Neural Comput. 9, 1627–1660 (1997).
    [CrossRef]
  16. B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive-field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
    [CrossRef]
  17. P. J. B. Hancock, R. J. Baddeley, L. S. Smith, “The principal components of natural images,” Network Comput. Neural Syst. 3, 61–70 (1992).
    [CrossRef]
  18. C. Fyfe, R. Baddeley, “Finding compact and sparse-distributed representations of visual images,” Network Comput. Neural Syst. 6, 333–344 (1995).
    [CrossRef]
  19. R. P. N. Rao, D. H. Ballard, “Dynamic-model of visual recognition predicts neural response properties in the visual-cortex,” Neural Comput. 9, 721–763 (1997).
    [CrossRef] [PubMed]
  20. R. P. N. Rao, D. H. Ballard, “Development of localized oriented receptive-fields by learning a translation-invariant code for natural images,” Network Comput. Neural Syst. 9, 219–234 (1998).
    [CrossRef]
  21. A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
    [CrossRef]
  22. J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London, Ser. B 265, 359–366 (1998).
    [CrossRef]
  23. C. Zetzsche, E. Barth, B. Wegmann, “The importance of intrinsically two-dimensional image features in biological vision and picture coding,” in Digital Images and Human Vision, A. B. Watson, ed. (MIT Press, Cambridge, Mass., 1993), pp. 109–138.
  24. D. L. Ruderman, “The statistics of natural images,” Network Comput. Neural Syst. 5, 517–548 (1994).
    [CrossRef]
  25. H. B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. A. Rosenbluth, ed. (MIT Press, Cambridge, Mass., 1961), pp. 217–234.
  26. H. B. Barlow, “Unsupervised learning,” Neural Comput. 1, 295–311 (1989).
    [CrossRef]
  27. W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipies in C: The Art of Scientific Programming, 2nd ed. (Cambridge U. Press, Cambridge, England, 1992).
  28. S. Marcelja, “Mathematical description of the responses of simple cortical cells,” J. Opt. Soc. Am. 70, 1297–1300 (1980).
    [CrossRef] [PubMed]
  29. R. L. De Valois, D. G. Albrecht, L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vision Res. 22, 545–559 (1982).
    [CrossRef] [PubMed]
  30. A. J. Parker, M. J. Hawken, “Two-dimensional spatial structure of receptive fields in monkey striate cortex,” J. Opt. Soc. Am. A 5, 598–605 (1988).
    [CrossRef] [PubMed]
  31. J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural images sequences yield spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
    [CrossRef]
  32. I. Daubechies, “Orthonormal bases of compactly supported wavelets,” Commun. Pure Appl. Math. XLI, 909–996 (1988).
    [CrossRef]
  33. R. W. Buccigrossi, E. P. Simoncelli, “Image compression via joint statistical characterization in the wavelet domain,” (University of Pennsylvania, Philadelphia, Penn., May1997).
  34. E. P. Simoncelli, E. H. Adelson, “Noise removal via Bayesian wavelet coring,” in Proceedings of International Conference IEEE on Image Processing, III Lausanne, Switzerland (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.
  35. S. Chen, “Basis pursuit,” Ph.D. dissertation (Stanford University, Stanford, Calif., 1995). Available at http://www-stat.stanford.edu/reports/chen.s
  36. R. Everson, L. Sirovich, “Karhunen–Loève procedure for gappy data,” J. Opt. Soc. Am. A 12, 1657–1664 (1995).
    [CrossRef]
  37. B. A. Pearlmutter, L. C. Parra, “Maximum likelihood blind source separation: a context-sensitive generalization of ICA,” in Advances in Neural and Information Processing SystemsM. C. Mozer, M. I. Jordan, T. Petsche, eds. (Morgan Kaufmann, Los Altos, Calif., 1997), Vol. 9, pp. 613–619.
  38. H. Attias, “Independent factor analysis,” Neural Comput. 11, 803–851 (1998).
    [CrossRef]
  39. B. D. Rao, K. Kreutz-Delgado, “An affine scaling methodology for best basis selection,” tech. rep. (Center for Information Engineering, University of California, San Diego, San Diego, Calif., 1997).
  40. R. M. Neal, Bayesian Learning for Neural Networks (Springer-Verlag, New York, 1996).
  41. J.-P. Nadal, N. Parga, “Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer,” Network 5, 565–581 (1994).
    [CrossRef]
  42. J.-P. Nadal, N. Parga, “Redundancy reduction and independent component analysis: conditions on cumulants and adaptive approaches,” Network 5, 565–581 (1994).
    [CrossRef]
  43. J-F. Cardoso, “Infomax and maximum likelihood for blind source separation,” IEEE Signal Process. Lett. 4, 109–111 (1997).
  44. G. E. Hinton, T. J. Sejnowski, “Learning and relearning in Boltzmann machines,” in Parallel Distributed Processing, D. E. Rumelhart, J. L. McClelland, eds. (MIT Press, Cambridge, Mass., 1986), Vol. 1, Chap. 7, pp. 282–317.
  45. R. Linsker, “Self-organization in a perceptual network,” Computer 21, 105–117 (1988).
    [CrossRef]
  46. J. J. Atick, “Could information-theory provide an ecological theory of sensory processing,” Network Comput. Neural Syst. 3, 213–251 (1992).
    [CrossRef]

1998

R. P. N. Rao, D. H. Ballard, “Development of localized oriented receptive-fields by learning a translation-invariant code for natural images,” Network Comput. Neural Syst. 9, 219–234 (1998).
[CrossRef]

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London, Ser. B 265, 359–366 (1998).
[CrossRef]

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural images sequences yield spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

H. Attias, “Independent factor analysis,” Neural Comput. 11, 803–851 (1998).
[CrossRef]

1997

J-F. Cardoso, “Infomax and maximum likelihood for blind source separation,” IEEE Signal Process. Lett. 4, 109–111 (1997).

R. P. N. Rao, D. H. Ballard, “Dynamic-model of visual recognition predicts neural response properties in the visual-cortex,” Neural Comput. 9, 721–763 (1997).
[CrossRef] [PubMed]

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

S. C. Zhu, Y. N. Wu, D. Mumford, “Minimax entropy principle and its application to texture modeling,” Neural Comput. 9, 1627–1660 (1997).
[CrossRef]

1996

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive-field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

T. S. Lee, “Image representation using 2D Gabor wavelets,” IEEE Trans. Pattern. Anal. Mach. Intell. 18, 959–971 (1996).
[CrossRef]

1995

A. J. Bell, T. J. Sejnowski, “An information maximization approach to blind separation and blind deconvolution,” Neural Comput. 7, 1129–1159 (1995).
[CrossRef] [PubMed]

C. Fyfe, R. Baddeley, “Finding compact and sparse-distributed representations of visual images,” Network Comput. Neural Syst. 6, 333–344 (1995).
[CrossRef]

R. Everson, L. Sirovich, “Karhunen–Loève procedure for gappy data,” J. Opt. Soc. Am. A 12, 1657–1664 (1995).
[CrossRef]

1994

J.-P. Nadal, N. Parga, “Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer,” Network 5, 565–581 (1994).
[CrossRef]

J.-P. Nadal, N. Parga, “Redundancy reduction and independent component analysis: conditions on cumulants and adaptive approaches,” Network 5, 565–581 (1994).
[CrossRef]

D. L. Ruderman, “The statistics of natural images,” Network Comput. Neural Syst. 5, 517–548 (1994).
[CrossRef]

P. Comon, “Independent component analysis, a new concept,” Signal Process. 36, 287–314 (1994).
[CrossRef]

D. J. Field, “What is the goal of sensory coding,” Neural Comput. 6, 559–601 (1994).
[CrossRef]

1993

S. G. Mallat, Z. F. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process. 41, 3397–3415 (1993).
[CrossRef]

1992

P. J. B. Hancock, R. J. Baddeley, L. S. Smith, “The principal components of natural images,” Network Comput. Neural Syst. 3, 61–70 (1992).
[CrossRef]

E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992).
[CrossRef]

R. R. Coifman, M. V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Inf. Theory 38, 713–718 (1992).
[CrossRef]

J. J. Atick, “Could information-theory provide an ecological theory of sensory processing,” Network Comput. Neural Syst. 3, 213–251 (1992).
[CrossRef]

1991

C. Jutten, J. Herault, “Blind separation of sources. 1. An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

1989

J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive-fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
[CrossRef] [PubMed]

H. B. Barlow, “Unsupervised learning,” Neural Comput. 1, 295–311 (1989).
[CrossRef]

1988

I. Daubechies, “Orthonormal bases of compactly supported wavelets,” Commun. Pure Appl. Math. XLI, 909–996 (1988).
[CrossRef]

R. Linsker, “Self-organization in a perceptual network,” Computer 21, 105–117 (1988).
[CrossRef]

A. J. Parker, M. J. Hawken, “Two-dimensional spatial structure of receptive fields in monkey striate cortex,” J. Opt. Soc. Am. A 5, 598–605 (1988).
[CrossRef] [PubMed]

J. G. Daugman, “Complete discrete 2-D Gabor transforms by neural networks for image-analysis and compression,” IEEE Trans. Acoust., Speech, Signal Process. 36, 1169–1179 (1988).
[CrossRef]

1985

1982

R. L. De Valois, D. G. Albrecht, L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vision Res. 22, 545–559 (1982).
[CrossRef] [PubMed]

1980

Adelson, E. H.

E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992).
[CrossRef]

E. P. Simoncelli, E. H. Adelson, “Noise removal via Bayesian wavelet coring,” in Proceedings of International Conference IEEE on Image Processing, III Lausanne, Switzerland (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.

Albrecht, D. G.

R. L. De Valois, D. G. Albrecht, L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vision Res. 22, 545–559 (1982).
[CrossRef] [PubMed]

Atick, J. J.

J. J. Atick, “Could information-theory provide an ecological theory of sensory processing,” Network Comput. Neural Syst. 3, 213–251 (1992).
[CrossRef]

Attias, H.

H. Attias, “Independent factor analysis,” Neural Comput. 11, 803–851 (1998).
[CrossRef]

Baddeley, R.

C. Fyfe, R. Baddeley, “Finding compact and sparse-distributed representations of visual images,” Network Comput. Neural Syst. 6, 333–344 (1995).
[CrossRef]

Baddeley, R. J.

P. J. B. Hancock, R. J. Baddeley, L. S. Smith, “The principal components of natural images,” Network Comput. Neural Syst. 3, 61–70 (1992).
[CrossRef]

Ballard, D. H.

R. P. N. Rao, D. H. Ballard, “Development of localized oriented receptive-fields by learning a translation-invariant code for natural images,” Network Comput. Neural Syst. 9, 219–234 (1998).
[CrossRef]

R. P. N. Rao, D. H. Ballard, “Dynamic-model of visual recognition predicts neural response properties in the visual-cortex,” Neural Comput. 9, 721–763 (1997).
[CrossRef] [PubMed]

Barlow, H. B.

H. B. Barlow, “Unsupervised learning,” Neural Comput. 1, 295–311 (1989).
[CrossRef]

H. B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. A. Rosenbluth, ed. (MIT Press, Cambridge, Mass., 1961), pp. 217–234.

Barth, E.

C. Zetzsche, E. Barth, B. Wegmann, “The importance of intrinsically two-dimensional image features in biological vision and picture coding,” in Digital Images and Human Vision, A. B. Watson, ed. (MIT Press, Cambridge, Mass., 1993), pp. 109–138.

Bell, A. J.

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

A. J. Bell, T. J. Sejnowski, “An information maximization approach to blind separation and blind deconvolution,” Neural Comput. 7, 1129–1159 (1995).
[CrossRef] [PubMed]

Buccigrossi, R. W.

R. W. Buccigrossi, E. P. Simoncelli, “Image compression via joint statistical characterization in the wavelet domain,” (University of Pennsylvania, Philadelphia, Penn., May1997).

Cardoso, J-F.

J-F. Cardoso, “Infomax and maximum likelihood for blind source separation,” IEEE Signal Process. Lett. 4, 109–111 (1997).

Chen, S.

S. Chen, “Basis pursuit,” Ph.D. dissertation (Stanford University, Stanford, Calif., 1995). Available at http://www-stat.stanford.edu/reports/chen.s

S. Chen, D. L. Donoho, M. A. Saunders, “Atomic decomposition by basis pursuit,” tech. rep. (Stanford University, Stanford, Calif., 1996).

Coifman, R. R.

R. R. Coifman, M. V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Inf. Theory 38, 713–718 (1992).
[CrossRef]

Comon, P.

P. Comon, “Independent component analysis, a new concept,” Signal Process. 36, 287–314 (1994).
[CrossRef]

Daubechies, I.

I. Daubechies, “Orthonormal bases of compactly supported wavelets,” Commun. Pure Appl. Math. XLI, 909–996 (1988).
[CrossRef]

Daugman, J. G.

J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive-fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
[CrossRef] [PubMed]

J. G. Daugman, “Complete discrete 2-D Gabor transforms by neural networks for image-analysis and compression,” IEEE Trans. Acoust., Speech, Signal Process. 36, 1169–1179 (1988).
[CrossRef]

J. G. Daugman, “Uncertainty relation for resolution in space, spatial-frequency, and orientation optimized by two-dimensional visual cortical filters,” J. Opt. Soc. Am. A 2, 1160–1169 (1985).
[CrossRef] [PubMed]

De Valois, R. L.

R. L. De Valois, D. G. Albrecht, L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vision Res. 22, 545–559 (1982).
[CrossRef] [PubMed]

Donoho, D. L.

S. Chen, D. L. Donoho, M. A. Saunders, “Atomic decomposition by basis pursuit,” tech. rep. (Stanford University, Stanford, Calif., 1996).

Everson, R.

Field, D. J.

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive-field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

D. J. Field, “What is the goal of sensory coding,” Neural Comput. 6, 559–601 (1994).
[CrossRef]

Flannery, B. P.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipies in C: The Art of Scientific Programming, 2nd ed. (Cambridge U. Press, Cambridge, England, 1992).

Freeman, W. T.

E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992).
[CrossRef]

Fyfe, C.

C. Fyfe, R. Baddeley, “Finding compact and sparse-distributed representations of visual images,” Network Comput. Neural Syst. 6, 333–344 (1995).
[CrossRef]

Hancock, P. J. B.

P. J. B. Hancock, R. J. Baddeley, L. S. Smith, “The principal components of natural images,” Network Comput. Neural Syst. 3, 61–70 (1992).
[CrossRef]

Hawken, M. J.

Heeger, D. J.

E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992).
[CrossRef]

Herault, J.

C. Jutten, J. Herault, “Blind separation of sources. 1. An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Hinton, G. E.

G. E. Hinton, T. J. Sejnowski, “Learning and relearning in Boltzmann machines,” in Parallel Distributed Processing, D. E. Rumelhart, J. L. McClelland, eds. (MIT Press, Cambridge, Mass., 1986), Vol. 1, Chap. 7, pp. 282–317.

Jutten, C.

C. Jutten, J. Herault, “Blind separation of sources. 1. An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Kreutz-Delgado, K.

B. D. Rao, K. Kreutz-Delgado, “An affine scaling methodology for best basis selection,” tech. rep. (Center for Information Engineering, University of California, San Diego, San Diego, Calif., 1997).

Lee, T. S.

T. S. Lee, “Image representation using 2D Gabor wavelets,” IEEE Trans. Pattern. Anal. Mach. Intell. 18, 959–971 (1996).
[CrossRef]

Lewicki, M. S.

M. S. Lewicki, T. J. Sejnowski, “Learning overcomplete representations,” Neural Comput. (to be published).

Linsker, R.

R. Linsker, “Self-organization in a perceptual network,” Computer 21, 105–117 (1988).
[CrossRef]

Mallat, S. G.

S. G. Mallat, Z. F. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process. 41, 3397–3415 (1993).
[CrossRef]

Marcelja, S.

Mumford, D.

S. C. Zhu, Y. N. Wu, D. Mumford, “Minimax entropy principle and its application to texture modeling,” Neural Comput. 9, 1627–1660 (1997).
[CrossRef]

Nadal, J.-P.

J.-P. Nadal, N. Parga, “Redundancy reduction and independent component analysis: conditions on cumulants and adaptive approaches,” Network 5, 565–581 (1994).
[CrossRef]

J.-P. Nadal, N. Parga, “Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer,” Network 5, 565–581 (1994).
[CrossRef]

Neal, R. M.

R. M. Neal, Bayesian Learning for Neural Networks (Springer-Verlag, New York, 1996).

Olshausen, B. A.

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive-field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

Parga, N.

J.-P. Nadal, N. Parga, “Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer,” Network 5, 565–581 (1994).
[CrossRef]

J.-P. Nadal, N. Parga, “Redundancy reduction and independent component analysis: conditions on cumulants and adaptive approaches,” Network 5, 565–581 (1994).
[CrossRef]

Parker, A. J.

Parra, L. C.

B. A. Pearlmutter, L. C. Parra, “Maximum likelihood blind source separation: a context-sensitive generalization of ICA,” in Advances in Neural and Information Processing SystemsM. C. Mozer, M. I. Jordan, T. Petsche, eds. (Morgan Kaufmann, Los Altos, Calif., 1997), Vol. 9, pp. 613–619.

Pearlmutter, B. A.

B. A. Pearlmutter, L. C. Parra, “Maximum likelihood blind source separation: a context-sensitive generalization of ICA,” in Advances in Neural and Information Processing SystemsM. C. Mozer, M. I. Jordan, T. Petsche, eds. (Morgan Kaufmann, Los Altos, Calif., 1997), Vol. 9, pp. 613–619.

Press, W. H.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipies in C: The Art of Scientific Programming, 2nd ed. (Cambridge U. Press, Cambridge, England, 1992).

Rao, B. D.

B. D. Rao, K. Kreutz-Delgado, “An affine scaling methodology for best basis selection,” tech. rep. (Center for Information Engineering, University of California, San Diego, San Diego, Calif., 1997).

Rao, R. P. N.

R. P. N. Rao, D. H. Ballard, “Development of localized oriented receptive-fields by learning a translation-invariant code for natural images,” Network Comput. Neural Syst. 9, 219–234 (1998).
[CrossRef]

R. P. N. Rao, D. H. Ballard, “Dynamic-model of visual recognition predicts neural response properties in the visual-cortex,” Neural Comput. 9, 721–763 (1997).
[CrossRef] [PubMed]

Ruderman, D. L.

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural images sequences yield spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

D. L. Ruderman, “The statistics of natural images,” Network Comput. Neural Syst. 5, 517–548 (1994).
[CrossRef]

Saunders, M. A.

S. Chen, D. L. Donoho, M. A. Saunders, “Atomic decomposition by basis pursuit,” tech. rep. (Stanford University, Stanford, Calif., 1996).

Sejnowski, T. J.

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

A. J. Bell, T. J. Sejnowski, “An information maximization approach to blind separation and blind deconvolution,” Neural Comput. 7, 1129–1159 (1995).
[CrossRef] [PubMed]

M. S. Lewicki, T. J. Sejnowski, “Learning overcomplete representations,” Neural Comput. (to be published).

G. E. Hinton, T. J. Sejnowski, “Learning and relearning in Boltzmann machines,” in Parallel Distributed Processing, D. E. Rumelhart, J. L. McClelland, eds. (MIT Press, Cambridge, Mass., 1986), Vol. 1, Chap. 7, pp. 282–317.

Simoncelli, E. P.

E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992).
[CrossRef]

R. W. Buccigrossi, E. P. Simoncelli, “Image compression via joint statistical characterization in the wavelet domain,” (University of Pennsylvania, Philadelphia, Penn., May1997).

E. P. Simoncelli, E. H. Adelson, “Noise removal via Bayesian wavelet coring,” in Proceedings of International Conference IEEE on Image Processing, III Lausanne, Switzerland (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.

Sirovich, L.

Smith, L. S.

P. J. B. Hancock, R. J. Baddeley, L. S. Smith, “The principal components of natural images,” Network Comput. Neural Syst. 3, 61–70 (1992).
[CrossRef]

Teukolsky, S. A.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipies in C: The Art of Scientific Programming, 2nd ed. (Cambridge U. Press, Cambridge, England, 1992).

Thorell, L. G.

R. L. De Valois, D. G. Albrecht, L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vision Res. 22, 545–559 (1982).
[CrossRef] [PubMed]

van der Schaaf, A.

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London, Ser. B 265, 359–366 (1998).
[CrossRef]

van Hateren, J. H.

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London, Ser. B 265, 359–366 (1998).
[CrossRef]

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural images sequences yield spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

Vetterling, W. T.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipies in C: The Art of Scientific Programming, 2nd ed. (Cambridge U. Press, Cambridge, England, 1992).

Wegmann, B.

C. Zetzsche, E. Barth, B. Wegmann, “The importance of intrinsically two-dimensional image features in biological vision and picture coding,” in Digital Images and Human Vision, A. B. Watson, ed. (MIT Press, Cambridge, Mass., 1993), pp. 109–138.

Wickerhauser, M. V.

R. R. Coifman, M. V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Inf. Theory 38, 713–718 (1992).
[CrossRef]

Wu, Y. N.

S. C. Zhu, Y. N. Wu, D. Mumford, “Minimax entropy principle and its application to texture modeling,” Neural Comput. 9, 1627–1660 (1997).
[CrossRef]

Zetzsche, C.

C. Zetzsche, E. Barth, B. Wegmann, “The importance of intrinsically two-dimensional image features in biological vision and picture coding,” in Digital Images and Human Vision, A. B. Watson, ed. (MIT Press, Cambridge, Mass., 1993), pp. 109–138.

Zhang, Z. F.

S. G. Mallat, Z. F. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process. 41, 3397–3415 (1993).
[CrossRef]

Zhu, S. C.

S. C. Zhu, Y. N. Wu, D. Mumford, “Minimax entropy principle and its application to texture modeling,” Neural Comput. 9, 1627–1660 (1997).
[CrossRef]

Commun. Pure Appl. Math.

I. Daubechies, “Orthonormal bases of compactly supported wavelets,” Commun. Pure Appl. Math. XLI, 909–996 (1988).
[CrossRef]

Computer

R. Linsker, “Self-organization in a perceptual network,” Computer 21, 105–117 (1988).
[CrossRef]

IEEE Signal Process. Lett.

J-F. Cardoso, “Infomax and maximum likelihood for blind source separation,” IEEE Signal Process. Lett. 4, 109–111 (1997).

IEEE Trans. Acoust., Speech, Signal Process.

J. G. Daugman, “Complete discrete 2-D Gabor transforms by neural networks for image-analysis and compression,” IEEE Trans. Acoust., Speech, Signal Process. 36, 1169–1179 (1988).
[CrossRef]

IEEE Trans. Biomed. Eng.

J. G. Daugman, “Entropy reduction and decorrelation in visual coding by oriented neural receptive-fields,” IEEE Trans. Biomed. Eng. 36, 107–114 (1989).
[CrossRef] [PubMed]

IEEE Trans. Inf. Theory

E. P. Simoncelli, W. T. Freeman, E. H. Adelson, D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory 38, 587–607 (1992).
[CrossRef]

R. R. Coifman, M. V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Inf. Theory 38, 713–718 (1992).
[CrossRef]

IEEE Trans. Pattern. Anal. Mach. Intell.

T. S. Lee, “Image representation using 2D Gabor wavelets,” IEEE Trans. Pattern. Anal. Mach. Intell. 18, 959–971 (1996).
[CrossRef]

IEEE Trans. Signal Process.

S. G. Mallat, Z. F. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process. 41, 3397–3415 (1993).
[CrossRef]

J. Opt. Soc. Am.

J. Opt. Soc. Am. A

Nature (London)

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive-field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

Network

J.-P. Nadal, N. Parga, “Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer,” Network 5, 565–581 (1994).
[CrossRef]

J.-P. Nadal, N. Parga, “Redundancy reduction and independent component analysis: conditions on cumulants and adaptive approaches,” Network 5, 565–581 (1994).
[CrossRef]

Network Comput. Neural Syst.

J. J. Atick, “Could information-theory provide an ecological theory of sensory processing,” Network Comput. Neural Syst. 3, 213–251 (1992).
[CrossRef]

P. J. B. Hancock, R. J. Baddeley, L. S. Smith, “The principal components of natural images,” Network Comput. Neural Syst. 3, 61–70 (1992).
[CrossRef]

C. Fyfe, R. Baddeley, “Finding compact and sparse-distributed representations of visual images,” Network Comput. Neural Syst. 6, 333–344 (1995).
[CrossRef]

R. P. N. Rao, D. H. Ballard, “Development of localized oriented receptive-fields by learning a translation-invariant code for natural images,” Network Comput. Neural Syst. 9, 219–234 (1998).
[CrossRef]

D. L. Ruderman, “The statistics of natural images,” Network Comput. Neural Syst. 5, 517–548 (1994).
[CrossRef]

Neural Comput.

H. B. Barlow, “Unsupervised learning,” Neural Comput. 1, 295–311 (1989).
[CrossRef]

R. P. N. Rao, D. H. Ballard, “Dynamic-model of visual recognition predicts neural response properties in the visual-cortex,” Neural Comput. 9, 721–763 (1997).
[CrossRef] [PubMed]

S. C. Zhu, Y. N. Wu, D. Mumford, “Minimax entropy principle and its application to texture modeling,” Neural Comput. 9, 1627–1660 (1997).
[CrossRef]

D. J. Field, “What is the goal of sensory coding,” Neural Comput. 6, 559–601 (1994).
[CrossRef]

A. J. Bell, T. J. Sejnowski, “An information maximization approach to blind separation and blind deconvolution,” Neural Comput. 7, 1129–1159 (1995).
[CrossRef] [PubMed]

H. Attias, “Independent factor analysis,” Neural Comput. 11, 803–851 (1998).
[CrossRef]

Proc. R. Soc. London Ser. B

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural images sequences yield spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

Proc. R. Soc. London, Ser. B

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London, Ser. B 265, 359–366 (1998).
[CrossRef]

Signal Process.

C. Jutten, J. Herault, “Blind separation of sources. 1. An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

P. Comon, “Independent component analysis, a new concept,” Signal Process. 36, 287–314 (1994).
[CrossRef]

Vision Res.

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

R. L. De Valois, D. G. Albrecht, L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vision Res. 22, 545–559 (1982).
[CrossRef] [PubMed]

Other

R. W. Buccigrossi, E. P. Simoncelli, “Image compression via joint statistical characterization in the wavelet domain,” (University of Pennsylvania, Philadelphia, Penn., May1997).

E. P. Simoncelli, E. H. Adelson, “Noise removal via Bayesian wavelet coring,” in Proceedings of International Conference IEEE on Image Processing, III Lausanne, Switzerland (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.

S. Chen, “Basis pursuit,” Ph.D. dissertation (Stanford University, Stanford, Calif., 1995). Available at http://www-stat.stanford.edu/reports/chen.s

C. Zetzsche, E. Barth, B. Wegmann, “The importance of intrinsically two-dimensional image features in biological vision and picture coding,” in Digital Images and Human Vision, A. B. Watson, ed. (MIT Press, Cambridge, Mass., 1993), pp. 109–138.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipies in C: The Art of Scientific Programming, 2nd ed. (Cambridge U. Press, Cambridge, England, 1992).

H. B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. A. Rosenbluth, ed. (MIT Press, Cambridge, Mass., 1961), pp. 217–234.

M. S. Lewicki, T. J. Sejnowski, “Learning overcomplete representations,” Neural Comput. (to be published).

S. Chen, D. L. Donoho, M. A. Saunders, “Atomic decomposition by basis pursuit,” tech. rep. (Stanford University, Stanford, Calif., 1996).

B. D. Rao, K. Kreutz-Delgado, “An affine scaling methodology for best basis selection,” tech. rep. (Center for Information Engineering, University of California, San Diego, San Diego, Calif., 1997).

R. M. Neal, Bayesian Learning for Neural Networks (Springer-Verlag, New York, 1996).

B. A. Pearlmutter, L. C. Parra, “Maximum likelihood blind source separation: a context-sensitive generalization of ICA,” in Advances in Neural and Information Processing SystemsM. C. Mozer, M. I. Jordan, T. Petsche, eds. (Morgan Kaufmann, Los Altos, Calif., 1997), Vol. 9, pp. 613–619.

G. E. Hinton, T. J. Sejnowski, “Learning and relearning in Boltzmann machines,” in Parallel Distributed Processing, D. E. Rumelhart, J. L. McClelland, eds. (MIT Press, Cambridge, Mass., 1986), Vol. 1, Chap. 7, pp. 282–317.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (7)

Fig. 1
Fig. 1

Results from training (a) complete and (b) 2×-overcomplete bases on natural scenes. The graphs plot and the odd-numbered basis functions in decreasing order of L2 norm.  

Fig. 2
Fig. 2

Basis function characteristics for (a) the complete case and (b) the 2×-overcomplete case. Each basis function was fitted by a Gabor function to characterize its position, spatial-frequency selectivity, and orientation. At the top are polar plots of the peak spatial-frequency tuning and orientation selectivity. Each dot denotes the center spatial frequency and orientation of a fitted basis function. The cross hairs indicate the 1/4-bandwidth in spatial frequency and orientation. The plots at the bottom show the spatial layout of the same set of basis functions. Each bar denotes the center position and orientation of a fitted basis function within the 12×12 grid. The thickness and length of each line denotes its spatial-frequency band (lower spatial frequencies are represented with thicker lines, and vice versa). Increasing the degree of overcompleteness results in a denser tiling of the joint four-dimensional space of position, orientation, and spatial frequency.

Fig. 3
Fig. 3

Histogram of peak spatial-frequency bandwidths (in cycles per pixel) for (a) complete and (b) 2×-overcomplete learned basis functions.

Fig. 4
Fig. 4

Some of the complete bases used for comparison with the learned basis.

Fig. 5
Fig. 5

Coefficient histograms for some of the bases used in the coding efficiency comparisons. Each histogram shows 97.5% of the coefficient range. The vertical axis is scaled so that each histogram peak falls within the plot. The sample kurtosis is shown for each histogram.

Fig. 6
Fig. 6

Demonstration of image denoising by use of the 1× (complete) basis set. Each image is shown tiled into nonoverlapping 12×12 blocks, to which the image model was then applied. The results (lower left) show both a qualitative and a quantitative improvement over Wiener filtering (lower right).

Fig. 7
Fig. 7

Reconstruction of missing information. From the original image, 71% of the pixels were removed and reconstructed with the methods described in the text. The reconstruction by the model is superior to the reconstruction based on spline interpolation, because the model can fill in actual structure in the image, whereas spline interpolation only smoothes between available pixels. This can be seen by comparing the image patches in row 2, column 1 and row 3, column 4.

Tables (3)

Tables Icon

Table 1 Bits per Pixel for Complete Bases on Natural Image Data Set

Tables Icon

Table 2 Bits per Pixel for Complete Bases on Pixel Data Set

Tables Icon

Table 3 Bits per Pixel for 2×-Overcomplete Bases on Images Data Set

Equations (63)

Equations on this page are rendered with MathJax. Learn more.

LH(p)=-p(x)log p(x).
L=E[l(X)]xp(x)log 1q(x),
=xp(x)log p(x)q(x)+xp(x)log 1p(x),
=DKL(pq)+H(p),
x=As+,
P(s|x, A)P(x|A, s)P(s).
sˆ=maxsP(s|x, A),
=maxs[log P(x|A, s)+log P(s)],
=minsλ2 |x-As|2+ θT|s|.
sˆ=A+x.
L=log P(x|A),
P(x|A)=dsP(x|A, s)P(s).
ΔALA=A log P(x|A),
=1P(x|A)  A P(x|s, A)P(s)ds,
=1P(x|A) λesTP(x|s, A)P(s)ds,
=λesTP(s|x, A)ds,
=λesTP(s|x, A),
Lconst.-λ2 |x-Asˆ|2+log P(sˆ)-12 log det H,
ΔAλesT-AH-1.
ΔA-A(zsT+ATAH-1),
g(x, y)=a exp-12 u(x, y)σu2+v(x, y)σv2×cos[2πfu(x, y)+ϕ],
u(x, y)=(x-x0)cos(θ)+(y-y0)sin(θ),
v(x, y)=-(x-x0)sin(θ)+(y-y0)cos(θ).
numberofbits-log2 P(x|A)-L log2(σx),
numberofbits-i niN log2 f[i],
sˆ=maxsP(s|x, A)
=minsλ2 |x-As|2+θT|s|.
xˆ=Asˆ.
P(x|A, s)exp-i λi2 |x-As|i2,
sˆ=minsi λi2 |x-As|i2+θT|s|,
xˆ=Asˆ,
log P(x|A)=const.+F(A, sˆ)+V(sˆ),
sˆ=argmaxsF(A, s),
F(A, s)=-λ2 |x-As|2+log P(s),
log P(s|x, A),
V(A, s)=-12 log|det H(s)|,
H(s)=-F(A, s).
e=x-As,
z= log P(s),
F=λATe+z,
B(s)=- log P(s),
H(s)=-F(s)=λATA+B(s),
yk=2kV=(H-1)kk dBkkdsk,
ds^kdAij=λ[eiHkj-1-(AH-1)iksj].
ddA F[A, sˆ(A)]=FA+F dsˆA,
dFdA=λes^T.
ddA V[A, sˆ(A)]=VA+V dsˆA.
VA=-λAH-1,
dVdA=λ-AH-1+12 (eyTH-1-AH-1ys^T).
ΔAλes^T-AH-1+12 (eyTH-1-AH-1ys^T).
ΔAλ(esT-AH-1).
AATΔAλ(AATesT-AATAH-1],
=-A(zsT+ATAH-1),
kFAij=λ(δkjei-Aiksj),
FAij-H dsˆdAij=0.
dsˆdAij=H-1 FAij.
ds^kdAij=λ[eiHkj-1-(AH-1)iksj].
H-1=(C+B)-1,
=(QVQT+B)-1,
=Q(V+QTBQ)-1QT.
C(C+B)-1=I-B(C+B)-1,
=I-BQ(V+QTBQ)-1QT,
I-BQ diag-1(V+QTBQ)QT,

Metrics