Abstract

Recently, different models of the statistical structure of natural images have been proposed. These models predict properties of biological visual systems and can be used as priors in Bayesian inference. The fundamental model is independent component analysis, which can be estimated by maximization of the sparsenesses of linear filter outputs. This leads to the emergence of principal simple cell properties. Alternatively, simple cell properties are obtained by maximizing the temporal coherence in natural image sequences. Taking account of the basic dependencies of linear filter outputs permit modeling of complex cells and topographic organization as well. We propose a unifying framework for these statistical properties, based on the concept of spatiotemporal activity “bubbles.” A bubble means here an activation of simple cells (linear filters) that is contiguous both in space (the cortical surface) and in time.

© 2003 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. H. B. Barlow, “Single units and sensation: a neuron doctrine for perceptual psychology?” Perception 1, 371–394 (1972).
    [CrossRef] [PubMed]
  2. D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
    [CrossRef]
  3. E. P. Simoncelli, B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216 (2001).
    [CrossRef] [PubMed]
  4. B. A. Olshausen, “Principles of image representation in visual cortex,” in The Visual Neurosciences, L. M. Chalupa, J. S. Werner, eds. (MIT Press, Cambridge, Mass., 2003).
  5. A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis (Wiley Interscience, New York, 2001).
  6. A. Hyvärinen, “Sparse code shrinkage: denoising of non-gaussian data by maximum likelihood estimation,” Neural Comput. 11, 1739–1768 (1999).
    [CrossRef]
  7. E. P. Simoncelli, E. H. Adelson, “Noise removal via bayesian wavelet coring,” in Proceedings of the Third IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.
  8. G. E. Hinton, Z. Ghahramani, “Generative models for discovering sparse distributed representations,” Philos. Trans. R. Soc. London Ser. B 352, 1177–1190 (1997).
    [CrossRef]
  9. D. C. Knill, W. Richards, eds., Perception as Bayesian Inference (Cambridge U. Press, Cambridge, UK, 1996).
  10. A. Hyvärinen, “Fast and robust fixed-point algorithms for independent component analysis,” IEEE Trans. Neural Netw. 10, 626–634 (1999).
    [CrossRef]
  11. B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
    [CrossRef]
  12. P. Földiák, “Learning invariance from transformation sequences,” Neural Comput. 3, 194–200 (1991).
    [CrossRef]
  13. C. Kayser, W. Einhäuser, O. Dümmer, P. König, K. Körding, “Extracting slow subspaces from natural videos leads to complex cells,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2001), (Springer-Verlag, Berlin, 2001), pp. 1075–1080.
  14. L. Wiskott, T. J. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770 (2002).
    [CrossRef] [PubMed]
  15. P. Berkes, L. Wiskott, “Applying slow feature analysis to image sequences yields a rich repertoire of complex cell properties,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2002) (Springer-Verlag, Berlin, 2002), pp. 81–86.
  16. J. Hurri, A. Hyvärinen, “Simple-cell-like receptive fields maximize temporal coherence in natural video,” Neural Comput. 15, 663–691 (2003).
    [CrossRef] [PubMed]
  17. A. Hyvärinen, “Blind source separation by nonstationarity of variance: cumulant-based approach,” IEEE Trans. Neural Netw. 12, 1471–1474 (2001).
    [CrossRef]
  18. C. Zetzsche, G. Krieger, “Nonlinear neurons and high-order statistics: new approaches to human vision and electronic image processing,” in Human Vision and Electronic Imaging IV, B. Rogowitz, T. V. Pappas, eds., Proc. SPIE3644, 2–33 (1999).
    [CrossRef]
  19. E. P. Simoncelli, O. Schwartz, “Modeling surround suppression in V1 neurons with a statistically-derived normalization model,” in Advances in Neural Information Processing Systems 11, M. S. Kearns, S. A. Solla, D. A. Cohn, eds. (MIT Press, Cambridge, Mass., 1999), pp. 153–159.
  20. A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspeces,” Neural Comput. 12, 1705–1720 (2000).
    [CrossRef]
  21. A. Hyvärinen, P. O. Hoyer, M. Inki, “Topographic independent component analysis,” Neural Comput. 13, 1527–1558 (2001).
    [CrossRef] [PubMed]
  22. O. Schwartz, E. P. Simoncelli, “Natural signal statistics and sensory gain control,” Nat. Neurosci. 4, 819–825 (2001).
    [CrossRef] [PubMed]
  23. M. J. Wainwright, E. Simoncelli, A. S. Willsky, “Random cascades on wavelet trees and their use in analyzing and modeling natural images,” Appl. Comput. Harmon. Anal. 11, 89–123 (2001).
    [CrossRef]
  24. D. H. Hubel, T. N. Wiesel, “Receptive fields and functional architecture of monkey striate cortex,” J. Physiol. (London) 195, 215–243 (1968).
  25. G. G. Blasdel, “Orientation selectivity, preference, and continuity in monkey striate cortex,” J. Neurosci. 12, 3139–3161 (1992).
    [PubMed]
  26. D. H. Hubel, T. N. Wiesel, “Functional architecture of macaque monkey visual cortex (Ferrier Lecture),” Proc. R. Soc. London Ser. B 198, 1–59 (1977).
    [CrossRef]
  27. R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
    [PubMed]
  28. T. Kohonen, Self-Organizing Maps (Springer, Berlin, 1995).
  29. N. V. Swindale, “The development of topography in the visual cortex: a review of models,” Network 7, 161–247 (1996).
    [CrossRef] [PubMed]
  30. C. von der Malsburg, “Self-organization of orientation-sensitive cells in the striate cortex,” Kybernetik 14, 85–100 (1973).
    [CrossRef] [PubMed]
  31. A. Hyvärinen, P. O. Hoyer, “A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images,” Vision Res. 41, 2413–2423 (2001).
    [CrossRef] [PubMed]
  32. A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
    [CrossRef]
  33. J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
    [CrossRef]
  34. P. Comon, “Independent component analysis—a new concept?” Signal Process. 36, 287–314 (1994).
    [CrossRef]
  35. C. Jutten, J. Hérault, “Blind separation of sources. Part I: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
    [CrossRef]
  36. J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
    [CrossRef]
  37. J. Hurri, A. Hyvärinen, “A two-layer temporal generative model of natural video exhibits complex-cell-like pooling of simple cell outputs,” in Computational Neuroscience: Trends in Research 2003, E. De Schutter, ed. (Elsevier, Amsterdam, The Netherlands, 2003).
  38. B. A. Olshausen, “Sparse codes and spikes,” in Statistical Theories of the Brain, R. Rao, B. A. Olshausen, eds. (MIT Press, Cambridge, Mass.2001).
  39. R. C. Emerson, J. R. Bergen, E. H. Adelson, “Directionally selective complex cells and the computation of motion energy in cat visual cortex,” Vision Res. 32, 203–218 (1992).
    [CrossRef] [PubMed]
  40. D. Pollen, S. Ronner, “Visual cortical neurons as localized spatial frequency filters,” IEEE Trans. Syst. Man Cybern. SMC-13, 907–916 (1983).
    [CrossRef]
  41. M. Welicky, W. H. Bosking, D. Fitzpatrick, “A systematic map of direction preference in primary visual cortex,” Nature (London) 379, 725–728 (1996).
    [CrossRef]
  42. P. O. Hoyer, A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vision Res. 42, 1593–1605 (2002).
    [CrossRef] [PubMed]
  43. J. Portilla, V. Strela, M. J. Wainwright, E. P. Simoncelli, “Adaptive Wiener denoising using a Gaussian scale mixture model in the wavelet domain,” in Proceedings of the International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 2001).
  44. R. Durbin, G. Mitchison, “A dimension reduction framework for understanding cortical maps,” Nature (London) 343, 644–647 (1990).
    [CrossRef]
  45. W. S. Geisler, D. G. Albrecht, “Cortical neurons: isolation of contrast gain control,” Vision Res. 32, 1409–1410 (1992).
    [CrossRef] [PubMed]
  46. D. Heeger, “Normalization of cell responses in cat striate cortex,” Visual Neurosci. 9, 181–198 (1992).
    [CrossRef]
  47. K. Matsuoka, M. Ohya, M. Kawamoto, “A neural net for blind separation of nonstationary signals,” Neural Networks 8, 411–419 (1995).
    [CrossRef]
  48. D.-T. Pham, J.-F. Cardoso, “Blind separation of instantaneous mixtures of non-stationary sources,” in Proceedings of the International Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000) (Helsinki University of Technology, Espoo, Finland, 2000), pp. 187–193.
  49. R. F. Engle, ed., ARCH: Selected Readings (Oxford U. Press, Oxford, UK, 1995).
  50. B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
    [CrossRef]
  51. A. Hyvärinen, M. Inki, “Estimating overcomplete independent component bases from image windows,” J. Math. Imaging Vision 17, 139–152 (2002).
    [CrossRef]
  52. A. Pece, “The problem of sparse image coding,” J. Math. Imaging Vision 17, 87–106 (2002).
    [CrossRef]
  53. B. A. Olshausen, P. Sallee, M. S. Lewicki, “Learning sparse image codes using a wavelet pyramid architecture,” in Advances in Neural Information Processing Systems, (MIT Press, Cambridge, Mass., 2001), Vol. 13, pp. 887–893.
  54. P. Paatero, U. Tapper, “Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values,” Environmetrics 5, 111–126 (1994).
    [CrossRef]
  55. D. D. Lee, H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature (London) 401, 788–791 (1999).
    [CrossRef]
  56. P. O. Hoyer, “Modeling receptive fields with non-negative sparse coding,” in Computational Neuroscience: Trends in Research 2003, E. De Schutter, ed. (Elsevier, Amsterdam, The Netherlands, 2003).
  57. D. R. Tailor, L. H. Finkel, G. Buchsbaum, “Color-opponent receptive fields derived from independent component analysis of natural images,” Vision Res. 40, 2671–2676 (2000).
    [CrossRef] [PubMed]
  58. T. Wachtler, T-W. Lee, T. J. Sejnowski, “Chromatic structure of natural scenes,” J. Opt. Soc. Am. A 18, 65–77 (2001).
    [CrossRef]
  59. P. O. Hoyer, A. Hyvärinen, “Independent component analysis applied to feature extraction from colour and stereo images,” Network Comput. Neural Syst. 11, 191–210 (2000).
    [CrossRef]

2003 (1)

J. Hurri, A. Hyvärinen, “Simple-cell-like receptive fields maximize temporal coherence in natural video,” Neural Comput. 15, 663–691 (2003).
[CrossRef] [PubMed]

2002 (4)

L. Wiskott, T. J. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770 (2002).
[CrossRef] [PubMed]

P. O. Hoyer, A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vision Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

A. Hyvärinen, M. Inki, “Estimating overcomplete independent component bases from image windows,” J. Math. Imaging Vision 17, 139–152 (2002).
[CrossRef]

A. Pece, “The problem of sparse image coding,” J. Math. Imaging Vision 17, 87–106 (2002).
[CrossRef]

2001 (7)

A. Hyvärinen, P. O. Hoyer, “A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images,” Vision Res. 41, 2413–2423 (2001).
[CrossRef] [PubMed]

A. Hyvärinen, P. O. Hoyer, M. Inki, “Topographic independent component analysis,” Neural Comput. 13, 1527–1558 (2001).
[CrossRef] [PubMed]

O. Schwartz, E. P. Simoncelli, “Natural signal statistics and sensory gain control,” Nat. Neurosci. 4, 819–825 (2001).
[CrossRef] [PubMed]

M. J. Wainwright, E. Simoncelli, A. S. Willsky, “Random cascades on wavelet trees and their use in analyzing and modeling natural images,” Appl. Comput. Harmon. Anal. 11, 89–123 (2001).
[CrossRef]

A. Hyvärinen, “Blind source separation by nonstationarity of variance: cumulant-based approach,” IEEE Trans. Neural Netw. 12, 1471–1474 (2001).
[CrossRef]

E. P. Simoncelli, B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216 (2001).
[CrossRef] [PubMed]

T. Wachtler, T-W. Lee, T. J. Sejnowski, “Chromatic structure of natural scenes,” J. Opt. Soc. Am. A 18, 65–77 (2001).
[CrossRef]

2000 (3)

D. R. Tailor, L. H. Finkel, G. Buchsbaum, “Color-opponent receptive fields derived from independent component analysis of natural images,” Vision Res. 40, 2671–2676 (2000).
[CrossRef] [PubMed]

P. O. Hoyer, A. Hyvärinen, “Independent component analysis applied to feature extraction from colour and stereo images,” Network Comput. Neural Syst. 11, 191–210 (2000).
[CrossRef]

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspeces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

1999 (3)

A. Hyvärinen, “Fast and robust fixed-point algorithms for independent component analysis,” IEEE Trans. Neural Netw. 10, 626–634 (1999).
[CrossRef]

A. Hyvärinen, “Sparse code shrinkage: denoising of non-gaussian data by maximum likelihood estimation,” Neural Comput. 11, 1739–1768 (1999).
[CrossRef]

D. D. Lee, H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature (London) 401, 788–791 (1999).
[CrossRef]

1998 (2)

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

1997 (3)

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

G. E. Hinton, Z. Ghahramani, “Generative models for discovering sparse distributed representations,” Philos. Trans. R. Soc. London Ser. B 352, 1177–1190 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

1996 (3)

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

N. V. Swindale, “The development of topography in the visual cortex: a review of models,” Network 7, 161–247 (1996).
[CrossRef] [PubMed]

M. Welicky, W. H. Bosking, D. Fitzpatrick, “A systematic map of direction preference in primary visual cortex,” Nature (London) 379, 725–728 (1996).
[CrossRef]

1995 (1)

K. Matsuoka, M. Ohya, M. Kawamoto, “A neural net for blind separation of nonstationary signals,” Neural Networks 8, 411–419 (1995).
[CrossRef]

1994 (3)

P. Paatero, U. Tapper, “Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values,” Environmetrics 5, 111–126 (1994).
[CrossRef]

P. Comon, “Independent component analysis—a new concept?” Signal Process. 36, 287–314 (1994).
[CrossRef]

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

1992 (4)

G. G. Blasdel, “Orientation selectivity, preference, and continuity in monkey striate cortex,” J. Neurosci. 12, 3139–3161 (1992).
[PubMed]

R. C. Emerson, J. R. Bergen, E. H. Adelson, “Directionally selective complex cells and the computation of motion energy in cat visual cortex,” Vision Res. 32, 203–218 (1992).
[CrossRef] [PubMed]

W. S. Geisler, D. G. Albrecht, “Cortical neurons: isolation of contrast gain control,” Vision Res. 32, 1409–1410 (1992).
[CrossRef] [PubMed]

D. Heeger, “Normalization of cell responses in cat striate cortex,” Visual Neurosci. 9, 181–198 (1992).
[CrossRef]

1991 (2)

C. Jutten, J. Hérault, “Blind separation of sources. Part I: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

P. Földiák, “Learning invariance from transformation sequences,” Neural Comput. 3, 194–200 (1991).
[CrossRef]

1990 (1)

R. Durbin, G. Mitchison, “A dimension reduction framework for understanding cortical maps,” Nature (London) 343, 644–647 (1990).
[CrossRef]

1988 (1)

R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
[PubMed]

1983 (1)

D. Pollen, S. Ronner, “Visual cortical neurons as localized spatial frequency filters,” IEEE Trans. Syst. Man Cybern. SMC-13, 907–916 (1983).
[CrossRef]

1977 (1)

D. H. Hubel, T. N. Wiesel, “Functional architecture of macaque monkey visual cortex (Ferrier Lecture),” Proc. R. Soc. London Ser. B 198, 1–59 (1977).
[CrossRef]

1973 (1)

C. von der Malsburg, “Self-organization of orientation-sensitive cells in the striate cortex,” Kybernetik 14, 85–100 (1973).
[CrossRef] [PubMed]

1972 (1)

H. B. Barlow, “Single units and sensation: a neuron doctrine for perceptual psychology?” Perception 1, 371–394 (1972).
[CrossRef] [PubMed]

1968 (1)

D. H. Hubel, T. N. Wiesel, “Receptive fields and functional architecture of monkey striate cortex,” J. Physiol. (London) 195, 215–243 (1968).

Adelson, E. H.

R. C. Emerson, J. R. Bergen, E. H. Adelson, “Directionally selective complex cells and the computation of motion energy in cat visual cortex,” Vision Res. 32, 203–218 (1992).
[CrossRef] [PubMed]

E. P. Simoncelli, E. H. Adelson, “Noise removal via bayesian wavelet coring,” in Proceedings of the Third IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.

Albrecht, D. G.

W. S. Geisler, D. G. Albrecht, “Cortical neurons: isolation of contrast gain control,” Vision Res. 32, 1409–1410 (1992).
[CrossRef] [PubMed]

Barlow, H. B.

H. B. Barlow, “Single units and sensation: a neuron doctrine for perceptual psychology?” Perception 1, 371–394 (1972).
[CrossRef] [PubMed]

Bell, A. J.

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

Bergen, J. R.

R. C. Emerson, J. R. Bergen, E. H. Adelson, “Directionally selective complex cells and the computation of motion energy in cat visual cortex,” Vision Res. 32, 203–218 (1992).
[CrossRef] [PubMed]

Berkes, P.

P. Berkes, L. Wiskott, “Applying slow feature analysis to image sequences yields a rich repertoire of complex cell properties,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2002) (Springer-Verlag, Berlin, 2002), pp. 81–86.

Blasdel, G. G.

G. G. Blasdel, “Orientation selectivity, preference, and continuity in monkey striate cortex,” J. Neurosci. 12, 3139–3161 (1992).
[PubMed]

Bosking, W. H.

M. Welicky, W. H. Bosking, D. Fitzpatrick, “A systematic map of direction preference in primary visual cortex,” Nature (London) 379, 725–728 (1996).
[CrossRef]

Buchsbaum, G.

D. R. Tailor, L. H. Finkel, G. Buchsbaum, “Color-opponent receptive fields derived from independent component analysis of natural images,” Vision Res. 40, 2671–2676 (2000).
[CrossRef] [PubMed]

Cardoso, J.-F.

D.-T. Pham, J.-F. Cardoso, “Blind separation of instantaneous mixtures of non-stationary sources,” in Proceedings of the International Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000) (Helsinki University of Technology, Espoo, Finland, 2000), pp. 187–193.

Comon, P.

P. Comon, “Independent component analysis—a new concept?” Signal Process. 36, 287–314 (1994).
[CrossRef]

De Valois, R. L.

R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
[PubMed]

Dümmer, O.

C. Kayser, W. Einhäuser, O. Dümmer, P. König, K. Körding, “Extracting slow subspaces from natural videos leads to complex cells,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2001), (Springer-Verlag, Berlin, 2001), pp. 1075–1080.

Durbin, R.

R. Durbin, G. Mitchison, “A dimension reduction framework for understanding cortical maps,” Nature (London) 343, 644–647 (1990).
[CrossRef]

Einhäuser, W.

C. Kayser, W. Einhäuser, O. Dümmer, P. König, K. Körding, “Extracting slow subspaces from natural videos leads to complex cells,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2001), (Springer-Verlag, Berlin, 2001), pp. 1075–1080.

Emerson, R. C.

R. C. Emerson, J. R. Bergen, E. H. Adelson, “Directionally selective complex cells and the computation of motion energy in cat visual cortex,” Vision Res. 32, 203–218 (1992).
[CrossRef] [PubMed]

Field, D. J.

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

Finkel, L. H.

D. R. Tailor, L. H. Finkel, G. Buchsbaum, “Color-opponent receptive fields derived from independent component analysis of natural images,” Vision Res. 40, 2671–2676 (2000).
[CrossRef] [PubMed]

Fitzpatrick, D.

M. Welicky, W. H. Bosking, D. Fitzpatrick, “A systematic map of direction preference in primary visual cortex,” Nature (London) 379, 725–728 (1996).
[CrossRef]

Földiák, P.

P. Földiák, “Learning invariance from transformation sequences,” Neural Comput. 3, 194–200 (1991).
[CrossRef]

Geisler, W. S.

W. S. Geisler, D. G. Albrecht, “Cortical neurons: isolation of contrast gain control,” Vision Res. 32, 1409–1410 (1992).
[CrossRef] [PubMed]

Ghahramani, Z.

G. E. Hinton, Z. Ghahramani, “Generative models for discovering sparse distributed representations,” Philos. Trans. R. Soc. London Ser. B 352, 1177–1190 (1997).
[CrossRef]

Hamilton, S. L.

R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
[PubMed]

Heeger, D.

D. Heeger, “Normalization of cell responses in cat striate cortex,” Visual Neurosci. 9, 181–198 (1992).
[CrossRef]

Hérault, J.

C. Jutten, J. Hérault, “Blind separation of sources. Part I: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Hinton, G. E.

G. E. Hinton, Z. Ghahramani, “Generative models for discovering sparse distributed representations,” Philos. Trans. R. Soc. London Ser. B 352, 1177–1190 (1997).
[CrossRef]

Hoyer, P. O.

P. O. Hoyer, A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vision Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

A. Hyvärinen, P. O. Hoyer, “A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images,” Vision Res. 41, 2413–2423 (2001).
[CrossRef] [PubMed]

A. Hyvärinen, P. O. Hoyer, M. Inki, “Topographic independent component analysis,” Neural Comput. 13, 1527–1558 (2001).
[CrossRef] [PubMed]

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspeces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

P. O. Hoyer, A. Hyvärinen, “Independent component analysis applied to feature extraction from colour and stereo images,” Network Comput. Neural Syst. 11, 191–210 (2000).
[CrossRef]

P. O. Hoyer, “Modeling receptive fields with non-negative sparse coding,” in Computational Neuroscience: Trends in Research 2003, E. De Schutter, ed. (Elsevier, Amsterdam, The Netherlands, 2003).

Hubel, D. H.

D. H. Hubel, T. N. Wiesel, “Functional architecture of macaque monkey visual cortex (Ferrier Lecture),” Proc. R. Soc. London Ser. B 198, 1–59 (1977).
[CrossRef]

D. H. Hubel, T. N. Wiesel, “Receptive fields and functional architecture of monkey striate cortex,” J. Physiol. (London) 195, 215–243 (1968).

Hurri, J.

J. Hurri, A. Hyvärinen, “Simple-cell-like receptive fields maximize temporal coherence in natural video,” Neural Comput. 15, 663–691 (2003).
[CrossRef] [PubMed]

J. Hurri, A. Hyvärinen, “A two-layer temporal generative model of natural video exhibits complex-cell-like pooling of simple cell outputs,” in Computational Neuroscience: Trends in Research 2003, E. De Schutter, ed. (Elsevier, Amsterdam, The Netherlands, 2003).

Hyvärinen, A.

J. Hurri, A. Hyvärinen, “Simple-cell-like receptive fields maximize temporal coherence in natural video,” Neural Comput. 15, 663–691 (2003).
[CrossRef] [PubMed]

A. Hyvärinen, M. Inki, “Estimating overcomplete independent component bases from image windows,” J. Math. Imaging Vision 17, 139–152 (2002).
[CrossRef]

P. O. Hoyer, A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vision Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

A. Hyvärinen, “Blind source separation by nonstationarity of variance: cumulant-based approach,” IEEE Trans. Neural Netw. 12, 1471–1474 (2001).
[CrossRef]

A. Hyvärinen, P. O. Hoyer, M. Inki, “Topographic independent component analysis,” Neural Comput. 13, 1527–1558 (2001).
[CrossRef] [PubMed]

A. Hyvärinen, P. O. Hoyer, “A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images,” Vision Res. 41, 2413–2423 (2001).
[CrossRef] [PubMed]

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspeces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

P. O. Hoyer, A. Hyvärinen, “Independent component analysis applied to feature extraction from colour and stereo images,” Network Comput. Neural Syst. 11, 191–210 (2000).
[CrossRef]

A. Hyvärinen, “Fast and robust fixed-point algorithms for independent component analysis,” IEEE Trans. Neural Netw. 10, 626–634 (1999).
[CrossRef]

A. Hyvärinen, “Sparse code shrinkage: denoising of non-gaussian data by maximum likelihood estimation,” Neural Comput. 11, 1739–1768 (1999).
[CrossRef]

A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis (Wiley Interscience, New York, 2001).

J. Hurri, A. Hyvärinen, “A two-layer temporal generative model of natural video exhibits complex-cell-like pooling of simple cell outputs,” in Computational Neuroscience: Trends in Research 2003, E. De Schutter, ed. (Elsevier, Amsterdam, The Netherlands, 2003).

Inki, M.

A. Hyvärinen, M. Inki, “Estimating overcomplete independent component bases from image windows,” J. Math. Imaging Vision 17, 139–152 (2002).
[CrossRef]

A. Hyvärinen, P. O. Hoyer, M. Inki, “Topographic independent component analysis,” Neural Comput. 13, 1527–1558 (2001).
[CrossRef] [PubMed]

Jutten, C.

C. Jutten, J. Hérault, “Blind separation of sources. Part I: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Karhunen, J.

A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis (Wiley Interscience, New York, 2001).

Kawamoto, M.

K. Matsuoka, M. Ohya, M. Kawamoto, “A neural net for blind separation of nonstationary signals,” Neural Networks 8, 411–419 (1995).
[CrossRef]

Kayser, C.

C. Kayser, W. Einhäuser, O. Dümmer, P. König, K. Körding, “Extracting slow subspaces from natural videos leads to complex cells,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2001), (Springer-Verlag, Berlin, 2001), pp. 1075–1080.

Kohonen, T.

T. Kohonen, Self-Organizing Maps (Springer, Berlin, 1995).

König, P.

C. Kayser, W. Einhäuser, O. Dümmer, P. König, K. Körding, “Extracting slow subspaces from natural videos leads to complex cells,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2001), (Springer-Verlag, Berlin, 2001), pp. 1075–1080.

Körding, K.

C. Kayser, W. Einhäuser, O. Dümmer, P. König, K. Körding, “Extracting slow subspaces from natural videos leads to complex cells,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2001), (Springer-Verlag, Berlin, 2001), pp. 1075–1080.

Krieger, G.

C. Zetzsche, G. Krieger, “Nonlinear neurons and high-order statistics: new approaches to human vision and electronic image processing,” in Human Vision and Electronic Imaging IV, B. Rogowitz, T. V. Pappas, eds., Proc. SPIE3644, 2–33 (1999).
[CrossRef]

Lee, D. D.

D. D. Lee, H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature (London) 401, 788–791 (1999).
[CrossRef]

Lee, T-W.

Lewicki, M. S.

B. A. Olshausen, P. Sallee, M. S. Lewicki, “Learning sparse image codes using a wavelet pyramid architecture,” in Advances in Neural Information Processing Systems, (MIT Press, Cambridge, Mass., 2001), Vol. 13, pp. 887–893.

Matsuoka, K.

K. Matsuoka, M. Ohya, M. Kawamoto, “A neural net for blind separation of nonstationary signals,” Neural Networks 8, 411–419 (1995).
[CrossRef]

Mitchison, G.

R. Durbin, G. Mitchison, “A dimension reduction framework for understanding cortical maps,” Nature (London) 343, 644–647 (1990).
[CrossRef]

Ohya, M.

K. Matsuoka, M. Ohya, M. Kawamoto, “A neural net for blind separation of nonstationary signals,” Neural Networks 8, 411–419 (1995).
[CrossRef]

Oja, E.

A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis (Wiley Interscience, New York, 2001).

Olshausen, B. A.

E. P. Simoncelli, B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216 (2001).
[CrossRef] [PubMed]

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

B. A. Olshausen, “Principles of image representation in visual cortex,” in The Visual Neurosciences, L. M. Chalupa, J. S. Werner, eds. (MIT Press, Cambridge, Mass., 2003).

B. A. Olshausen, “Sparse codes and spikes,” in Statistical Theories of the Brain, R. Rao, B. A. Olshausen, eds. (MIT Press, Cambridge, Mass.2001).

B. A. Olshausen, P. Sallee, M. S. Lewicki, “Learning sparse image codes using a wavelet pyramid architecture,” in Advances in Neural Information Processing Systems, (MIT Press, Cambridge, Mass., 2001), Vol. 13, pp. 887–893.

Paatero, P.

P. Paatero, U. Tapper, “Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values,” Environmetrics 5, 111–126 (1994).
[CrossRef]

Pece, A.

A. Pece, “The problem of sparse image coding,” J. Math. Imaging Vision 17, 87–106 (2002).
[CrossRef]

Pham, D.-T.

D.-T. Pham, J.-F. Cardoso, “Blind separation of instantaneous mixtures of non-stationary sources,” in Proceedings of the International Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000) (Helsinki University of Technology, Espoo, Finland, 2000), pp. 187–193.

Pollen, D.

D. Pollen, S. Ronner, “Visual cortical neurons as localized spatial frequency filters,” IEEE Trans. Syst. Man Cybern. SMC-13, 907–916 (1983).
[CrossRef]

Portilla, J.

J. Portilla, V. Strela, M. J. Wainwright, E. P. Simoncelli, “Adaptive Wiener denoising using a Gaussian scale mixture model in the wavelet domain,” in Proceedings of the International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 2001).

Ronner, S.

D. Pollen, S. Ronner, “Visual cortical neurons as localized spatial frequency filters,” IEEE Trans. Syst. Man Cybern. SMC-13, 907–916 (1983).
[CrossRef]

Ruderman, D. L.

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

Sallee, P.

B. A. Olshausen, P. Sallee, M. S. Lewicki, “Learning sparse image codes using a wavelet pyramid architecture,” in Advances in Neural Information Processing Systems, (MIT Press, Cambridge, Mass., 2001), Vol. 13, pp. 887–893.

Schwartz, O.

O. Schwartz, E. P. Simoncelli, “Natural signal statistics and sensory gain control,” Nat. Neurosci. 4, 819–825 (2001).
[CrossRef] [PubMed]

E. P. Simoncelli, O. Schwartz, “Modeling surround suppression in V1 neurons with a statistically-derived normalization model,” in Advances in Neural Information Processing Systems 11, M. S. Kearns, S. A. Solla, D. A. Cohn, eds. (MIT Press, Cambridge, Mass., 1999), pp. 153–159.

Sejnowski, T. J.

L. Wiskott, T. J. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770 (2002).
[CrossRef] [PubMed]

T. Wachtler, T-W. Lee, T. J. Sejnowski, “Chromatic structure of natural scenes,” J. Opt. Soc. Am. A 18, 65–77 (2001).
[CrossRef]

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

Seung, H. S.

D. D. Lee, H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature (London) 401, 788–791 (1999).
[CrossRef]

Silverman, M. S.

R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
[PubMed]

Simoncelli, E.

M. J. Wainwright, E. Simoncelli, A. S. Willsky, “Random cascades on wavelet trees and their use in analyzing and modeling natural images,” Appl. Comput. Harmon. Anal. 11, 89–123 (2001).
[CrossRef]

Simoncelli, E. P.

O. Schwartz, E. P. Simoncelli, “Natural signal statistics and sensory gain control,” Nat. Neurosci. 4, 819–825 (2001).
[CrossRef] [PubMed]

E. P. Simoncelli, B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216 (2001).
[CrossRef] [PubMed]

E. P. Simoncelli, E. H. Adelson, “Noise removal via bayesian wavelet coring,” in Proceedings of the Third IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.

E. P. Simoncelli, O. Schwartz, “Modeling surround suppression in V1 neurons with a statistically-derived normalization model,” in Advances in Neural Information Processing Systems 11, M. S. Kearns, S. A. Solla, D. A. Cohn, eds. (MIT Press, Cambridge, Mass., 1999), pp. 153–159.

J. Portilla, V. Strela, M. J. Wainwright, E. P. Simoncelli, “Adaptive Wiener denoising using a Gaussian scale mixture model in the wavelet domain,” in Proceedings of the International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 2001).

Strela, V.

J. Portilla, V. Strela, M. J. Wainwright, E. P. Simoncelli, “Adaptive Wiener denoising using a Gaussian scale mixture model in the wavelet domain,” in Proceedings of the International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 2001).

Swindale, N. V.

N. V. Swindale, “The development of topography in the visual cortex: a review of models,” Network 7, 161–247 (1996).
[CrossRef] [PubMed]

Switkes, E.

R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
[PubMed]

Tailor, D. R.

D. R. Tailor, L. H. Finkel, G. Buchsbaum, “Color-opponent receptive fields derived from independent component analysis of natural images,” Vision Res. 40, 2671–2676 (2000).
[CrossRef] [PubMed]

Tapper, U.

P. Paatero, U. Tapper, “Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values,” Environmetrics 5, 111–126 (1994).
[CrossRef]

Tootell, R. B. H.

R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
[PubMed]

van der Schaaf, A.

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

van Hateren, J. H.

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

von der Malsburg, C.

C. von der Malsburg, “Self-organization of orientation-sensitive cells in the striate cortex,” Kybernetik 14, 85–100 (1973).
[CrossRef] [PubMed]

Wachtler, T.

Wainwright, M. J.

M. J. Wainwright, E. Simoncelli, A. S. Willsky, “Random cascades on wavelet trees and their use in analyzing and modeling natural images,” Appl. Comput. Harmon. Anal. 11, 89–123 (2001).
[CrossRef]

J. Portilla, V. Strela, M. J. Wainwright, E. P. Simoncelli, “Adaptive Wiener denoising using a Gaussian scale mixture model in the wavelet domain,” in Proceedings of the International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 2001).

Welicky, M.

M. Welicky, W. H. Bosking, D. Fitzpatrick, “A systematic map of direction preference in primary visual cortex,” Nature (London) 379, 725–728 (1996).
[CrossRef]

Wiesel, T. N.

D. H. Hubel, T. N. Wiesel, “Functional architecture of macaque monkey visual cortex (Ferrier Lecture),” Proc. R. Soc. London Ser. B 198, 1–59 (1977).
[CrossRef]

D. H. Hubel, T. N. Wiesel, “Receptive fields and functional architecture of monkey striate cortex,” J. Physiol. (London) 195, 215–243 (1968).

Willsky, A. S.

M. J. Wainwright, E. Simoncelli, A. S. Willsky, “Random cascades on wavelet trees and their use in analyzing and modeling natural images,” Appl. Comput. Harmon. Anal. 11, 89–123 (2001).
[CrossRef]

Wiskott, L.

L. Wiskott, T. J. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770 (2002).
[CrossRef] [PubMed]

P. Berkes, L. Wiskott, “Applying slow feature analysis to image sequences yields a rich repertoire of complex cell properties,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2002) (Springer-Verlag, Berlin, 2002), pp. 81–86.

Zetzsche, C.

C. Zetzsche, G. Krieger, “Nonlinear neurons and high-order statistics: new approaches to human vision and electronic image processing,” in Human Vision and Electronic Imaging IV, B. Rogowitz, T. V. Pappas, eds., Proc. SPIE3644, 2–33 (1999).
[CrossRef]

Annu. Rev. Neurosci. (1)

E. P. Simoncelli, B. A. Olshausen, “Natural image statistics and neural representation,” Annu. Rev. Neurosci. 24, 1193–1216 (2001).
[CrossRef] [PubMed]

Appl. Comput. Harmon. Anal. (1)

M. J. Wainwright, E. Simoncelli, A. S. Willsky, “Random cascades on wavelet trees and their use in analyzing and modeling natural images,” Appl. Comput. Harmon. Anal. 11, 89–123 (2001).
[CrossRef]

Environmetrics (1)

P. Paatero, U. Tapper, “Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values,” Environmetrics 5, 111–126 (1994).
[CrossRef]

IEEE Trans. Neural Netw. (2)

A. Hyvärinen, “Fast and robust fixed-point algorithms for independent component analysis,” IEEE Trans. Neural Netw. 10, 626–634 (1999).
[CrossRef]

A. Hyvärinen, “Blind source separation by nonstationarity of variance: cumulant-based approach,” IEEE Trans. Neural Netw. 12, 1471–1474 (2001).
[CrossRef]

IEEE Trans. Syst. Man Cybern. (1)

D. Pollen, S. Ronner, “Visual cortical neurons as localized spatial frequency filters,” IEEE Trans. Syst. Man Cybern. SMC-13, 907–916 (1983).
[CrossRef]

J. Math. Imaging Vision (2)

A. Hyvärinen, M. Inki, “Estimating overcomplete independent component bases from image windows,” J. Math. Imaging Vision 17, 139–152 (2002).
[CrossRef]

A. Pece, “The problem of sparse image coding,” J. Math. Imaging Vision 17, 87–106 (2002).
[CrossRef]

J. Neurosci. (2)

G. G. Blasdel, “Orientation selectivity, preference, and continuity in monkey striate cortex,” J. Neurosci. 12, 3139–3161 (1992).
[PubMed]

R. B. H. Tootell, M. S. Silverman, S. L. Hamilton, E. Switkes, R. L. De Valois, “Functional anatomy of macaque striate cortex. V. Spatial frequency,” J. Neurosci. 8, 1610–1624 (1988).
[PubMed]

J. Opt. Soc. Am. A (1)

J. Physiol. (London) (1)

D. H. Hubel, T. N. Wiesel, “Receptive fields and functional architecture of monkey striate cortex,” J. Physiol. (London) 195, 215–243 (1968).

Kybernetik (1)

C. von der Malsburg, “Self-organization of orientation-sensitive cells in the striate cortex,” Kybernetik 14, 85–100 (1973).
[CrossRef] [PubMed]

Nat. Neurosci. (1)

O. Schwartz, E. P. Simoncelli, “Natural signal statistics and sensory gain control,” Nat. Neurosci. 4, 819–825 (2001).
[CrossRef] [PubMed]

Nature (London) (4)

B. A. Olshausen, D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature (London) 381, 607–609 (1996).
[CrossRef]

D. D. Lee, H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature (London) 401, 788–791 (1999).
[CrossRef]

M. Welicky, W. H. Bosking, D. Fitzpatrick, “A systematic map of direction preference in primary visual cortex,” Nature (London) 379, 725–728 (1996).
[CrossRef]

R. Durbin, G. Mitchison, “A dimension reduction framework for understanding cortical maps,” Nature (London) 343, 644–647 (1990).
[CrossRef]

Network (1)

N. V. Swindale, “The development of topography in the visual cortex: a review of models,” Network 7, 161–247 (1996).
[CrossRef] [PubMed]

Network Comput. Neural Syst. (1)

P. O. Hoyer, A. Hyvärinen, “Independent component analysis applied to feature extraction from colour and stereo images,” Network Comput. Neural Syst. 11, 191–210 (2000).
[CrossRef]

Neural Comput. (7)

P. Földiák, “Learning invariance from transformation sequences,” Neural Comput. 3, 194–200 (1991).
[CrossRef]

L. Wiskott, T. J. Sejnowski, “Slow feature analysis: unsupervised learning of invariances,” Neural Comput. 14, 715–770 (2002).
[CrossRef] [PubMed]

D. J. Field, “What is the goal of sensory coding?” Neural Comput. 6, 559–601 (1994).
[CrossRef]

A. Hyvärinen, “Sparse code shrinkage: denoising of non-gaussian data by maximum likelihood estimation,” Neural Comput. 11, 1739–1768 (1999).
[CrossRef]

J. Hurri, A. Hyvärinen, “Simple-cell-like receptive fields maximize temporal coherence in natural video,” Neural Comput. 15, 663–691 (2003).
[CrossRef] [PubMed]

A. Hyvärinen, P. O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspeces,” Neural Comput. 12, 1705–1720 (2000).
[CrossRef]

A. Hyvärinen, P. O. Hoyer, M. Inki, “Topographic independent component analysis,” Neural Comput. 13, 1527–1558 (2001).
[CrossRef] [PubMed]

Neural Networks (1)

K. Matsuoka, M. Ohya, M. Kawamoto, “A neural net for blind separation of nonstationary signals,” Neural Networks 8, 411–419 (1995).
[CrossRef]

Perception (1)

H. B. Barlow, “Single units and sensation: a neuron doctrine for perceptual psychology?” Perception 1, 371–394 (1972).
[CrossRef] [PubMed]

Philos. Trans. R. Soc. London Ser. B (1)

G. E. Hinton, Z. Ghahramani, “Generative models for discovering sparse distributed representations,” Philos. Trans. R. Soc. London Ser. B 352, 1177–1190 (1997).
[CrossRef]

Proc. R. Soc. London Ser. B (3)

D. H. Hubel, T. N. Wiesel, “Functional architecture of macaque monkey visual cortex (Ferrier Lecture),” Proc. R. Soc. London Ser. B 198, 1–59 (1977).
[CrossRef]

J. H. van Hateren, A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 359–366 (1998).
[CrossRef]

J. H. van Hateren, D. L. Ruderman, “Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex,” Proc. R. Soc. London Ser. B 265, 2315–2320 (1998).
[CrossRef]

Signal Process. (2)

P. Comon, “Independent component analysis—a new concept?” Signal Process. 36, 287–314 (1994).
[CrossRef]

C. Jutten, J. Hérault, “Blind separation of sources. Part I: An adaptive algorithm based on neuromimetic architecture,” Signal Process. 24, 1–10 (1991).
[CrossRef]

Vision Res. (7)

A. Hyvärinen, P. O. Hoyer, “A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images,” Vision Res. 41, 2413–2423 (2001).
[CrossRef] [PubMed]

A. J. Bell, T. J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res. 37, 3327–3338 (1997).
[CrossRef]

B. A. Olshausen, D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?” Vision Res. 37, 3311–3325 (1997).
[CrossRef]

D. R. Tailor, L. H. Finkel, G. Buchsbaum, “Color-opponent receptive fields derived from independent component analysis of natural images,” Vision Res. 40, 2671–2676 (2000).
[CrossRef] [PubMed]

R. C. Emerson, J. R. Bergen, E. H. Adelson, “Directionally selective complex cells and the computation of motion energy in cat visual cortex,” Vision Res. 32, 203–218 (1992).
[CrossRef] [PubMed]

W. S. Geisler, D. G. Albrecht, “Cortical neurons: isolation of contrast gain control,” Vision Res. 32, 1409–1410 (1992).
[CrossRef] [PubMed]

P. O. Hoyer, A. Hyvärinen, “A multi-layer sparse coding network learns contour coding from natural images,” Vision Res. 42, 1593–1605 (2002).
[CrossRef] [PubMed]

Visual Neurosci. (1)

D. Heeger, “Normalization of cell responses in cat striate cortex,” Visual Neurosci. 9, 181–198 (1992).
[CrossRef]

Other (16)

J. Portilla, V. Strela, M. J. Wainwright, E. P. Simoncelli, “Adaptive Wiener denoising using a Gaussian scale mixture model in the wavelet domain,” in Proceedings of the International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 2001).

P. O. Hoyer, “Modeling receptive fields with non-negative sparse coding,” in Computational Neuroscience: Trends in Research 2003, E. De Schutter, ed. (Elsevier, Amsterdam, The Netherlands, 2003).

D.-T. Pham, J.-F. Cardoso, “Blind separation of instantaneous mixtures of non-stationary sources,” in Proceedings of the International Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000) (Helsinki University of Technology, Espoo, Finland, 2000), pp. 187–193.

R. F. Engle, ed., ARCH: Selected Readings (Oxford U. Press, Oxford, UK, 1995).

B. A. Olshausen, P. Sallee, M. S. Lewicki, “Learning sparse image codes using a wavelet pyramid architecture,” in Advances in Neural Information Processing Systems, (MIT Press, Cambridge, Mass., 2001), Vol. 13, pp. 887–893.

T. Kohonen, Self-Organizing Maps (Springer, Berlin, 1995).

J. Hurri, A. Hyvärinen, “A two-layer temporal generative model of natural video exhibits complex-cell-like pooling of simple cell outputs,” in Computational Neuroscience: Trends in Research 2003, E. De Schutter, ed. (Elsevier, Amsterdam, The Netherlands, 2003).

B. A. Olshausen, “Sparse codes and spikes,” in Statistical Theories of the Brain, R. Rao, B. A. Olshausen, eds. (MIT Press, Cambridge, Mass.2001).

D. C. Knill, W. Richards, eds., Perception as Bayesian Inference (Cambridge U. Press, Cambridge, UK, 1996).

E. P. Simoncelli, E. H. Adelson, “Noise removal via bayesian wavelet coring,” in Proceedings of the Third IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 1996), pp. 379–382.

B. A. Olshausen, “Principles of image representation in visual cortex,” in The Visual Neurosciences, L. M. Chalupa, J. S. Werner, eds. (MIT Press, Cambridge, Mass., 2003).

A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis (Wiley Interscience, New York, 2001).

P. Berkes, L. Wiskott, “Applying slow feature analysis to image sequences yields a rich repertoire of complex cell properties,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2002) (Springer-Verlag, Berlin, 2002), pp. 81–86.

C. Zetzsche, G. Krieger, “Nonlinear neurons and high-order statistics: new approaches to human vision and electronic image processing,” in Human Vision and Electronic Imaging IV, B. Rogowitz, T. V. Pappas, eds., Proc. SPIE3644, 2–33 (1999).
[CrossRef]

E. P. Simoncelli, O. Schwartz, “Modeling surround suppression in V1 neurons with a statistically-derived normalization model,” in Advances in Neural Information Processing Systems 11, M. S. Kearns, S. A. Solla, D. A. Cohn, eds. (MIT Press, Cambridge, Mass., 1999), pp. 153–159.

C. Kayser, W. Einhäuser, O. Dümmer, P. König, K. Körding, “Extracting slow subspaces from natural videos leads to complex cells,” in Proceedings of the International Conference on Artificial Neural Networks (ICANN2001), (Springer-Verlag, Berlin, 2001), pp. 1075–1080.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (9)

Fig. 1
Fig. 1

Illustration of a sparse probability density. The vertical axis is the probability density, and the horizontal axis is the (absolute) value of random variable s. The sparse exponential density function is given by the solid curve. For comparison, the density of the absolute value of a Gaussian random variable of the same variance is given by the dashed curve.

Fig. 2
Fig. 2

Illustration of the energy correlation in the probability density in Eq. (8). (a) Two-dimensional conditional density of sj (vertical axis) given si (horizontal axis). The conditional density is obtained by taking vertical slices of the density function, and then normalizing each slice so that it integrates to 1, and thus defines a proper probability density function. Black means low probability density, and white means high probability density. We see that the conditional distribution gets broader as si goes further from zero in either direction.19 This leads to correlation of energies, since the expectation of the square is nothing but the variance. (b) Conditional variance of sj (vertical axis) for given si (horizontal axis).

Fig. 3
Fig. 3

Illustration of a temporal bubble. The original signal z(t) (top) is multiplied by a variance (activity) signal v(t) (middle) to obtain the observed signal s(t) (bottom). The observed signal is both sparse and temporally coherent.

Fig. 4
Fig. 4

Output of a filter estimated by ICA when the input consists of an image sequence. A temporal bubble structure is clearly visible. For details on the data, see Subsection 3.C.

Fig. 5
Fig. 5

Log error in separation of artificial signal mixtures as a function of the size of the interval of temporal integration. This reaches a minimum at approximately 7. Size 1 would correspond to ordinary sparseness, i.e., no temporal integration. Standard errors of the mean are shown as well.

Fig. 6
Fig. 6

Four types of representation. The plots show the outputs of filters as a function of time (horizontal axis) and the position of the filter on the topographic grid (vertical axis). Each pixel is the output of one unit at a given time point, gray being zero, white and black meaning positive and negative outputs. For simplicity, the topography is here one dimensional. In the basic sparse representation, the filters are independent. In the topographic representation, the activations of the filters are also spatially grouped. In the representation that has temporal coherence, they are temporally grouped. The bubble representation combines all these aspects, leading to spatiotemporal activity bubbles. Note that the two latter types of representation require that the data have a temporal structure, unlike the two former ones.

Fig. 7
Fig. 7

Combination of temporal and spatial (i.e., topographic) energy correlation. The two signals are caricatures of what the outputs of two simple cells with strong energy correlation could look like. They are uncorrelated, both from each other and temporally. Nevertheless, we see temporal bubbles of activity in the outputs, and these bubbles are simultaneous, which eventually leads to spatiotemporal bubbles when there are many cells arranged topographically. Note that a very similar figure was used to illustrate basic energy correlation in topographic ICA.21 In that context, the temporal energy correlation was added for the purposes of illustration only, whereas here it is an essential part of the model.

Fig. 8
Fig. 8

Spatial basis vectors estimated by our model from natural image sequences. The results are very similar to what was found by topographic ICA.31

Fig. 9
Fig. 9

Bubbles that emerge from image sequences. (a) We used a representation with one-dimensional topography to be able to visualize the results (shown arranged on a two-dimensional grid for reasons of space). (b) Outputs of the cells for two different input image sequences, coded as gray-scale values (gray=0). The vertical axis is the cell index, and the horizontal axis is the time index. One can clearly see the bubblelike quality of the data. (c) Image sequences used as input in (b) (again, shown in two dimensions for reasons of space).

Equations (45)

Equations on this page are rendered with MathJax. Learn more.

p(s)=12exp(-2|s|).
G(s2)=-αs2+β=-α|s|+β.
cov[s(t), s(t-τ)]=E{s(t)s(t-τ)},
cov{[s(t)]2, [s(t-τ)]2}=E{[s(t)]2[s(t-τ)]2}-E{[s(t)]2}E{[s(t-τ)]2}.
cov(si2, sj2)=E{si2sj2}-E{si2}E{sj2}0.
si=ziv,
sj=zjv,
p(si, sj)=23πexp(-3si2+sj2).
E{G(si2+sj2)},
I(x, y)=i=1nai(x, y)si
si=wi, I=x,ywi(x, y)I(x, y).
s(t)=v(t)z(t).
v(t)=f(ϕ(t)*u(t))=fτϕ(τ)u(t-τ),
kurt[s(t)]=E{[s(t)]4}-3(E{[s(t)]2})2=E{[v(t)]4[z(t)]4}-3(E{[v(t)]2[z(t)]2})2=3[E{[v(t)]4}-3(E{[v(t)]2})2],
I(x, y, t)=i=1nai(x, y)si(t),
log p(s(1),, s(T))t=0TG(b(t)),
b(t)=ϕ(t)*[s(t)]2=τϕ(τ)[s(t-τ)]2.
log L(w1,, wn; I(x, y, t))i=1nt=0TG(bi(t)),
bi(t)=τϕ(τ)wi,It-τ2=τϕ(τ)x,ywi(x, y)I(x, y, t-τ)2
G(b)=-αb+β,
vi(t)=fjh(i, j)[ϕ(t)*uj(t)],
si(t)=vi(t)zi(t).
I(x, y, t)=i=1nai(x, y)si(t).
bi(t)=τj=1nh(i, j)ϕ(τ)wj, It-τ2.
log L(w1,, wn; I(x, y, t))t=0Ti=1nG(bi(t)).
bik=j=1nh(i, j)t=0Tϕ(T/2-t)wj, Ikt2,
wj(x, y)  wj(x, y)+μk=1Kt=0TIk(x, y, t)i=1nh(i, j)wj, Iktg(bik),
W  (WWT)-1/2W.
I(x, y, t)=i=1nτai(x, y, τ)si(t-τ).
si(k, 1)=x,yτ=1Twi(x, y, τ)Ik(x, y, τ),
si(k, 2)=x,yτ=1Twi(x, y, τ)Ik(x, y, T+τ),
log L(w1,, wn; I(x, y, t))
=k=1Ki=1nGj=1nh(i, j){[sj(k, 1)]2+[sj(k, 2)2]}.
 
p(s(t), u(t); t=1,, T)=tpss(t)v(t)pu(u(t))v(v),
12πTexp-12t[s(t)]2τϕ(τ)u(t-τ)
×tpu(u(t))τϕ(τ)u(t-τ)1/2du(1)du(T),
 
12πTexp-12tu(t)t{[s(t)]2ϕ(t-t)}
×tpu(u(t))τϕ(τ)u(t-τ)1/2du(1)du(T).
 
τϕ(τ)u(t-τ)1/2ϕ(0)u(t).
p(s(t); t=1,, T)texpGτϕ(τ)[s(t-τ)]2,
G(y)=log12πexp-12 uypu(u)ϕ(0)udu.
G0(y)=-log(1+y)+12log π2ϕ(0).

Metrics