Abstract

Self-organization and learning is a distinctive feature of neural nets and processors that sets them apart from conventional approaches to signal processing. It leads to self-programmability which alleviates the problem of programming complexity in artificial neural nets. In this paper architectures for partitioning an optoelectronic analog of a neural net into distinct layers with prescribed interconnectivity pattern to enable stochastic learning by simulated annealing in the context of a Boltzmann machine are presented. Stochastic learning is of interest because of its relevance to the role of noise in biological neural nets. Practical considerations and methodologies for appreciably accelerating stochastic learning in such a multilayered net are described. These include the use of parallel optical computing of the global energy of the net, the use of fast nonvolatile programmable spatial light modulators to realize fast plasticity, optical generation of random number arrays, and an adaptive noisy thresholding scheme that also makes stochastic learning more biologically plausible. The findings reported predict optoelectronic chips that can be used in the realization of optical learning machines.

© 1987 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. B. Widrow, M. E. Hoff, “Adaptive Switching Circuits,” in WESCON Convention Board, Part 4 (1960), pp. 96–104;in Self-Organizing Systems, M. C. Yovitz et al., Eds. (Spartan, Washington, DC, 1962).
  2. K. Nakano, “Associatron—a Model of Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-2, 380 (1972).
    [CrossRef]
  3. T. Kohonen, “Correlation Matrix Memories,” IEEE Trans. Comput. C-21, 353 (1972).
    [CrossRef]
  4. T. Kohonen, Associative Memory (Springer-Verlag, Heidelberg, 1978);Self-Organization and Associative Memory (Springer-Verlag, New York, 1984).
  5. D. J. Willshaw, “A Simple Network Capable of Inductive Generalization,” Proc. R. Soc. London 182, 233 (1972).
    [CrossRef]
  6. J. A. Anderson, J. W. Silverstein, S. A. Ritz, R. S. Jones, “Distinctive Features, Categorial Perception, and Probability Learning: Some Applications of a Neural Model,” Physiol. Rev. 34, 413 (1977).
  7. J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982);“Neurons with Graded Response Have Collective Computational Properties Like Those of Two-State Neurons,” Proc. Natl. Acad. Sci. U.S.A. 81, 3088, (1984).
    [CrossRef] [PubMed]
  8. S. Grossberg, Studies of Mind and Brain (Reidel, Boston, 1982).
    [CrossRef]
  9. A. C. Sanderson, Y. Y. Zeevi, Eds., Special Issue on Neural and Sensory Information Processing,” IEEE Trans. Syst. Man Cybern. SMC-13 (1983).
    [CrossRef]
  10. D. Psaltis, N. Farhat, “A New Approach to Optical Information Processing Based on the Hopfield Model,” in Digest of the Thirteenth Congress of the International Commission on Optics (ICO-13), Sapporo, Japan (1984), p. 24.
  11. D. Psaltis, N. Farhat, “Optical Information Processing Based on an Associative-Memory Model of Neural Nets with Thresholding and Feedback,” Opt. Lett. 10, 98 (1985).
    [CrossRef] [PubMed]
  12. N. H. Farhat, D. Psaltis, A. Prata, E. Paek, “Optical Implementation of the Hopfield Model,” Appl. Opt. 24, 1469 (1985).
    [CrossRef] [PubMed]
  13. N. H. Farhat, D. Psaltis, “Architectures for Optical Implementation of 2-D Content Addressable Memories,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT3.
  14. K. S. Lee, N. H. Farhat, “Content Addressable Memory with Smooth Transition and Adaptive Thresholding,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WJ35.
  15. D. Psaltis, E. Paek, J. Hong, “Acoustooptic Implementation of Neural Network Models,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT6.
  16. J. D. Condon, “Optical Window Addressable Memory,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper.
  17. A. D. Fisher, C. L. Giles, J. N. Lee, “Associative Processor Architectures for Optical Computing,” J. Opt. Soc. Am. A 1, 1337 (1984);“An Adaptive, Associative Optical Computing Element,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), paper WB4.
  18. A. D. Fisher, C. L. Giles, “Optical Adaptive Associative Computer Architectures,” in Proceedings, IEEE COPCOM Spring Meeting, IEEE catalog no. CH2135-2/85/0000/0342 (1985), p. 342.
  19. A. D. Fisher, “Implementation of Adaptive Associative Optical Computing Elements,” Proc. Soc. Photo-Opt. Instrum. Eng. 625, 196 (1986).
  20. B. H. Soffer, G. J. Dunning, Y. Owechko, E. Marom, “Associative Holographic Memory with Feedback Using Phase-Conjugate Mirrors,” Opt. Lett. 11, 118 (1986).
    [CrossRef] [PubMed]
  21. D. Z. Anderson, “Coherent Optical Eigenstate Memory,” Opt.Lett. 11, 56 (1986).
    [CrossRef] [PubMed]
  22. A. Yariv, S-K. Kwong, “Associative Memories Based on Message-Bearing Optical Modes in Phase-Conjugate Resonators,” Opt. Lett. 11, 186 (1986).
    [CrossRef] [PubMed]
  23. N. Farhat, S. Miyahara, K. S. Lee, “Optical Implementation of 2-D Neural Nets and Their Application in Recognition of Radar Targets,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 146.
  24. N. Farhat, D. Psaltis, “Optical Implementation of Associative Memory Based on Models of Neural Networks,” in Optical Signal Processing, J. L. Horner, Ed. (Academic, New York, 1987), p. 129, in press.
  25. B. Kosko, C. Guest, “Optical Bidirectional Associative Memory,” Proc. Soc. Photo-Opt. Instrum. Eng.758, (1987), in press.
  26. N. Metropolis, A. M. Rosenbluth, M. N. Rosenbluth, A. H. Teller, “Equations of State Calculations by Fast Computing Machines,” J. Chem. Phys. 21, 1087 (1953).
    [CrossRef]
  27. S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, “Optimization by Simulated Annealing,” Science 220, 671 (1983).
    [CrossRef] [PubMed]
  28. D. H. Ackley, G. E. Hinton, T. J. Seinowski, “A Learning Algorithm for Boltzmann Machines,” Cognitive Sci. 1, 147 (1985).
  29. T. J. Sejnowski, C. R. Rosenberg, “NETtalk: A Parallel Network That Learns to Read Loud,” Johns Hopkins U., Electrical Engineering & Computer Science Technical Report JHU/EECS-96/01 (1986).
  30. D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learning Internal Representations by Error Propagation,” in Parallel Distributed Processing, D. F. Rumelhart, J. L. McClelland, Eds. (MIT Press, Cambridge, MA, 1986), Vol. 1, p. 318.
  31. S. Miyahara, “Automated Radar Target Recognition Based on Models of Neural Networks,” U. Pennsylvania Dissertation (1987).
  32. R. J. McEliece, E. C. Posner, E. R. Rodemich, S. S. Venkatesh, “The Capacity of the Hopfield Associative Memory,” IEEE Trans. Inf. Theory IT-33, 461, March1987.
    [CrossRef]
  33. G. B. Ermentrout, J. D. Cowan, “Large Scale Spatially Organized Activity in Neural Nets,” SIAM (Soc. Ind. Appl. Math.) J. Appl. Math. 38, 1 (1980).
    [CrossRef]
  34. N. H. Farhat, “Architectures for Opto-Electronic Analogs of Self-Organizing Neural Networks,” Opt. Lett. 12, 448 (1987), accepted for publication.
    [CrossRef] [PubMed]
  35. G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for Self-Organizing Neural Pattern Recognition Machines,” Comput. Vision Graphics Image Process. 37, 54 (1987).
    [CrossRef]
  36. M. L. Minsky, S. Papert, Perceptrons (MIT Press, Cambridge, MA, 1969).
  37. F. Rosenblatt, Principles of Neuro-Dynamics: Perceptions and the Theory of Brain Mechanisms (Spartan Books, Washington, DC, 1962).
  38. M. A. Cohen, S. Grossberg, “Absolute Stability of Global Pattern Formation and Parallel Memory Storage by Competitive Neural Networks,” IEEE Trans. Syst. Man Cybern. SMC-13, 815 (1983).
    [CrossRef]
  39. S. Kirkpatrick, IBM Watson Research Center; private communication (1987).
  40. A. J. Ticknor, H. H. Barrett, R. L. Easton, “Optical Boltzmann Machines,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper PD3.
  41. G. M. Morris, “Optical Computing by Monte Carlo Methods,” Opt. Eng. 24, 86 (1985).
    [CrossRef]
  42. J. Marron, A. J. Martino, G. M. Morris, “Generation of Random Arrays Using Clipped Laser Speckle,” Appl. Opt. 25, 26 (1986).
    [CrossRef] [PubMed]
  43. F. Devos, P. Garda, P. Chavel, “Optical Generation of Random-Number Arrays for On-Chip Massively Parallel Monte Carlo Cellular Processors,” Opt. Lett. 12, 152 (1987).
    [CrossRef] [PubMed]
  44. Y. Tsuchiya, E. Inuzuka, T. Kurono, M. Hosada, “Photon Counting Imaging and Its Application,” Advances in Electronics and Electron Physics, B. L. Morgan, Ed. 64A, 21 (Academic Press, London, 1988).
  45. J. Alspector, R. B. Allen, “A Neuromorphic VLSI Learning System,” in Advanced Research in VLSI, Paul Losleben, Ed. (MIT Press, Cambridge, MA, 1987), to be published.
  46. W. E. Ross, D. Psaltis, R. H. Anderson, “Two-Dimensional Magneto-Optic Spatial Light Modulator for Signal Processing,” Opt. Eng. 22, 485 (1983).
    [CrossRef]
  47. R. M. Karp, “Combinatorics, Complexity and Randomness,” Commun. ACM 29, 98 (Feb.1986).
    [CrossRef]
  48. M. Takeda, J. W. Goodman, “Neural Networks for Computation: Number Representations and Programming Complexity,” Appl. Opt. 25, 3033 (1986).
    [CrossRef] [PubMed]
  49. D. G. Bounds, “Numerical Simulations of Boltzmann Machines,” in Neural Networks For Computing, J. S. Denker, Ed., Vol. 151, (American Institute of Physics, New York, 1986), p. 59.

1987

R. J. McEliece, E. C. Posner, E. R. Rodemich, S. S. Venkatesh, “The Capacity of the Hopfield Associative Memory,” IEEE Trans. Inf. Theory IT-33, 461, March1987.
[CrossRef]

N. H. Farhat, “Architectures for Opto-Electronic Analogs of Self-Organizing Neural Networks,” Opt. Lett. 12, 448 (1987), accepted for publication.
[CrossRef] [PubMed]

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for Self-Organizing Neural Pattern Recognition Machines,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[CrossRef]

F. Devos, P. Garda, P. Chavel, “Optical Generation of Random-Number Arrays for On-Chip Massively Parallel Monte Carlo Cellular Processors,” Opt. Lett. 12, 152 (1987).
[CrossRef] [PubMed]

1986

1985

A. D. Fisher, C. L. Giles, “Optical Adaptive Associative Computer Architectures,” in Proceedings, IEEE COPCOM Spring Meeting, IEEE catalog no. CH2135-2/85/0000/0342 (1985), p. 342.

D. Psaltis, N. Farhat, “Optical Information Processing Based on an Associative-Memory Model of Neural Nets with Thresholding and Feedback,” Opt. Lett. 10, 98 (1985).
[CrossRef] [PubMed]

N. H. Farhat, D. Psaltis, A. Prata, E. Paek, “Optical Implementation of the Hopfield Model,” Appl. Opt. 24, 1469 (1985).
[CrossRef] [PubMed]

D. H. Ackley, G. E. Hinton, T. J. Seinowski, “A Learning Algorithm for Boltzmann Machines,” Cognitive Sci. 1, 147 (1985).

G. M. Morris, “Optical Computing by Monte Carlo Methods,” Opt. Eng. 24, 86 (1985).
[CrossRef]

1984

A. D. Fisher, C. L. Giles, J. N. Lee, “Associative Processor Architectures for Optical Computing,” J. Opt. Soc. Am. A 1, 1337 (1984);“An Adaptive, Associative Optical Computing Element,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), paper WB4.

1983

A. C. Sanderson, Y. Y. Zeevi, Eds., Special Issue on Neural and Sensory Information Processing,” IEEE Trans. Syst. Man Cybern. SMC-13 (1983).
[CrossRef]

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, “Optimization by Simulated Annealing,” Science 220, 671 (1983).
[CrossRef] [PubMed]

M. A. Cohen, S. Grossberg, “Absolute Stability of Global Pattern Formation and Parallel Memory Storage by Competitive Neural Networks,” IEEE Trans. Syst. Man Cybern. SMC-13, 815 (1983).
[CrossRef]

W. E. Ross, D. Psaltis, R. H. Anderson, “Two-Dimensional Magneto-Optic Spatial Light Modulator for Signal Processing,” Opt. Eng. 22, 485 (1983).
[CrossRef]

1982

J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982);“Neurons with Graded Response Have Collective Computational Properties Like Those of Two-State Neurons,” Proc. Natl. Acad. Sci. U.S.A. 81, 3088, (1984).
[CrossRef] [PubMed]

1980

G. B. Ermentrout, J. D. Cowan, “Large Scale Spatially Organized Activity in Neural Nets,” SIAM (Soc. Ind. Appl. Math.) J. Appl. Math. 38, 1 (1980).
[CrossRef]

1977

J. A. Anderson, J. W. Silverstein, S. A. Ritz, R. S. Jones, “Distinctive Features, Categorial Perception, and Probability Learning: Some Applications of a Neural Model,” Physiol. Rev. 34, 413 (1977).

1972

K. Nakano, “Associatron—a Model of Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-2, 380 (1972).
[CrossRef]

T. Kohonen, “Correlation Matrix Memories,” IEEE Trans. Comput. C-21, 353 (1972).
[CrossRef]

D. J. Willshaw, “A Simple Network Capable of Inductive Generalization,” Proc. R. Soc. London 182, 233 (1972).
[CrossRef]

1960

B. Widrow, M. E. Hoff, “Adaptive Switching Circuits,” in WESCON Convention Board, Part 4 (1960), pp. 96–104;in Self-Organizing Systems, M. C. Yovitz et al., Eds. (Spartan, Washington, DC, 1962).

1953

N. Metropolis, A. M. Rosenbluth, M. N. Rosenbluth, A. H. Teller, “Equations of State Calculations by Fast Computing Machines,” J. Chem. Phys. 21, 1087 (1953).
[CrossRef]

Ackley, D. H.

D. H. Ackley, G. E. Hinton, T. J. Seinowski, “A Learning Algorithm for Boltzmann Machines,” Cognitive Sci. 1, 147 (1985).

Allen, R. B.

J. Alspector, R. B. Allen, “A Neuromorphic VLSI Learning System,” in Advanced Research in VLSI, Paul Losleben, Ed. (MIT Press, Cambridge, MA, 1987), to be published.

Alspector, J.

J. Alspector, R. B. Allen, “A Neuromorphic VLSI Learning System,” in Advanced Research in VLSI, Paul Losleben, Ed. (MIT Press, Cambridge, MA, 1987), to be published.

Anderson, D. Z.

D. Z. Anderson, “Coherent Optical Eigenstate Memory,” Opt.Lett. 11, 56 (1986).
[CrossRef] [PubMed]

Anderson, J. A.

J. A. Anderson, J. W. Silverstein, S. A. Ritz, R. S. Jones, “Distinctive Features, Categorial Perception, and Probability Learning: Some Applications of a Neural Model,” Physiol. Rev. 34, 413 (1977).

Anderson, R. H.

W. E. Ross, D. Psaltis, R. H. Anderson, “Two-Dimensional Magneto-Optic Spatial Light Modulator for Signal Processing,” Opt. Eng. 22, 485 (1983).
[CrossRef]

Barrett, H. H.

A. J. Ticknor, H. H. Barrett, R. L. Easton, “Optical Boltzmann Machines,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper PD3.

Bounds, D. G.

D. G. Bounds, “Numerical Simulations of Boltzmann Machines,” in Neural Networks For Computing, J. S. Denker, Ed., Vol. 151, (American Institute of Physics, New York, 1986), p. 59.

Carpenter, G. A.

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for Self-Organizing Neural Pattern Recognition Machines,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[CrossRef]

Chavel, P.

Cohen, M. A.

M. A. Cohen, S. Grossberg, “Absolute Stability of Global Pattern Formation and Parallel Memory Storage by Competitive Neural Networks,” IEEE Trans. Syst. Man Cybern. SMC-13, 815 (1983).
[CrossRef]

Condon, J. D.

J. D. Condon, “Optical Window Addressable Memory,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper.

Cowan, J. D.

G. B. Ermentrout, J. D. Cowan, “Large Scale Spatially Organized Activity in Neural Nets,” SIAM (Soc. Ind. Appl. Math.) J. Appl. Math. 38, 1 (1980).
[CrossRef]

Devos, F.

Dunning, G. J.

Easton, R. L.

A. J. Ticknor, H. H. Barrett, R. L. Easton, “Optical Boltzmann Machines,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper PD3.

Ermentrout, G. B.

G. B. Ermentrout, J. D. Cowan, “Large Scale Spatially Organized Activity in Neural Nets,” SIAM (Soc. Ind. Appl. Math.) J. Appl. Math. 38, 1 (1980).
[CrossRef]

Farhat, N.

D. Psaltis, N. Farhat, “Optical Information Processing Based on an Associative-Memory Model of Neural Nets with Thresholding and Feedback,” Opt. Lett. 10, 98 (1985).
[CrossRef] [PubMed]

N. Farhat, D. Psaltis, “Optical Implementation of Associative Memory Based on Models of Neural Networks,” in Optical Signal Processing, J. L. Horner, Ed. (Academic, New York, 1987), p. 129, in press.

N. Farhat, S. Miyahara, K. S. Lee, “Optical Implementation of 2-D Neural Nets and Their Application in Recognition of Radar Targets,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 146.

D. Psaltis, N. Farhat, “A New Approach to Optical Information Processing Based on the Hopfield Model,” in Digest of the Thirteenth Congress of the International Commission on Optics (ICO-13), Sapporo, Japan (1984), p. 24.

Farhat, N. H.

N. H. Farhat, “Architectures for Opto-Electronic Analogs of Self-Organizing Neural Networks,” Opt. Lett. 12, 448 (1987), accepted for publication.
[CrossRef] [PubMed]

N. H. Farhat, D. Psaltis, A. Prata, E. Paek, “Optical Implementation of the Hopfield Model,” Appl. Opt. 24, 1469 (1985).
[CrossRef] [PubMed]

N. H. Farhat, D. Psaltis, “Architectures for Optical Implementation of 2-D Content Addressable Memories,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT3.

K. S. Lee, N. H. Farhat, “Content Addressable Memory with Smooth Transition and Adaptive Thresholding,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WJ35.

Fisher, A. D.

A. D. Fisher, “Implementation of Adaptive Associative Optical Computing Elements,” Proc. Soc. Photo-Opt. Instrum. Eng. 625, 196 (1986).

A. D. Fisher, C. L. Giles, “Optical Adaptive Associative Computer Architectures,” in Proceedings, IEEE COPCOM Spring Meeting, IEEE catalog no. CH2135-2/85/0000/0342 (1985), p. 342.

A. D. Fisher, C. L. Giles, J. N. Lee, “Associative Processor Architectures for Optical Computing,” J. Opt. Soc. Am. A 1, 1337 (1984);“An Adaptive, Associative Optical Computing Element,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), paper WB4.

Garda, P.

Gelatt, C. D.

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, “Optimization by Simulated Annealing,” Science 220, 671 (1983).
[CrossRef] [PubMed]

Giles, C. L.

A. D. Fisher, C. L. Giles, “Optical Adaptive Associative Computer Architectures,” in Proceedings, IEEE COPCOM Spring Meeting, IEEE catalog no. CH2135-2/85/0000/0342 (1985), p. 342.

A. D. Fisher, C. L. Giles, J. N. Lee, “Associative Processor Architectures for Optical Computing,” J. Opt. Soc. Am. A 1, 1337 (1984);“An Adaptive, Associative Optical Computing Element,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), paper WB4.

Goodman, J. W.

Grossberg, S.

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for Self-Organizing Neural Pattern Recognition Machines,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[CrossRef]

M. A. Cohen, S. Grossberg, “Absolute Stability of Global Pattern Formation and Parallel Memory Storage by Competitive Neural Networks,” IEEE Trans. Syst. Man Cybern. SMC-13, 815 (1983).
[CrossRef]

S. Grossberg, Studies of Mind and Brain (Reidel, Boston, 1982).
[CrossRef]

Guest, C.

B. Kosko, C. Guest, “Optical Bidirectional Associative Memory,” Proc. Soc. Photo-Opt. Instrum. Eng.758, (1987), in press.

Hinton, G. E.

D. H. Ackley, G. E. Hinton, T. J. Seinowski, “A Learning Algorithm for Boltzmann Machines,” Cognitive Sci. 1, 147 (1985).

D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learning Internal Representations by Error Propagation,” in Parallel Distributed Processing, D. F. Rumelhart, J. L. McClelland, Eds. (MIT Press, Cambridge, MA, 1986), Vol. 1, p. 318.

Hoff, M. E.

B. Widrow, M. E. Hoff, “Adaptive Switching Circuits,” in WESCON Convention Board, Part 4 (1960), pp. 96–104;in Self-Organizing Systems, M. C. Yovitz et al., Eds. (Spartan, Washington, DC, 1962).

Hong, J.

D. Psaltis, E. Paek, J. Hong, “Acoustooptic Implementation of Neural Network Models,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT6.

Hopfield, J. J.

J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982);“Neurons with Graded Response Have Collective Computational Properties Like Those of Two-State Neurons,” Proc. Natl. Acad. Sci. U.S.A. 81, 3088, (1984).
[CrossRef] [PubMed]

Hosada, M.

Y. Tsuchiya, E. Inuzuka, T. Kurono, M. Hosada, “Photon Counting Imaging and Its Application,” Advances in Electronics and Electron Physics, B. L. Morgan, Ed. 64A, 21 (Academic Press, London, 1988).

Inuzuka, E.

Y. Tsuchiya, E. Inuzuka, T. Kurono, M. Hosada, “Photon Counting Imaging and Its Application,” Advances in Electronics and Electron Physics, B. L. Morgan, Ed. 64A, 21 (Academic Press, London, 1988).

Jones, R. S.

J. A. Anderson, J. W. Silverstein, S. A. Ritz, R. S. Jones, “Distinctive Features, Categorial Perception, and Probability Learning: Some Applications of a Neural Model,” Physiol. Rev. 34, 413 (1977).

Karp, R. M.

R. M. Karp, “Combinatorics, Complexity and Randomness,” Commun. ACM 29, 98 (Feb.1986).
[CrossRef]

Kirkpatrick, S.

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, “Optimization by Simulated Annealing,” Science 220, 671 (1983).
[CrossRef] [PubMed]

S. Kirkpatrick, IBM Watson Research Center; private communication (1987).

Kohonen, T.

T. Kohonen, “Correlation Matrix Memories,” IEEE Trans. Comput. C-21, 353 (1972).
[CrossRef]

T. Kohonen, Associative Memory (Springer-Verlag, Heidelberg, 1978);Self-Organization and Associative Memory (Springer-Verlag, New York, 1984).

Kosko, B.

B. Kosko, C. Guest, “Optical Bidirectional Associative Memory,” Proc. Soc. Photo-Opt. Instrum. Eng.758, (1987), in press.

Kurono, T.

Y. Tsuchiya, E. Inuzuka, T. Kurono, M. Hosada, “Photon Counting Imaging and Its Application,” Advances in Electronics and Electron Physics, B. L. Morgan, Ed. 64A, 21 (Academic Press, London, 1988).

Kwong, S-K.

Lee, J. N.

A. D. Fisher, C. L. Giles, J. N. Lee, “Associative Processor Architectures for Optical Computing,” J. Opt. Soc. Am. A 1, 1337 (1984);“An Adaptive, Associative Optical Computing Element,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), paper WB4.

Lee, K. S.

K. S. Lee, N. H. Farhat, “Content Addressable Memory with Smooth Transition and Adaptive Thresholding,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WJ35.

N. Farhat, S. Miyahara, K. S. Lee, “Optical Implementation of 2-D Neural Nets and Their Application in Recognition of Radar Targets,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 146.

Marom, E.

Marron, J.

Martino, A. J.

McEliece, R. J.

R. J. McEliece, E. C. Posner, E. R. Rodemich, S. S. Venkatesh, “The Capacity of the Hopfield Associative Memory,” IEEE Trans. Inf. Theory IT-33, 461, March1987.
[CrossRef]

Metropolis, N.

N. Metropolis, A. M. Rosenbluth, M. N. Rosenbluth, A. H. Teller, “Equations of State Calculations by Fast Computing Machines,” J. Chem. Phys. 21, 1087 (1953).
[CrossRef]

Minsky, M. L.

M. L. Minsky, S. Papert, Perceptrons (MIT Press, Cambridge, MA, 1969).

Miyahara, S.

N. Farhat, S. Miyahara, K. S. Lee, “Optical Implementation of 2-D Neural Nets and Their Application in Recognition of Radar Targets,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 146.

S. Miyahara, “Automated Radar Target Recognition Based on Models of Neural Networks,” U. Pennsylvania Dissertation (1987).

Morris, G. M.

Nakano, K.

K. Nakano, “Associatron—a Model of Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-2, 380 (1972).
[CrossRef]

Owechko, Y.

Paek, E.

N. H. Farhat, D. Psaltis, A. Prata, E. Paek, “Optical Implementation of the Hopfield Model,” Appl. Opt. 24, 1469 (1985).
[CrossRef] [PubMed]

D. Psaltis, E. Paek, J. Hong, “Acoustooptic Implementation of Neural Network Models,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT6.

Papert, S.

M. L. Minsky, S. Papert, Perceptrons (MIT Press, Cambridge, MA, 1969).

Posner, E. C.

R. J. McEliece, E. C. Posner, E. R. Rodemich, S. S. Venkatesh, “The Capacity of the Hopfield Associative Memory,” IEEE Trans. Inf. Theory IT-33, 461, March1987.
[CrossRef]

Prata, A.

Psaltis, D.

N. H. Farhat, D. Psaltis, A. Prata, E. Paek, “Optical Implementation of the Hopfield Model,” Appl. Opt. 24, 1469 (1985).
[CrossRef] [PubMed]

D. Psaltis, N. Farhat, “Optical Information Processing Based on an Associative-Memory Model of Neural Nets with Thresholding and Feedback,” Opt. Lett. 10, 98 (1985).
[CrossRef] [PubMed]

W. E. Ross, D. Psaltis, R. H. Anderson, “Two-Dimensional Magneto-Optic Spatial Light Modulator for Signal Processing,” Opt. Eng. 22, 485 (1983).
[CrossRef]

D. Psaltis, N. Farhat, “A New Approach to Optical Information Processing Based on the Hopfield Model,” in Digest of the Thirteenth Congress of the International Commission on Optics (ICO-13), Sapporo, Japan (1984), p. 24.

N. H. Farhat, D. Psaltis, “Architectures for Optical Implementation of 2-D Content Addressable Memories,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT3.

N. Farhat, D. Psaltis, “Optical Implementation of Associative Memory Based on Models of Neural Networks,” in Optical Signal Processing, J. L. Horner, Ed. (Academic, New York, 1987), p. 129, in press.

D. Psaltis, E. Paek, J. Hong, “Acoustooptic Implementation of Neural Network Models,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT6.

Ritz, S. A.

J. A. Anderson, J. W. Silverstein, S. A. Ritz, R. S. Jones, “Distinctive Features, Categorial Perception, and Probability Learning: Some Applications of a Neural Model,” Physiol. Rev. 34, 413 (1977).

Rodemich, E. R.

R. J. McEliece, E. C. Posner, E. R. Rodemich, S. S. Venkatesh, “The Capacity of the Hopfield Associative Memory,” IEEE Trans. Inf. Theory IT-33, 461, March1987.
[CrossRef]

Rosenberg, C. R.

T. J. Sejnowski, C. R. Rosenberg, “NETtalk: A Parallel Network That Learns to Read Loud,” Johns Hopkins U., Electrical Engineering & Computer Science Technical Report JHU/EECS-96/01 (1986).

Rosenblatt, F.

F. Rosenblatt, Principles of Neuro-Dynamics: Perceptions and the Theory of Brain Mechanisms (Spartan Books, Washington, DC, 1962).

Rosenbluth, A. M.

N. Metropolis, A. M. Rosenbluth, M. N. Rosenbluth, A. H. Teller, “Equations of State Calculations by Fast Computing Machines,” J. Chem. Phys. 21, 1087 (1953).
[CrossRef]

Rosenbluth, M. N.

N. Metropolis, A. M. Rosenbluth, M. N. Rosenbluth, A. H. Teller, “Equations of State Calculations by Fast Computing Machines,” J. Chem. Phys. 21, 1087 (1953).
[CrossRef]

Ross, W. E.

W. E. Ross, D. Psaltis, R. H. Anderson, “Two-Dimensional Magneto-Optic Spatial Light Modulator for Signal Processing,” Opt. Eng. 22, 485 (1983).
[CrossRef]

Rumelhart, D. E.

D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learning Internal Representations by Error Propagation,” in Parallel Distributed Processing, D. F. Rumelhart, J. L. McClelland, Eds. (MIT Press, Cambridge, MA, 1986), Vol. 1, p. 318.

Seinowski, T. J.

D. H. Ackley, G. E. Hinton, T. J. Seinowski, “A Learning Algorithm for Boltzmann Machines,” Cognitive Sci. 1, 147 (1985).

Sejnowski, T. J.

T. J. Sejnowski, C. R. Rosenberg, “NETtalk: A Parallel Network That Learns to Read Loud,” Johns Hopkins U., Electrical Engineering & Computer Science Technical Report JHU/EECS-96/01 (1986).

Silverstein, J. W.

J. A. Anderson, J. W. Silverstein, S. A. Ritz, R. S. Jones, “Distinctive Features, Categorial Perception, and Probability Learning: Some Applications of a Neural Model,” Physiol. Rev. 34, 413 (1977).

Soffer, B. H.

Takeda, M.

Teller, A. H.

N. Metropolis, A. M. Rosenbluth, M. N. Rosenbluth, A. H. Teller, “Equations of State Calculations by Fast Computing Machines,” J. Chem. Phys. 21, 1087 (1953).
[CrossRef]

Ticknor, A. J.

A. J. Ticknor, H. H. Barrett, R. L. Easton, “Optical Boltzmann Machines,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper PD3.

Tsuchiya, Y.

Y. Tsuchiya, E. Inuzuka, T. Kurono, M. Hosada, “Photon Counting Imaging and Its Application,” Advances in Electronics and Electron Physics, B. L. Morgan, Ed. 64A, 21 (Academic Press, London, 1988).

Vecchi, M. P.

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, “Optimization by Simulated Annealing,” Science 220, 671 (1983).
[CrossRef] [PubMed]

Venkatesh, S. S.

R. J. McEliece, E. C. Posner, E. R. Rodemich, S. S. Venkatesh, “The Capacity of the Hopfield Associative Memory,” IEEE Trans. Inf. Theory IT-33, 461, March1987.
[CrossRef]

Widrow, B.

B. Widrow, M. E. Hoff, “Adaptive Switching Circuits,” in WESCON Convention Board, Part 4 (1960), pp. 96–104;in Self-Organizing Systems, M. C. Yovitz et al., Eds. (Spartan, Washington, DC, 1962).

Williams, R. J.

D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learning Internal Representations by Error Propagation,” in Parallel Distributed Processing, D. F. Rumelhart, J. L. McClelland, Eds. (MIT Press, Cambridge, MA, 1986), Vol. 1, p. 318.

Willshaw, D. J.

D. J. Willshaw, “A Simple Network Capable of Inductive Generalization,” Proc. R. Soc. London 182, 233 (1972).
[CrossRef]

Yariv, A.

Appl. Opt.

Cognitive Sci.

D. H. Ackley, G. E. Hinton, T. J. Seinowski, “A Learning Algorithm for Boltzmann Machines,” Cognitive Sci. 1, 147 (1985).

Commun. ACM

R. M. Karp, “Combinatorics, Complexity and Randomness,” Commun. ACM 29, 98 (Feb.1986).
[CrossRef]

Comput. Vision Graphics Image Process.

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for Self-Organizing Neural Pattern Recognition Machines,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[CrossRef]

IEEE Trans. Comput.

T. Kohonen, “Correlation Matrix Memories,” IEEE Trans. Comput. C-21, 353 (1972).
[CrossRef]

IEEE Trans. Inf. Theory IT-33

R. J. McEliece, E. C. Posner, E. R. Rodemich, S. S. Venkatesh, “The Capacity of the Hopfield Associative Memory,” IEEE Trans. Inf. Theory IT-33, 461, March1987.
[CrossRef]

IEEE Trans. Syst. Man Cybern.

M. A. Cohen, S. Grossberg, “Absolute Stability of Global Pattern Formation and Parallel Memory Storage by Competitive Neural Networks,” IEEE Trans. Syst. Man Cybern. SMC-13, 815 (1983).
[CrossRef]

K. Nakano, “Associatron—a Model of Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-2, 380 (1972).
[CrossRef]

A. C. Sanderson, Y. Y. Zeevi, Eds., Special Issue on Neural and Sensory Information Processing,” IEEE Trans. Syst. Man Cybern. SMC-13 (1983).
[CrossRef]

J. Chem. Phys.

N. Metropolis, A. M. Rosenbluth, M. N. Rosenbluth, A. H. Teller, “Equations of State Calculations by Fast Computing Machines,” J. Chem. Phys. 21, 1087 (1953).
[CrossRef]

J. Opt. Soc. Am. A

A. D. Fisher, C. L. Giles, J. N. Lee, “Associative Processor Architectures for Optical Computing,” J. Opt. Soc. Am. A 1, 1337 (1984);“An Adaptive, Associative Optical Computing Element,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), paper WB4.

Opt. Eng.

G. M. Morris, “Optical Computing by Monte Carlo Methods,” Opt. Eng. 24, 86 (1985).
[CrossRef]

W. E. Ross, D. Psaltis, R. H. Anderson, “Two-Dimensional Magneto-Optic Spatial Light Modulator for Signal Processing,” Opt. Eng. 22, 485 (1983).
[CrossRef]

Opt. Lett.

Opt.Lett.

D. Z. Anderson, “Coherent Optical Eigenstate Memory,” Opt.Lett. 11, 56 (1986).
[CrossRef] [PubMed]

Physiol. Rev.

J. A. Anderson, J. W. Silverstein, S. A. Ritz, R. S. Jones, “Distinctive Features, Categorial Perception, and Probability Learning: Some Applications of a Neural Model,” Physiol. Rev. 34, 413 (1977).

Proc. Natl. Acad. Sci. U.S.A.

J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982);“Neurons with Graded Response Have Collective Computational Properties Like Those of Two-State Neurons,” Proc. Natl. Acad. Sci. U.S.A. 81, 3088, (1984).
[CrossRef] [PubMed]

Proc. R. Soc. London

D. J. Willshaw, “A Simple Network Capable of Inductive Generalization,” Proc. R. Soc. London 182, 233 (1972).
[CrossRef]

Proc. Soc. Photo-Opt. Instrum. Eng.

A. D. Fisher, “Implementation of Adaptive Associative Optical Computing Elements,” Proc. Soc. Photo-Opt. Instrum. Eng. 625, 196 (1986).

Proceedings, IEEE COPCOM Spring Meeting

A. D. Fisher, C. L. Giles, “Optical Adaptive Associative Computer Architectures,” in Proceedings, IEEE COPCOM Spring Meeting, IEEE catalog no. CH2135-2/85/0000/0342 (1985), p. 342.

Science

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, “Optimization by Simulated Annealing,” Science 220, 671 (1983).
[CrossRef] [PubMed]

SIAM (Soc. Ind. Appl. Math.) J. Appl. Math.

G. B. Ermentrout, J. D. Cowan, “Large Scale Spatially Organized Activity in Neural Nets,” SIAM (Soc. Ind. Appl. Math.) J. Appl. Math. 38, 1 (1980).
[CrossRef]

WESCON Convention Board

B. Widrow, M. E. Hoff, “Adaptive Switching Circuits,” in WESCON Convention Board, Part 4 (1960), pp. 96–104;in Self-Organizing Systems, M. C. Yovitz et al., Eds. (Spartan, Washington, DC, 1962).

Other

T. Kohonen, Associative Memory (Springer-Verlag, Heidelberg, 1978);Self-Organization and Associative Memory (Springer-Verlag, New York, 1984).

D. Psaltis, N. Farhat, “A New Approach to Optical Information Processing Based on the Hopfield Model,” in Digest of the Thirteenth Congress of the International Commission on Optics (ICO-13), Sapporo, Japan (1984), p. 24.

N. H. Farhat, D. Psaltis, “Architectures for Optical Implementation of 2-D Content Addressable Memories,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT3.

K. S. Lee, N. H. Farhat, “Content Addressable Memory with Smooth Transition and Adaptive Thresholding,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WJ35.

D. Psaltis, E. Paek, J. Hong, “Acoustooptic Implementation of Neural Network Models,” in Technical Digest, Optical Society of America Annual Meeting (Optical Society of America, Washington, DC, 1985), paper WT6.

J. D. Condon, “Optical Window Addressable Memory,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper.

S. Grossberg, Studies of Mind and Brain (Reidel, Boston, 1982).
[CrossRef]

N. Farhat, S. Miyahara, K. S. Lee, “Optical Implementation of 2-D Neural Nets and Their Application in Recognition of Radar Targets,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 146.

N. Farhat, D. Psaltis, “Optical Implementation of Associative Memory Based on Models of Neural Networks,” in Optical Signal Processing, J. L. Horner, Ed. (Academic, New York, 1987), p. 129, in press.

B. Kosko, C. Guest, “Optical Bidirectional Associative Memory,” Proc. Soc. Photo-Opt. Instrum. Eng.758, (1987), in press.

T. J. Sejnowski, C. R. Rosenberg, “NETtalk: A Parallel Network That Learns to Read Loud,” Johns Hopkins U., Electrical Engineering & Computer Science Technical Report JHU/EECS-96/01 (1986).

D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learning Internal Representations by Error Propagation,” in Parallel Distributed Processing, D. F. Rumelhart, J. L. McClelland, Eds. (MIT Press, Cambridge, MA, 1986), Vol. 1, p. 318.

S. Miyahara, “Automated Radar Target Recognition Based on Models of Neural Networks,” U. Pennsylvania Dissertation (1987).

Y. Tsuchiya, E. Inuzuka, T. Kurono, M. Hosada, “Photon Counting Imaging and Its Application,” Advances in Electronics and Electron Physics, B. L. Morgan, Ed. 64A, 21 (Academic Press, London, 1988).

J. Alspector, R. B. Allen, “A Neuromorphic VLSI Learning System,” in Advanced Research in VLSI, Paul Losleben, Ed. (MIT Press, Cambridge, MA, 1987), to be published.

S. Kirkpatrick, IBM Watson Research Center; private communication (1987).

A. J. Ticknor, H. H. Barrett, R. L. Easton, “Optical Boltzmann Machines,” in Technical Digest of Topical Meeting on Optical Computing (Optical Society of America, Washington, DC, 1985), postdeadline paper PD3.

M. L. Minsky, S. Papert, Perceptrons (MIT Press, Cambridge, MA, 1969).

F. Rosenblatt, Principles of Neuro-Dynamics: Perceptions and the Theory of Brain Mechanisms (Spartan Books, Washington, DC, 1962).

D. G. Bounds, “Numerical Simulations of Boltzmann Machines,” in Neural Networks For Computing, J. S. Denker, Ed., Vol. 151, (American Institute of Physics, New York, 1986), p. 59.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (8)

Fig. 1
Fig. 1

Outer-product (distributed) storage and recall scheme.

Fig. 2
Fig. 2

Direct storage and inner-product recall scheme.

Fig. 3
Fig. 3

Reflexive inner-product scheme.

Fig. 4
Fig. 4

Two equivalent neural net analogs: (a) outer-product distributed storage and recall with external feedback; (b) reflexive inner-product direct storage and recall with internal feedback.

Fig. 5
Fig. 5

Concept of nonlinear resonator content addressable memory.

Fig. 6
Fig. 6

Optoelectronic analog of self-organizing neural net partitioned into three layers capable of stochastic self-programming and learning.

Fig. 7
Fig. 7

Two schemes for parallel computing of the global energy in an optoelectronic analog of a multilayered self-organizing net: (a) electronic scheme; (b) optoelectronic scheme.

Fig. 8
Fig. 8

Optoelectronic neural chip.

Equations (9)

Equations on this page are rendered with MathJax. Learn more.

v ( q + 1 i = sgn { j i T ij v ( q ) ij } ,
T ij = m = 1 M v i ( m ) v j ( m ) ,
v ( q + 1 i = sgn { m = 1 M C ( q ) m v i ( m ) } ,
C ( q ) m = j = 1 N v j ( m ) v ( q ) j
P k = exp Δ E k T ,
E = 1 2 i u i v i ,
u i = j i T ij v j θ i + I i
u i = j i T ij v j .
E = 1 2 i [ ( j i T ij + T ij ) v j ] v i = 1 2 i ( j i T ij v j ) v i = 1 2 i u i v i ,

Metrics