Abstract

Most associative memory work has concentrated on autoassociative memories (AAMs). These associative processors provide reduced noise and error correction in their output data. We will consider heteroassociative (HAMs), which are needed to provide decisions on the class of the input data and inferences for subsequent processing. We derive new equations for the storage capacity and noise performance of HAMs, emphasize how they differ from those derived for AAMs, suggest new performance measures to be used, and show how different recollection vector encodings can improve HAM performance.

© 1989 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. T. Kohonen, Self-Organization and Associative Memory (Springer-Verlag, Berlin, 1988).
    [CrossRef]
  2. J. Hong, D. Psaltis, “Storage Capacity of Holographic Associative Memories,” Opt. Lett. 11, 812 (1986).
    [CrossRef] [PubMed]
  3. G. S. Stiles, D. L. Denq, “On the Effect of Noise on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Pattern Anal. Machine Intell. PAMI-7, 358 (1985).
    [CrossRef]
  4. J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982).
    [CrossRef] [PubMed]
  5. Y. Abu-Mostafa, J. St. Jacques, “Information Capacity of the Hopfield Model,” IEEE Trans. Inf. Theory IT-31, 461 (1985).
    [CrossRef]
  6. G. Palm, “On Associative Memory,” Biol. Cybern. 36, 19 (1Feb.1980).
    [CrossRef] [PubMed]
  7. Y. Owechko, G. J. Dunning, E. Marom, B. H. Soffer, “Holographic Associative Memory with Nonlinearities in the Correlation Domain,” Appl. Opt. 26, 1900 (1987).
    [CrossRef] [PubMed]
  8. K. Murakami, T. Aibara, “An Improvement on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-17, 699 (1987).
    [CrossRef]
  9. C. L. Giles, T. Maxwell, “Learning, Invariance, and Generalization in High-Order Neural Networks,” Appl. Opt. 26, 4972 (1987).
    [CrossRef] [PubMed]
  10. C. R. Rao, S. K. Mitra, Generalized Inverse of Matrices and Its Applications (Wiley, New York, 1971).
  11. D. Casasent, B. Telfer, “Distortion-Invariant Associative Memories and Processors,” Proc. Soc. Photo-Opt. Instrum. Eng. 697, 60 (1986).
  12. D. Casasent, B. Telfer, “Associative Memory Synthesis, Performance, Storage Capacity and Updating: New Heteroassociative Memory Results,” Proc. Soc. Photo-Opt. Instrum. Eng. 848, 313 (1987).
  13. A. D. Fisher, W. L. Lippincott, J. N. Lee, “Optical Implementations of Associative Networks with Versatile Adaptive Learning Capabilities,” Appl. Opt. 26, 5039 (1987).
    [CrossRef] [PubMed]
  14. B. Montgomery, B.V.K. Vijaya Kumar, “An Evaluation of the Use of the Hopfield Neural Network Model as a Nearest-Neighbor Algorithm,” Appl. Opt. 25, 3759 (1986).
    [CrossRef] [PubMed]
  15. D. Casasent, “Scene Analysis Research: Optical Pattern Recognition and Artificial Intelligence,” Proc. Soc. Photo-Opt. Instrum. Eng. 634, 439 (1986).
  16. M. J. Little, C. S. Bak, “Enhanced Memory Capacity of a Hopfield Neural Network,” Proc. Soc. Photo-Opt. Instrum. Eng. 698, 150 (1986).
  17. D. Psaltis, C. H. Park, “Nonlinear Discriminant Functions and Associative Memories,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 370.
  18. D. Psaltis, N. Farhat, “Optical Information Processing Based on an Associative-Memory Model of Neural Nets with Thresholding and Feedback,” Opt. Lett. 10, 98 (1985).
    [CrossRef] [PubMed]
  19. J. A. Anderson, “Cognitive and Psychological Computation with Neural Models,” IEEE Trans. Syst. Man Cybern. SMC-13, 799 (1983).
    [CrossRef]
  20. D. Casasent, “Unified Synthetic Discriminant Function Computational Formulation,” Appl. Opt. 23, 1620 (1984).
    [CrossRef] [PubMed]
  21. J. Goodman et al., “Parallel Incoherent Optical Vector-Matrix Multiplier,” Tech. Rep. L-723-1, BMD (Feb.1979).
  22. D. Casasent, J. Jackson, C. Neuman, “Frequency-Multiplexed and Pipelined Iterative Optical Systolic Processors,” Appl. Opt. 22, 115 (1983).
    [CrossRef] [PubMed]
  23. D. Casasent, J. Jackson, “Space and Frequency-Multiplexed Optical Linear Algebra Processor,” Appl. Opt. 25, 2258 (1986).
    [CrossRef] [PubMed]
  24. K. Wagner, D. Psaltis, “A Space Integrating Acousto-Optic Matrix-Matrix Multiplier,” Opt. Commun. 52, 173 (1984).
    [CrossRef]
  25. G. H. Golub, C. Reinsch, “Singular Value Decomposition and Least Squares Solutions,” Numer. Math. 14, 403 (1970).
    [CrossRef]
  26. S. Liebowitz, D. Casasent, “Error-Correction Coding in an Associative Processor,” Appl. Opt. 26, 999 (1987).
    [CrossRef] [PubMed]

1987 (6)

1986 (6)

B. Montgomery, B.V.K. Vijaya Kumar, “An Evaluation of the Use of the Hopfield Neural Network Model as a Nearest-Neighbor Algorithm,” Appl. Opt. 25, 3759 (1986).
[CrossRef] [PubMed]

D. Casasent, “Scene Analysis Research: Optical Pattern Recognition and Artificial Intelligence,” Proc. Soc. Photo-Opt. Instrum. Eng. 634, 439 (1986).

M. J. Little, C. S. Bak, “Enhanced Memory Capacity of a Hopfield Neural Network,” Proc. Soc. Photo-Opt. Instrum. Eng. 698, 150 (1986).

D. Casasent, J. Jackson, “Space and Frequency-Multiplexed Optical Linear Algebra Processor,” Appl. Opt. 25, 2258 (1986).
[CrossRef] [PubMed]

D. Casasent, B. Telfer, “Distortion-Invariant Associative Memories and Processors,” Proc. Soc. Photo-Opt. Instrum. Eng. 697, 60 (1986).

J. Hong, D. Psaltis, “Storage Capacity of Holographic Associative Memories,” Opt. Lett. 11, 812 (1986).
[CrossRef] [PubMed]

1985 (3)

G. S. Stiles, D. L. Denq, “On the Effect of Noise on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Pattern Anal. Machine Intell. PAMI-7, 358 (1985).
[CrossRef]

Y. Abu-Mostafa, J. St. Jacques, “Information Capacity of the Hopfield Model,” IEEE Trans. Inf. Theory IT-31, 461 (1985).
[CrossRef]

D. Psaltis, N. Farhat, “Optical Information Processing Based on an Associative-Memory Model of Neural Nets with Thresholding and Feedback,” Opt. Lett. 10, 98 (1985).
[CrossRef] [PubMed]

1984 (2)

K. Wagner, D. Psaltis, “A Space Integrating Acousto-Optic Matrix-Matrix Multiplier,” Opt. Commun. 52, 173 (1984).
[CrossRef]

D. Casasent, “Unified Synthetic Discriminant Function Computational Formulation,” Appl. Opt. 23, 1620 (1984).
[CrossRef] [PubMed]

1983 (2)

D. Casasent, J. Jackson, C. Neuman, “Frequency-Multiplexed and Pipelined Iterative Optical Systolic Processors,” Appl. Opt. 22, 115 (1983).
[CrossRef] [PubMed]

J. A. Anderson, “Cognitive and Psychological Computation with Neural Models,” IEEE Trans. Syst. Man Cybern. SMC-13, 799 (1983).
[CrossRef]

1982 (1)

J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982).
[CrossRef] [PubMed]

1980 (1)

G. Palm, “On Associative Memory,” Biol. Cybern. 36, 19 (1Feb.1980).
[CrossRef] [PubMed]

1970 (1)

G. H. Golub, C. Reinsch, “Singular Value Decomposition and Least Squares Solutions,” Numer. Math. 14, 403 (1970).
[CrossRef]

Abu-Mostafa, Y.

Y. Abu-Mostafa, J. St. Jacques, “Information Capacity of the Hopfield Model,” IEEE Trans. Inf. Theory IT-31, 461 (1985).
[CrossRef]

Aibara, T.

K. Murakami, T. Aibara, “An Improvement on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-17, 699 (1987).
[CrossRef]

Anderson, J. A.

J. A. Anderson, “Cognitive and Psychological Computation with Neural Models,” IEEE Trans. Syst. Man Cybern. SMC-13, 799 (1983).
[CrossRef]

Bak, C. S.

M. J. Little, C. S. Bak, “Enhanced Memory Capacity of a Hopfield Neural Network,” Proc. Soc. Photo-Opt. Instrum. Eng. 698, 150 (1986).

Casasent, D.

D. Casasent, B. Telfer, “Associative Memory Synthesis, Performance, Storage Capacity and Updating: New Heteroassociative Memory Results,” Proc. Soc. Photo-Opt. Instrum. Eng. 848, 313 (1987).

S. Liebowitz, D. Casasent, “Error-Correction Coding in an Associative Processor,” Appl. Opt. 26, 999 (1987).
[CrossRef] [PubMed]

D. Casasent, J. Jackson, “Space and Frequency-Multiplexed Optical Linear Algebra Processor,” Appl. Opt. 25, 2258 (1986).
[CrossRef] [PubMed]

D. Casasent, B. Telfer, “Distortion-Invariant Associative Memories and Processors,” Proc. Soc. Photo-Opt. Instrum. Eng. 697, 60 (1986).

D. Casasent, “Scene Analysis Research: Optical Pattern Recognition and Artificial Intelligence,” Proc. Soc. Photo-Opt. Instrum. Eng. 634, 439 (1986).

D. Casasent, “Unified Synthetic Discriminant Function Computational Formulation,” Appl. Opt. 23, 1620 (1984).
[CrossRef] [PubMed]

D. Casasent, J. Jackson, C. Neuman, “Frequency-Multiplexed and Pipelined Iterative Optical Systolic Processors,” Appl. Opt. 22, 115 (1983).
[CrossRef] [PubMed]

Denq, D. L.

G. S. Stiles, D. L. Denq, “On the Effect of Noise on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Pattern Anal. Machine Intell. PAMI-7, 358 (1985).
[CrossRef]

Dunning, G. J.

Farhat, N.

Fisher, A. D.

Giles, C. L.

Golub, G. H.

G. H. Golub, C. Reinsch, “Singular Value Decomposition and Least Squares Solutions,” Numer. Math. 14, 403 (1970).
[CrossRef]

Goodman, J.

J. Goodman et al., “Parallel Incoherent Optical Vector-Matrix Multiplier,” Tech. Rep. L-723-1, BMD (Feb.1979).

Hong, J.

Hopfield, J. J.

J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982).
[CrossRef] [PubMed]

Jackson, J.

Jacques, J. St.

Y. Abu-Mostafa, J. St. Jacques, “Information Capacity of the Hopfield Model,” IEEE Trans. Inf. Theory IT-31, 461 (1985).
[CrossRef]

Kohonen, T.

T. Kohonen, Self-Organization and Associative Memory (Springer-Verlag, Berlin, 1988).
[CrossRef]

Lee, J. N.

Liebowitz, S.

Lippincott, W. L.

Little, M. J.

M. J. Little, C. S. Bak, “Enhanced Memory Capacity of a Hopfield Neural Network,” Proc. Soc. Photo-Opt. Instrum. Eng. 698, 150 (1986).

Marom, E.

Maxwell, T.

Mitra, S. K.

C. R. Rao, S. K. Mitra, Generalized Inverse of Matrices and Its Applications (Wiley, New York, 1971).

Montgomery, B.

Murakami, K.

K. Murakami, T. Aibara, “An Improvement on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-17, 699 (1987).
[CrossRef]

Neuman, C.

Owechko, Y.

Palm, G.

G. Palm, “On Associative Memory,” Biol. Cybern. 36, 19 (1Feb.1980).
[CrossRef] [PubMed]

Park, C. H.

D. Psaltis, C. H. Park, “Nonlinear Discriminant Functions and Associative Memories,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 370.

Psaltis, D.

J. Hong, D. Psaltis, “Storage Capacity of Holographic Associative Memories,” Opt. Lett. 11, 812 (1986).
[CrossRef] [PubMed]

D. Psaltis, N. Farhat, “Optical Information Processing Based on an Associative-Memory Model of Neural Nets with Thresholding and Feedback,” Opt. Lett. 10, 98 (1985).
[CrossRef] [PubMed]

K. Wagner, D. Psaltis, “A Space Integrating Acousto-Optic Matrix-Matrix Multiplier,” Opt. Commun. 52, 173 (1984).
[CrossRef]

D. Psaltis, C. H. Park, “Nonlinear Discriminant Functions and Associative Memories,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 370.

Rao, C. R.

C. R. Rao, S. K. Mitra, Generalized Inverse of Matrices and Its Applications (Wiley, New York, 1971).

Reinsch, C.

G. H. Golub, C. Reinsch, “Singular Value Decomposition and Least Squares Solutions,” Numer. Math. 14, 403 (1970).
[CrossRef]

Soffer, B. H.

Stiles, G. S.

G. S. Stiles, D. L. Denq, “On the Effect of Noise on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Pattern Anal. Machine Intell. PAMI-7, 358 (1985).
[CrossRef]

Telfer, B.

D. Casasent, B. Telfer, “Associative Memory Synthesis, Performance, Storage Capacity and Updating: New Heteroassociative Memory Results,” Proc. Soc. Photo-Opt. Instrum. Eng. 848, 313 (1987).

D. Casasent, B. Telfer, “Distortion-Invariant Associative Memories and Processors,” Proc. Soc. Photo-Opt. Instrum. Eng. 697, 60 (1986).

Vijaya Kumar, B.V.K.

Wagner, K.

K. Wagner, D. Psaltis, “A Space Integrating Acousto-Optic Matrix-Matrix Multiplier,” Opt. Commun. 52, 173 (1984).
[CrossRef]

Appl. Opt. (8)

Biol. Cybern. (1)

G. Palm, “On Associative Memory,” Biol. Cybern. 36, 19 (1Feb.1980).
[CrossRef] [PubMed]

IEEE Trans. Inf. Theory (1)

Y. Abu-Mostafa, J. St. Jacques, “Information Capacity of the Hopfield Model,” IEEE Trans. Inf. Theory IT-31, 461 (1985).
[CrossRef]

IEEE Trans. Pattern Anal. Machine Intell. (1)

G. S. Stiles, D. L. Denq, “On the Effect of Noise on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Pattern Anal. Machine Intell. PAMI-7, 358 (1985).
[CrossRef]

IEEE Trans. Syst. Man Cybern. (2)

K. Murakami, T. Aibara, “An Improvement on the Moore-Penrose Generalized Inverse Associative Memory,” IEEE Trans. Syst. Man Cybern. SMC-17, 699 (1987).
[CrossRef]

J. A. Anderson, “Cognitive and Psychological Computation with Neural Models,” IEEE Trans. Syst. Man Cybern. SMC-13, 799 (1983).
[CrossRef]

Numer. Math. (1)

G. H. Golub, C. Reinsch, “Singular Value Decomposition and Least Squares Solutions,” Numer. Math. 14, 403 (1970).
[CrossRef]

Opt. Commun. (1)

K. Wagner, D. Psaltis, “A Space Integrating Acousto-Optic Matrix-Matrix Multiplier,” Opt. Commun. 52, 173 (1984).
[CrossRef]

Opt. Lett. (2)

Proc. Natl. Acad. Sci. U.S.A. (1)

J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982).
[CrossRef] [PubMed]

Proc. Soc. Photo-Opt. Instrum. Eng. (4)

D. Casasent, “Scene Analysis Research: Optical Pattern Recognition and Artificial Intelligence,” Proc. Soc. Photo-Opt. Instrum. Eng. 634, 439 (1986).

M. J. Little, C. S. Bak, “Enhanced Memory Capacity of a Hopfield Neural Network,” Proc. Soc. Photo-Opt. Instrum. Eng. 698, 150 (1986).

D. Casasent, B. Telfer, “Distortion-Invariant Associative Memories and Processors,” Proc. Soc. Photo-Opt. Instrum. Eng. 697, 60 (1986).

D. Casasent, B. Telfer, “Associative Memory Synthesis, Performance, Storage Capacity and Updating: New Heteroassociative Memory Results,” Proc. Soc. Photo-Opt. Instrum. Eng. 848, 313 (1987).

Other (4)

C. R. Rao, S. K. Mitra, Generalized Inverse of Matrices and Its Applications (Wiley, New York, 1971).

D. Psaltis, C. H. Park, “Nonlinear Discriminant Functions and Associative Memories,” in Neural Networks for Computing, J. S. Denker, Ed. (American Institute of Physics, New York, 1986), p. 370.

T. Kohonen, Self-Organization and Associative Memory (Springer-Verlag, Berlin, 1988).
[CrossRef]

J. Goodman et al., “Parallel Incoherent Optical Vector-Matrix Multiplier,” Tech. Rep. L-723-1, BMD (Feb.1979).

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (6)

Fig. 1
Fig. 1

Analog optical matrix–vector multiplier for pseudoinverse associative memory.

Fig. 2
Fig. 2

Autoassociative memory image recollection: partial phantom input (a) and output obtained (b); partial DC10 input (c) and output obtained (d); noisy phantom input (e); and output obtained (f).

Fig. 3
Fig. 3

Unit vector HAM 36-class demonstration. The input orientation is noted on the left, the object is shown in the center, and the output vector obtained is given on the right.

Fig. 4
Fig. 4

Binary vector HAM two-class demonstration. The recollection vectors [1,0]T and [0,1]T denote a Phantom and DC-10, respectively. The input orientation is noted on the left, the object is shown in the center, and the output vector obtained is shown to the right.

Fig. 5
Fig. 5

Probability of correct recognition PC vs output noise standard deviation σo for four different recollection vector encoding schemes.

Fig. 6
Fig. 6

Probability of correct recognition PC vs input noise standard deviation σi for four different recollection vector encoding schemes.

Tables (3)

Tables Icon

Table I σ o 2 / σ i 2 for AAM and HAM

Tables Icon

Table II SNRo/SNRi for AAM and HAM

Tables Icon

Table III Associative Memory SNRo/SNRi Performance as M Increases

Equations (37)

Equations on this page are rendered with MathJax. Learn more.

MX = Y .
M = Y X +
X + = ( X T X ) - 1 X T ,
h = Σ a j x j = Xa ,
h k T = u k T ( X T X ) - 1 X T = u k T X + .
σ o 2 / σ i 2 = M / N
σ o 2 / σ i 2 = M / N .
σ o 2 / σ i 2 = E { y i j 2 } E { Tr ( V - 1 ) } ,
σ o 2 / σ i 2 = ( c 2 / K ) Tr [ V - 1 ] .
σ o 2 / σ i 2 = E { y i j 2 } Tr [ V - 1 ] ,
σ o 2 / σ i 2 = ( 1 / K ) i m k v m k - 1 y i m y i ,
s i 2 = E { [ x k i ] 2 } - E { x k i } 2 ,
s o 2 = E { [ y k i ] 2 } - E { y k i } 2 ,
SNR o SNR i = s o 2 σ i 2 s i 2 σ o 2 .
SNR o SNR i = s o 2 K s i 2 i m k v m k - 1 y i m y i k ,
SNR o SNR i = s o 2 s i 2 E { y i j 2 } Tr { V - 1 } .
SNR o SNR i = M Tr [ V ] Tr [ V - 1 ] = 1 M ,
p = 0.5 - erf ( 0.5 / σ o ) ,
P C = ( 1 - p ) K = [ 0.5 + erf ( 0.5 / σ o ) ] K .
P C = P ( T ) { 0.5 + erf [ T / σ o ] } K - 1 d T .
P C = ( 1 - p ) K + K p ( 1 - p ) K - 1 ,
σ i 2 = σ o 2 / 0.0218 = ( 0.2 ) 2 / 0.0218 1.83
y = y k + Mn .
σ o 2 = E { [ ( Mn ) i ] 2 } = j k E { m i j m i j } E { n j n k } .
σ o 2 = σ i 2 N E { m i j 2 }
M = X X + = X ( X T X ) - 1 X T .
Tr ( M M T ) = i ( M M T ) i i = i j m i j 2 = Tr ( M ) ,
Tr [ M ] = i j m i j 2 = M .
E { m i j 2 } = Tr ( M ) N 2 = M N 2 .
Tr [ M M T ] = i ( M M T ) i i = i j m i j 2 ,
E { Tr [ M M T ] } = i j E { m i j 2 } .
E { Tr [ M M T ] } = KNE { m i j 2 } .
M M T = Y V - 1 Y T .
( M M T ) i i = m k v m k - 1 y i m y i k ,
Tr ( M M T ) = i m k v m k - 1 y i m y i k .
E { Tr [ M M T ] } = i m k E { v m k - 1 } E { y i m 2 } δ k m = i m E { v m m - 1 } E { y i m 2 } = m E { v m m - 1 } i E { y i m 2 } ,
E { Tr [ M M T ] } = KE { y i m 2 } E { Tr [ X T X ] - 1 } .

Metrics