Abstract

Adaptive resonance architectures are neural networks that self-organize stable pattern recognition codes in real-time in response to arbitrary sequences of input patterns. This article introduces ART 2, a class of adaptive resonance architectures which rapidly self-organize pattern recognition categories in response to arbitrary sequences of either analog or binary input patterns. In order to cope with arbitrary sequences of analog input patterns, ART 2 architectures embody solutions to a number of design principles, such as the stability-plasticity tradeoff, the search-direct access tradeoff, and the match-reset tradeoff. In these architectures, top-down learned expectation and matching mechanisms are critical in self-stabilizing the code learning process. A parallel search scheme updates itself adaptively as the learning process unfolds, and realizes a form of real-time hypothesis discovery, testing, learning, and recognition. After learning self-stabilizes, the search process is automatically disengaged. Thereafter input patterns directly access their recognition codes without any search. Thus recognition time for familiar inputs does not increase with the complexity of the learned code. A novel input pattern can directly access a category if it shares invariant properties with the set of familiar exemplars of that category. A parameter called the attentional vigilance parameter determines how fine the categories will be. If vigilance increases (decreases) due to environmental feedback, then the system automatically searches for and learns finer (coarser) recognition categories. Gain control parameters enable the architecture to suppress noise up to a prescribed level. The architecture's global design enables it to learn effectively despite the high degree of nonlinearity of such mechanisms.

© 1987 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. S. Grossberg, “Adaptive Pattern Classification and Universal Recoding, II: Feedback, Expectation, Olfaction, and Illusions,” Biol. Cybern. 23, 187 (1976).
    [PubMed]
  2. G. A. Carpenter, S. Grossberg, “Category Learning and Adaptive Pattern Recognition: a Neural Network Model,” in Proceedings, Third Army Conference on Applied Mathematics and Computing, ARO Report 86-1 (1985), pp. 37–56.
  3. G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine,” Comput. Vision Graphics Image Process. 37, 54 (1987).
    [Crossref]
  4. S. Grossberg, “Competitive Learning: from Interactive Activation to Adaptive Resonance,” Cognitive Sci. 11, 23 (1987).
    [Crossref]
  5. G. A. Carpenter, S. Grossberg, “ART 2: Stable Self-Organization of Pattern Recognition Codes for Analog Input Patterns,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).
  6. G. A. Carpenter, S. Grossberg, “Invariant Pattern Recognition and Recall by an Attentive Self-Organizing ART Architecture in a Nonstationary World,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).
  7. K. Hartley, “Seeing the Need for ART”, Sci. News 132, 14 (1987).
    [Crossref]
  8. P. Kolodzy, “Multidimensional Machine Vision Using Neural Networks,” in Proceedings, First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).
  9. S. Grossberg, Studies of Mind and Brain: Neural Principles of Learning, Perception, Development, Cognition, and Motor Control (Reidel, Boston, 1982).
  10. A. L. Hodgkin, A. F. Huxley, “A Quantitative Description of Membrane Current and Its Applications to Conduction and Excitation in Nerve,” J. Physiol. London 117, 500 (1952).

1987 (3)

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[Crossref]

S. Grossberg, “Competitive Learning: from Interactive Activation to Adaptive Resonance,” Cognitive Sci. 11, 23 (1987).
[Crossref]

K. Hartley, “Seeing the Need for ART”, Sci. News 132, 14 (1987).
[Crossref]

1976 (1)

S. Grossberg, “Adaptive Pattern Classification and Universal Recoding, II: Feedback, Expectation, Olfaction, and Illusions,” Biol. Cybern. 23, 187 (1976).
[PubMed]

1952 (1)

A. L. Hodgkin, A. F. Huxley, “A Quantitative Description of Membrane Current and Its Applications to Conduction and Excitation in Nerve,” J. Physiol. London 117, 500 (1952).

Carpenter, G. A.

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[Crossref]

G. A. Carpenter, S. Grossberg, “Category Learning and Adaptive Pattern Recognition: a Neural Network Model,” in Proceedings, Third Army Conference on Applied Mathematics and Computing, ARO Report 86-1 (1985), pp. 37–56.

G. A. Carpenter, S. Grossberg, “Invariant Pattern Recognition and Recall by an Attentive Self-Organizing ART Architecture in a Nonstationary World,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

G. A. Carpenter, S. Grossberg, “ART 2: Stable Self-Organization of Pattern Recognition Codes for Analog Input Patterns,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

Grossberg, S.

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[Crossref]

S. Grossberg, “Competitive Learning: from Interactive Activation to Adaptive Resonance,” Cognitive Sci. 11, 23 (1987).
[Crossref]

S. Grossberg, “Adaptive Pattern Classification and Universal Recoding, II: Feedback, Expectation, Olfaction, and Illusions,” Biol. Cybern. 23, 187 (1976).
[PubMed]

G. A. Carpenter, S. Grossberg, “ART 2: Stable Self-Organization of Pattern Recognition Codes for Analog Input Patterns,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

S. Grossberg, Studies of Mind and Brain: Neural Principles of Learning, Perception, Development, Cognition, and Motor Control (Reidel, Boston, 1982).

G. A. Carpenter, S. Grossberg, “Category Learning and Adaptive Pattern Recognition: a Neural Network Model,” in Proceedings, Third Army Conference on Applied Mathematics and Computing, ARO Report 86-1 (1985), pp. 37–56.

G. A. Carpenter, S. Grossberg, “Invariant Pattern Recognition and Recall by an Attentive Self-Organizing ART Architecture in a Nonstationary World,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

Hartley, K.

K. Hartley, “Seeing the Need for ART”, Sci. News 132, 14 (1987).
[Crossref]

Hodgkin, A. L.

A. L. Hodgkin, A. F. Huxley, “A Quantitative Description of Membrane Current and Its Applications to Conduction and Excitation in Nerve,” J. Physiol. London 117, 500 (1952).

Huxley, A. F.

A. L. Hodgkin, A. F. Huxley, “A Quantitative Description of Membrane Current and Its Applications to Conduction and Excitation in Nerve,” J. Physiol. London 117, 500 (1952).

Kolodzy, P.

P. Kolodzy, “Multidimensional Machine Vision Using Neural Networks,” in Proceedings, First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

Biol. Cybern. (1)

S. Grossberg, “Adaptive Pattern Classification and Universal Recoding, II: Feedback, Expectation, Olfaction, and Illusions,” Biol. Cybern. 23, 187 (1976).
[PubMed]

Cognitive Sci. (1)

S. Grossberg, “Competitive Learning: from Interactive Activation to Adaptive Resonance,” Cognitive Sci. 11, 23 (1987).
[Crossref]

Comput. Vision Graphics Image Process. (1)

G. A. Carpenter, S. Grossberg, “A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine,” Comput. Vision Graphics Image Process. 37, 54 (1987).
[Crossref]

J. Physiol. London (1)

A. L. Hodgkin, A. F. Huxley, “A Quantitative Description of Membrane Current and Its Applications to Conduction and Excitation in Nerve,” J. Physiol. London 117, 500 (1952).

Sci. News (1)

K. Hartley, “Seeing the Need for ART”, Sci. News 132, 14 (1987).
[Crossref]

Other (5)

P. Kolodzy, “Multidimensional Machine Vision Using Neural Networks,” in Proceedings, First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

S. Grossberg, Studies of Mind and Brain: Neural Principles of Learning, Perception, Development, Cognition, and Motor Control (Reidel, Boston, 1982).

G. A. Carpenter, S. Grossberg, “ART 2: Stable Self-Organization of Pattern Recognition Codes for Analog Input Patterns,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

G. A. Carpenter, S. Grossberg, “Invariant Pattern Recognition and Recall by an Attentive Self-Organizing ART Architecture in a Nonstationary World,” in Proceedings First International Conference on Neural Networks, San Diego (IEEE, New York, 1987).

G. A. Carpenter, S. Grossberg, “Category Learning and Adaptive Pattern Recognition: a Neural Network Model,” in Proceedings, Third Army Conference on Applied Mathematics and Computing, ARO Report 86-1 (1985), pp. 37–56.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1

Category grouping of fifty analog input patterns into thirty-four recognition categories. Each input pattern I is depicted as a function of i (i = 1…M), with successive Ii values connected by straight lines. The category structure established on one complete presentation of the fifty inputs remains stable thereafter if the same inputs are presented again.

Fig. 2
Fig. 2

Typical ART 1 architecture. Rectangles represent fields where STM patterns are stored. Semicircles represent adaptive filter pathways and arrows represent paths which are not adaptive. Filled circles represent gain control nuclei, which sum input signals. Their output paths are nonspecific in the sense that at any given time a uniform signal is sent to all nodes in a receptor field. Gain control at F1 and F2 coordinates STM processing with input presentation rate.

Fig. 3
Fig. 3

Lower vigilance implies coarser grouping. The same ART 2 system as used in Fig. 1 has here grouped the same fifty inputs into twenty recognition categories. Note, for example, that categories 1 and 2 of Fig. 1 are here joined in category 1; categories 14, 15, and 32 are here joined in category 10; and categories 19–22 are here joined in category 13.

Fig. 4
Fig. 4

Category learning by an ART 2 model without an orienting subsystem, (a) The same ART 2 system as used in Figs. 1 and 3, but with vigilance level set equal to zero, has here grouped the same fifty inputs into six recognition categories after one presentation of each pattern. Without the full ART 2 system's ability to reset on mismatch, transitory groupings occur, as in category 1. (b) By the third presentation of each input, a coarse but stable category structure has been established.

Fig. 5
Fig. 5

Search for a correct F2 code, (a) The input pattern I generates the specific STM activity pattern X at F1 as it nonspecifically activates A. Pattern X both inhibits A and generates the output signal pattern S. Signal pattern S is transformed into the input pattern T, which activates the STM pattern Y across F2. (b) Pattern Y generates the top-down signal pattern U which is transformed into the template pattern V. If V mismatches I at F1, a new STM activity pattern X* is generated at F1. The reduction in total STM activity which occurs when X is transformed into X* causes a decrease in the total inhibition from F1 to A. (c) Then the input-driven activation of A can release a nonspecific arousal wave to F2, which resets the STM pattern Y at F2. (d) After Y is inhibited, its top-down template is eliminated, and X can be reinstated at F1. Now X once again generates input pattern T to F2, but since Y remains inhibited T can activate a different STM pattern Y* at F2. If the top-down template due to Y* also mismatches I at F1, the rapid search for an appropriate F2 code continues.

Fig. 6
Fig. 6

Typical ART 2 architecture. Open arrows indicate specific patterned inputs to target nodes. Filled arrows indicate nonspecific gain control inputs. The gain control nuclei (large filled circles) nonspecifically inhibit target nodes in proportion to the L2 norm of STM activity in their source fields [Eqs. (5), (6), (9), (20), and (21)]. When F2 makes a choice, g(yJ) = d if the Jth F2 node is active and g(yJ) = 0 otherwise. As in ART 1, gain control (not shown) coordinates STM processing with an input presentation rate.

Fig. 7
Fig. 7

Graph of ‖r‖ as a function of ‖cdzJ‖ for values of cos (u, zJ) between 0 and 1 and for c = 0.1 and d = 0.9. F2 reset occurs whenever ‖r‖ falls below the vigilance parameter ρ.

Fig. 8
Fig. 8

ART 2 matching processes. The ART 2 system of Fig. 6 was used to generate the five simulations shown in columns (a)–(e). Each column shows the first ten simulation trials, in which four input patterns (A, B, C, D) are presented in order ABCAD on trials 1–5, and again on trials 6–10. Details are given in the text (Sec. XII). (a) The full ART 2 system, with ρ = 0.95, separates the four inputs into four categories. Search occurs on trials 1–5; thereafter each input directly accesses its category representation. Parameters a = 10, b = 10, c = 0.1, d = 0.9, θ = 0.2, and M = 25. The initial zij (0) values, 1, are half of the maximum, 2, allowed by constraint (27). The piecewise linear signal function (11) is used throughout, (b) Vigilance is here set so low (ρ = 0) that no search can ever occur. The coarse category structure established on trials 6–10 is, however, stable and consistent. All system parameters except ρ are as in (a). (c) With b = 0, the ART 2 system here generates an unstable, or inconsistent, category structure. Namely, input A goes alternatively to categories 1 and 2, and will continue to do so for as long as the sequence ABCAD repeats. All parameters except b are as in (b). Similar instability can occur when d is close to 0. (d) With a = 0, the ART 2 matching process differs from that which occurs when a is large; namely, the input pattern I is stronger, relative to the top-down pattern zJ, than in (b). All parameters except a are as in (b). Similar processing occurs when d is small (∼0.1) but not close to 0. (e) With θ = 0, the F1 signal function f becomes linear. Without the noise suppression/contrast enhancement provided by a nonlinear f, the completely different inputs B and D are here placed in a single category. All parameters except θ are as in (b).

Fig. 9
Fig. 9

Alternative ART 2 architecture.

Fig. 10
Fig. 10

Alternative ART 2 architecture.

Fig. 11
Fig. 11

Recognition category summary for the ART 2 system in Fig. 10. System parameters and vigilance level are the same as in Fig. 3, which was generated using the ART 2 model of Fig. 6. Because of the constant I input to the orienting subsystem, the ART 2 system of Fig. 10 is here seen to be slightly more sensitive to pattern mismatch at a given vigilance level, all other things being equal.

Tables (1)

Tables Icon

Table I Corresponding Categories

Equations (29)

Equations on this page are rendered with MathJax. Learn more.

d d t V i = A V i + ( 1 B V i ) J i + ( C + D V i ) J i
0 < 1 .
V i = J i + A + D J i .
p i = u i + j g ( y i ) z j i
q i = p i e + p ,
u i = υ i e + v ,
υ i = f ( x i ) + b f ( q i ) ,
w i = I i + a u i ,
x i = w i e + w ,
f ( x ) = { 2 θ x 2 ( x 2 + θ 2 ) if 0 x θ , x if x θ ,
f ( x ) = { 0 if 0 x < θ , x if x θ ,
T j = i p i z i j
T J = max { T j : j = M + 1 N } .
g ( y J ) = { d if T J = max { T j : the j th F 2 node has not been reset on the current trial } , 0 otherwise .
p i = { u i if F 2 is inactive , u i + d z J i if the J th F 2 node is active .
top - down ( F 2 F 1 ) : d d t z j i = g ( y j ) [ p i z j i ] ,
bottom - up ( F 1 F 2 ) : d d t z i j = g ( y j ) [ p i z i j ] .
d d t z J i = d [ p i z J i ] = d ( 1 d ) [ u i 1 d z J i ] ,
d d t z i J = d [ p i z i J ] = d ( 1 d ) [ u i 1 d z i J ] ,
r i = u i + c p i e + u + c p .
ρ e + r > 1 ,
r = [ 1 + 2 c p cos ( u , p ) + c p 2 ] ½ 1 + c p ,
p cos ( u , p ) = 1 + d z J cos ( u , z J ) .
p = [ 1 + 2 d z J cos ( u , z J ) + d z J 2 ] ½ .
r = [ ( 1 + c ) 2 + 2 ( 1 + c ) c d z J cos ( u , z J ) + c d z J 2 ] ½ 1 + [ c 2 + 2 c c d z J cos ( u , z J ) + c d z J 2 ] ½ .
z j i ( 0 ) = 0 ,
c d 1 d 1 .
z J ( 0 ) 1 1 d .
z i j ( 0 ) 1 ( 1 d ) M

Metrics