Abstract

There is currently a strong push in the research community to develop biological scale implementations of neuron based vision models. Systems at this scale are computationally demanding and generally utilize more accurate neuron models, such as the Izhikevich and the Hodgkin–Huxley models, in favor of the more popular integrate and fire model. We examine the feasibility of using graphics processing units (GPUs) to accelerate a spiking neural network based character recognition network to enable such large scale systems. Two versions of the network utilizing the Izhikevich and Hodgkin–Huxley models are implemented. Three NVIDIA general-purpose (GP) GPU platforms are examined, including the GeForce 9800 GX2, the Tesla C1060, and the Tesla S1070. Our results show that the GPGPUs can provide significant speedup over conventional processors. In particular, the fastest GPGPU utilized, the Tesla S1070, provided a speedup of 5.6 and 84.4 over highly optimized implementations on the fastest central processing unit (CPU) tested, a quadcore 2.67GHz Xeon processor, for the Izhikevich and the Hodgkin–Huxley models, respectively. The CPU implementation utilized all four cores and the vector data parallelism offered by the processor. The results indicate that GPUs are well suited for this application domain.

© 2010 Optical Society of America

Full Article  |  PDF Article

References

  • View by:
  • |
  • |
  • |

  1. E. Izhikevich, “Which model to use for cortical spiking neurons?,” IEEE Trans. Neural Networks 15, 1063-1070 (2004).
    [CrossRef]
  2. R. Ananthanarayanan and D. Modha, “Anatomy of a cortical simulator,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,” (IEEE, 2007).
    [PubMed]
  3. H. Markram, “The blue brain project,” Nat. Rev. Neurosci. 7, 153-160 (2006).
    [CrossRef] [PubMed]
  4. A. Delorme and S. J. Thorpe, “SpikeNET: an event-driven simulation package for modelling large networks of spiking neurons,” Network Comput. Neural Syst. 14, 613-627 (Nov. 2003).
    [CrossRef]
  5. Y. Dan and M. Poo, “Spike time dependent plasticity of neural circuits,” Neuron , 44, 23-30 (2004).
    [CrossRef] [PubMed]
  6. M. A. Bhuiyan, R. Jalasutram, and T. M. Taha, “Character recognition with two spiking neural network models on multicore architectures,” in IEEE Symposium on Computational Intelligence for Multimedia Signal and Vision Processing (IEEE, 2009).
    [CrossRef]
  7. A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and application to conduction and excitation in nerve,” J. Physiol. 117, 500-544 (1952).
    [PubMed]
  8. E. M. Izhikevich, “Simple Model of Spiking Neurons,” IEEE Trans. Neural Networks 14, 1569-1572 (2003).
    [CrossRef]
  9. GeForce 9800 GX2 Specifications, http://www.nvidia.com/object/product_geforce_9800_gx2_us.html
  10. Tesla C1060 Specifications, http://www.nvidia.com/object/product_tesla_c1060_us.html
  11. C. Johansson and A. Lansner, “Towards cortex sized artificial neural systems,” Neural Networks 20, 48-61 (2007).
    [CrossRef]
  12. A. R. Baig, “Spatial-temporal artificial neurons applied to online cursive handwritten character recognition,” in European Symposium on Artificial Neural Networks (d-side, 2004), pp. 561-566.
  13. A. Gupta and L. Long, “Character recognition using spiking neural networks,” in Proceedings of the International Joint Conference on Neural Networks (IEEE, 2007).
    [CrossRef]
  14. C. Panchev and S. Wermter, “Temporal sequence detection with spiking neurons: towards recognizing robot language instructions,” Connect. Sci. 18, 1-22 (2006).
    [CrossRef]
  15. D. V. Buonomano and M. M. Merzenich, “A neural network model of temporal code generation and position invariant pattern recognition,” Neural Comput. 11, 103-116 (1999).
    [CrossRef] [PubMed]
  16. T. Ichishita and R. Fujii, “Performance evaluation of a temporal sequence learning spiking neural network,” in Proceedings of the 7th IEEE International Conference on Computer and Information Technology (IEEE, 2007).
  17. K-Team, Inc.; available online at http://www.k-team.com/.
  18. W. Rall, “Branching dendritic trees and motoneuron membrane resistivity,” Exp. Neurol. 1, 491-527 (1959).
    [CrossRef] [PubMed]
  19. M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
    [CrossRef]
  20. J.-P. Tiesel and A. S. Maida, “Using parallel GPU architecture for simulation of planar I/F networks,” in International Joint Conference on Neural Networks (IEEE, 2009).
    [CrossRef]
  21. J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
    [CrossRef]
  22. N. Satish, M. Harris, and M. Garland, “Designing efficient sorting algorithms for manycore GPUs,” in IEEE International Symposium on Parallel and Distributed Processing (IEEE, 2009).
    [CrossRef]
  23. M. Harris, “Optimizing parallel reduction in CUDA,” http://developer.download.nvidia.com/compute/cuda/1_1/Website/projects/reduction/doc/reduction.pdf.

2009 (1)

J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
[CrossRef]

2008 (1)

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

2007 (1)

C. Johansson and A. Lansner, “Towards cortex sized artificial neural systems,” Neural Networks 20, 48-61 (2007).
[CrossRef]

2006 (2)

C. Panchev and S. Wermter, “Temporal sequence detection with spiking neurons: towards recognizing robot language instructions,” Connect. Sci. 18, 1-22 (2006).
[CrossRef]

H. Markram, “The blue brain project,” Nat. Rev. Neurosci. 7, 153-160 (2006).
[CrossRef] [PubMed]

2004 (2)

Y. Dan and M. Poo, “Spike time dependent plasticity of neural circuits,” Neuron , 44, 23-30 (2004).
[CrossRef] [PubMed]

E. Izhikevich, “Which model to use for cortical spiking neurons?,” IEEE Trans. Neural Networks 15, 1063-1070 (2004).
[CrossRef]

2003 (1)

E. M. Izhikevich, “Simple Model of Spiking Neurons,” IEEE Trans. Neural Networks 14, 1569-1572 (2003).
[CrossRef]

1999 (1)

D. V. Buonomano and M. M. Merzenich, “A neural network model of temporal code generation and position invariant pattern recognition,” Neural Comput. 11, 103-116 (1999).
[CrossRef] [PubMed]

1959 (1)

W. Rall, “Branching dendritic trees and motoneuron membrane resistivity,” Exp. Neurol. 1, 491-527 (1959).
[CrossRef] [PubMed]

1952 (1)

A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and application to conduction and excitation in nerve,” J. Physiol. 117, 500-544 (1952).
[PubMed]

Ananthanarayanan, R.

R. Ananthanarayanan and D. Modha, “Anatomy of a cortical simulator,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,” (IEEE, 2007).
[PubMed]

Baig, A. R.

A. R. Baig, “Spatial-temporal artificial neurons applied to online cursive handwritten character recognition,” in European Symposium on Artificial Neural Networks (d-side, 2004), pp. 561-566.

Bhuiyan, M. A.

M. A. Bhuiyan, R. Jalasutram, and T. M. Taha, “Character recognition with two spiking neural network models on multicore architectures,” in IEEE Symposium on Computational Intelligence for Multimedia Signal and Vision Processing (IEEE, 2009).
[CrossRef]

Buonomano, D. V.

D. V. Buonomano and M. M. Merzenich, “A neural network model of temporal code generation and position invariant pattern recognition,” Neural Comput. 11, 103-116 (1999).
[CrossRef] [PubMed]

Dan, Y.

Y. Dan and M. Poo, “Spike time dependent plasticity of neural circuits,” Neuron , 44, 23-30 (2004).
[CrossRef] [PubMed]

Delorme, A.

A. Delorme and S. J. Thorpe, “SpikeNET: an event-driven simulation package for modelling large networks of spiking neurons,” Network Comput. Neural Syst. 14, 613-627 (Nov. 2003).
[CrossRef]

Djurfeldt, M.

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

Dutt, N.

J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
[CrossRef]

Ekeberg, O.

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

Fujii, R.

T. Ichishita and R. Fujii, “Performance evaluation of a temporal sequence learning spiking neural network,” in Proceedings of the 7th IEEE International Conference on Computer and Information Technology (IEEE, 2007).

Garland, M.

N. Satish, M. Harris, and M. Garland, “Designing efficient sorting algorithms for manycore GPUs,” in IEEE International Symposium on Parallel and Distributed Processing (IEEE, 2009).
[CrossRef]

Gupta, A.

A. Gupta and L. Long, “Character recognition using spiking neural networks,” in Proceedings of the International Joint Conference on Neural Networks (IEEE, 2007).
[CrossRef]

Harris, M.

N. Satish, M. Harris, and M. Garland, “Designing efficient sorting algorithms for manycore GPUs,” in IEEE International Symposium on Parallel and Distributed Processing (IEEE, 2009).
[CrossRef]

M. Harris, “Optimizing parallel reduction in CUDA,” http://developer.download.nvidia.com/compute/cuda/1_1/Website/projects/reduction/doc/reduction.pdf.

Hodgkin, A. L.

A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and application to conduction and excitation in nerve,” J. Physiol. 117, 500-544 (1952).
[PubMed]

Huxley, A. F.

A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and application to conduction and excitation in nerve,” J. Physiol. 117, 500-544 (1952).
[PubMed]

Ichishita, T.

T. Ichishita and R. Fujii, “Performance evaluation of a temporal sequence learning spiking neural network,” in Proceedings of the 7th IEEE International Conference on Computer and Information Technology (IEEE, 2007).

Izhikevich, E.

E. Izhikevich, “Which model to use for cortical spiking neurons?,” IEEE Trans. Neural Networks 15, 1063-1070 (2004).
[CrossRef]

Izhikevich, E. M.

E. M. Izhikevich, “Simple Model of Spiking Neurons,” IEEE Trans. Neural Networks 14, 1569-1572 (2003).
[CrossRef]

Jalasutram, R.

M. A. Bhuiyan, R. Jalasutram, and T. M. Taha, “Character recognition with two spiking neural network models on multicore architectures,” in IEEE Symposium on Computational Intelligence for Multimedia Signal and Vision Processing (IEEE, 2009).
[CrossRef]

Johansson, C.

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

C. Johansson and A. Lansner, “Towards cortex sized artificial neural systems,” Neural Networks 20, 48-61 (2007).
[CrossRef]

Krichmar, J. L.

J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
[CrossRef]

Lansner, A.

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

C. Johansson and A. Lansner, “Towards cortex sized artificial neural systems,” Neural Networks 20, 48-61 (2007).
[CrossRef]

Long, L.

A. Gupta and L. Long, “Character recognition using spiking neural networks,” in Proceedings of the International Joint Conference on Neural Networks (IEEE, 2007).
[CrossRef]

Lundqvist, M.

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

Maida, A. S.

J.-P. Tiesel and A. S. Maida, “Using parallel GPU architecture for simulation of planar I/F networks,” in International Joint Conference on Neural Networks (IEEE, 2009).
[CrossRef]

Markram, H.

H. Markram, “The blue brain project,” Nat. Rev. Neurosci. 7, 153-160 (2006).
[CrossRef] [PubMed]

Merzenich, M. M.

D. V. Buonomano and M. M. Merzenich, “A neural network model of temporal code generation and position invariant pattern recognition,” Neural Comput. 11, 103-116 (1999).
[CrossRef] [PubMed]

Modha, D.

R. Ananthanarayanan and D. Modha, “Anatomy of a cortical simulator,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,” (IEEE, 2007).
[PubMed]

Nageswarana, J. M.

J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
[CrossRef]

Nicolau, A.

J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
[CrossRef]

Panchev, C.

C. Panchev and S. Wermter, “Temporal sequence detection with spiking neurons: towards recognizing robot language instructions,” Connect. Sci. 18, 1-22 (2006).
[CrossRef]

Poo, M.

Y. Dan and M. Poo, “Spike time dependent plasticity of neural circuits,” Neuron , 44, 23-30 (2004).
[CrossRef] [PubMed]

Rall, W.

W. Rall, “Branching dendritic trees and motoneuron membrane resistivity,” Exp. Neurol. 1, 491-527 (1959).
[CrossRef] [PubMed]

Rehn, M.

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

Satish, N.

N. Satish, M. Harris, and M. Garland, “Designing efficient sorting algorithms for manycore GPUs,” in IEEE International Symposium on Parallel and Distributed Processing (IEEE, 2009).
[CrossRef]

Taha, T. M.

M. A. Bhuiyan, R. Jalasutram, and T. M. Taha, “Character recognition with two spiking neural network models on multicore architectures,” in IEEE Symposium on Computational Intelligence for Multimedia Signal and Vision Processing (IEEE, 2009).
[CrossRef]

Thorpe, S. J.

A. Delorme and S. J. Thorpe, “SpikeNET: an event-driven simulation package for modelling large networks of spiking neurons,” Network Comput. Neural Syst. 14, 613-627 (Nov. 2003).
[CrossRef]

Tiesel, J.-P.

J.-P. Tiesel and A. S. Maida, “Using parallel GPU architecture for simulation of planar I/F networks,” in International Joint Conference on Neural Networks (IEEE, 2009).
[CrossRef]

Veidenbauma, A. V.

J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
[CrossRef]

Wermter, S.

C. Panchev and S. Wermter, “Temporal sequence detection with spiking neurons: towards recognizing robot language instructions,” Connect. Sci. 18, 1-22 (2006).
[CrossRef]

Connect. Sci. (1)

C. Panchev and S. Wermter, “Temporal sequence detection with spiking neurons: towards recognizing robot language instructions,” Connect. Sci. 18, 1-22 (2006).
[CrossRef]

Exp. Neurol. (1)

W. Rall, “Branching dendritic trees and motoneuron membrane resistivity,” Exp. Neurol. 1, 491-527 (1959).
[CrossRef] [PubMed]

IBM J. Res. Dev. (1)

M. Djurfeldt, M. Lundqvist, C. Johansson, M. Rehn, O. Ekeberg, and A. Lansner, “Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer,” IBM J. Res. Dev. 52, 31-41 (2008).
[CrossRef]

IEEE Trans. Neural Networks (2)

E. Izhikevich, “Which model to use for cortical spiking neurons?,” IEEE Trans. Neural Networks 15, 1063-1070 (2004).
[CrossRef]

E. M. Izhikevich, “Simple Model of Spiking Neurons,” IEEE Trans. Neural Networks 14, 1569-1572 (2003).
[CrossRef]

J. Physiol. (1)

A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and application to conduction and excitation in nerve,” J. Physiol. 117, 500-544 (1952).
[PubMed]

Nat. Rev. Neurosci. (1)

H. Markram, “The blue brain project,” Nat. Rev. Neurosci. 7, 153-160 (2006).
[CrossRef] [PubMed]

Network Comput. Neural Syst. (1)

A. Delorme and S. J. Thorpe, “SpikeNET: an event-driven simulation package for modelling large networks of spiking neurons,” Network Comput. Neural Syst. 14, 613-627 (Nov. 2003).
[CrossRef]

Neural Comput. (1)

D. V. Buonomano and M. M. Merzenich, “A neural network model of temporal code generation and position invariant pattern recognition,” Neural Comput. 11, 103-116 (1999).
[CrossRef] [PubMed]

Neural Networks (2)

C. Johansson and A. Lansner, “Towards cortex sized artificial neural systems,” Neural Networks 20, 48-61 (2007).
[CrossRef]

J. M. Nageswarana, N. Dutt, J. L. Krichmar, A. Nicolau, and A. V. Veidenbauma, “A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors,” Neural Networks 22, 791-800 (2009).
[CrossRef]

Neuron (1)

Y. Dan and M. Poo, “Spike time dependent plasticity of neural circuits,” Neuron , 44, 23-30 (2004).
[CrossRef] [PubMed]

Other (11)

M. A. Bhuiyan, R. Jalasutram, and T. M. Taha, “Character recognition with two spiking neural network models on multicore architectures,” in IEEE Symposium on Computational Intelligence for Multimedia Signal and Vision Processing (IEEE, 2009).
[CrossRef]

GeForce 9800 GX2 Specifications, http://www.nvidia.com/object/product_geforce_9800_gx2_us.html

Tesla C1060 Specifications, http://www.nvidia.com/object/product_tesla_c1060_us.html

R. Ananthanarayanan and D. Modha, “Anatomy of a cortical simulator,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,” (IEEE, 2007).
[PubMed]

A. R. Baig, “Spatial-temporal artificial neurons applied to online cursive handwritten character recognition,” in European Symposium on Artificial Neural Networks (d-side, 2004), pp. 561-566.

A. Gupta and L. Long, “Character recognition using spiking neural networks,” in Proceedings of the International Joint Conference on Neural Networks (IEEE, 2007).
[CrossRef]

J.-P. Tiesel and A. S. Maida, “Using parallel GPU architecture for simulation of planar I/F networks,” in International Joint Conference on Neural Networks (IEEE, 2009).
[CrossRef]

T. Ichishita and R. Fujii, “Performance evaluation of a temporal sequence learning spiking neural network,” in Proceedings of the 7th IEEE International Conference on Computer and Information Technology (IEEE, 2007).

K-Team, Inc.; available online at http://www.k-team.com/.

N. Satish, M. Harris, and M. Garland, “Designing efficient sorting algorithms for manycore GPUs,” in IEEE International Symposium on Parallel and Distributed Processing (IEEE, 2009).
[CrossRef]

M. Harris, “Optimizing parallel reduction in CUDA,” http://developer.download.nvidia.com/compute/cuda/1_1/Website/projects/reduction/doc/reduction.pdf.

Cited By

OSA participates in CrossRef's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1
Fig. 1

Spikes produced with the Hodgkin–Huxley model.

Fig. 2
Fig. 2

Spikes produced with the Izhikevich model.

Fig. 3
Fig. 3

Simplified CUDA GPU architecture.

Fig. 4
Fig. 4

Network used to test spiking models.

Fig. 5
Fig. 5

Algorithm flow chart.

Fig. 6
Fig. 6

Training images.

Fig. 7
Fig. 7

Additional test images.

Fig. 8
Fig. 8

Neurons/second throughput of GPGPUs and processors for the (a)  Izhikevich and (b)  Hodgkin–Huxley models.

Fig. 9
Fig. 9

Timing breakdown by percentage of overall runtime for processing the 768 × 768 image using the (a) Izhikevich and (b) Hodgkin–Huxley models.

Fig. 10
Fig. 10

Variation in speedup of GPU implementations with a firing rate in the level 1 neurons for the (a) Izhikevich and (b) Hodgkin–Huxley models.

Tables (4)

Tables Icon

Table 1 Spiking Network Configurations Evaluated

Tables Icon

Table 2 Speedup Achieved on Four GPU Platforms for the Izhikevich Model for Different Image Sizes

Tables Icon

Table 3 Speedup Achieved on Four GPU Platforms for the Hodgkin–Huxley Model for Different Image Sizes

Tables Icon

Table 4 Bytes/Flop Requirement of the Models and Capabilities of the GPGPU Cards

Equations (8)

Equations on this page are rendered with MathJax. Learn more.

d v d t = ( 1 C ) { I g K n 4 ( V E K ) g Na m 3 h ( V E Na ) g L ( V E L ) } ,
d n d t = ( n ( V ) n ) / τ n ( V ) ,
d m d t = ( m ( V ) m ) / τ m ( V ) ,
d h d t = ( h ( V ) h ) / τ h ( V ) .
d V d t = 0.04 V 2 + 5 V + 140 u + I ,
d u d t = a ( b V u ) ,
if    V 30 mV , then     { V c u u + d .
I j = i w ( i , j ) f ( i ) ,

Metrics