A bio-inspired hierarchical spiking neural network with biological synaptic plasticity for event camera object recognition_Journal of Biomedical Engineering

Authors：

 ZHOU Qian ^1,2,3 , ZHENG Peng ^1,2,3 , LI Xiaohu ^1,2,3

1. State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300130, P. R. China;
2. Hebei Key Laboratory of Bioelectromagnetics and Neural Engineering, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin 300130, P. R. China;
3. Tianjin Key Laboratory of Bioelectricity and Intelligent Health, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin 300130, P. R. China;

Corresponding author：

ZHOU Qian, Email: qzhou@hebut.edu.cn

Keywords：

Event camera; Object recognition; Spiking timing dependent plasticity (STDP); Reward-modulated STDP; Spiking neural network

DOI：

10.7507/1001-5515.202207040

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

With inherent sparse spike-based coding and asynchronous event-driven computation, spiking neural network (SNN) is naturally suitable for processing event stream data of event cameras. In order to improve the feature extraction and classification performance of bio-inspired hierarchical SNNs, in this paper an event camera object recognition system based on biological synaptic plasticity is proposed. In our system input event streams were firstly segmented adaptively using spiking neuron potential to improve computational efficiency of the system. Multi-layer feature learning and classification are implemented by our bio-inspired hierarchical SNN with synaptic plasticity. After Gabor filter-based event-driven convolution layer which extracted primary visual features of event streams, we used a feature learning layer with unsupervised spiking timing dependent plasticity (STDP) rule to help the network extract frequent salient features, and a feature learning layer with reward-modulated STDP rule to help the network learn diagnostic features. The classification accuracies of the network proposed in this paper on the four benchmark event stream datasets were better than the existing bio-inspired hierarchical SNNs. Moreover, our method showed good classification ability for short event stream input data, and was robust to input event stream noise. The results show that our method can improve the feature extraction and classification performance of this kind of SNNs for event camera object recognition.

Citation： ZHOU Qian, ZHENG Peng, LI Xiaohu. A bio-inspired hierarchical spiking neural network with biological synaptic plasticity for event camera object recognition. Journal of Biomedical Engineering, 2023, 40(4): 692-699. doi: 10.7507/1001-5515.202207040 Copy

1.	Bloss R. Latest in VISION SENSOR technology as well as innovations in sensing, pressure, force, medical, particle size and many other applications. Sensor Rev, 2017, 37(1): 7-11.
2.	Posch C, Serrano-Gotarredona T, Linares-Barranco B, et al. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc IEEE, 2014, 102(10): 1470-1484.
3.	Duwek H C, Shalumov A, Tsur E E. Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks// 2021 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 2021: 1333-1341.
4.	Gallego G, Delbrück T, Orchard G, et al. Event-based vision: A survey. IEEE Trans Pattern Anal Mach Intell, 2020, 44(1): 154-180.
5.	Chen G, Hong L, Dong J, et al. EDDD: Event-based drowsiness driving detection through facial motion analysis with neuromorphic vision sensor. IEEE Sens J, 2020, 20(11): 6170-6181.
6.	Lee C, Kosta A K, Zhu A Z, et al. Spike-FlowNet: event-based optical flow estimation with energy-efficient hybrid neural networks// Vedaldi A, Bischof H, Brox T. 2020 European Conference on Computer Vision (ECCV). Cham: Springer, 2020: 366-382.
7.	Glover A, Vasco V, Bartolozzi C. A controlled-delay event camera framework for on-line robotics// 2018 IEEE International Conference on Robotics and Automation(ICRA). Brisbane: IEEE, 2018: 2178-2183.
8.	Chen G, Cao H, Conradt J, et al. Event-based neuromorphic vision for autonomous driving: a paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Process Mag, 2020, 37(4): 34-49.
9.	Neil D, Liu S C. Effective sensor fusion with event-based sensors and deep network architectures// 2016 IEEE International Symposium on Circuits and Systems (ISCAS). Montreal: IEEE, 2016: 2282-2285.
10.	Zhu A Z, Yuan L, Chaney K, et al. EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv, 2018: 1802.06898.
11.	李杰. 基于深度卷积神经网络的动态手势识别. 济南: 山东大学, 2019.
12.	Fang Y, Wang Z, Gomez J, et al. A swarm optimization solver based on ferroelectric spiking neural networks. Front Neurosci, 2019, 13: 855.
13.	Shrestha S B, Orchard G. Slayer: Spike layer error reassignment in time. ArXiv preprint arXiv, 2018: 1810.08646.
14.	She X, Mukhopadhyay S. SPEED: Spiking neural network with event-driven unsupervised learning and near-real-time inference for event-based vision. IEEE Sens J, 2021, 21(18): 20578-20588.
15.	Zhao B, Ding R, Chen S, et al. Feedforward categorization on AER motion events using cortex-like features in a spiking neural network. IEEE Trans Neural Netw Learn Syst, 2014, 26(9): 1963-1978.
16.	Xiao R, Tang H, Ma Y, et al. An event-driven categorization model for aer image sensors using multispike encoding and learning. IEEE Trans Neural Netw Learn Syst, 2019, 31(9): 3649-3657.
17.	肖蓉. 脉冲神经网络编码和学习算法及应用研究. 成都: 四川大学, 2021.
18.	Liu Q, Pan G, Ruan H, et al. Unsupervised AER object recognition based on multiscale spatio-temporal features and spiking neurons. IEEE Trans Neural Netw Learn Syst, 2020, 31(12): 5300-5311.
19.	Orchard G, Meyer C, Etienne-Cummings R, et al. HFirst: A temporal approach to object recognition. IEEE Trans Pattern Anal Mach Intell, 2015, 37(10): 2028-2040.
20.	Mozafari M, Kheradpisheh S R, Masquelier T, et al. First-spike-based visual categorization using reward-modulated STDP. IEEE Trans Neural Netw Learn Syst, 2018, 29(12): 6178-6190.
21.	Meliza C D, Dan Y. Receptive-field modification in rat visual cortex induced by paired visual stimulation and single-cell spiking. Neuron, 2006, 49(2): 183-189.
22.	Masquelier T, Guyonneau R, Thorpe S J. Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains. PLoS One, 2008, 3(1): e1377.
23.	Florian R V. Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput, 2007, 19(6): 1468-1502.
24.	吕梦雨. 基于类脑的脉冲神经网络研究及其应用. 桂林: 桂林电子科技大学, 2021.
25.	Serre T, Wolf L, Bileschi S, et al. Robust object recognition with cortex-like mechanisms. IEEE Trans Pattern Anal Mach Intell, 2007, 29(3): 411-426.
26.	郭磊, 吕欢, 黄凤荣, 等. 基于突触可塑性的无标度脉冲神经网络的动态特性研究. 生物医学工程学杂志, 2019, 36(6): 902-910.
27.	Markram H, Gerstner W, Sjöström P J. Spike-timing-dependent plasticity: a comprehensive overview. Front Synaptic Neurosci, 2012, 4: 2.
28.	张慧港, 徐桂芝, 郭嘉荣, 等. 类脑脉冲神经网络及其神经形态芯片研究综述. 生物医学工程学杂志, 2021, 38(5): 986-994, 1002.
29.	Brzosko Z, Zannone S, Schultz W, et al. Sequential neuromodulation of Hebbian plasticity offers mechanism for effective reward-based navigation. Elife, 2017, 6: e27756.
30.	Srivastava N, Hinton G, Krizhevsky A, et al. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res, 2014, 15(1): 1929-1958.
31.	Serrano-Gotarredona T, Linares-Barranco B. Poker-DVS and MNIST-DVS. Their history, how they were made, and other details. Front Neurosci, 2015, 9: 481.
32.	Orchard G, Jayawant A, Cohen G K, et al. Converting static image datasets to spiking neuromorphic datasets using saccades. Front Neurosci, 2015, 9: 437.
33.	Sironi A, Brambilla M, Bourdis N, et al. HATS: Histograms of averaged time surfaces for robust event-based object classification// 2018 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City: IEEE, 2018: 1731-1740.
34.	Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proc IEEE, 1998, 86(11): 2278-2324.

1. Bloss R. Latest in VISION SENSOR technology as well as innovations in sensing, pressure, force, medical, particle size and many other applications. Sensor Rev, 2017, 37(1): 7-11.
2. Posch C, Serrano-Gotarredona T, Linares-Barranco B, et al. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc IEEE, 2014, 102(10): 1470-1484.
3. Duwek H C, Shalumov A, Tsur E E. Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks// 2021 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 2021: 1333-1341.
4. Gallego G, Delbrück T, Orchard G, et al. Event-based vision: A survey. IEEE Trans Pattern Anal Mach Intell, 2020, 44(1): 154-180.
5. Chen G, Hong L, Dong J, et al. EDDD: Event-based drowsiness driving detection through facial motion analysis with neuromorphic vision sensor. IEEE Sens J, 2020, 20(11): 6170-6181.
6. Lee C, Kosta A K, Zhu A Z, et al. Spike-FlowNet: event-based optical flow estimation with energy-efficient hybrid neural networks// Vedaldi A, Bischof H, Brox T. 2020 European Conference on Computer Vision (ECCV). Cham: Springer, 2020: 366-382.
7. Glover A, Vasco V, Bartolozzi C. A controlled-delay event camera framework for on-line robotics// 2018 IEEE International Conference on Robotics and Automation(ICRA). Brisbane: IEEE, 2018: 2178-2183.
8. Chen G, Cao H, Conradt J, et al. Event-based neuromorphic vision for autonomous driving: a paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Process Mag, 2020, 37(4): 34-49.
9. Neil D, Liu S C. Effective sensor fusion with event-based sensors and deep network architectures// 2016 IEEE International Symposium on Circuits and Systems (ISCAS). Montreal: IEEE, 2016: 2282-2285.
10. Zhu A Z, Yuan L, Chaney K, et al. EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv, 2018: 1802.06898.
11. 李杰. 基于深度卷积神经网络的动态手势识别. 济南: 山东大学, 2019.
12. Fang Y, Wang Z, Gomez J, et al. A swarm optimization solver based on ferroelectric spiking neural networks. Front Neurosci, 2019, 13: 855.
13. Shrestha S B, Orchard G. Slayer: Spike layer error reassignment in time. ArXiv preprint arXiv, 2018: 1810.08646.
14. She X, Mukhopadhyay S. SPEED: Spiking neural network with event-driven unsupervised learning and near-real-time inference for event-based vision. IEEE Sens J, 2021, 21(18): 20578-20588.
15. Zhao B, Ding R, Chen S, et al. Feedforward categorization on AER motion events using cortex-like features in a spiking neural network. IEEE Trans Neural Netw Learn Syst, 2014, 26(9): 1963-1978.
16. Xiao R, Tang H, Ma Y, et al. An event-driven categorization model for aer image sensors using multispike encoding and learning. IEEE Trans Neural Netw Learn Syst, 2019, 31(9): 3649-3657.
17. 肖蓉. 脉冲神经网络编码和学习算法及应用研究. 成都: 四川大学, 2021.
18. Liu Q, Pan G, Ruan H, et al. Unsupervised AER object recognition based on multiscale spatio-temporal features and spiking neurons. IEEE Trans Neural Netw Learn Syst, 2020, 31(12): 5300-5311.
19. Orchard G, Meyer C, Etienne-Cummings R, et al. HFirst: A temporal approach to object recognition. IEEE Trans Pattern Anal Mach Intell, 2015, 37(10): 2028-2040.
20. Mozafari M, Kheradpisheh S R, Masquelier T, et al. First-spike-based visual categorization using reward-modulated STDP. IEEE Trans Neural Netw Learn Syst, 2018, 29(12): 6178-6190.
21. Meliza C D, Dan Y. Receptive-field modification in rat visual cortex induced by paired visual stimulation and single-cell spiking. Neuron, 2006, 49(2): 183-189.
22. Masquelier T, Guyonneau R, Thorpe S J. Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains. PLoS One, 2008, 3(1): e1377.
23. Florian R V. Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput, 2007, 19(6): 1468-1502.
24. 吕梦雨. 基于类脑的脉冲神经网络研究及其应用. 桂林: 桂林电子科技大学, 2021.
25. Serre T, Wolf L, Bileschi S, et al. Robust object recognition with cortex-like mechanisms. IEEE Trans Pattern Anal Mach Intell, 2007, 29(3): 411-426.
26. 郭磊, 吕欢, 黄凤荣, 等. 基于突触可塑性的无标度脉冲神经网络的动态特性研究. 生物医学工程学杂志, 2019, 36(6): 902-910.
27. Markram H, Gerstner W, Sjöström P J. Spike-timing-dependent plasticity: a comprehensive overview. Front Synaptic Neurosci, 2012, 4: 2.
28. 张慧港, 徐桂芝, 郭嘉荣, 等. 类脑脉冲神经网络及其神经形态芯片研究综述. 生物医学工程学杂志, 2021, 38(5): 986-994, 1002.
29. Brzosko Z, Zannone S, Schultz W, et al. Sequential neuromodulation of Hebbian plasticity offers mechanism for effective reward-based navigation. Elife, 2017, 6: e27756.
30. Srivastava N, Hinton G, Krizhevsky A, et al. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res, 2014, 15(1): 1929-1958.
31. Serrano-Gotarredona T, Linares-Barranco B. Poker-DVS and MNIST-DVS. Their history, how they were made, and other details. Front Neurosci, 2015, 9: 481.
32. Orchard G, Jayawant A, Cohen G K, et al. Converting static image datasets to spiking neuromorphic datasets using saccades. Front Neurosci, 2015, 9: 437.
33. Sironi A, Brambilla M, Bourdis N, et al. HATS: Histograms of averaged time surfaces for robust event-based object classification// 2018 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City: IEEE, 2018: 1731-1740.
34. Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proc IEEE, 1998, 86(11): 2278-2324.

Previous Article
Recognition of high-frequency steady-state visual evoked potential for brain-computer interface
Next Article
Alterations of β-γ coupling of scalp electroencephalography during epilepsy

Journal of Biomedical Engineering

A bio-inspired hierarchical spiking neural network with biological synaptic plasticity for event camera object recognition

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content