Convolutional neural network based on temporal-spatial feature learning for motor imagery electroencephalogram signal decoding_Journal of Biomedical Engineering

Authors：

CHU Yaqi ^1,2,3 , ZHU Bo ^1,2,3 ,  ZHAO Xingang ^1,2 , ZHAO Yiwen ^1,2

1. State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, P.R.China;
2. Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110016, P.R.China;
3. University of Chinese Academy of Sciences (UCAS), Beijing 100049, P.R.China;

Corresponding author：

ZHAO Xingang, Email: zhaoxingang@sia.cn

Keywords：

motor imagery electroencephalogram; brain-computer interface; temporal-spatial feature; convolutional neural network; signal decoding

DOI：

10.7507/1001-5515.202007006

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

With the advantage of providing more natural and flexible control manner, brain-computer interface systems based on motor imagery electroencephalogram (EEG) have been widely used in the field of human-machine interaction. However, due to the lower signal-noise ratio and poor spatial resolution of EEG signals, the decoding accuracy is relative low. To solve this problem, a novel convolutional neural network based on temporal-spatial feature learning (TSCNN) was proposed for motor imagery EEG decoding. Firstly, for the EEG signals preprocessed by band-pass filtering, a temporal-wise convolution layer and a spatial-wise convolution layer were respectively designed, and temporal-spatial features of motor imagery EEG were constructed. Then, 2-layer two-dimensional convolutional structures were adopted to learn abstract features from the raw temporal-spatial features. Finally, the softmax layer combined with the fully connected layer were used to perform decoding task from the extracted abstract features. The experimental results of the proposed method on the open dataset showed that the average decoding accuracy was 80.09%, which is approximately 13.75% and 10.99% higher than that of the state-of-the-art common spatial pattern (CSP) + support vector machine (SVM) and filter bank CSP (FBCSP) + SVM recognition methods, respectively. This demonstrates that the proposed method can significantly improve the reliability of motor imagery EEG decoding.

Citation： CHU Yaqi, ZHU Bo, ZHAO Xingang, ZHAO Yiwen. Convolutional neural network based on temporal-spatial feature learning for motor imagery electroencephalogram signal decoding. Journal of Biomedical Engineering, 2021, 38(1): 1-9. doi: 10.7507/1001-5515.202007006 Copy

1.	Rohm M, Schneiders M, Müller C, et al. Hybrid brain-computer interfaces and hybrid neuroprostheses for restoration of upper limb functions in individuals with high-level spinal cord injury. Artif Intell Med, 2013, 59(2): 133-142.
2.	López-Larraz E, Trincado-Alonso F, Rajasekaran V, et al. Control of an ambulatory exoskeleton with a brain-machine interface for spinal cord injury gait rehabilitation. Front Neurosci, 2016, 10: 359.
3.	Meng J, Zhang S, Bekyo A, et al. Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks. Sci Rep, 2016, 6: 38565.
4.	Doud A J, Lucas J P, Pisansky M T, et al. Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface. PloS One, 2011, 6(10): 263-296.
5.	Chaudhary U, Birbaumer N, Ramos-Murguialday A. Brain–computer interfaces for communication and rehabilitation. Nat Rev Neurol, 2016, 12(9): 513.
6.	Pfurtscheller G, Brunner C, Schlögl A, et al. Mu rhythm (de) synchronization and EEG single-trial classification of different motor imagery tasks. NeuroImage, 2006, 31(1): 153-159.
7.	Kato K, Takahashi K, Mizuguchi N, et al. Online detection of amplitude modulation of motor-related EEG desynchronization using a lock-in amplifier: Comparison with a fast Fourier transform, a continuous wavelet transform, and an autoregressive algorithm. J Neurosci Meth, 2018, 293: 289-298.
8.	Taran S, Bajaj V. Motor imagery tasks-based EEG signals classification using tunable-Q wavelet transform. Neural Comput Appl, 2019, 31: 6925-6932.
9.	Kevric J, Subasi A. Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomed Signal Proces, 2017, 31: 398-406.
10.	Lotte F, Guan C. Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. IEEE Trans Biomed Eng, 2010, 58(2): 355-362.
11.	Lotte F, Bougrain L, Cichocki A, et al. A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng, 2018, 15(3): 031005.
12.	Chu Yaqi, Zhao Xingang, Zou Yijun, et al. Decoding multiclass motor imagery EEG from the same upper limb by combining Riemannian geometry features and partial least squares regression. J Neural Eng, 2020, 17(4): 046029.
13.	Duan Lijuan, Bao Menghu, Cui Song, et al. Motor imagery EEG classification based on kernel hierarchical extreme learning machine. Cogn Comput, 2017, 9(6): 758-765.
14.	Shin H C, Roth H R, Gao M, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298.
15.	Al-Ayyoub M, Nuseir A, Alsmearat K, et al. Deep learning for Arabic NLP: A survey. J Comput Sci, 2018, 26: 522-531.
16.	Garland J, Gregg D. Low complexity multiply accumulate unit for weight-sharing convolutional neural networks. IEEE Comput Archit L, 2017, 16(2): 132-135.
17.	Tang Z, Li C, Sun S. Single-trial EEG classification of motor imagery using deep convolutional neural networks. Optik, 2017, 130: 11-18.
18.	孔祥浩, 马琳, 薄洪健, 等. CNN 与 CSP 相结合的脑电特征提取与识别方法研究. 信号处理, 2018, 34(2): 164-173.
19.	Pérez-Zapata A F, Cardona-Escobar A F, Jaramillo-Garzón J A, et al. Deep Convolutional Neural Networks and power spectral density features for Motor Imagery classification of EEG Signals// 2018 International Conference on Augmented Cognition. Cham: Springer, 2018: 158-169.
20.	Sakhavi S, Guan C, Yan S. Learning temporal information for brain-computer interface using convolutional neural networks. IEEE Trans Neur Net Lear, 2018, 29(11): 5619-5629.
21.	胡章芳, 张力, 黄丽嘉, 等. 基于时频域的卷积神经网络运动想象脑电信号识别方法. 计算机应用, 2019, 39(8): 2480-2483.
22.	Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
23.	Tabar Y R, Halici U. A novel deep learning approach for classification of EEG motor imagery signals. J Neural Eng, 2016, 14(1): 016003.
24.	Delorme A, Mullen T, Kothe C, et al. EEGLAB, SIFT, NFT, BCILAB, and ERICA: new tools for advanced EEG processing. Comput Intel Neurosc, 2011, 2011: 10.
25.	Ang K K, Chin Z Y, Wang C, et al. Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front Neurosci, 2012, 6: 39.
26.	Wu Wei, Chen Zhe, Gao Xiaorong, et al. Probabilistic common spatial patterns for multichannel EEG analysis. IEEE Trans Pattern Anal, 2014, 37(3): 639-653.
27.	Quitadamo L R, Cavrini F, Sbernini L, et al. Support vector machines to detect physiological patterns for EEG and EMG-based human–computer interaction: a review. J Neural Eng, 2017, 14(1): 011001.
28.	LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436.
29.	Billinger M, Daly I, Kaiser V, et al. Is it significant? Guidelines for reporting BCI performance// 2012 Towards Practical Brain-Computer Interfaces. Berlin: Springer, 2012: 333-354.
30.	Wright S J. Coordinate descent algorithms. Math Program, 2015, 151(1): 3-34.

1. Rohm M, Schneiders M, Müller C, et al. Hybrid brain-computer interfaces and hybrid neuroprostheses for restoration of upper limb functions in individuals with high-level spinal cord injury. Artif Intell Med, 2013, 59(2): 133-142.
2. López-Larraz E, Trincado-Alonso F, Rajasekaran V, et al. Control of an ambulatory exoskeleton with a brain-machine interface for spinal cord injury gait rehabilitation. Front Neurosci, 2016, 10: 359.
3. Meng J, Zhang S, Bekyo A, et al. Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks. Sci Rep, 2016, 6: 38565.
4. Doud A J, Lucas J P, Pisansky M T, et al. Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface. PloS One, 2011, 6(10): 263-296.
5. Chaudhary U, Birbaumer N, Ramos-Murguialday A. Brain–computer interfaces for communication and rehabilitation. Nat Rev Neurol, 2016, 12(9): 513.
6. Pfurtscheller G, Brunner C, Schlögl A, et al. Mu rhythm (de) synchronization and EEG single-trial classification of different motor imagery tasks. NeuroImage, 2006, 31(1): 153-159.
7. Kato K, Takahashi K, Mizuguchi N, et al. Online detection of amplitude modulation of motor-related EEG desynchronization using a lock-in amplifier: Comparison with a fast Fourier transform, a continuous wavelet transform, and an autoregressive algorithm. J Neurosci Meth, 2018, 293: 289-298.
8. Taran S, Bajaj V. Motor imagery tasks-based EEG signals classification using tunable-Q wavelet transform. Neural Comput Appl, 2019, 31: 6925-6932.
9. Kevric J, Subasi A. Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomed Signal Proces, 2017, 31: 398-406.
10. Lotte F, Guan C. Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. IEEE Trans Biomed Eng, 2010, 58(2): 355-362.
11. Lotte F, Bougrain L, Cichocki A, et al. A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng, 2018, 15(3): 031005.
12. Chu Yaqi, Zhao Xingang, Zou Yijun, et al. Decoding multiclass motor imagery EEG from the same upper limb by combining Riemannian geometry features and partial least squares regression. J Neural Eng, 2020, 17(4): 046029.
13. Duan Lijuan, Bao Menghu, Cui Song, et al. Motor imagery EEG classification based on kernel hierarchical extreme learning machine. Cogn Comput, 2017, 9(6): 758-765.
14. Shin H C, Roth H R, Gao M, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging, 2016, 35(5): 1285-1298.
15. Al-Ayyoub M, Nuseir A, Alsmearat K, et al. Deep learning for Arabic NLP: A survey. J Comput Sci, 2018, 26: 522-531.
16. Garland J, Gregg D. Low complexity multiply accumulate unit for weight-sharing convolutional neural networks. IEEE Comput Archit L, 2017, 16(2): 132-135.
17. Tang Z, Li C, Sun S. Single-trial EEG classification of motor imagery using deep convolutional neural networks. Optik, 2017, 130: 11-18.
18. 孔祥浩, 马琳, 薄洪健, 等. CNN 与 CSP 相结合的脑电特征提取与识别方法研究. 信号处理, 2018, 34(2): 164-173.
19. Pérez-Zapata A F, Cardona-Escobar A F, Jaramillo-Garzón J A, et al. Deep Convolutional Neural Networks and power spectral density features for Motor Imagery classification of EEG Signals// 2018 International Conference on Augmented Cognition. Cham: Springer, 2018: 158-169.
20. Sakhavi S, Guan C, Yan S. Learning temporal information for brain-computer interface using convolutional neural networks. IEEE Trans Neur Net Lear, 2018, 29(11): 5619-5629.
21. 胡章芳, 张力, 黄丽嘉, 等. 基于时频域的卷积神经网络运动想象脑电信号识别方法. 计算机应用, 2019, 39(8): 2480-2483.
22. Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
23. Tabar Y R, Halici U. A novel deep learning approach for classification of EEG motor imagery signals. J Neural Eng, 2016, 14(1): 016003.
24. Delorme A, Mullen T, Kothe C, et al. EEGLAB, SIFT, NFT, BCILAB, and ERICA: new tools for advanced EEG processing. Comput Intel Neurosc, 2011, 2011: 10.
25. Ang K K, Chin Z Y, Wang C, et al. Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front Neurosci, 2012, 6: 39.
26. Wu Wei, Chen Zhe, Gao Xiaorong, et al. Probabilistic common spatial patterns for multichannel EEG analysis. IEEE Trans Pattern Anal, 2014, 37(3): 639-653.
27. Quitadamo L R, Cavrini F, Sbernini L, et al. Support vector machines to detect physiological patterns for EEG and EMG-based human–computer interaction: a review. J Neural Eng, 2017, 14(1): 011001.
28. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436.
29. Billinger M, Daly I, Kaiser V, et al. Is it significant? Guidelines for reporting BCI performance// 2012 Towards Practical Brain-Computer Interfaces. Berlin: Springer, 2012: 333-354.
30. Wright S J. Coordinate descent algorithms. Math Program, 2015, 151(1): 3-34.

Next Article
A heart sound classification method based on joint decision of extreme gradient boosting and deep neural network

Journal of Biomedical Engineering

Convolutional neural network based on temporal-spatial feature learning for motor imagery electroencephalogram signal decoding

Abstract Full text Figures/Tables Video References Cited by

Next Article

Format

Content