Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks_Journal of Biomedical Engineering

Authors：

WANG Jiaheng ¹ , WANG Yueming ^1,2,3,4 ,  YAO Lin ^1,2

1. School of Computer Science, Zhejiang Universty, Hangzhou 310000, P.R.China;
2. Frontiers Science Center for Brain & Brain-machine Integration, Zhejiang Universty, Hangzhou 310000, P.R.China;
3. Qiushi Academy for Advanced Studies, Zhejiang Universty, Hangzhou 310000, P.R.China;
4. Zhejiang Lab, Hangzhou 310000, P.R.China;

Corresponding author：

YAO Lin, Email: ly329@zju.edu.cn

Keywords：

emotion recognition; feature extraction; long short-term memory; electroencephalogram

DOI：

10.7507/1001-5515.202012054

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Emotion plays an important role in people's cognition and communication. By analyzing electroencephalogram (EEG) signals to identify internal emotions and feedback emotional information in an active or passive way, affective brain-computer interactions can effectively promote human-computer interaction. This paper focuses on emotion recognition using EEG. We systematically evaluate the performance of state-of-the-art feature extraction and classification methods with a public-available dataset for emotion analysis using physiological signals (DEAP). The common random split method will lead to high correlation between training and testing samples. Thus, we use block-wise K fold cross validation. Moreover, we compare the accuracy of emotion recognition with different time window length. The experimental results indicate that 4 s time window is appropriate for sampling. Filter-bank long short-term memory networks (FBLSTM) using differential entropy features as input was proposed. The average accuracy of low and high in valance dimension, arousal dimension and combination of the four in valance-arousal plane is 78.8%, 78.4% and 70.3%, respectively. These results demonstrate the advantage of our emotion recognition model over the current studies in terms of classification accuracy. Our model might provide a novel method for emotion recognition in affective brain-computer interactions.

Citation： WANG Jiaheng, WANG Yueming, YAO Lin. Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks. Journal of Biomedical Engineering, 2021, 38(3): 447-454. doi: 10.7507/1001-5515.202012054 Copy

1.	Gunes H, Schuller B, Pantic M, et al. Emotion representation, analysis and synthesis in continuous space: a survey//2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG 2011). Santa Barbara: IEEE, 2011: 827-834.
2.	Alarcão S M, Fonseca M J. Emotions recognition using EEG signals: a survey. IEEE Trans Affect Comput, 2019, 10(3): 374-393.
3.	Katsigiannis S, Ramzan N. DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform, 2018, 22(1): 98-107.
4.	Koelstra S, Mühl C, Soleymani M, et al. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 2012, 3(1): 18-31.
5.	Zheng Weilong, Lu Baoliang. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev, 2015, 7(3): 162-175.
6.	Petrantonakis P C, Hadjileontiadis L J. Emotion recognition from EEG using higher order crossings. IEEE Trans Inf Technol Biomed, 2010, 14(2): 186-197.
7.	Zheng Weilong, Zhu Jiayi, Lu Baoliang. Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput, 2019, 10(3): 417-429.
8.	Cui Heng, Liu Aiping, Zhang Xu, et al. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowledge-Based Systems, 2020, 205: 106243.
9.	Zheng W L, Liu W, Lu Y, et al. EmotionMeter: a multimodal framework for recognizing human emotions. IEEE Trans Cybern, 2019, 49(3): 1110-1122.
10.	Liu S, Wang X, Zhao L, et al. Subject-independent emotion recognition of EEG signals based on dynamic empirical convolutional neural network. IEEE/ACM Trans Comput Biol Bioinform, 2020.
11.	Liu Yongjin, Yu Minjing, Zhao Guozhen, et al. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE Trans Affect Comput, 2018, 9(4): 550-562.
12.	Li J, Qiu S, Shen Y Y, et al. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Trans Cybern, 2020, 50(7): 3281-3293.
13.	Russell J A. A circumplex model of affect. J Pers Soc Psychol, 1980, 39(6): 1161-1178.
14.	Jenke R, Peer A, Buss M. Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput, 2014, 5(3): 327-339.
15.	Shi Lichen, Jiao Yingying, Lu Baoliang. Differential entropy feature for EEG-based vigilance estimation//2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Osaka: IEEE, 2013: 6627-6630.
16.	Blankertz B, Dornhege G, Krauledat M, et al. The non-invasive Berlin brain-computer interface: fast acquisition of effective performance in untrained subjects. Neuroimage, 2007, 37(2): 539-550.
17.	Gramfort A, Luessi M, Larson E, et al. MNE software for processing MEG and EEG data. Neuroimage, 2014, 86(1): 446-460.
18.	Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput, 1997, 9(8): 1735-1780.
19.	Mühl C, Allison B, Nijholt A, et al. A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges. Brain-Computer Interfaces, 2014, 1(2): 66-84.
20.	Atkinson J, Campos D. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Systems with Applications, 2016, 47(1): 35-41.
21.	Mohammadi Z, Frounchi J, Amiri M. Wavelet-based emotion recognition system using EEG signal. Neural Computing and Applications, 2017, 28(8): 1985-1990.
22.	Zhang Xiaowei, Hu Bin, Chen Jing, et al. Ontology-based context modeling for emotion recognition in an intelligent web. World Wide Web, 2013, 16(4): 497-513.

1. Gunes H, Schuller B, Pantic M, et al. Emotion representation, analysis and synthesis in continuous space: a survey//2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG 2011). Santa Barbara: IEEE, 2011: 827-834.
2. Alarcão S M, Fonseca M J. Emotions recognition using EEG signals: a survey. IEEE Trans Affect Comput, 2019, 10(3): 374-393.
3. Katsigiannis S, Ramzan N. DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform, 2018, 22(1): 98-107.
4. Koelstra S, Mühl C, Soleymani M, et al. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 2012, 3(1): 18-31.
5. Zheng Weilong, Lu Baoliang. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev, 2015, 7(3): 162-175.
6. Petrantonakis P C, Hadjileontiadis L J. Emotion recognition from EEG using higher order crossings. IEEE Trans Inf Technol Biomed, 2010, 14(2): 186-197.
7. Zheng Weilong, Zhu Jiayi, Lu Baoliang. Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput, 2019, 10(3): 417-429.
8. Cui Heng, Liu Aiping, Zhang Xu, et al. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowledge-Based Systems, 2020, 205: 106243.
9. Zheng W L, Liu W, Lu Y, et al. EmotionMeter: a multimodal framework for recognizing human emotions. IEEE Trans Cybern, 2019, 49(3): 1110-1122.
10. Liu S, Wang X, Zhao L, et al. Subject-independent emotion recognition of EEG signals based on dynamic empirical convolutional neural network. IEEE/ACM Trans Comput Biol Bioinform, 2020.
11. Liu Yongjin, Yu Minjing, Zhao Guozhen, et al. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE Trans Affect Comput, 2018, 9(4): 550-562.
12. Li J, Qiu S, Shen Y Y, et al. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Trans Cybern, 2020, 50(7): 3281-3293.
13. Russell J A. A circumplex model of affect. J Pers Soc Psychol, 1980, 39(6): 1161-1178.
14. Jenke R, Peer A, Buss M. Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput, 2014, 5(3): 327-339.
15. Shi Lichen, Jiao Yingying, Lu Baoliang. Differential entropy feature for EEG-based vigilance estimation//2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Osaka: IEEE, 2013: 6627-6630.
16. Blankertz B, Dornhege G, Krauledat M, et al. The non-invasive Berlin brain-computer interface: fast acquisition of effective performance in untrained subjects. Neuroimage, 2007, 37(2): 539-550.
17. Gramfort A, Luessi M, Larson E, et al. MNE software for processing MEG and EEG data. Neuroimage, 2014, 86(1): 446-460.
18. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput, 1997, 9(8): 1735-1780.
19. Mühl C, Allison B, Nijholt A, et al. A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges. Brain-Computer Interfaces, 2014, 1(2): 66-84.
20. Atkinson J, Campos D. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Systems with Applications, 2016, 47(1): 35-41.
21. Mohammadi Z, Frounchi J, Amiri M. Wavelet-based emotion recognition system using EEG signal. Neural Computing and Applications, 2017, 28(8): 1985-1990.
22. Zhang Xiaowei, Hu Bin, Chen Jing, et al. Ontology-based context modeling for emotion recognition in an intelligent web. World Wide Web, 2013, 16(4): 497-513.

Journal of Biomedical Engineering

Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content