Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals_Journal of Biomedical Engineering

Authors：

 YANG Wenyang ^1,2 , XU Kexin ¹

1. College of Computer, Xi'an Shiyou University, Xi’an 710065, P. R. China;
2. Key Laboratory of Modern Teaching Technology, Ministry of Education, Xi’an 710062, P. R. China;

Corresponding author：

Keywords：

Emotion recognition; Electroencephalogram signal; Virtual reality technology; Machine learning

DOI：

10.7507/1001-5515.202310045

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Emotion recognition refers to the process of determining and identifying an individual's current emotional state by analyzing various signals such as voice, facial expressions, and physiological indicators etc. Using electroencephalogram (EEG) signals and virtual reality (VR) technology for emotion recognition research helps to better understand human emotional changes, enabling applications in areas such as psychological therapy, education, and training to enhance people’s quality of life. However, there is a lack of comprehensive review literature summarizing the combined researches of EEG signals and VR environments for emotion recognition. Therefore, this paper summarizes and synthesizes relevant research from the past five years. Firstly, it introduces the relevant theories of VR and EEG signal emotion recognition. Secondly, it focuses on the analysis of emotion induction, feature extraction, and classification methods in emotion recognition using EEG signals within VR environments. The article concludes by summarizing the research’s application directions and providing an outlook on future development trends, aiming to serve as a reference for researchers in related fields.

Citation： YANG Wenyang, XU Kexin. Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals. Journal of Biomedical Engineering, 2024, 41(2): 389-397. doi: 10.7507/1001-5515.202310045 Copy

1.	Jafari M, Shoeibi A, Khodatars M, et al. Emotion recognition in EEG signals using deep learning methods: a review. Computers in Biology and Medicine, 2023, 165: 107450.
2.	Valentina M, Francesca B, Chiara S, et al. How do emotions elicited in virtual reality affect our memory? a systematic review. Computers in Human Behavior, 2023, 146: 107812.
3.	杨俊峰. 虚拟现实产业迈入快车道. 人民日报海外版, 2022. [2022-11-23(8)]. DOI: 10.28656/n.cnki.nrmrh.2022.004035.
4.	Hofmann S M, Klotzsche F, Mariola A, et al. Decoding subjective emotional arousal from EEG during an immersive virtual reality experience. Elife, 2021, 10: e64812.
5.	Ekman, P. Basic emotions. Handbook of Cognition and Emotion, 1999, 98(16): 45-60.
6.	Xie J, Lan P, Wang S, et al. Brain activation differences of six basic emotions between 2D screen and virtual reality modalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022. DOI: 10.1109/TNSRE.2022.3229389.
7.	Russell J A. A circumplex model of affect. Journal of Personality and Social Psychology, 1980, 39(6): 1161-1178.
8.	Ricci G, De Crescenzio F, Santhosh S, et al. Relationship between electroencephalographic data and comfort perception captured in a virtual reality design environment of an aircraft cabin. Scientific Reports, 2022, 12(1): 10938.
9.	Piacrd R W, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Ma chine Intelligence, 2001, 23(10): 1175-1191.
10.	Zhang J, Xu Z, Zhou Y, et al. An empirical comparative study on the two methods of eliciting singers’ emotions in singing: self-imagination and VR training. Frontiers in Neuroscience, 2021, 15: 693468.
11.	Sofian N S, James M, Jason T. A dataset for emotion recognition using virtual reality and EEG (DER-VREEG): emotional state classification using low-cost wearable VR-EEG headsets. Big Data and Cognitive Computing, 2022, 6(1): 16.
12.	聂玮, 贾江旭, 王敉敉, 等. 基于脑电实验的虚拟现实环境全景绿视率对人体愉悦度的影响研究. 景观设计学(中英文), 2022, 10(2): 36-51.
13.	Namazi H, Babini M H, Kuca K, et al. Information and memory-based analysis for decoding of the human learning between normal and virtual reality (VR) conditions. Fractals, 2021, 29(3): 2150163.
14.	王晨. 不同沉浸度的虚拟自然场景对人情绪和恢复性效益的影响研究. 北京: 北京建筑大学, 2023.
15.	Peng Y, Jin F, Kong W, et al. OGSSL: a semi-supervised classification model coupled with optimal graph learning for EEG emotion recognition. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 1288-1297.
16.	Raheel A, Majid M, Anwar S M, et al. Emotion classification in response to tactile enhanced multimedia using frequency domain features of brain signals//2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin: IEEE, 2019: 1201-1204.
17.	Cao X, Zhao K, Xu D. Emotion recognition of single-electrode EEG based on multi-feature combination in time-frequency domain. Journal of Physics: Conference Series, 2021, 1827: 012031.
18.	毛玲, 魏士松, 张灵维, 等. 面向飞行模拟器的立体视觉刺激脑机接口系统. 中南大学学报(自然科学版), 2022, 53(8): 2946-2954.
19.	莫云. 基于CSP变换和滤波器组的对数带通功率特征提取方法. 电子测量技术, 2021, 44(10): 33-38.
20.	Hou H R, Zhang X N, Meng Q H. Odor-induced emotion recognition based on average frequency band division of EEG signals. Journal of Neuroscience Methods, 2020, 334: 108599.
21.	Suhaimi N S, Teo J, Mountstephens J. Emotional state classification in virtual reality using wearable electroencephalography. IOP Conference Series: Materials Science and Engineering, 2018, 341: 012016.
22.	王发旺, 陈睿, 伏云发. 基于DBN和RF的跨被试情绪识别研究. 南京大学学报(自然科学), 2021, 57(4): 617-626.
23.	Ömer T, Siraç M Ö. The convolutional neural network approach from electroencephalogram signals in emotional detection. Concurrency and Computation: Practice and Experience, 2021, 33(20): e6356.
24.	Algarni M, Saeed F, Al-Hadhrami T, et al. Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM). Sensors, 2022, 22(8): 2976.
25.	Kalpana M C, J. A, Jude D H. Emotion recognition from EEG signals using recurrent neural networks. Electronics, 2022, 11(15): 2387.
26.	赵丹丹, 赵倩, 董宜先, 等. 基于EEG和DE-CNN-GRU的情绪识别. 计算机系统应用, 2023, 32(4): 206-213.
27.	Tan W, Xu Y, Liu P, et al. A method of VR-EEG scene cognitive rehabilitation training. Health Information Science and Systems, 2021, 9(1): 4.
28.	Liu C, Zhang Y L, Sun L M, et al. The effect of classroom wall color on learning performance: a virtual reality experiment. Building Simulation, 2022, 15(12): 2019-2030.
29.	杨晓哲, 任友群. 虚拟现实与脑电联动系统的开发及其教育研究功能探索. 远程教育杂志, 2019, 37(1): 45-52.
30.	Yokota Y, Naruse Y. Temporal fluctuation of mood in gaming task modulates feedback negativity: EEG study with virtual reality. Frontiers in Human Neuroscience, 2021, 15: 536288.
31.	陆钰鑫. VR游戏类型、交互方式以及网络时延对玩家认知情绪体验和眩晕感的影响研究. 天津: 天津师范大学, 2021.
32.	Armin M, Cruz-Garza J G, Saleh K. Enhancing lighting design through the investigation of illuminance and correlated color temperature's effects on brain activity: an EEG-VR approach. Journal of Building Engineering, 2023, 75: 106776.
33.	Schnack A, Wright J M, Holdershaw L J. Does the locomotion technique matter in an immersive virtual store environment?-Comparing motion-tracked walking and instant teleportation. Journal of Retailing and Consumer Services, 2021, 58: 102266.
34.	Byoungho J, Gwia K, Marguerite M, et al. Consumer store experience through virtual reality: its effect on emotional states and perceived store attractiveness. Fashion and Textiles, 2021, 8(1): 1-21.
35.	Li J P, Qiu S, Shen Y Y, et al. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Transactions on Cybernetics, 2019, 50(7): 3281-3293.
36.	郭苗苗, 陈昕彤, 王磊, 等. 子域自适应网络跨被试情绪识别算法. 信号处理, 2022, 38(10): 2211-2220.
37.	杨文阳, 张文瑄. 基于脑电信号的注意力水平评价研究进展. 生物医学工程学杂志, 2023, 40(4): 820-828.
38.	Cai J, Xiao R, Cui W, et al. Application of electroencephalography-based machine learning in emotion recognition: a review. Frontiers in Systems Neuroscience, 2021, 15: 729707.

1. Jafari M, Shoeibi A, Khodatars M, et al. Emotion recognition in EEG signals using deep learning methods: a review. Computers in Biology and Medicine, 2023, 165: 107450.
2. Valentina M, Francesca B, Chiara S, et al. How do emotions elicited in virtual reality affect our memory? a systematic review. Computers in Human Behavior, 2023, 146: 107812.
3. 杨俊峰. 虚拟现实产业迈入快车道. 人民日报海外版, 2022. [2022-11-23(8)]. DOI: 10.28656/n.cnki.nrmrh.2022.004035.
4. Hofmann S M, Klotzsche F, Mariola A, et al. Decoding subjective emotional arousal from EEG during an immersive virtual reality experience. Elife, 2021, 10: e64812.
5. Ekman, P. Basic emotions. Handbook of Cognition and Emotion, 1999, 98(16): 45-60.
6. Xie J, Lan P, Wang S, et al. Brain activation differences of six basic emotions between 2D screen and virtual reality modalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022. DOI: 10.1109/TNSRE.2022.3229389.
7. Russell J A. A circumplex model of affect. Journal of Personality and Social Psychology, 1980, 39(6): 1161-1178.
8. Ricci G, De Crescenzio F, Santhosh S, et al. Relationship between electroencephalographic data and comfort perception captured in a virtual reality design environment of an aircraft cabin. Scientific Reports, 2022, 12(1): 10938.
9. Piacrd R W, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Ma chine Intelligence, 2001, 23(10): 1175-1191.
10. Zhang J, Xu Z, Zhou Y, et al. An empirical comparative study on the two methods of eliciting singers’ emotions in singing: self-imagination and VR training. Frontiers in Neuroscience, 2021, 15: 693468.
11. Sofian N S, James M, Jason T. A dataset for emotion recognition using virtual reality and EEG (DER-VREEG): emotional state classification using low-cost wearable VR-EEG headsets. Big Data and Cognitive Computing, 2022, 6(1): 16.
12. 聂玮, 贾江旭, 王敉敉, 等. 基于脑电实验的虚拟现实环境全景绿视率对人体愉悦度的影响研究. 景观设计学(中英文), 2022, 10(2): 36-51.
13. Namazi H, Babini M H, Kuca K, et al. Information and memory-based analysis for decoding of the human learning between normal and virtual reality (VR) conditions. Fractals, 2021, 29(3): 2150163.
14. 王晨. 不同沉浸度的虚拟自然场景对人情绪和恢复性效益的影响研究. 北京: 北京建筑大学, 2023.
15. Peng Y, Jin F, Kong W, et al. OGSSL: a semi-supervised classification model coupled with optimal graph learning for EEG emotion recognition. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 1288-1297.
16. Raheel A, Majid M, Anwar S M, et al. Emotion classification in response to tactile enhanced multimedia using frequency domain features of brain signals//2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin: IEEE, 2019: 1201-1204.
17. Cao X, Zhao K, Xu D. Emotion recognition of single-electrode EEG based on multi-feature combination in time-frequency domain. Journal of Physics: Conference Series, 2021, 1827: 012031.
18. 毛玲, 魏士松, 张灵维, 等. 面向飞行模拟器的立体视觉刺激脑机接口系统. 中南大学学报(自然科学版), 2022, 53(8): 2946-2954.
19. 莫云. 基于CSP变换和滤波器组的对数带通功率特征提取方法. 电子测量技术, 2021, 44(10): 33-38.
20. Hou H R, Zhang X N, Meng Q H. Odor-induced emotion recognition based on average frequency band division of EEG signals. Journal of Neuroscience Methods, 2020, 334: 108599.
21. Suhaimi N S, Teo J, Mountstephens J. Emotional state classification in virtual reality using wearable electroencephalography. IOP Conference Series: Materials Science and Engineering, 2018, 341: 012016.
22. 王发旺, 陈睿, 伏云发. 基于DBN和RF的跨被试情绪识别研究. 南京大学学报(自然科学), 2021, 57(4): 617-626.
23. Ömer T, Siraç M Ö. The convolutional neural network approach from electroencephalogram signals in emotional detection. Concurrency and Computation: Practice and Experience, 2021, 33(20): e6356.
24. Algarni M, Saeed F, Al-Hadhrami T, et al. Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM). Sensors, 2022, 22(8): 2976.
25. Kalpana M C, J. A, Jude D H. Emotion recognition from EEG signals using recurrent neural networks. Electronics, 2022, 11(15): 2387.
26. 赵丹丹, 赵倩, 董宜先, 等. 基于EEG和DE-CNN-GRU的情绪识别. 计算机系统应用, 2023, 32(4): 206-213.
27. Tan W, Xu Y, Liu P, et al. A method of VR-EEG scene cognitive rehabilitation training. Health Information Science and Systems, 2021, 9(1): 4.
28. Liu C, Zhang Y L, Sun L M, et al. The effect of classroom wall color on learning performance: a virtual reality experiment. Building Simulation, 2022, 15(12): 2019-2030.
29. 杨晓哲, 任友群. 虚拟现实与脑电联动系统的开发及其教育研究功能探索. 远程教育杂志, 2019, 37(1): 45-52.
30. Yokota Y, Naruse Y. Temporal fluctuation of mood in gaming task modulates feedback negativity: EEG study with virtual reality. Frontiers in Human Neuroscience, 2021, 15: 536288.
31. 陆钰鑫. VR游戏类型、交互方式以及网络时延对玩家认知情绪体验和眩晕感的影响研究. 天津: 天津师范大学, 2021.
32. Armin M, Cruz-Garza J G, Saleh K. Enhancing lighting design through the investigation of illuminance and correlated color temperature's effects on brain activity: an EEG-VR approach. Journal of Building Engineering, 2023, 75: 106776.
33. Schnack A, Wright J M, Holdershaw L J. Does the locomotion technique matter in an immersive virtual store environment?-Comparing motion-tracked walking and instant teleportation. Journal of Retailing and Consumer Services, 2021, 58: 102266.
34. Byoungho J, Gwia K, Marguerite M, et al. Consumer store experience through virtual reality: its effect on emotional states and perceived store attractiveness. Fashion and Textiles, 2021, 8(1): 1-21.
35. Li J P, Qiu S, Shen Y Y, et al. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Transactions on Cybernetics, 2019, 50(7): 3281-3293.
36. 郭苗苗, 陈昕彤, 王磊, 等. 子域自适应网络跨被试情绪识别算法. 信号处理, 2022, 38(10): 2211-2220.
37. 杨文阳, 张文瑄. 基于脑电信号的注意力水平评价研究进展. 生物医学工程学杂志, 2023, 40(4): 820-828.
38. Cai J, Xiao R, Cui W, et al. Application of electroencephalography-based machine learning in emotion recognition: a review. Frontiers in Systems Neuroscience, 2021, 15: 729707.

Previous Article
Progress in the analysis of hemolysis and coagulation models for interventional micro-axial flow blood pumps
Next Article
A review on electroencephalogram based channel selection

Journal of Biomedical Engineering

Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content