Mental fatigue state recognition method based on convolution neural network and long short-term memory_Journal of Biomedical Engineering

Authors：

WANG Hui ¹ , ZHANG Pin ¹ , JIN Fenghu ² , ZHAO Baoyong ¹ , ZENG Qinbo ² ,  XIAO Wendong ^1,3,4

1. School of Automation, University of Science And Technology Beijing, Beijing 100083, P. R. China;
2. China Ordnance Equipment Group Automation Research Institute Co., Mianyang, Sichuan 621000, P. R. China;
3. Shunde Innovation School, University of Science, and Technology Beijing, Shunde, Guangdong 528399, P. R. China;
4. Beijing Engineering Research Center of Industrial Spectrum Imaging, Beijing 100083, P. R. China;

Corresponding author：

XIAO Wendong, Email: wdxiao@ustb.edu.cn

Keywords：

Psychological fatigue; Electrocardiogram signals; Convolution neural network; Long short-term memory

DOI：

10.7507/1001-5515.202306016

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The pace of modern life is accelerating, the pressure of life is gradually increasing, and the long-term accumulation of mental fatigue poses a threat to health. By analyzing physiological signals and parameters, this paper proposes a method that can identify the state of mental fatigue, which helps to maintain a healthy life. The method proposed in this paper is a new recognition method of psychological fatigue state of electrocardiogram signals based on convolutional neural network and long short-term memory. Firstly, the convolution layer of one-dimensional convolutional neural network model is used to extract local features, the key information is extracted through pooling layer, and some redundant data is removed. Then, the extracted features are used as input to the long short-term memory model to further fuse the ECG features. Finally, by integrating the key information through the full connection layer, the accurate recognition of mental fatigue state is successfully realized. The results show that compared with traditional machine learning algorithms, the proposed method significantly improves the accuracy of mental fatigue recognition to 96.3%, which provides a reliable basis for the early warning and evaluation of mental fatigue.

Citation： WANG Hui, ZHANG Pin, JIN Fenghu, ZHAO Baoyong, ZENG Qinbo, XIAO Wendong. Mental fatigue state recognition method based on convolution neural network and long short-term memory. Journal of Biomedical Engineering, 2024, 41(1): 34-40. doi: 10.7507/1001-5515.202306016 Copy

1.	Lipovac D, Žitnik J, Burnard M D. A pilot study examining the suitability of the mental arithmetic task and single-item measures of affective states to assess affective, physiological, and attention restoration at a wooden desk. Journal of Wood Science, 2022, 68: 35.
2.	Lee K F A, Chan E, Car J, et al. Lowering the sampling rate: Heart rate response during cognitive fatigue. Biosensors, 2022, 12(5): 315.
3.	Matsuura Y, Ochi G. The potential of heart rate variability monitoring for mental health assessment in top wheel gymnastics athletes: a single case design. Applied Psychophysiology and Biofeedback, 2023, 48(3): 335-343.
4.	Wang H, Han M, Avouka T, et al. Research on fatigue identification methods based on low-load wearable ECG monitoring devices. Review of Scientific Instruments, 2023, 94(4): 045103.
5.	Lu K, Sjörs Dahlman A, Karlsson J, et al. Detecting driver fatigue using heart rate variability: a systematic review. Accident Analysis & Prevention, 2022, 178: 106830.
6.	Matuz A, van der Linden D, Darnai G, et al. Generalisable machine learning models trained on heart rate variability data to predict mental fatigue. Scientific Reports, 2022, 12(1): 20023.
7.	Mehmood I, Li H, Qarout Y, et al. Deep learning-based construction equipment operators’ mental fatigue classification using wearable EEG sensor data. Advanced Engineering Informatics, 2023, 56: 101978.
8.	Zorzos I, Kakkos I, Miloulis S T, et al. Applying neural networks with time-frequency features for the detection of mental fatigue. Applied Sciences, 2023, 13(3): 1512.
9.	Wang F, Wan Y, Li M, et al. Recent advances in fatigue detection algorithm based on EEG. Intelligent Automation & Soft Computing, 2023, 35(3): 3573-3586.
10.	Ettahiri H, Vicente J M F, Fechtali T. EEG signals in mental fatigue detection: a comparing study of machine learning technics vs deep learning//International Work-Conference on the Interplay Between Natural and Artificial Computation. Cham: Springer International Publishing, 2022: 625-633.
11.	Ye C, Yin Z, Zhao M, et al. Identification of mental fatigue levels in a language understanding task based on multi-domain EEG features and an ensemble convolutional neural network. Biomedical Signal Processing and Control, 2022, 72: 103360.
12.	Zhang Y, Chen Y, Pan Z. A deep temporal model for mental fatigue detection//2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2018: 1879-1884.
13.	Schmidt P, Reiss A, Duerichen R, et al. Introducing wesad, a multimodal dataset for wearable stress and affect detection//Proceedings of the 20th ACM International Conference on Multimodal Interaction. Boulder: ICMI, 2018: 400-408.
14.	毛玲, 孙即祥, 张国敏, 等. 基于形态滤波的心电信号基线矫正算法. 信号处理, 2008, 24(4): 582-585.
15.	王晓娜, 宋世德. 基于 NI ELVISⅡ 平台设计 Bainter 陷波器抑制工频干扰实验. 实验室科学, 2020, 23(4): 35-38.
16.	梁莹, 马小龙, 朝乐蒙, 等. 基于经验小波变换的心电信号基线漂移噪声去除预处理方法研究. 医疗卫生装备, 2022, 43(6): 7-13.
17.	Wang Y, Deepu C J, Lian Y. A computationally efficient QRS detection algorithm for wearable ECG sensors//2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2011: 5641-5644.
18.	邵宁宁. 基于卷积神经网络的心律失常分类研究. 秦皇岛: 燕山大学, 2021.
19.	武东辉, 许静, 陈继斌等. 基于融合注意力机制与CNN-LSTM的人体行为识别算法. 科学技术与工程, 2023, 23(2): 681-689.
20.	郑涛, 刘辉, 陈薇, 等. 基于CNN-LSTM的水泥熟料f-CaO预测模型. 控制工程, 2022. DOI: 10.14107/j.cnki.kzgc.20220261.
21.	王愈轩, 刘尔佳, 黄永章. 基于CNN-LSTM-lightGBM组合的超短期风电功率预测方法. 科学技术与工程, 2022, 22(36): 16067-16074.

1. Lipovac D, Žitnik J, Burnard M D. A pilot study examining the suitability of the mental arithmetic task and single-item measures of affective states to assess affective, physiological, and attention restoration at a wooden desk. Journal of Wood Science, 2022, 68: 35.
2. Lee K F A, Chan E, Car J, et al. Lowering the sampling rate: Heart rate response during cognitive fatigue. Biosensors, 2022, 12(5): 315.
3. Matsuura Y, Ochi G. The potential of heart rate variability monitoring for mental health assessment in top wheel gymnastics athletes: a single case design. Applied Psychophysiology and Biofeedback, 2023, 48(3): 335-343.
4. Wang H, Han M, Avouka T, et al. Research on fatigue identification methods based on low-load wearable ECG monitoring devices. Review of Scientific Instruments, 2023, 94(4): 045103.
5. Lu K, Sjörs Dahlman A, Karlsson J, et al. Detecting driver fatigue using heart rate variability: a systematic review. Accident Analysis & Prevention, 2022, 178: 106830.
6. Matuz A, van der Linden D, Darnai G, et al. Generalisable machine learning models trained on heart rate variability data to predict mental fatigue. Scientific Reports, 2022, 12(1): 20023.
7. Mehmood I, Li H, Qarout Y, et al. Deep learning-based construction equipment operators’ mental fatigue classification using wearable EEG sensor data. Advanced Engineering Informatics, 2023, 56: 101978.
8. Zorzos I, Kakkos I, Miloulis S T, et al. Applying neural networks with time-frequency features for the detection of mental fatigue. Applied Sciences, 2023, 13(3): 1512.
9. Wang F, Wan Y, Li M, et al. Recent advances in fatigue detection algorithm based on EEG. Intelligent Automation & Soft Computing, 2023, 35(3): 3573-3586.
10. Ettahiri H, Vicente J M F, Fechtali T. EEG signals in mental fatigue detection: a comparing study of machine learning technics vs deep learning//International Work-Conference on the Interplay Between Natural and Artificial Computation. Cham: Springer International Publishing, 2022: 625-633.
11. Ye C, Yin Z, Zhao M, et al. Identification of mental fatigue levels in a language understanding task based on multi-domain EEG features and an ensemble convolutional neural network. Biomedical Signal Processing and Control, 2022, 72: 103360.
12. Zhang Y, Chen Y, Pan Z. A deep temporal model for mental fatigue detection//2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2018: 1879-1884.
13. Schmidt P, Reiss A, Duerichen R, et al. Introducing wesad, a multimodal dataset for wearable stress and affect detection//Proceedings of the 20th ACM International Conference on Multimodal Interaction. Boulder: ICMI, 2018: 400-408.
14. 毛玲, 孙即祥, 张国敏, 等. 基于形态滤波的心电信号基线矫正算法. 信号处理, 2008, 24(4): 582-585.
15. 王晓娜, 宋世德. 基于 NI ELVISⅡ 平台设计 Bainter 陷波器抑制工频干扰实验. 实验室科学, 2020, 23(4): 35-38.
16. 梁莹, 马小龙, 朝乐蒙, 等. 基于经验小波变换的心电信号基线漂移噪声去除预处理方法研究. 医疗卫生装备, 2022, 43(6): 7-13.
17. Wang Y, Deepu C J, Lian Y. A computationally efficient QRS detection algorithm for wearable ECG sensors//2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2011: 5641-5644.
18. 邵宁宁. 基于卷积神经网络的心律失常分类研究. 秦皇岛: 燕山大学, 2021.
19. 武东辉, 许静, 陈继斌等. 基于融合注意力机制与CNN-LSTM的人体行为识别算法. 科学技术与工程, 2023, 23(2): 681-689.
20. 郑涛, 刘辉, 陈薇, 等. 基于CNN-LSTM的水泥熟料f-CaO预测模型. 控制工程, 2022. DOI: 10.14107/j.cnki.kzgc.20220261.
21. 王愈轩, 刘尔佳, 黄永章. 基于CNN-LSTM-lightGBM组合的超短期风电功率预测方法. 科学技术与工程, 2022, 22(36): 16067-16074.

Journal of Biomedical Engineering

Mental fatigue state recognition method based on convolution neural network and long short-term memory

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content