Multi-modal physiological time-frequency feature extraction network for accurate sleep stage classification_Journal of Biomedical Engineering

Authors：

HU Kailei ,  CHEN Jingxia , ZHANG Pengwei , XUE Wen , XIE Jia

1. School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi’an 710021, P. R. China;

Corresponding author：

CHEN Jingxia, Email: chenjingxia@sust.edu.cn

Keywords：

Sleep stage classification; Physiological signal; Multi-modal; Multi-scale; Bidirectional gated recurrent units network

DOI：

10.7507/1001-5515.202306010

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Sleep stage classification is essential for clinical disease diagnosis and sleep quality assessment. Most of the existing methods for sleep stage classification are based on single-channel or single-modal signal, and extract features using a single-branch, deep convolutional network, which not only hinders the capture of the diversity features related to sleep and increase the computational cost, but also has a certain impact on the accuracy of sleep stage classification. To solve this problem, this paper proposes an end-to-end multi-modal physiological time-frequency feature extraction network (MTFF-Net) for accurate sleep stage classification. First, multi-modal physiological signal containing electroencephalogram (EEG), electrocardiogram (ECG), electrooculogram (EOG) and electromyogram (EMG) are converted into two-dimensional time-frequency images containing time-frequency features by using short time Fourier transform (STFT). Then, the time-frequency feature extraction network combining multi-scale EEG compact convolution network (Ms-EEGNet) and bidirectional gated recurrent units (Bi-GRU) network is used to obtain multi-scale spectral features related to sleep feature waveforms and time series features related to sleep stage transition. According to the American Academy of Sleep Medicine (AASM) EEG sleep stage classification criterion, the model achieved 84.3% accuracy in the five-classification task on the third subgroup of the Institute of Systems and Robotics of the University of Coimbra Sleep Dataset (ISRUC-S3), with 83.1% macro F1 score value and 79.8% Cohen’s Kappa coefficient. The experimental results show that the proposed model achieves higher classification accuracy and promotes the application of deep learning algorithms in assisting clinical decision-making.

Citation： HU Kailei, CHEN Jingxia, ZHANG Pengwei, XUE Wen, XIE Jia. Multi-modal physiological time-frequency feature extraction network for accurate sleep stage classification. Journal of Biomedical Engineering, 2024, 41(1): 26-33. doi: 10.7507/1001-5515.202306010 Copy

1.	Wolpert E A. A Manual of standardized terminology, techniques and scoring system for sleep stages of human subjects. Arch Gen Psychiatry, 1969, 20(2): 246-247.
2.	Choi J H, Kim E J, Kim Y S, et al. Validation study of portable device for the diagnosis of obstructive sleep apnea according to the new AASM scoring criteria: Watch-PAT 100. Acta Otolaryngol, 2010, 130(7): 838-843.
3.	Dimitriadis S I, Salis C, Linden D. A novel, fast and efficient single-sensor automatic sleep-stage classification based on complementary cross-frequency coupling estimates. Clin Neurophysiol, 2018, 129(4): 815-828.
4.	Alickovic E, Subasi A. Ensemble SVM method for automatic sleep stage classification. IEEE Trans Instrum Meas. 2018, 67(6): 1258–1265.
5.	Klok A B, Edin J, Cesari M, et al. A new fully automated random-forest algorithm for sleep staging//2018 40th Annual International Conference of the ieee Engineering in Medicine and Biology Society (EMBC). IEEE, 2018: 4920-4923.
6.	Seifpour S, Niknazar H, Mikaeili M , et al. A new automatic sleep staging system based on statistical behavior of local extrema using single channel EEG signal. Expert Syst Appl, 2018, 104: 277-293.
7.	Memar P, Faradji F. A novel multi-class EEG-based sleep stage classification system. IEEE Trans Neural Syst Rehabil Eng, 2018, 26(1): 84-95.
8.	Ghimatgar H, Kazemi K, Helfroush M S, et al. An automatic single-channel EEG-based sleep stage scoring method based on hidden Markov model. J Neurosci Methods, 2019, 324: 108320.
9.	Li C, Qi Y, Ding X, et al. A deep learning method approach for sleep stage classification with EEG spectrogram. Int J Environ Res Public Health. 2022; 19(10): 6322.
10.	Fan J, Sun C, Long M, et al. EOGNET: a novel deep learning model for sleep stage classification based on single-channel EOG signal. Front Neurosci, 2021, 15: 573194.
11.	Pei W, Li W, Siuly S, et al. A hybrid deep learning scheme for multi-channel sleep stage classification. Comput Mater Contin, 2022, 71(1): 889-905.
12.	Jia Ziyu, Lin Youfang, Wang Jing, et al. GraphSleepNet: adaptive spatial-temporal graph convolution network for sleep stage classification//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI’20), 2021: 1324-1330.
13.	Jia Ziyu, Lin Youfang, Wang Jing, et al. Multi-view spatial-temporal graph convolutional networks with domain generalization for sleep stage classification. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 1977-1986.
14.	Li Y, Chen J, Ma W, et al. MVF-SleepNet: multi-view fusion network for sleep stage classification. IEEE J Biomed Health Inform, 2022. DOI: 10.1109/JBHI.2022.3208314.
15.	Supratak A, Dong H, Wu C, et al. DeepSleepNet: a model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Trans Neural Syst Rehabil Eng, 2017, 25(11): 1998-2008.
16.	Huang J, Ren L, Zhou X, et al. An improved neural network based on SENet for sleep stage classification. IEEE J Biomed Health Inform, 2022, 26(10): 4948-4956.
17.	Wang H, Lu C, Zhang Q, et al. A novel sleep staging network based on multi-scale dual attention. Biomed Signal Process Control, 2022, 74: 103486.
18.	Huang C, Xiao Y, Xu G. Predicting human intention-behavior through EEG signal analysis using multi-scale CNN. IEEE/ACM Trans Comput Biol Bioinform, 2020, 18(5): 1722-1729.
19.	Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
20.	Altuwaijri G A, Muhammad G. A multibranch of convolutional neural network models for electroencephalogram-based motor imagery classification. Biosensors (Basel), 2022, 12(1): 22.
21.	Cho K, van Merrienboer B, Bahdanau D, et al. On the properties of neural machine translation: encoder-decoder approaches// Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014, arXiv: 1409. 1259.
22.	卢伊虹, 吴礼祝, 潘家辉. 基于CNN-BiLSTM的自动睡眠分期算法. 计算机系统应用, 2022, 31(4): 180-187.
23.	Khalighi S, Sousa T, Santos J M, et al. ISRUC-Sleep: a comprehensive public dataset for sleep researchers. Computer Methods Programs Biomed, 2016, 124: 180-192.

1. Wolpert E A. A Manual of standardized terminology, techniques and scoring system for sleep stages of human subjects. Arch Gen Psychiatry, 1969, 20(2): 246-247.
2. Choi J H, Kim E J, Kim Y S, et al. Validation study of portable device for the diagnosis of obstructive sleep apnea according to the new AASM scoring criteria: Watch-PAT 100. Acta Otolaryngol, 2010, 130(7): 838-843.
3. Dimitriadis S I, Salis C, Linden D. A novel, fast and efficient single-sensor automatic sleep-stage classification based on complementary cross-frequency coupling estimates. Clin Neurophysiol, 2018, 129(4): 815-828.
4. Alickovic E, Subasi A. Ensemble SVM method for automatic sleep stage classification. IEEE Trans Instrum Meas. 2018, 67(6): 1258–1265.
5. Klok A B, Edin J, Cesari M, et al. A new fully automated random-forest algorithm for sleep staging//2018 40th Annual International Conference of the ieee Engineering in Medicine and Biology Society (EMBC). IEEE, 2018: 4920-4923.
6. Seifpour S, Niknazar H, Mikaeili M , et al. A new automatic sleep staging system based on statistical behavior of local extrema using single channel EEG signal. Expert Syst Appl, 2018, 104: 277-293.
7. Memar P, Faradji F. A novel multi-class EEG-based sleep stage classification system. IEEE Trans Neural Syst Rehabil Eng, 2018, 26(1): 84-95.
8. Ghimatgar H, Kazemi K, Helfroush M S, et al. An automatic single-channel EEG-based sleep stage scoring method based on hidden Markov model. J Neurosci Methods, 2019, 324: 108320.
9. Li C, Qi Y, Ding X, et al. A deep learning method approach for sleep stage classification with EEG spectrogram. Int J Environ Res Public Health. 2022; 19(10): 6322.
10. Fan J, Sun C, Long M, et al. EOGNET: a novel deep learning model for sleep stage classification based on single-channel EOG signal. Front Neurosci, 2021, 15: 573194.
11. Pei W, Li W, Siuly S, et al. A hybrid deep learning scheme for multi-channel sleep stage classification. Comput Mater Contin, 2022, 71(1): 889-905.
12. Jia Ziyu, Lin Youfang, Wang Jing, et al. GraphSleepNet: adaptive spatial-temporal graph convolution network for sleep stage classification//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI’20), 2021: 1324-1330.
13. Jia Ziyu, Lin Youfang, Wang Jing, et al. Multi-view spatial-temporal graph convolutional networks with domain generalization for sleep stage classification. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 1977-1986.
14. Li Y, Chen J, Ma W, et al. MVF-SleepNet: multi-view fusion network for sleep stage classification. IEEE J Biomed Health Inform, 2022. DOI: 10.1109/JBHI.2022.3208314.
15. Supratak A, Dong H, Wu C, et al. DeepSleepNet: a model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Trans Neural Syst Rehabil Eng, 2017, 25(11): 1998-2008.
16. Huang J, Ren L, Zhou X, et al. An improved neural network based on SENet for sleep stage classification. IEEE J Biomed Health Inform, 2022, 26(10): 4948-4956.
17. Wang H, Lu C, Zhang Q, et al. A novel sleep staging network based on multi-scale dual attention. Biomed Signal Process Control, 2022, 74: 103486.
18. Huang C, Xiao Y, Xu G. Predicting human intention-behavior through EEG signal analysis using multi-scale CNN. IEEE/ACM Trans Comput Biol Bioinform, 2020, 18(5): 1722-1729.
19. Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
20. Altuwaijri G A, Muhammad G. A multibranch of convolutional neural network models for electroencephalogram-based motor imagery classification. Biosensors (Basel), 2022, 12(1): 22.
21. Cho K, van Merrienboer B, Bahdanau D, et al. On the properties of neural machine translation: encoder-decoder approaches// Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014, arXiv: 1409. 1259.
22. 卢伊虹, 吴礼祝, 潘家辉. 基于CNN-BiLSTM的自动睡眠分期算法. 计算机系统应用, 2022, 31(4): 180-187.
23. Khalighi S, Sousa T, Santos J M, et al. ISRUC-Sleep: a comprehensive public dataset for sleep researchers. Computer Methods Programs Biomed, 2016, 124: 180-192.

Journal of Biomedical Engineering

Multi-modal physiological time-frequency feature extraction network for accurate sleep stage classification

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content