Visual object detection system based on augmented reality and steady-state visual evoked potential_Journal of Biomedical Engineering

Authors：

GUO Meng’ao ¹ ,  YANG Banghua ¹ , GENG Yiting ¹ , JIE Rongxin ¹ , ZHANG Yonghuai ² , ZHENG Yanyan ³

1. School of Mechanical and Electrical Engineering and Automation, Shanghai University, Shanghai 200444, P. R. China;
2. Shanghai Sensing Technology Company, Shanghai 201900, P. R. China;
3. Wenzhou People’s Hospital of Zhejiang Province, Wenzhou, Zhejiang 325041, P. R. China;

Corresponding author：

YANG Banghua, Email: yangbanghua@shu.edu.cn

Keywords：

Brain-computer interface; Steady-state visual evoked potential; Augmented reality; Target recognition

DOI：

10.7507/1001-5515.202403041

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

This study investigates a brain-computer interface (BCI) system based on an augmented reality (AR) environment and steady-state visual evoked potentials (SSVEP). The system is designed to facilitate the selection of real-world objects through visual gaze in real-life scenarios. By integrating object detection technology and AR technology, the system augmented real objects with visual enhancements, providing users with visual stimuli that induced corresponding brain signals. SSVEP technology was then utilized to interpret these brain signals and identify the objects that users focused on. Additionally, an adaptive dynamic time-window-based filter bank canonical correlation analysis was employed to rapidly parse the subjects’ brain signals. Experimental results indicated that the system could effectively recognize SSVEP signals, achieving an average accuracy rate of 90.6% in visual target identification. This system extends the application of SSVEP signals to real-life scenarios, demonstrating feasibility and efficacy in assisting individuals with mobility impairments and physical disabilities in object selection tasks.

Citation： GUO Meng’ao, YANG Banghua, GENG Yiting, JIE Rongxin, ZHANG Yonghuai, ZHENG Yanyan. Visual object detection system based on augmented reality and steady-state visual evoked potential. Journal of Biomedical Engineering, 2024, 41(4): 684-691. doi: 10.7507/1001-5515.202403041 Copy

1.	Wolpaw J R, Birbaumer N, Heetderks W J, et al. Brain-computer interface technology: a review of the first international meeting. IEEE Trans Neural Syst Rehabil Eng, 2000, 8(2): 164-173.
2.	Wolpaw J R, Loeb G E, Allison B Z, et al. BCI meeting 2005-workshop on signals and recording methods. IEEE Trans Neural Syst Rehabil Eng, 2006, 14(2): 138-141.
3.	Wang C, Phua K S, Ang K K, et al. A feasibility study of non-invasive motor-imagery BCI-based robotic rehabilitation for stroke patients// 2009 4th International IEEE/EMBS Conference on Neural Engineering. Antalya: IEEE, 2009: 271-274.
4.	Zhang Y, Guo D, Li F, et al. Correlated component analysis for enhancing the performance of SSVEP-based brain-computer interface. IEEE Trans Neural Syst Rehabil Eng, 2018, PP(99): 1.
5.	Gao S, Gao X. The design and implementation of visual braincomputer interfaces// 2014 International Winter Workshop on Brain-Computer Interface (BCI). Gangwon: IEEE, 2014: 1-3.
6.	Angrisani L, Arpaia P, De Benedetto E, et al. Expanding the frontiers of wearable brain-computer interfaces combining augmented reality and visually evoked potentials// 2023 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE). Milano: IEEE, 2023: 58-62.
7.	Angrisani L, Arpaia P, De Benedetto E, et al. Wearable brain–computer interfaces based on steady-state visually evoked potentials and augmented reality: a review. IEEE Sens J, 2023, 23(15): 16501-16514.
8.	张力新, 张裕坤, 柯余峰, 等. 基于Hololens的增强现实脑-机接口研究. 中国生物医学工程学报, 2019, 38(1): 51-58.
9.	Wang Y, Li K, Zhang X, et al. Research on the application of augmented reality in SSVEP-BCI// International Conference on Computing and Artificial Intelligence (ICCAI’20). New York: ACM, 2020: 505-509.
10.	Horii S, Nakauchi S, Kitazaki M. AR-SSVEP for brain-machine interface: estimating user’s gaze in head-mounted display with USB camera// 2015 IEEE Virtual Reality (VR). Arles: IEEE, 2015: 193-194.
11.	Park S, Cha H S, Kwon J, et al. Development of an online home appliance control system using augmented reality and an SSVEP-based brain-computer interface// 2020 8th International Winter Conference on Brain-Computer Interface (BCI). Gangwon: IEEE, 2020: 1-2.
12.	Sakkalis V, Krana M, Farmaki C, et al. Augmented reality driven steady-state visual evoked potentials for wheelchair navigation. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 2960-2969.
13.	Zhang S, Chen Y, Zhang L, et al. Study on robot grasping system of SSVEP-BCI based on augmented reality stimulus. Tsinghua Sci Technol, 2023, 28(2): 322-329.
14.	Fang B, Ding W, Sun F, et al. Brain–computer interface integrated with augmented reality for human–robot interaction. IEEE Trans Cogn Dev Syst, 2023, 15(4): 1702-1711.
15.	Yao Y, Yang B, Xia X, et al. Design of upper limb rehabilitation training system combining BCI and AR technology// 2021 40th Chinese Control Conference (CCC). Shanghai: IEEE, 2021: 7131-7134.
16.	Zhu D, Dai L, Du P. CCE-YOLOv5s: An improved Yolov5 model for UAV small target detection// 2023 IEEE 5th International Conference on Civil Aviation Safety and Information Technology (ICCASIT). Dali: IEEE, 2023: 824-829.
17.	Samara M, Farmaki C, Zacharioudakis N, et al. Comparison between dry and wet EEG electrodes in an SSVEP-based BCI for robot navigation// 2022 IEEE 22nd International Conference on Bioinformatics and Bioengineering (BIBE). Taiwan: IEEE, 2022: 333-338.
18.	Cavallaro M. Technical tips: another way to approach the 10-20 system of electrode placement. Am J EEG Technol, 1992, 32(3): 225-232.
19.	Wang H, Zhang Y, Waytowich N R, et al. Discriminative feature extraction via multivariate linear regression for ssvep-based bci. IEEE Trans Neural Syst Rehabil Eng, 2016, 24(5): 532-541.
20.	Chen X, Wang Y, Nakanishi M, et al. High-speed spelling with a noninvasive brain-computer interface. Proc Natl Acad Sci, 2015, 112(44): E6058-E6067.
21.	Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface. Neural Eng, 2015, 12(4): 046008.
22.	Lin Z, Zhang C, Wu Wei, et al. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng, 2007, 54(6): 1172-1176.
23.	Yang C, Li X, Shi N, et al. A dynamic window recognition algorithm for SSVEP-based brain–computer interfaces using a spatio-temporal equalizer. Int J Neural Syst, 2018, 28(10): 1850028.
24.	Chen Y, Yang C, Chen X, et al. A novel training-free recognition method for SSVEP-based BCIs using dynamic window strategy. J Neural Eng, 2021, 18(3): 036007.
25.	Lee T, Nam S, Hyun D J. Adaptive window method based on FBCCA for optimal SSVEP recognition. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 78-86.
26.	Kübler A, Neumann N, Kaiser J, et al. Brain-computer communication: self-regulation of slow cortical potentials for verbal communication. Arch Phys Med Rehab, 2001, 82(11): 1533-1539.
27.	迟新一, 崔红岩, 陈小刚. 结合稳态视觉诱发电位的多模态脑机接口研究进展. 中国生物医学工程学报, 2022, 41(2): 204-213.
28.	何柳诗, 谢俊, 于鸿伟, 等. 融合眼动追踪和目标动态可调的稳态视觉诱发电位脑机接口系统设计. 西安交通大学学报, 2021, 55(10): 87-95.

1. Wolpaw J R, Birbaumer N, Heetderks W J, et al. Brain-computer interface technology: a review of the first international meeting. IEEE Trans Neural Syst Rehabil Eng, 2000, 8(2): 164-173.
2. Wolpaw J R, Loeb G E, Allison B Z, et al. BCI meeting 2005-workshop on signals and recording methods. IEEE Trans Neural Syst Rehabil Eng, 2006, 14(2): 138-141.
3. Wang C, Phua K S, Ang K K, et al. A feasibility study of non-invasive motor-imagery BCI-based robotic rehabilitation for stroke patients// 2009 4th International IEEE/EMBS Conference on Neural Engineering. Antalya: IEEE, 2009: 271-274.
4. Zhang Y, Guo D, Li F, et al. Correlated component analysis for enhancing the performance of SSVEP-based brain-computer interface. IEEE Trans Neural Syst Rehabil Eng, 2018, PP(99): 1.
5. Gao S, Gao X. The design and implementation of visual braincomputer interfaces// 2014 International Winter Workshop on Brain-Computer Interface (BCI). Gangwon: IEEE, 2014: 1-3.
6. Angrisani L, Arpaia P, De Benedetto E, et al. Expanding the frontiers of wearable brain-computer interfaces combining augmented reality and visually evoked potentials// 2023 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE). Milano: IEEE, 2023: 58-62.
7. Angrisani L, Arpaia P, De Benedetto E, et al. Wearable brain–computer interfaces based on steady-state visually evoked potentials and augmented reality: a review. IEEE Sens J, 2023, 23(15): 16501-16514.
8. 张力新, 张裕坤, 柯余峰, 等. 基于Hololens的增强现实脑-机接口研究. 中国生物医学工程学报, 2019, 38(1): 51-58.
9. Wang Y, Li K, Zhang X, et al. Research on the application of augmented reality in SSVEP-BCI// International Conference on Computing and Artificial Intelligence (ICCAI’20). New York: ACM, 2020: 505-509.
10. Horii S, Nakauchi S, Kitazaki M. AR-SSVEP for brain-machine interface: estimating user’s gaze in head-mounted display with USB camera// 2015 IEEE Virtual Reality (VR). Arles: IEEE, 2015: 193-194.
11. Park S, Cha H S, Kwon J, et al. Development of an online home appliance control system using augmented reality and an SSVEP-based brain-computer interface// 2020 8th International Winter Conference on Brain-Computer Interface (BCI). Gangwon: IEEE, 2020: 1-2.
12. Sakkalis V, Krana M, Farmaki C, et al. Augmented reality driven steady-state visual evoked potentials for wheelchair navigation. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 2960-2969.
13. Zhang S, Chen Y, Zhang L, et al. Study on robot grasping system of SSVEP-BCI based on augmented reality stimulus. Tsinghua Sci Technol, 2023, 28(2): 322-329.
14. Fang B, Ding W, Sun F, et al. Brain–computer interface integrated with augmented reality for human–robot interaction. IEEE Trans Cogn Dev Syst, 2023, 15(4): 1702-1711.
15. Yao Y, Yang B, Xia X, et al. Design of upper limb rehabilitation training system combining BCI and AR technology// 2021 40th Chinese Control Conference (CCC). Shanghai: IEEE, 2021: 7131-7134.
16. Zhu D, Dai L, Du P. CCE-YOLOv5s: An improved Yolov5 model for UAV small target detection// 2023 IEEE 5th International Conference on Civil Aviation Safety and Information Technology (ICCASIT). Dali: IEEE, 2023: 824-829.
17. Samara M, Farmaki C, Zacharioudakis N, et al. Comparison between dry and wet EEG electrodes in an SSVEP-based BCI for robot navigation// 2022 IEEE 22nd International Conference on Bioinformatics and Bioengineering (BIBE). Taiwan: IEEE, 2022: 333-338.
18. Cavallaro M. Technical tips: another way to approach the 10-20 system of electrode placement. Am J EEG Technol, 1992, 32(3): 225-232.
19. Wang H, Zhang Y, Waytowich N R, et al. Discriminative feature extraction via multivariate linear regression for ssvep-based bci. IEEE Trans Neural Syst Rehabil Eng, 2016, 24(5): 532-541.
20. Chen X, Wang Y, Nakanishi M, et al. High-speed spelling with a noninvasive brain-computer interface. Proc Natl Acad Sci, 2015, 112(44): E6058-E6067.
21. Chen X, Wang Y, Gao S, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface. Neural Eng, 2015, 12(4): 046008.
22. Lin Z, Zhang C, Wu Wei, et al. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng, 2007, 54(6): 1172-1176.
23. Yang C, Li X, Shi N, et al. A dynamic window recognition algorithm for SSVEP-based brain–computer interfaces using a spatio-temporal equalizer. Int J Neural Syst, 2018, 28(10): 1850028.
24. Chen Y, Yang C, Chen X, et al. A novel training-free recognition method for SSVEP-based BCIs using dynamic window strategy. J Neural Eng, 2021, 18(3): 036007.
25. Lee T, Nam S, Hyun D J. Adaptive window method based on FBCCA for optimal SSVEP recognition. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 78-86.
26. Kübler A, Neumann N, Kaiser J, et al. Brain-computer communication: self-regulation of slow cortical potentials for verbal communication. Arch Phys Med Rehab, 2001, 82(11): 1533-1539.
27. 迟新一, 崔红岩, 陈小刚. 结合稳态视觉诱发电位的多模态脑机接口研究进展. 中国生物医学工程学报, 2022, 41(2): 204-213.
28. 何柳诗, 谢俊, 于鸿伟, 等. 融合眼动追踪和目标动态可调的稳态视觉诱发电位脑机接口系统设计. 西安交通大学学报, 2021, 55(10): 87-95.

Journal of Biomedical Engineering

Visual object detection system based on augmented reality and steady-state visual evoked potential

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content