An improved Vision Transformer model for the recognition of blood cells_Journal of Biomedical Engineering

Authors：

SUN Tianyu ¹ , ZHU Qingtao ¹ , YANG Jian ¹ ,  ZENG Liang ²

1. Department of Electronic Engineering, Tsinghua University, Beijing 100084, P. R. China;
2. Beijing Institute of Technology, Beijing 100081, P. R. China;

Corresponding author：

ZENG Liang, Email: liang@bit.edu.cn

Keywords：

Blood cell recognition; Vision Transformer; Self attention; Leukemia

DOI：

10.7507/1001-5515.202203008

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Leukemia is a common, multiple and dangerous blood disease, whose early diagnosis and treatment are very important. At present, the diagnosis of leukemia heavily relies on morphological examination of blood cell images by pathologists, which is tedious and time-consuming. Meanwhile, the diagnostic results are highly subjective, which may lead to misdiagnosis and missed diagnosis. To address the gap above, we proposed an improved Vision Transformer model for blood cell recognition. First, a faster R-CNN network was used to locate and extract individual blood cell slices from original images. Then, we split the single-cell image into multiple image patches and put them into the encoder layer for feature extraction. Based on the self-attention mechanism of the Transformer, we proposed a sparse attention module which could focus on the discriminative parts of blood cell images and improve the fine-grained feature representation ability of the model. Finally, a contrastive loss function was adopted to further increase the inter-class difference and intra-class consistency of the extracted features. Experimental results showed that the proposed module outperformed the other approaches and significantly improved the accuracy to 91.96% on the Munich single-cell morphological dataset of leukocytes, which is expected to provide a reference for physicians’ clinical diagnosis.

Citation： SUN Tianyu, ZHU Qingtao, YANG Jian, ZENG Liang. An improved Vision Transformer model for the recognition of blood cells. Journal of Biomedical Engineering, 2022, 39(6): 1097-1107. doi: 10.7507/1001-5515.202203008 Copy

1.	黄治虎, 陈宝安, 欧阳建, 等. 我国白血病流行病学调查的现状和对策. 临床血液学杂志, 2009, 22(2): 166-167.
2.	Musto P, Statuto T, Valvano L, et al. An update on biology, diagnosis and treatment of primary plasma cell leukemia. Expert Rev Hematol, 2019, 12(4): 245-253.
3.	伍柏青, 傅新文. 当代五分类血细胞分析仪技术原理分析. 实验与检验医学, 2011, 29(4): 391-394.
4.	Bennett J M, Catovsky D, Daniel M T, et al. Proposals for the classification of the acute leukaemias French-American-British (FAB) co-operative group. Brit J Haematol, 1976, 33(4): 451-458.
5.	Shen D, Wu G, Suk H I. Deep learning in medical image analysis. Annu Rev Biomed Eng, 2017, 19: 221-248.
6.	Litjens G, Kooi T, Bejnordi B E, et al. A survey on deep learning in medical image analysis. Med Image Anal, 2017, 42: 60-88.
7.	Dhieb N, Ghazzai H, Besbes H, et al. An automated blood cells counting and classification framework using mask R-CNN deep learning model// 2019 31st International Conference on Microelectronics (ICM). Cairo: IEEE, 2019: 300-303.
8.	Tobias R R, De Jesus L C, Mital M E, et al. Faster R-CNN model with momentum optimizer for RBC and WBC variants classification// 2020 IEEE 2nd Global Conference on Life Sciences and Technologies (LifeTech). Kyoto: IEEE, 2020: 235-239.
9.	Shakarami A, Menhaj M B, Mahdavi-Hormat A, et al. A fast and yet efficient YOLOv3 for blood cell detection. Biomed Signal Proces, 2021, 66: 102495.
10.	鞠孟汐, 李欣蔚, 李章勇. 基于深度主动学习的白带白细胞智能检测方法研究. 生物医学工程学杂志, 2020, 37(3): 519-526.
11.	Xia T, Jiang R, Fu Y Q, et al. Automated blood cell detection and counting via deep learning for microfluidic point-of-care medical devices. IOP Conf Ser Mater Sci Eng, 2019, 646: 012048.
12.	Novoselnik F, Grbić R, Galić I, et al. Automatic white blood cell detection and identification using convolutional neural network// 2018 International Conference on Smart Systems and Technologies (SST). Osijek: IEEE, 2018: 163-167.
13.	Matek C, Schwarz S, Spiekermann K, et al. Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks. Nat Mach Intell, 2019, 1(11): 538-544.
14.	Fu X, Fu M, Li Q, et al. Morphogo: an automatic bone marrow cell classification system on digital images analyzed by artificial intelligence. Acta Cytol, 2020, 64(6): 588-596.
15.	Huang P, Wang J, Zhang J, et al. Attention-aware residual network based manifold learning for white blood cells classification. IEEE J Biomed Health Inform, 2020, 25(4): 1206-1214.
16.	Mori J, Kaji S, Kawai H, et al. Assessment of dysplasia in bone marrow smear with convolutional neural network. Sci Rep, 2020, 10(1): 1-8.
17.	Ghosh M, Das D, Mandal S, et al. Statistical pattern analysis of white blood cell nuclei morphometry// 2010 IEEE Students Technology Symposium (TechSym). Kharagpur: IEEE, 2010: 59-66.
18.	Rezatofighi S H, Soltanian-Zadeh H. Automatic recognition of five types of white blood cells in peripheral blood. Comput Med Imaging Graph, 2011, 35(4): 333-343.
19.	Zhu Q, Lu D, Zhang T, et al. Fine-grained classification of neutrophils with hybrid loss// International Conference on Image and Graphics. Haikou: Springer, 2021: 102-113.
20.	Zhou Z, Siddiquee M M R, Tajbakhsh N, et al. Unet++: A nested U-Net architecture for medical image segmentation. Deep Learn Med Image Anal Multimodal Learn Clin Decis Support, 2018, 11045: 3-11.
21.	Lu Y, Qin X, Fan H, et al. WBC-Net: A white blood cell segmentation network based on UNet++ and ResNet. Appl Soft Comput, 2021, 101: 107006.
22.	Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation// International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer, 2015: 234-241.
23.	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for image recognition at scale// International Conference on Learning Representations. New Orleans: ICLR, 2021: 1-22.
24.	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). California: Curran Associates Inc, 2017: 6000-6010.
25.	Hadsell R, Chopra S, LeCun Y. Dimensionality reduction by learning an invariant mapping// 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). New York: IEEE, 2006, 2: 1735-1742.
26.	Matek C, Schwarz S, Marr C, et al. A single-cell morphological dataset of leukocytes from AML patients and non-malignant controls [2022-03-05]. https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=61080958.
27.	Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 2016, 39(6): 1137-1149.
28.	Settles B. Active learning literature survey. Madison: University of Wisconsin-Madison, 2010.
29.	Van Dyk D A, Meng X L. The art of data augmentation. J Comput Graph Stat, 2001, 10(1): 1-50.
30.	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition// 3rd International Conference on Learning Representations (ICLR 2015). San Diego: ICLR, 2015: 1-14.
31.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
32.	Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011-2023.
33.	Xie S, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE 2017: 1492-1500.
34.	Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks// International Conference on Machine Learning. California: IMLS, 2019: 6105-6114.
35.	van der Maaten L, Hinton G. Visualizing data using t-SNE. J Mach Learn Res, 2008, 9(86): 2579-2605.

1. 黄治虎, 陈宝安, 欧阳建, 等. 我国白血病流行病学调查的现状和对策. 临床血液学杂志, 2009, 22(2): 166-167.
2. Musto P, Statuto T, Valvano L, et al. An update on biology, diagnosis and treatment of primary plasma cell leukemia. Expert Rev Hematol, 2019, 12(4): 245-253.
3. 伍柏青, 傅新文. 当代五分类血细胞分析仪技术原理分析. 实验与检验医学, 2011, 29(4): 391-394.
4. Bennett J M, Catovsky D, Daniel M T, et al. Proposals for the classification of the acute leukaemias French-American-British (FAB) co-operative group. Brit J Haematol, 1976, 33(4): 451-458.
5. Shen D, Wu G, Suk H I. Deep learning in medical image analysis. Annu Rev Biomed Eng, 2017, 19: 221-248.
6. Litjens G, Kooi T, Bejnordi B E, et al. A survey on deep learning in medical image analysis. Med Image Anal, 2017, 42: 60-88.
7. Dhieb N, Ghazzai H, Besbes H, et al. An automated blood cells counting and classification framework using mask R-CNN deep learning model// 2019 31st International Conference on Microelectronics (ICM). Cairo: IEEE, 2019: 300-303.
8. Tobias R R, De Jesus L C, Mital M E, et al. Faster R-CNN model with momentum optimizer for RBC and WBC variants classification// 2020 IEEE 2nd Global Conference on Life Sciences and Technologies (LifeTech). Kyoto: IEEE, 2020: 235-239.
9. Shakarami A, Menhaj M B, Mahdavi-Hormat A, et al. A fast and yet efficient YOLOv3 for blood cell detection. Biomed Signal Proces, 2021, 66: 102495.
10. 鞠孟汐, 李欣蔚, 李章勇. 基于深度主动学习的白带白细胞智能检测方法研究. 生物医学工程学杂志, 2020, 37(3): 519-526.
11. Xia T, Jiang R, Fu Y Q, et al. Automated blood cell detection and counting via deep learning for microfluidic point-of-care medical devices. IOP Conf Ser Mater Sci Eng, 2019, 646: 012048.
12. Novoselnik F, Grbić R, Galić I, et al. Automatic white blood cell detection and identification using convolutional neural network// 2018 International Conference on Smart Systems and Technologies (SST). Osijek: IEEE, 2018: 163-167.
13. Matek C, Schwarz S, Spiekermann K, et al. Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks. Nat Mach Intell, 2019, 1(11): 538-544.
14. Fu X, Fu M, Li Q, et al. Morphogo: an automatic bone marrow cell classification system on digital images analyzed by artificial intelligence. Acta Cytol, 2020, 64(6): 588-596.
15. Huang P, Wang J, Zhang J, et al. Attention-aware residual network based manifold learning for white blood cells classification. IEEE J Biomed Health Inform, 2020, 25(4): 1206-1214.
16. Mori J, Kaji S, Kawai H, et al. Assessment of dysplasia in bone marrow smear with convolutional neural network. Sci Rep, 2020, 10(1): 1-8.
17. Ghosh M, Das D, Mandal S, et al. Statistical pattern analysis of white blood cell nuclei morphometry// 2010 IEEE Students Technology Symposium (TechSym). Kharagpur: IEEE, 2010: 59-66.
18. Rezatofighi S H, Soltanian-Zadeh H. Automatic recognition of five types of white blood cells in peripheral blood. Comput Med Imaging Graph, 2011, 35(4): 333-343.
19. Zhu Q, Lu D, Zhang T, et al. Fine-grained classification of neutrophils with hybrid loss// International Conference on Image and Graphics. Haikou: Springer, 2021: 102-113.
20. Zhou Z, Siddiquee M M R, Tajbakhsh N, et al. Unet++: A nested U-Net architecture for medical image segmentation. Deep Learn Med Image Anal Multimodal Learn Clin Decis Support, 2018, 11045: 3-11.
21. Lu Y, Qin X, Fan H, et al. WBC-Net: A white blood cell segmentation network based on UNet++ and ResNet. Appl Soft Comput, 2021, 101: 107006.
22. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation// International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer, 2015: 234-241.
23. Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for image recognition at scale// International Conference on Learning Representations. New Orleans: ICLR, 2021: 1-22.
24. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). California: Curran Associates Inc, 2017: 6000-6010.
25. Hadsell R, Chopra S, LeCun Y. Dimensionality reduction by learning an invariant mapping// 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). New York: IEEE, 2006, 2: 1735-1742.
26. Matek C, Schwarz S, Marr C, et al. A single-cell morphological dataset of leukocytes from AML patients and non-malignant controls [2022-03-05]. https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=61080958.
27. Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 2016, 39(6): 1137-1149.
28. Settles B. Active learning literature survey. Madison: University of Wisconsin-Madison, 2010.
29. Van Dyk D A, Meng X L. The art of data augmentation. J Comput Graph Stat, 2001, 10(1): 1-50.
30. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition// 3rd International Conference on Learning Representations (ICLR 2015). San Diego: ICLR, 2015: 1-14.
31. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
32. Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011-2023.
33. Xie S, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE 2017: 1492-1500.
34. Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks// International Conference on Machine Learning. California: IMLS, 2019: 6105-6114.
35. van der Maaten L, Hinton G. Visualizing data using t-SNE. J Mach Learn Res, 2008, 9(86): 2579-2605.

Journal of Biomedical Engineering

An improved Vision Transformer model for the recognition of blood cells

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content