Application of a parallel branches network based on Transformer for skin melanoma segmentation_Journal of Biomedical Engineering

Authors：

YI Sanli ^1,2 , ZHANG Gang ^1,2 ,  HE Jianfeng ¹

1. School of Information Engineering and Automation, Kunming University of Scienceand Technology, Kunming 650500, P. R. China;
2. Key Laboratory of Computer Technology Application of Yunnan Province, Kunming 650500, P. R. China;

Corresponding author：

HE Jianfeng, Email: 120112624@qq.com

Keywords：

Deep learning; Skin segmentation; Transformer; Computer vision

DOI：

10.7507/1001-5515.202110073

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Cutaneous malignant melanoma is a common malignant tumor. Accurate segmentation of the lesion area is extremely important for early diagnosis of the disease. In order to achieve more effective and accurate segmentation of skin lesions, a parallel network architecture based on Transformer is proposed in this paper. This network is composed of two parallel branches: the former is the newly constructed multiple residual frequency channel attention network (MFC), and the latter is the visual transformer network (ViT). First, in the MFC network branch, the multiple residual module and the frequency channel attention module (FCA) module are fused to improve the robustness of the network and enhance the capability of extracting image detailed features. Second, in the ViT network branch, multiple head self-attention (MSA) in Transformer is used to preserve the global features of the image. Finally, the feature information extracted from the two branches are combined in parallel to realize image segmentation more effectively. To verify the proposed algorithm, we conducted experiments on the dermoscopy image dataset published by the International Skin Imaging Collaboration (ISIC) in 2018. The results show that the intersection-over-union (IoU) and Dice coefficients of the proposed algorithm achieve 90.15% and 94.82%, respectively, which are better than the latest skin melanoma segmentation networks. Therefore, the proposed network can better segment the lesion area and provide dermatologists with more accurate lesion data.

Citation： YI Sanli, ZHANG Gang, HE Jianfeng. Application of a parallel branches network based on Transformer for skin melanoma segmentation. Journal of Biomedical Engineering, 2022, 39(5): 937-944, 957. doi: 10.7507/1001-5515.202110073 Copy

1.	Al-Masni M A, Kim D H, Kim T S. Multiple skin lesions diagnostics via integrated deep convolutional networks for segmentation and classification. Comput Methods Programs Biomed, 2020, 190: 105351.
2.	Jaisakthi S, Mirunalini P, Aravindan C. Automated skin lesion segmentation of dermoscopic images using GrabCut and k-means algorithms. IET Comput Vis, 2018, 12(8): 1088-1095.
3.	蒋新辉, 李喆. 基于U型结构上下文编码解码网络的皮肤病变分割研究. 激光与光电子学进展, 2021, 58(12): 122-129.
4.	Hasan M, Roy S, Mondal C, et al. Dermo-DOCTOR: a framework for concurrent skin lesion detection and recognition using a deep convolutional neural network with end-to-end dual encoders. Biomedical Signal Processing and Control, 2021, 68: 102661.
5.	杨国亮, 赖振东, 喻丁玲. 一种改进UNet++网络用于检测黑色素瘤皮肤病变. 中国医学影像技术, 2020, 36(12): 1877-1881.
6.	Badrinarayanan V, Kendall A, Cipolla R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans Pattern Anal Mach Intell, 2017, 39(12): 2481-2495.
7.	易三莉, 王天伟, 杨雪莲, 等. 基于改进U-Net的肺野分割算法. 激光与光电子学进展, 2022, 59(2): 175-183.
8.	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE T Pattern Anal. 2015, 39(4): 640-651.
9.	Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1-9.
10.	Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation// 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Munich: Springer, 2015: 234-241.
11.	Valanarasu J M J, Oza P, Hacihalilu I, et al. Medical transformer: gated axial-attention for medical image segmentation// Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), Strasbourg: MICCAI, 2021. https://doi.org/10.48550/arXiv.2102.10662.
12.	Schlemper J, Oktay O, Schaap M, et al. Attention gated networks: learning to leverage salient regions in medical images. Med Image Anal, 2019, 53: 197-207.
13.	Oktay O, Schlemper J, Folgoc L, et al. Attention U-Net: learning where to look for the pancreas// Medical Imaging with Deep Learning (MIDL). Amsterdam: Academic Press. 2018. https: //doi.org/10.48550/arXiv.1804.03999.
14.	Thomas E, Pawan SJ, Kumar S, et al. Multi-Res-Attention UNet: A CNN Model for the Segmentation of Focal Cortical Dysplasia Lesions from Magnetic Resonance Images. IEEE J Biomed Health Inform, 2021, 25(5): 1724-1734.
15.	Chen L, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation// Proceedings of the European Conference on Computer Vision (ECCV). Munich: Springer. 2018: 801-818.
16.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 770–778.
17.	Ibtehaz N, Rahman MS. MultiResUNet : rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Netw, 2020, 121: 74-87.
18.	Qin Zequn, Zhang Pengyi, Wu Fei, et al. FcaNet: frequency channel attention networks// 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal: IEEE, 2021: 763-772.
19.	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach: NIPS, 2017: 6000-6010.
20.	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: transformers for image recognition at scale// International Conference on Learning Representations (ICLR 2022), 2021: 2010.11929v2. https: //doi.org/10.48550/arXiv.2010.11929.
21.	Hu Jie, Shen Li, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011-2023.
22.	Zhou Z, Siddiquee M, Tajbakhsh N, et al. UNet++: a nested U-Net architecture for medical image segmentation// 2018 International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Granada: Springer, 2018, 11045:3–11.
23.	Wang Haonan, Cao Peng, Wang Jiaqi, et al. UCTransNet: rethinking the skip connections in U-Net from a channel-wise perspective with transformer// AAAI Conference on Artificial Intelligence (AAAI 2022), AAAI, 2022. https://doi.org/10.48550/arXiv.2109.04335.
24.	Wang J, Wei L, Wang L, et al. Boundary-aware transformers for skin lesion segmentation// 2018 International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Strasbourg: Springer, 2021: 206-216.
25.	Gu Z, Cheng J, Fu H, et al. CE-Net: context encoder network for 2D medical image segmentation. IEEE Trans Med Imaging, 2019, 38(10): 2281-2292.
26.	Lee H, Kim J, Lee S, et al. Structure boundary preserving segmentation for medical image with ambiguous boundary// 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle: IEEE, 2020: 4816–4825.

1. Al-Masni M A, Kim D H, Kim T S. Multiple skin lesions diagnostics via integrated deep convolutional networks for segmentation and classification. Comput Methods Programs Biomed, 2020, 190: 105351.
2. Jaisakthi S, Mirunalini P, Aravindan C. Automated skin lesion segmentation of dermoscopic images using GrabCut and k-means algorithms. IET Comput Vis, 2018, 12(8): 1088-1095.
3. 蒋新辉, 李喆. 基于U型结构上下文编码解码网络的皮肤病变分割研究. 激光与光电子学进展, 2021, 58(12): 122-129.
4. Hasan M, Roy S, Mondal C, et al. Dermo-DOCTOR: a framework for concurrent skin lesion detection and recognition using a deep convolutional neural network with end-to-end dual encoders. Biomedical Signal Processing and Control, 2021, 68: 102661.
5. 杨国亮, 赖振东, 喻丁玲. 一种改进UNet++网络用于检测黑色素瘤皮肤病变. 中国医学影像技术, 2020, 36(12): 1877-1881.
6. Badrinarayanan V, Kendall A, Cipolla R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans Pattern Anal Mach Intell, 2017, 39(12): 2481-2495.
7. 易三莉, 王天伟, 杨雪莲, 等. 基于改进U-Net的肺野分割算法. 激光与光电子学进展, 2022, 59(2): 175-183.
8. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE T Pattern Anal. 2015, 39(4): 640-651.
9. Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1-9.
10. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation// 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Munich: Springer, 2015: 234-241.
11. Valanarasu J M J, Oza P, Hacihalilu I, et al. Medical transformer: gated axial-attention for medical image segmentation// Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), Strasbourg: MICCAI, 2021. https://doi.org/10.48550/arXiv.2102.10662.
12. Schlemper J, Oktay O, Schaap M, et al. Attention gated networks: learning to leverage salient regions in medical images. Med Image Anal, 2019, 53: 197-207.
13. Oktay O, Schlemper J, Folgoc L, et al. Attention U-Net: learning where to look for the pancreas// Medical Imaging with Deep Learning (MIDL). Amsterdam: Academic Press. 2018. https: //doi.org/10.48550/arXiv.1804.03999.
14. Thomas E, Pawan SJ, Kumar S, et al. Multi-Res-Attention UNet: A CNN Model for the Segmentation of Focal Cortical Dysplasia Lesions from Magnetic Resonance Images. IEEE J Biomed Health Inform, 2021, 25(5): 1724-1734.
15. Chen L, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation// Proceedings of the European Conference on Computer Vision (ECCV). Munich: Springer. 2018: 801-818.
16. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016: 770–778.
17. Ibtehaz N, Rahman MS. MultiResUNet : rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Netw, 2020, 121: 74-87.
18. Qin Zequn, Zhang Pengyi, Wu Fei, et al. FcaNet: frequency channel attention networks// 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal: IEEE, 2021: 763-772.
19. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need// 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach: NIPS, 2017: 6000-6010.
20. Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: transformers for image recognition at scale// International Conference on Learning Representations (ICLR 2022), 2021: 2010.11929v2. https: //doi.org/10.48550/arXiv.2010.11929.
21. Hu Jie, Shen Li, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011-2023.
22. Zhou Z, Siddiquee M, Tajbakhsh N, et al. UNet++: a nested U-Net architecture for medical image segmentation// 2018 International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Granada: Springer, 2018, 11045:3–11.
23. Wang Haonan, Cao Peng, Wang Jiaqi, et al. UCTransNet: rethinking the skip connections in U-Net from a channel-wise perspective with transformer// AAAI Conference on Artificial Intelligence (AAAI 2022), AAAI, 2022. https://doi.org/10.48550/arXiv.2109.04335.
24. Wang J, Wei L, Wang L, et al. Boundary-aware transformers for skin lesion segmentation// 2018 International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Strasbourg: Springer, 2021: 206-216.
25. Gu Z, Cheng J, Fu H, et al. CE-Net: context encoder network for 2D medical image segmentation. IEEE Trans Med Imaging, 2019, 38(10): 2281-2292.
26. Lee H, Kim J, Lee S, et al. Structure boundary preserving segmentation for medical image with ambiguous boundary// 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle: IEEE, 2020: 4816–4825.

Previous Article
Research on grading algorithm of diabetic retinopathy based on cross-layer bilinear pooling
Next Article
Confocal probe localization algorithm based on region growing and endoscope size prior

Journal of Biomedical Engineering

Application of a parallel branches network based on Transformer for skin melanoma segmentation

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content