Medical image instance segmentation: from candidate region to no candidate region_Journal of Biomedical Engineering

Authors：

ZHOU Tao ^1,2 ,  ZHAO Yanan ¹ , LU Huiling ³ , HOU Senbao ¹ , ZHENG Xiaomin ⁴

1. School of Computer Science and Engineering, North Minzu University, Yinchuan 750021, P. R. China;
2. Key Laboratory of Image and Graphics Intelligent Processing of State Ethnic Affairs Commission, North Minzu University, Yinchuan 750021, P. R. China;
3. School of Science, Ningxia Medical University, Yinchuan 750004, P. R. China;
4. Research Institute for Reproductive Medicine and Genetic Diseases, Wuxi Maternity and Child Health Hospital, Wuxi, Jiangsu 214002, P. R. China;

Corresponding author：

ZHAO Yanan, Email: zyn1644o@163.com

Keywords：

Medical image instance segmentation; Candidate region; No candidate region; Two-stage instance segmentation; Single-stage instance segmentation

DOI：

10.7507/1001-5515.202201034

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

In recent years, the task of object detection and segmentation in medical image is the research hotspot and difficulty in the field of image processing. Instance segmentation provides instance-level labels for different objects belonging to the same class, so it is widely used in the field of medical image processing. In this paper, medical image instance segmentation was summarized from the following aspects: First, the basic principle of instance segmentation was described, the instance segmentation models were classified into three categories, the development context of the instance segmentation algorithm was displayed in two-dimensional space, and six classic model diagrams of instance segmentation were given. Second, from the perspective of the three models of two-stage instance segmentation, single-stage instance segmentation and three-dimensional (3D) instance segmentation, we summarized the ideas of the three types of models, discussed the advantages and disadvantages, and sorted out the latest developments. Third, the application status of instance segmentation in six medical images such as colon tissue image, cervical image, bone imaging image, pathological section image of gastric cancer, computed tomography (CT) image of lung nodule and X-ray image of breast was summarized. Fourth, the main challenges in the field of medical image instance segmentation were discussed and the future development direction was prospected. In this paper, the principle, models and characteristics of instance segmentation are systematically summarized, as well as the application of instance segmentation in the field of medical image processing, which is of positive guiding significance to the study of instance segmentation.

Citation： ZHOU Tao, ZHAO Yanan, LU Huiling, HOU Senbao, ZHENG Xiaomin. Medical image instance segmentation: from candidate region to no candidate region. Journal of Biomedical Engineering, 2022, 39(6): 1218-1232. doi: 10.7507/1001-5515.202201034 Copy

1.	Hafiz A M, Bhat G M. A survey on instance segmentation: state of the art. Int J Multimed Inf R, 2020, 9: 171-189.
2.	Jiang Lei, Chen Wenkai, Dong Bao, et al. A deep learning-based approach for glomeruli instance segmentation from multistained renal biopsy pathologic images. Am J Pathol, 2021, 191(8): 1431-1441.
3.	Vania M, Lee D. Intervertebral disc instance segmentation using a multistage optimization mask-RCNN (MOM-RCNN). J Comput Des Eng, 2021, 8(4): 1023-1036.
4.	李佳昇. 基于深度学习的肝脏及肝脏肿瘤分割和检测的研究. 吉林: 吉林大学, 2020.
5.	Hariharan B, Arbeláez P, Girshick R, et al. Simultaneous detection and segmentation// Fleet D, Pajdla T, Schiele B, et al. European Conference on Computer Vision (ECCV). ECCV 2014. Cham: Springer, 2014: 297-312.
6.	梁新宇, 林洗坤, 权冀川, 等. 基于深度学习的图像实例分割技术研究进展. 电子学报, 2020, 48(12): 2476-2486.
7.	Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 580-587.
8.	Pinheiro P, Collobert R, Dollar P. Learning to segment object candidates// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1990-1998.
9.	Pinheiro P O, Lin T Y, Collobert R, et al. Learning to refine object segments// Leibe B, Matas J, Sebe N, et al. European Conference on Computer Vision (ECCV). ECCV 2016. Cham: Springer, 2016: 75-91.
10.	Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE T Pattern Anal, 2017, 39(6): 1137-1149.
11.	Li Y, Qi H, Dai J, et al. Fully convolutional instance-aware semantic segmentation// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 4438-4446.
12.	Ge W, Huang W, Guo S, et al. Label-PEnet: sequential label propagation and enhancement networks for weakly supervised instance segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 3344-3353.
13.	He K, Gkioxari G, Dollár P, et al. Mask R-CNN// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 2980-2988.
14.	Chen L C, Hermans A, Papandreou G, et al. MaskLab: instance segmentation by refining object detection with semantic and direction features// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City: IEEE, 2018: 4013-4022.
15.	Zhang Xiangyi, An Gaoyun, Liu Yutian. Mask R-CNN with feature pyramid attention for instance segmentation// 2018 14th IEEE International Conference on Signal Processing (ICSP). Wuhan: IEEE, 2018: 1194-1197.
16.	Huang Z, Huang L, Gong Y, et al. Mask Scoring R-CNN// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 6402-6411.
17.	Homayounfar N, Xiong Y, Liang J, et al. LevelSet R-CNN: a deep variational method for instance segmentation// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 555-571.
18.	Lin Yuan, Zhao Qiu. Mask-RCNN with spatial attention for pedestrian segmentation in cyber–physical systems. Comput Commun, 2021, 180: 109-114.
19.	Long Kun, Tang Lei, Pu Xiaorong, et al. Probability-based Mask R-CNN for pulmonary embolism detection. Neurocomputing, 2021, 422: 345-353.
20.	Zhou H, Lei L, Xu Y, et al. Dual-supervised instance segmentation network combined with priori corner information// 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC). Xiamen: IEEE, 2019: 55-60.
21.	Hu M, Li Y, Fang L, et al. A²-FPN: attention aggregation based feature pyramid network for instance segmentation// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 15338-15347.
22.	Sofiiuk K, Barinova O, Konushin A. AdaptIS: adaptive instance selection network// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 7354-7362.
23.	Wang W, Feng R, Chen J, et al. Nodule-Plus R-CNN and deep self-paced active learning for 3D instance segmentation of pulmonary nodules. IEEE Access, 2019, 7: 128796-128805.
24.	Kai C, Pang J, Wang J, et al. Hybrid task cascade for instance segmentation// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 4969-4978.
25.	Cai Z, Vasconcelos N. Cascade R-CNN: high quality object detection and instance segmentation. IEEE T Pattern Anal, 2019, 43(5): 1483-1498.
26.	Wang K, Liew J H, Zou Y, et al. PANet: few-shot image semantic segmentation with prototype alignment// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 9196-9205.
27.	Cao J, Cholakkal H, Rao M A, et al. D2Det: towards high quality object detection and instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 11482-11491.
28.	Fan Zhibo, Yu Jingang, Liang Zhihao, et al. FGN: fully guided network for few-shot instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9169-9178.
29.	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE T Pattern Anal, 2015, 39(4): 640-651.
30.	Hayder Z, He X, Salzmann M. Boundary-aware instance segmentation// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 587-595.
31.	Xu W, Wang H, Qi F, et al. Explicit shape encoding for real-time instance segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 5167-5176.
32.	Li Z, Ma Y, Chen Y, et al. Joint COCO and Mapillary Workshop at ICCV 2019: COCO Instance Segmentation Challenge Track. (2020-10-06)[2022-10-16]. https: //arxiv.org/pdf/2010.02475.pdf.
33.	Peng S, Jiang W, Pi H, et al. Deep Snake for real-time instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 8530-8539.
34.	Chen X, Lian Y, Jiao L, et al. Supervised edge attention network for accurate image instance segmentation// European Conference on Computer Vision (ECCV). Glasgow: European, 2020: 617-631.
35.	Kang B R, Kim H Y. BshapeNet: object detection and instance segmentation with bounding shape masks. Pattern Recogn Lett, 2020, 131: 449-455.
36.	Cheng T, Wang X, Huang L, et al. Boundary-preserving Mask R-CNN// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 660-676.
37.	Ding H, Qiao S, Yuille A, et al. Deeply shape-guided cascade for instance segmentation// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 8274-8284.
38.	Tang C, Chen H, Li X, et al. Look closer to segment better: boundary patch refinement for instance segmentation// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 13921-13930.
39.	Shu L, Jia J, Fidler S, et al. SGN: sequential grouping networks for instance segmentation// 2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE, 2017: 3516-3524.
40.	Min B, Urtasun R. Deep watershed transform for instance segmentation// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 2858-2866.
41.	Kirillov A, Levinkov E, Andres B, et al. InstanceCut: from edges to instances with multiCut// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 7322-7331.
42.	Gao N, Shan Y, Wang Y, et al. SSAP: Single-shot instance segmentation with affinity pyramid// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 642-651.
43.	Laradji I H, Rostamzadeh N, Pinheiro P O, et al. Proposal-based instance segmentation with point supervision// 2020 IEEE International Conference on Image Processing (ICIP). Abu Dhabi: IEEE, 2020: 2126-2130.
44.	Zhang G, Lu X, Tan J, et al. RefineMask: towards high-quality instance segmentation with fine-grained features// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 6857-6865.
45.	Bolya D, Zhou C, Xiao F, et al. YOLACT: real-time instance segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 9156-9165.
46.	Fan R, Cheng M M, Hou Q, et al. S4Net: single stage salient-instance segmentation// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 6096-6105.
47.	Chen X, Girshick R, He K, et al. TensorMask: a foundation for dense object segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 2061-2069.
48.	Bolya D, Zhou C, Xiao F, et al. YOLACT++: Better real-time instance segmentation. IEEE T Pattern Anal, 2020, 44(2): 1108-1121.
49.	Lee S, Kim H. MPQ-YOLACT: mixed-precision quantization for lightweight YOLACT// 2020 IEEE International Conference on Consumer Electronics - Asia (ICCE-Asia). Busan: IEEE, 2020: 1-4.
50.	Liang J, Homayounfar N, Ma W C, et al. PolyTransform: deep polygon transformer for instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9128-9137.
51.	Zhang H, Tian Y, Wang K, et al. Mask SSD: An effective single-stage approach to object instance segmentation. IEEE T Image Process, 2020, 29: 2078-2093.
52.	Huang C, Wu W, Lei Z. Efficient instance segmentation network// 2020 IEEE International Conference on Artificial Intelligence and Information Systems (ICAIIS). Dalian: IEEE, 2020: 93-101.
53.	周涛, 刘赟璨, 陆惠玲, 等. ResNet及其在医学图像处理领域的应用: 研究进展与挑战. 电子与信息学报, 2022, 44(1): 149-167.
54.	Wang X, Kong T, Shen C, et al. SOLO: segmenting objects by locations// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 649-665.
55.	Wang X, Zhang R, Kong T, et al. SOLOv2: dynamic and fast instance segmentation. (2020-03-23)[2022-10-16]. https://arxiv.org/pdf/2003.10152.pdf.
56.	Ying H, Huang Z, Liu S , et al. EmbedMask: embedding coupling for one-stage instance segmentation. (2019-12-04)[2022-10-16]. https: //arxiv.org/pdf/1912.01954v2.pdf.
57.	Xie E, Sun P, Song X, et al. PolarMask: single shot instance segmentation with polar representation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 12190-12199.
58.	Chen H, Sun K, Tian Z, et al. BlendMask: top-down meets bottom-up for instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 8570-8578.
59.	Lee Y, Park J. CenterMask: real-time anchor-free instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 13903-13912.
60.	Tian Z, Shen C, Chen H. Conditional convolutions for instance segmentation// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 282-298.
61.	Yu J, Yao J, Zhang J, et al. SPRNet: single-pixel reconstruction for one-stage instance segmentation. IEEE T Cybernetics, 2021, 51(4): 1731-1742.
62.	Kirillov A, Wu Y, He K, et al. PointRend: image segmentation as rendering// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9796-9805.
63.	Yang Hanqing, Zheng Liyang, Barzegar S G, et al. BorderPointsMask: one-stage instance segmentation with boundary points representation. Neurocomputing, 2022, 467: 348-359.
64.	Zhang R, Tian Z, Shen C, et al. Mask encoding for single shot instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 10223-10232.
65.	Dijkstra K, Loosdrecht J, Atsma W A, et al. CentroidNetV2: a hybrid deep neural network for small-object segmentation and counting. Neurocomputing, 2020, 423: 490-505.
66.	Bohm A, Ucker A, Jager T, et al. ISOO_DL: instance segmentation of overlapping biological objects using deep learning// 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). Washington: IEEE, 2018: 1225-1229.
67.	Stahl F, Meyer M, Schwanecke U. IST - Style transfer with instance segmentation// 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA). Dubrovnik: IEEE, 2019: 277-281.
68.	周涛, 侯森宝, 陆惠玲, 等. 探析 U-Net 的改进机制及其在医学图像分割的应用. 生物医学工程学杂志, 2022, 39(4): 806-825.
69.	Wang W, Yu R, Huang Q, et al. SGPN: similarity group proposal network for 3D point cloud instance segmentation// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City: IEEE, 2018: 2569-2578.
70.	Yi L, Zhao W, Wang H, et al. GSPN: generative shape proposal network for 3D instance segmentation in point cloud// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 3942-3951.
71.	Hou J, Dai A, Nießner M. 3D-SIS: 3D semantic instance segmentation of RGB-D scans// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 4416-4425.
72.	Yang B, Wang J, Clark R, et al. Learning object bounding boxes for 3D instance segmentation on point clouds. (2019-09-05)[2022-10-16]. https: //arxiv.org/pdf/1906.01140.pdf.
73.	Zanjani F G, Pourtaherian A, Zinger S, et al. Mask-MCNet: tooth instance segmentation in 3D point clouds of intra-oral scans. Neurocomputing, 2021, 453: 286-298.
74.	Pham Q H, Nguyen D T, Hua B S, et al. JSIS3D: joint semantic-instance segmentation of 3D point clouds with multi-task pointwise networks and multi-value conditional random fields// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 8819-8828.
75.	Wang X, Liu S, Shen X, et al. Associatively segmenting instances and semantics in point clouds// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 4091-4100.
76.	Jiang L, Zhao H, Shi S, et al. PointGroup: dual-set point grouping for 3D instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 4866-4875.
77.	Han L, Zheng T, Xu L, et al. OccuSeg: occupancy-aware 3D instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 2937-2946.
78.	Tan J, Chen L, Wang K, et al. SASO: joint 3D semantic-instance segmentation via multi-scale semantic association and salient point clustering optimization// 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020, 15(5): 366-379.
79.	Engelmann F, Bokeloh M, Fathi A, et al. 3D-MPA: multi-proposal aggregation for 3D semantic instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9028-9037.
80.	Chen Feng, Wu Fei, Gao Guangwei, et al. JSPNet: learning joint semantic & instance segmentation of point clouds via feature self-similarity and cross-task probability. Pattern Recogn, 2022, 122: 108250.
81.	Elich C, Engelmann F, Kontogianni T, et al. 3D bird’s-eye-view instance segmentation// German Conference on Pattern Recognition (GCPR). Dortmund: DAGM, 2019: 11824.
82.	Du L, Tan J, Xue X, et al. 3DCFS: fast and robust joint 3D semantic-instance segmentation via coupled feature selection// 2020 IEEE International Conference on Robotics and Automation (ICRA). Paris: IEEE, 2020: 6868-6875.
83.	Jiang H, Yan F, Cai J, et al. End-to-end 3D point cloud instance segmentation without detection// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 12793-12802.
84.	Sun F, Xu Y, Sun W. SPSN: seed point selection network in point cloud instance segmentation// 2020 International Joint Conference on Neural Networks (IJCNN). Glasgow: IEEE, 2020: 1-8.
85.	Liu C, Furukawa Y. MASC: multi-scale affinity with sparse convolution for 3D instance segmentation. (2019-02-12)[2022-10-16]. https: //arxiv.org/pdf/1902.04478.pdf.
86.	Graham S, Chen H, Gamper J, et al. MILD-Net: minimal information loss dilated network for gland instance segmentation in colon histology images. Med Image Anal, 2019, 52: 199-211.
87.	Said M, Moustafa M, Wahba A. Color instance segmentation and classification of cervix images// 2019 IEEE 10th GCC Conference & Exhibition (GCC). Kuwait: IEEE, 2019: 1-6.
88.	Apiparakoon T, Rakratchatakul N, Chantadisai M, et al. MaligNet: semisupervised learning for bone lesion instance segmentation using bone scintigraphy. IEEE Access, 2020, 8: 27047-27066.
89.	Cao G, Song W, Zhao Z. Gastric cancer diagnosis with Mask R-CNN// 2019 11th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC). Hangzhou: IEEE, 2019: 60-63.
90.	Yan H, Lu H, Ye M, et al. Improved Mask R-CNN for lung nodule segmentation// 2019 10th International Conference on Information Technology in Medicine and Education (ITME). Xiamen: IEEE, 2019: 137-141.
91.	Bhatti H, Li J, Siddeeq S, et al. Multi-detection and segmentation of breast lesions based on Mask RCNN-FPN// 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul: IEEE, 2020: 2698-2704.

1. Hafiz A M, Bhat G M. A survey on instance segmentation: state of the art. Int J Multimed Inf R, 2020, 9: 171-189.
2. Jiang Lei, Chen Wenkai, Dong Bao, et al. A deep learning-based approach for glomeruli instance segmentation from multistained renal biopsy pathologic images. Am J Pathol, 2021, 191(8): 1431-1441.
3. Vania M, Lee D. Intervertebral disc instance segmentation using a multistage optimization mask-RCNN (MOM-RCNN). J Comput Des Eng, 2021, 8(4): 1023-1036.
4. 李佳昇. 基于深度学习的肝脏及肝脏肿瘤分割和检测的研究. 吉林: 吉林大学, 2020.
5. Hariharan B, Arbeláez P, Girshick R, et al. Simultaneous detection and segmentation// Fleet D, Pajdla T, Schiele B, et al. European Conference on Computer Vision (ECCV). ECCV 2014. Cham: Springer, 2014: 297-312.
6. 梁新宇, 林洗坤, 权冀川, 等. 基于深度学习的图像实例分割技术研究进展. 电子学报, 2020, 48(12): 2476-2486.
7. Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 580-587.
8. Pinheiro P, Collobert R, Dollar P. Learning to segment object candidates// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1990-1998.
9. Pinheiro P O, Lin T Y, Collobert R, et al. Learning to refine object segments// Leibe B, Matas J, Sebe N, et al. European Conference on Computer Vision (ECCV). ECCV 2016. Cham: Springer, 2016: 75-91.
10. Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE T Pattern Anal, 2017, 39(6): 1137-1149.
11. Li Y, Qi H, Dai J, et al. Fully convolutional instance-aware semantic segmentation// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 4438-4446.
12. Ge W, Huang W, Guo S, et al. Label-PEnet: sequential label propagation and enhancement networks for weakly supervised instance segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 3344-3353.
13. He K, Gkioxari G, Dollár P, et al. Mask R-CNN// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 2980-2988.
14. Chen L C, Hermans A, Papandreou G, et al. MaskLab: instance segmentation by refining object detection with semantic and direction features// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City: IEEE, 2018: 4013-4022.
15. Zhang Xiangyi, An Gaoyun, Liu Yutian. Mask R-CNN with feature pyramid attention for instance segmentation// 2018 14th IEEE International Conference on Signal Processing (ICSP). Wuhan: IEEE, 2018: 1194-1197.
16. Huang Z, Huang L, Gong Y, et al. Mask Scoring R-CNN// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 6402-6411.
17. Homayounfar N, Xiong Y, Liang J, et al. LevelSet R-CNN: a deep variational method for instance segmentation// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 555-571.
18. Lin Yuan, Zhao Qiu. Mask-RCNN with spatial attention for pedestrian segmentation in cyber–physical systems. Comput Commun, 2021, 180: 109-114.
19. Long Kun, Tang Lei, Pu Xiaorong, et al. Probability-based Mask R-CNN for pulmonary embolism detection. Neurocomputing, 2021, 422: 345-353.
20. Zhou H, Lei L, Xu Y, et al. Dual-supervised instance segmentation network combined with priori corner information// 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC). Xiamen: IEEE, 2019: 55-60.
21. Hu M, Li Y, Fang L, et al. A²-FPN: attention aggregation based feature pyramid network for instance segmentation// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 15338-15347.
22. Sofiiuk K, Barinova O, Konushin A. AdaptIS: adaptive instance selection network// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 7354-7362.
23. Wang W, Feng R, Chen J, et al. Nodule-Plus R-CNN and deep self-paced active learning for 3D instance segmentation of pulmonary nodules. IEEE Access, 2019, 7: 128796-128805.
24. Kai C, Pang J, Wang J, et al. Hybrid task cascade for instance segmentation// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 4969-4978.
25. Cai Z, Vasconcelos N. Cascade R-CNN: high quality object detection and instance segmentation. IEEE T Pattern Anal, 2019, 43(5): 1483-1498.
26. Wang K, Liew J H, Zou Y, et al. PANet: few-shot image semantic segmentation with prototype alignment// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 9196-9205.
27. Cao J, Cholakkal H, Rao M A, et al. D2Det: towards high quality object detection and instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 11482-11491.
28. Fan Zhibo, Yu Jingang, Liang Zhihao, et al. FGN: fully guided network for few-shot instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9169-9178.
29. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE T Pattern Anal, 2015, 39(4): 640-651.
30. Hayder Z, He X, Salzmann M. Boundary-aware instance segmentation// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 587-595.
31. Xu W, Wang H, Qi F, et al. Explicit shape encoding for real-time instance segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 5167-5176.
32. Li Z, Ma Y, Chen Y, et al. Joint COCO and Mapillary Workshop at ICCV 2019: COCO Instance Segmentation Challenge Track. (2020-10-06)[2022-10-16]. https: //arxiv.org/pdf/2010.02475.pdf.
33. Peng S, Jiang W, Pi H, et al. Deep Snake for real-time instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 8530-8539.
34. Chen X, Lian Y, Jiao L, et al. Supervised edge attention network for accurate image instance segmentation// European Conference on Computer Vision (ECCV). Glasgow: European, 2020: 617-631.
35. Kang B R, Kim H Y. BshapeNet: object detection and instance segmentation with bounding shape masks. Pattern Recogn Lett, 2020, 131: 449-455.
36. Cheng T, Wang X, Huang L, et al. Boundary-preserving Mask R-CNN// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 660-676.
37. Ding H, Qiao S, Yuille A, et al. Deeply shape-guided cascade for instance segmentation// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 8274-8284.
38. Tang C, Chen H, Li X, et al. Look closer to segment better: boundary patch refinement for instance segmentation// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 13921-13930.
39. Shu L, Jia J, Fidler S, et al. SGN: sequential grouping networks for instance segmentation// 2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE, 2017: 3516-3524.
40. Min B, Urtasun R. Deep watershed transform for instance segmentation// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 2858-2866.
41. Kirillov A, Levinkov E, Andres B, et al. InstanceCut: from edges to instances with multiCut// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 7322-7331.
42. Gao N, Shan Y, Wang Y, et al. SSAP: Single-shot instance segmentation with affinity pyramid// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 642-651.
43. Laradji I H, Rostamzadeh N, Pinheiro P O, et al. Proposal-based instance segmentation with point supervision// 2020 IEEE International Conference on Image Processing (ICIP). Abu Dhabi: IEEE, 2020: 2126-2130.
44. Zhang G, Lu X, Tan J, et al. RefineMask: towards high-quality instance segmentation with fine-grained features// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Online: IEEE, 2021: 6857-6865.
45. Bolya D, Zhou C, Xiao F, et al. YOLACT: real-time instance segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 9156-9165.
46. Fan R, Cheng M M, Hou Q, et al. S4Net: single stage salient-instance segmentation// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 6096-6105.
47. Chen X, Girshick R, He K, et al. TensorMask: a foundation for dense object segmentation// 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 2061-2069.
48. Bolya D, Zhou C, Xiao F, et al. YOLACT++: Better real-time instance segmentation. IEEE T Pattern Anal, 2020, 44(2): 1108-1121.
49. Lee S, Kim H. MPQ-YOLACT: mixed-precision quantization for lightweight YOLACT// 2020 IEEE International Conference on Consumer Electronics - Asia (ICCE-Asia). Busan: IEEE, 2020: 1-4.
50. Liang J, Homayounfar N, Ma W C, et al. PolyTransform: deep polygon transformer for instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9128-9137.
51. Zhang H, Tian Y, Wang K, et al. Mask SSD: An effective single-stage approach to object instance segmentation. IEEE T Image Process, 2020, 29: 2078-2093.
52. Huang C, Wu W, Lei Z. Efficient instance segmentation network// 2020 IEEE International Conference on Artificial Intelligence and Information Systems (ICAIIS). Dalian: IEEE, 2020: 93-101.
53. 周涛, 刘赟璨, 陆惠玲, 等. ResNet及其在医学图像处理领域的应用: 研究进展与挑战. 电子与信息学报, 2022, 44(1): 149-167.
54. Wang X, Kong T, Shen C, et al. SOLO: segmenting objects by locations// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 649-665.
55. Wang X, Zhang R, Kong T, et al. SOLOv2: dynamic and fast instance segmentation. (2020-03-23)[2022-10-16]. https://arxiv.org/pdf/2003.10152.pdf.
56. Ying H, Huang Z, Liu S , et al. EmbedMask: embedding coupling for one-stage instance segmentation. (2019-12-04)[2022-10-16]. https: //arxiv.org/pdf/1912.01954v2.pdf.
57. Xie E, Sun P, Song X, et al. PolarMask: single shot instance segmentation with polar representation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 12190-12199.
58. Chen H, Sun K, Tian Z, et al. BlendMask: top-down meets bottom-up for instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 8570-8578.
59. Lee Y, Park J. CenterMask: real-time anchor-free instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 13903-13912.
60. Tian Z, Shen C, Chen H. Conditional convolutions for instance segmentation// Vedaldi A, Bischof H, Brox T, et al. European Conference on Computer Vision (ECCV). ECCV 2020. Cham: Springer, 2020: 282-298.
61. Yu J, Yao J, Zhang J, et al. SPRNet: single-pixel reconstruction for one-stage instance segmentation. IEEE T Cybernetics, 2021, 51(4): 1731-1742.
62. Kirillov A, Wu Y, He K, et al. PointRend: image segmentation as rendering// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9796-9805.
63. Yang Hanqing, Zheng Liyang, Barzegar S G, et al. BorderPointsMask: one-stage instance segmentation with boundary points representation. Neurocomputing, 2022, 467: 348-359.
64. Zhang R, Tian Z, Shen C, et al. Mask encoding for single shot instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 10223-10232.
65. Dijkstra K, Loosdrecht J, Atsma W A, et al. CentroidNetV2: a hybrid deep neural network for small-object segmentation and counting. Neurocomputing, 2020, 423: 490-505.
66. Bohm A, Ucker A, Jager T, et al. ISOO_DL: instance segmentation of overlapping biological objects using deep learning// 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). Washington: IEEE, 2018: 1225-1229.
67. Stahl F, Meyer M, Schwanecke U. IST - Style transfer with instance segmentation// 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA). Dubrovnik: IEEE, 2019: 277-281.
68. 周涛, 侯森宝, 陆惠玲, 等. 探析 U-Net 的改进机制及其在医学图像分割的应用. 生物医学工程学杂志, 2022, 39(4): 806-825.
69. Wang W, Yu R, Huang Q, et al. SGPN: similarity group proposal network for 3D point cloud instance segmentation// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City: IEEE, 2018: 2569-2578.
70. Yi L, Zhao W, Wang H, et al. GSPN: generative shape proposal network for 3D instance segmentation in point cloud// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 3942-3951.
71. Hou J, Dai A, Nießner M. 3D-SIS: 3D semantic instance segmentation of RGB-D scans// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 4416-4425.
72. Yang B, Wang J, Clark R, et al. Learning object bounding boxes for 3D instance segmentation on point clouds. (2019-09-05)[2022-10-16]. https: //arxiv.org/pdf/1906.01140.pdf.
73. Zanjani F G, Pourtaherian A, Zinger S, et al. Mask-MCNet: tooth instance segmentation in 3D point clouds of intra-oral scans. Neurocomputing, 2021, 453: 286-298.
74. Pham Q H, Nguyen D T, Hua B S, et al. JSIS3D: joint semantic-instance segmentation of 3D point clouds with multi-task pointwise networks and multi-value conditional random fields// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 8819-8828.
75. Wang X, Liu S, Shen X, et al. Associatively segmenting instances and semantics in point clouds// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 4091-4100.
76. Jiang L, Zhao H, Shi S, et al. PointGroup: dual-set point grouping for 3D instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 4866-4875.
77. Han L, Zheng T, Xu L, et al. OccuSeg: occupancy-aware 3D instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 2937-2946.
78. Tan J, Chen L, Wang K, et al. SASO: joint 3D semantic-instance segmentation via multi-scale semantic association and salient point clustering optimization// 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020, 15(5): 366-379.
79. Engelmann F, Bokeloh M, Fathi A, et al. 3D-MPA: multi-proposal aggregation for 3D semantic instance segmentation// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 9028-9037.
80. Chen Feng, Wu Fei, Gao Guangwei, et al. JSPNet: learning joint semantic & instance segmentation of point clouds via feature self-similarity and cross-task probability. Pattern Recogn, 2022, 122: 108250.
81. Elich C, Engelmann F, Kontogianni T, et al. 3D bird’s-eye-view instance segmentation// German Conference on Pattern Recognition (GCPR). Dortmund: DAGM, 2019: 11824.
82. Du L, Tan J, Xue X, et al. 3DCFS: fast and robust joint 3D semantic-instance segmentation via coupled feature selection// 2020 IEEE International Conference on Robotics and Automation (ICRA). Paris: IEEE, 2020: 6868-6875.
83. Jiang H, Yan F, Cai J, et al. End-to-end 3D point cloud instance segmentation without detection// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 12793-12802.
84. Sun F, Xu Y, Sun W. SPSN: seed point selection network in point cloud instance segmentation// 2020 International Joint Conference on Neural Networks (IJCNN). Glasgow: IEEE, 2020: 1-8.
85. Liu C, Furukawa Y. MASC: multi-scale affinity with sparse convolution for 3D instance segmentation. (2019-02-12)[2022-10-16]. https: //arxiv.org/pdf/1902.04478.pdf.
86. Graham S, Chen H, Gamper J, et al. MILD-Net: minimal information loss dilated network for gland instance segmentation in colon histology images. Med Image Anal, 2019, 52: 199-211.
87. Said M, Moustafa M, Wahba A. Color instance segmentation and classification of cervix images// 2019 IEEE 10th GCC Conference & Exhibition (GCC). Kuwait: IEEE, 2019: 1-6.
88. Apiparakoon T, Rakratchatakul N, Chantadisai M, et al. MaligNet: semisupervised learning for bone lesion instance segmentation using bone scintigraphy. IEEE Access, 2020, 8: 27047-27066.
89. Cao G, Song W, Zhao Z. Gastric cancer diagnosis with Mask R-CNN// 2019 11th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC). Hangzhou: IEEE, 2019: 60-63.
90. Yan H, Lu H, Ye M, et al. Improved Mask R-CNN for lung nodule segmentation// 2019 10th International Conference on Information Technology in Medicine and Education (ITME). Xiamen: IEEE, 2019: 137-141.
91. Bhatti H, Li J, Siddeeq S, et al. Multi-detection and segmentation of breast lesions based on Mask RCNN-FPN// 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Seoul: IEEE, 2020: 2698-2704.

Previous Article
Design and preliminary application of outdoor flying pigeon-robot
Next Article
The current applicating state of neural network-based electroencephalogram diagnosis of Alzheimer’s disease

Journal of Biomedical Engineering

Medical image instance segmentation: from candidate region to no candidate region

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content