Skip to main content
SpringerLink
Log in

Examining the effect of synthetic data augmentation in polyp detection and segmentation

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

As with several medical image analysis tasks based on deep learning, gastrointestinal image analysis is plagued with data scarcity, privacy concerns and an insufficient number of pathology samples. This study examines the generation and utility of synthetic samples of colonoscopy images with polyps for data augmentation.

Methods

We modify and train a pix2pix model to generate synthetic colonoscopy samples with polyps to augment the original dataset. Subsequently, we create a variety of datasets by varying the quantity of synthetic samples and traditional augmentation samples, to train a U-Net network and Faster R-CNN model for segmentation and detection of polyps, respectively. We compare the performance of the models when trained with the resulting datasets in terms of F1 score, intersection over union, precision and recall. Further, we compare the performances of the models with unseen polyp datasets to assess their generalization ability.

Results

The average F1 coefficient and intersection over union are improved with increasing number of synthetic samples in U-Net over all test datasets. The performance of the Faster R-CNN model is also improved in terms of polyp detection, while decreasing the false-negative rate. Further, the experimental results for polyp detection outperform similar studies in the literature on the ETIS-PolypLaribDB dataset.

Conclusion

By varying the quantity of synthetic and traditional augmentation, there is the potential to control the sensitivity of deep learning models in polyp segmentation and detection. Further, GAN-based augmentation is a viable option for improving the performance of models for polyp segmentation and detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and material

Datasets used are all publicly available online. The synthetic data produced by the proposed framework will be made available online.

Code availability

The implementation code will be made available online. The link is provided in the paper.

Notes

  1. https://github.com/sing-group/deep-learning-colonoscopy.

  2. https://github.com/facebookresearch/detectron2.

  3. https://github.com/PrinceEAdjei/SyntheticGI-Imaging.

References

  1. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van der Laak JAWM, Van Ginneken B, Sánchez CI (2017) A survey of deep learning in medical image analysis. Med Image Anal 42:60–88. https://doi.org/10.1016/j.media.2017.07.006

    Article  PubMed  Google Scholar 

  2. Wu J, Chen JM, Cai JT (2021) Application of artificial intelligence in gastrointestinal endoscopy. J Clin Gastroenterol 55(2):110–120. https://doi.org/10.1097/MGC.0000000000001423

  3. Kudo SE, Mori Y, Misawa M, Takeda K, Kudo T, Itoh H, Oda M, Mori K (2019) Artificial intelligence and colonoscopy: current status and future perspectives. Dig Endosc 31(4):363–371. https://doi.org/10.1111/den.13340

    Article  PubMed  Google Scholar 

  4. Sánchez-Peralta LF, Bote-Curiel L, Picón A, Sánchez-Margallo FM, Pagador JB (2020) Deep learning to find colorectal polyps in colonoscopy: a systematic literature review. Artif Intell Med 108:101923. https://doi.org/10.1016/j.artmed.2020.101923

  5. Haggar FA, Boushey RP (2009) Colorectal cancer epidemiology: Incidence, mortality, survival, and risk factors. Clin Colon Rectal Surg 22(4):191–197. https://doi.org/10.1055/s-0029-1242458

  6. Tsuboi A, Oka S, Aoyama K, Saito H, Aoki T, Yamada A, Matsuda T, Fujishiro M, Ishihara S, Nakahori M, Koike K, Tanaka S, Tada T (2020) Artificial intelligence using a convolutional neural network for automatic detection of small-bowel angioectasia in capsule endoscopy images. Digestive Endosc 32(3):382–390. https://doi.org/10.1111/den.13507

  7. Liu D, Jiang H, Rao N, Du W, Luo C, Li Z, Zhu L, Gan T (2020) Depth information-based automatic annotation of early esophageal cancers in gastroscopic images using deep learning techniques. IEEE Access 8:97907–97919. https://doi.org/10.1109/ACCESS.2020.2996631

    Article  Google Scholar 

  8. Guo X, Zhang N, Guo J, Zhang H, Hao Y, Hang J (2009) Automated polyp segmentation for colonoscopy images: a method based on convolutional neural networks and ensemble learning. Med Phys 46(12):5666–5676. https://doi.org/10.1002/mp.13865

    Article  Google Scholar 

  9. Fan DP, Ji GP, Zhou T, Chen G, Fu H, Shen J, Shao L (2020) PraNet: Parallel reverse attention network for polyp segmentation. In: Medical Image Computing and Computer Assisted Intervention—MICCAI 2020. Lecture Notes in Computer Science, volume 12266, pp 263–273. https://doi.org/10.1007/978-3-030-59725-2_26

  10. Qadir HA, Shin Y, Solhusvik J, Bergsland J, Aabakken L, Balasingham I (2021) Toward real-time polyp detection using fully CNNs for 2D Gaussian shapes prediction. Med Image Anal 68:101897. https://doi.org/10.1016/j.media.2020.101897

  11. Liew WS, Tang TB, Lin CH, Lu CK (2021) Automatic colonic polyp detection using integration of modified deep residual convolutional neural network and ensemble learning approaches. Comp Methods Programs Biomed 206:106114. https://doi.org/10.1016/j.cmpb.2021.106114

  12. Becq A, Chandnani M, Bharadwaj S, Baran B, Ernest-Suarez K, Gabr M, Glissen-Brown J, Sawhney M, Pleskow DK, Berzin TM (2020) Effectiveness of a deep-learning polyp detection system in prospectively collected colonoscopy videos with variable bowel preparation quality. J Clin Gastroenterol 54(6):554–557. https://doi.org/10.1097/MCG.0000000000001271

    Article  PubMed  Google Scholar 

  13. Skrede OJ, De Raedt S, Kleppe A, Hveem TS, Liestøl K, Maddison J, Askautrud HA, Pradhan M, Nesheim JA, Albregtsen F, Farstad IN, Domingo E, Church DN, Nesbakken A, Shepherd NA, Tomlinson I, Kerr R, Novelli M, Kerr DJ, Danielsen HE (2020) Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. Lancet 395:350–360. https://doi.org/10.1016/S0140-6736(19)32998-8

  14. Du W, Rao N, Liu D, Jiang H, Luo C, Li Z, Gan T, Zeng B (2019) Review on the applications of deep learning in the analysis of gastrointestinal endoscopy images. IEEE Access 7:142053–142069. https://doi.org/10.1109/ACCESS.2019.2944676

    Article  Google Scholar 

  15. Borgli H, Thambawita V, Smedsrud PH, Hicks S, Jha D, Eskeland SL, Randel KR, Pogorelov K, Lux M, Nguyen DTD, Johansen D, Griwodz C, Stensland HK, Garcia-Ceja E, Schmidt PT, Hammer HL, Riegler MA, Halvorsen P, De Lange T (2020) HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific Data 7(28):1–14. https://doi.org/10.1038/s41597-020-00622-y

    Article  Google Scholar 

  16. Deng J, Dong W, Socher R, Li Li-Jia, Li K, Li Fei-Fei (2009) ImageNet: A large-scale hierarchical image database. IEEE conference on computer vision and pattern recognition (CVPR), 2009. Published online: https://doi.org/10.1109/CVPR.2009.5206848

  17. Sánchez-Peralta LF, Pagador JB, Picón A, José AC, Polo F, Andraka A, Bilbao R, Glove B, Saratxaga CL (2020) Sánchez-Margallo FMPICCOLO, white-light and narrow-band imaging colonoscopic dataset: a performance comparative models and datasets. Appl Sci 10(23):8501. https://doi.org/10.3390/app10238501

  18. Bernal J, Sánchez FJ, Fernández-Esparrach G, Gil D, Rodríguez C, Vilariño F (2015) WM-DOVA maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Comp Med Imag Graphics 43:99–111. https://doi.org/10.1016/j.compmedimag.2015.02.007

  19. Bernal J, Sánchez FJ, Vilariño F (2012) Towards automatic polyp detection with a polyp appearance model. Pattern Recogn 45(9):3166–3182. https://doi.org/10.1016/j.patcog.2012.03.002

    Article  Google Scholar 

  20. Sharib Ali, Debesh Jha, Noha Ghatwary, Stefano Realdon, Renato Cannizzaro, Osama E. Salem, Dominique Lamarque, Christian Daul, Kim V. Anonsen, Michael A. Riegler, Pal Halvorsen, Jens Rittscher, Thomas de Lange, James E. East (2021) PolypGen: A multi-center polyp detection and segmentation dataset for generalizability assessment. arXiv:2106.04463v1 [eess.IV], 8 June 2021. https://tinyurl.com/yhnb82v7

  21. Nogueira-Rodríguez A, Domínguez-Carbajales R, López-Fernández H, Iglesias A, Cubiella J, Fdez-Riverola F, Reboiro-Jato M, Glez-Peña D (2021) Deep Neural Networks approaches for detecting and classifying colorectal polyps. Neurocomputing 423(29):721–734. https://doi.org/10.1016/j.neucom.2020.02.123

    Article  Google Scholar 

  22. Shorten C, Khoshgoftaar TM (2019) A survey on Image Data augmentation for deep learning. J Big Data, Volume 6, article number 60. https://doi.org/10.1186/s40537-019-0197-0

  23. Sánchez-Peralta LF, Bote-Curiel L, Picón A, Sánchez-Margallo FM, Pagador JB (2020) Unravelling the effect of data augmentation transformations in polyp segmentation. Int J Comp Assist Radiol Surg 15:1975–1988. https://doi.org/10.1007/s11548-020-02262-4

  24. Rau A, Edwards PJE, Ahmad OF, Riordan P, Janatka M, Lovat LB, Stoyanov D (2019) Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy. Int J Comp Assist Radiol Surg 14:1167–1176. https://doi.org/10.1007/s11548-019-01962-w

  25. De Almeida Thomaz V, Sierra-Franco CA and Raposo AB (2021) Training data enhancements for improving colonic polyp detection using deep convolutional neural networks. Artif Intell Med 111:101988. https://doi.org/10.1016/j.artmed.2020.101988

  26. He F, Chen S, Li S, Zhou L, Zhang H, Peng H, Huang X (2021) Colonoscopic Image Synthesis For Polyp Detector Enhancement Via Gan And Adversarial Training. In: IEEE 18th International Symposium on Biomedical Imaging (ISBI), April 2021, pp 1887–1891. https://doi.org/10.1109/ISBI48211.2021.9434050.

  27. Jha D, Smedsrud PH, Riegler MA, Halvorsen P, De Lange T, Johansen D, Johansen HD (2020) Kvasir-SEG: a segmented polyp dataset. In: Multimedia Modeling. MMM 2020. Lecture Notes in Computer Science, volume 11962, 2020, pp 451–462. https://doi.org/10.1007/978-3-030-37734-2_37

  28. Silva J, Histace A, Romain O, Dray X, Granado B (2014) Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. Int J Comput Assist Radiol Surg 9:283–293. https://doi.org/10.1007/s11548-013-0926-3

    Article  PubMed  Google Scholar 

  29. Ronneberger O, Fischer P, Brox I (2015) U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A. (eds) Medical image computing and computer-assisted intervention—MICCAI 2015. Lecture Notes on Computer Science, volume 9351, 2015, pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

  30. Shaoqin Ren, Kaiming He, Ross Girshick, Jian Sun (2015) Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In: Advances in Neural Information Processing Systems 28 (NIPS 2015) https://arxiv.org/abs/1506.01497

  31. Zheng Y, Zhang R, Yu R, Jiang Y, Mak TWC, Wong S H, Lau JYW, Poon CCY (2018) Localisation of colorectal polyps by convolutional neural network features learnt from white light and narrow band endoscopic images of multiple databases. In: Proceedings of annual international conference IEEE Eng. Med. Biol. Soc. EMBS, July 2018, pp 4142–4145. DOI:https://doi.org/10.1109/EMBC.2018.8513337

  32. Vázquez D, Bernal J, Sánchez FJ, Fernández-Esparrach G, López AM, Romero A, Drozdzal M, Courville A (2017) A benchmark for Endoluminal scene segmentation of colonoscopy images. J Healthcare Eng Special Issue: Computer Vision in Healthcare Applications, Volume 2017, Article ID 4037190 https://doi.org/10.1155/2017/4037190

  33. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. In: Advances in Neural Information Processing Systems, volume 3, pp 2672–2680. https://arxiv.org/abs/1406.2661v1

  34. Armanious K, Jiang C, Fischer M, küstner T, Hepp T, Nikolaou K, Gatidis S, Yang B (2020) MedGAN: Medical image translation using GANS. Computerized Medical Imaging and Graphics, vol 79, pp 101684. https://doi.org/10.1016/j.compmedimag.2019.101684

  35. Isola P, J.Y. Zhu, Tinghui Zhou, Alexei A Efros (2017) Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), January 2017, pp 5967–5976. https://arxiv.org/abs/1611.07004

  36. Zhang R, Isola P, Alexei A Efros, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp 586–595. https://arxiv.org/abs/1801.03924

  37. Wilcox RR (2003) Comparing two independent groups. In: Wilcox RR (ed) Applying contemporary statistical techniques. Elsevier Science, Berlin, pp 237–284

    Chapter  Google Scholar 

  38. Ali B (2019) Pros and cons of GAN evaluation measures. Comput Vis Image Underst 179:41–65. https://doi.org/10.1016/j.cviu.2018.10.009

    Article  Google Scholar 

  39. Koehn C, Rex DK, Allen J, Bhatti U, Bhavsar-Burke I, Chandrasekar VT, Challa A, Duvvuri A, Dakhoul L, Ha J, Hamade N, Hicks SB, Jansson-Knodell C, Krajicek E, Kundumadam SD, Nutalapati V, Phatharacharukuk PP, Razmdjou S, Saito A, Sarkis F, Sutton R, Wehbeh A, Sharma P, Desai M (2021) Optical diagnosis of colorectal polyps using novel blue light imaging (BLI) classification among trainee endoscopists. Digestive Endoscopy, Published online: 30 May 2021. https://doi.org/10.1111/den.14050

  40. Bernal J, Tajkbaksh N, Sánchez FJ, Matuszewski BJ, Chen H, Yu L, Angermann Q, Romain O, Rustad B, Balasingham I, Pogorelov K, Choi S, Debard Q, Maier-Hein L, Speidel S, Stoyanov D, Brandao P, Córdova H, Sánchez-Montes C, Gurudu SR, Fernández-Esparrach G, Dray X, Liang J, Histace A (2017) Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 endoscopic vision challenge. IEEE Trans Med Imaging 6:1231–1249. https://doi.org/10.1109/TMI.2017.2664042

Download references

Funding

The present study was funded by the National Natural Science Foundation of China (61872405 and 61720106004) and the Key R&D Project of Sichuan Province (2020YFS0243).

Author information

Authors and Affiliations

  1. Key Laboratory for Neuroinformation of Ministry of Education, University of Electronic Science and Technology of China, Chengdu, 610054, China

    Prince Ebenezer Adjei, Zenebe Markos Lonseko, Wenju Du, Han Zhang & Nini Rao

  2. School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China

    Prince Ebenezer Adjei, Zenebe Markos Lonseko, Wenju Du, Han Zhang & Nini Rao

  3. Department of Computer Engineering, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana

    Prince Ebenezer Adjei

Authors
  1. Prince Ebenezer Adjei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zenebe Markos Lonseko
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenju Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Han Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nini Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

PEA helped in conception and design. PEA, NR, HZ performed acquisition, analysis and interpretation of data. PEA, NR drafted the article. WD, HZ, NR, ZML contributed to revision of the article critically for important intellectual content. PEA, ZML, WD, HZ and NR helped in final approval of the article.

Corresponding author

Correspondence to Nini Rao.

Ethics declarations

Conflict of interest

Prince Ebenezer Adjei declares no conflict of interest. Zenebe Markos Lonseko declares no conflict of interest. Wenju Du declares no conflict of interest. Han Zhang declares no conflict of interest. Nini Rao declares no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the Ethics Committee of West China Hospital of Sichuan University and University of Electronic Science and Technology of China, and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Consent to participate

Informed consent was obtained from all patients included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Adjei, P.E., Lonseko, Z.M., Du, W. et al. Examining the effect of synthetic data augmentation in polyp detection and segmentation. Int J CARS 17, 1289–1302 (2022). https://doi.org/10.1007/s11548-022-02651-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02651-x

Keywords

Search

Navigation