Skip to main content
Springer Nature Link
Log in

TongueMobile: automated tongue segmentation and diagnosis on smartphones

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Tongue diagnosis is a useful process in traditional Chinese medicine to assess diseases non-invasively by visually inspecting the tongue and its various properties. In this study, we developed an automated tongue diagnosis system with a mobile app for the general public. The image-segmentation component extracts the tongue body image from an input photograph taken by a smartphone. The tongue-coating color classification component predicts the category of the coating color. The segmented image and diagnosis results are returned to the app and shown to the user. Experimental results show that Mask R-CNN is the optimal choice for tongue-image segmentation under various input image conditions based on the mean interaction over union value of \(91\%\) and the Dice score of \(95\%\). ResNeXt outperformed other baseline tongue-coating color classification models. In addition, when the input image is adjusted with our color-correction modules in advance, the classification accuracy of ResNeXt101 is improved by approximately \(12\%\).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availibility

The datasets generated during and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. https://cloudtcm.com/article/84 (2020)

  2. Lin B, Xie J, Li C, Qu Y (2018) DeepTongue: tongue segmentation via ResNet. In: Proceedings of 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1035–1039

  3. Wei YK, Fan P, Zeng G (2014) Application of improved grabcut method in tongue diagnosis system. Transducer Microsyst Technol 33:157–160

    Google Scholar 

  4. Chen S, Fu H, Wang Y (2012) Application of improved graph theory image segmentation algorithm in tongue image segmentation. Jisuanji Gongcheng yu Yingyong (Comput Eng Appl) 48(5):201–203

    Google Scholar 

  5. Guo J, Yang Y, Wu Q, Su J, Ma F (2016) Adaptive active contour model based automatic tongue image segmentation. In: Proceedings of 9th international congress on image and signal processing, BioMedical engineering and informatics (CISP-BMEI), pp 1386–1390

  6. Shi M, Li G, Li F (2013) C2G2FSnake: automatic tongue image segmentation utilizing prior knowledge. Sci China Inf Sci 56(9):1–14

    Article  Google Scholar 

  7. Ling Z, Jian Q (2010) Tongue-image segmentation based on gray projection and threshold-adaptive method. Chin J Tissue Eng Res 14(9):1638–1641

    Google Scholar 

  8. Yu-ke W (2011) Tongue image segmentation method based on adaptive thresholds. Comput Technol Dev 09:63–65

    Google Scholar 

  9. Fu ZC, Li XQ, Li FF (2009) Tongue image segmentation based on snake model and radial edge detection. J Image Graphics 14(4):688–693

    MathSciNet  Google Scholar 

  10. Qing-Li L, Yong-Qi X, Jian-Yu W, Xiao-Qiang Y (2007) Automated tongue segmentation algorithm based on hyperspectral image. J Infrared Millim Waves 26(1):77–80

    Google Scholar 

  11. Pinheiro O, Pedro O, Collobert R, Dollar P (2015) Learning to segment object candidates. In: Proceedings of advances in neural information processing systems (NeurIPS), vol 28

  12. Pinheiro T-Y, Pedro O, Lin C, Ronanand Dollár P (2016) Learning to refine object segments. In: Proceedings of European conference on computer vision (ECCV), pp 75–91

  13. Wang X, Kong T, Shen C, Jiang Y, Li L (2020) SOLO: segmenting objects by locations. In: Proceedings of European conference on computer vision (ECCV), pp 649–665

  14. He K, Gkioxari G, Dollár P, Girshick RB (2017) Mask R-CNN. In: Proceedings of IEEE international conference on computer vision (ICCV), pp 2980–2988

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  16. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5987–5995

  17. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3431–3440

  18. Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26(3):297–302

    Article  Google Scholar 

  19. Eelbode T, Bertels J, Berman M, Vandermeulen D, Maes F, Bisschops R, Blaschko MB (2020) Optimization for medical image segmentation: theory and practice when evaluating with dice score or Jaccard index. IEEE Trans Med Imaging 39(11):3679–3690

    Article  Google Scholar 

  20. Zhou J, Zhang Q, Zhang B, Chen X (2019) TongueNet: a precise and fast tongue segmentation system using u-net with a morphological processing layer. Appl Sci 9(15):3128

    Article  Google Scholar 

  21. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of 2017 IEEE conference on computer vision and pattern recognition (CVPR). CVPR’17, pp 936–944

  22. Ryu I, Siio I (2014) TongueDx: a tongue diagnosis for health care on smartphones. In: Proceedings of 5th augmented human international conference (AH), pp 25–1252

  23. Li X, Yang D, Wang Y, Yang S, Qi L, Li F, Gan Z, Zhang W (2019) Automatic tongue image segmentation for real-time remote diagnosis. In: 2019 IEEE international conference on bioinformatics and biomedicine (BIBM), pp 409–414

  24. Liu W, Zhou C, Li Z, Hu Z (2020) Patch-driven tongue image segmentation using sparse representation. IEEE Access 8:41372–41383

    Article  Google Scholar 

  25. Huang Y, Lai Z, Wang W (2021) TU-Net: a precise network for tongue segmentation. In: Proceedings of the 2020 9th international conference on computing and pattern recognition (ICCPR), pp 244–249

  26. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of medical image computing and computer-assisted intervention (MICCAI), pp 234–241

  27. Wei LIU, Jinming CHEN, Bo LIU, Wei HU, Xingjin WU, Hui ZHOU (2022) Tongue image segmentation and tongue color classification based on deep learning. Digit Chin Med 5(3):253–263

    Article  Google Scholar 

  28. Yang Z, Zhao Y, Yu J, Mao X, Xu H, Huang L (2022) An intelligent tongue diagnosis system via deep learning on the android platform. Diagnostics 12(10):2451

    Article  Google Scholar 

  29. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of 2017 IEEE international conference on computer vision (ICCV), pp 618–626

  30. Li J, Zhang Z, Zhu X, Zhao Y, Ma Y, Zang J, Li B, Cao X, Xue C (2022) Automatic classification framework of tongue feature based on convolutional neural networks. Micromachines 13(4):501

    Article  Google Scholar 

  31. Girshick R (2015) Fast R-CNN. In: Proceedings of the 2015 IEEE international conference on computer vision (ICCV), pp 1440–1448

  32. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Proceedings of the 28th international conference on neural information processing systems (NeurIPS), pp 91–99

  33. Uijlings JRR, van de Sande KEA, Gevers T, Smeulders AWM (2013) Selective search for object recognition. Int J Comput Vis 104:154–171

    Article  Google Scholar 

  34. Zheng S, Lu J, Zhao H, Zhu X, Yabiao Wang ZL, Fu Y, Feng J, Xiang T, Torr PHS, Zhang L (2021) Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: IEEE conference on computer vision and pattern recognition, (CVPR), pp 6881–6890

  35. Ranftl R, Bochkovskiy A, Koltun V (2021) Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp 12179–12188

  36. Strudel R, Garcia R, Laptev I, Schmid C (2021) Segmenter: transformer for semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp 7262–7272

  37. Zhang B, Tian Z, Tang Q, Chu X, Wei X, Shen C, liu Y (2022) SegViT: semantic segmentation with plain vision transformers. In: Proceedings of advances in neural information processing systems (NeurIPS)

  38. Afifi M (2018) Semantic white balance: semantic color constancy using convolutional neural network. CoRR abs/1802.00153

  39. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of 26th advances in neural information processing systems (NeurIPS), pp 1097–1105

  40. Afifi M, Price B, Cohen S, Brown MS (2019) When color constancy goes wrong: correcting improperly white-balanced images. In: Proceedings of 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1535–1544

  41. Chen Y, Biookaghazadeh S, Zhao M (2019) Exploring the capabilities of mobile devices in supportingdeep learning. In: Proceedings of the 4th ACM/IEEE symposium on edge computing (SEC), pp 127–138

  42. Zhou H, Zhang W, Wang C, Ma X, Yu H (2021) BBNet: a novel convolutional neural network structure in edge-cloud collaborative inference. Sensors 21(13):4494

    Article  Google Scholar 

  43. Xia C, Zhao J, Cui H, Feng X, Xue J (2019) DNNTune: automatic benchmarking dnn models for mobile-cloud computing. ACM Trans Archit Code Optim 16(4):1–26

    Article  Google Scholar 

  44. Kang Y, Hauswald J, Gao C, Rovinski A, Mudge T, Mars J, Tang L (2017) Neurosurgeon: collaborative intelligence between the cloud and mobile edge. In: Proceedings of the twenty-second international conference on architectural support for programming languages and operating systems (ASPLOS), pp 615–629

  45. Zhou Z, Chen X, Li E, Zeng L, Luo K, Zhang J (2019) Edge intelligence: paving the last mile of artificial intelligence with edge computing. In: Proceedings of the IEEE, vol 107, pp 1738–1762

  46. Wu J, Wang L, Pei Q, Cui X, Liu F, Yang T (2022) HiTDL: high-throughput deep learning inference at the hybrid mobile edge. IEEE Trans Parallel Distrib Syst 33(12):4499–4514

    Article  Google Scholar 

  47. Zhang X, Yang Y, Feng Y, Chen Z (2019) Software engineering practice in the development of deep learning applications. CoRR 1910.03156

  48. Gu R, Niu C, Wu F, Chen G, Hu C, Lyu C, Wu Z (2021) From server-based to client-based machine learning: a comprehensive survey. ACM Comput Surv 54(1):1–36

    Article  Google Scholar 

  49. Dhar S, Guo J, Liu JJ, Tripathi S, Kurup U, Shah M (2021) A survey of on-device machine learning: an algorithms and learning theory perspective. ACM Trans Internet Things 2(3):1–49

    Article  Google Scholar 

  50. Bianco S, Cadene R, Celona L, Napoletano P (2018) Benchmark analysis of representative deep neural network architectures. IEEE Access 6:64270–64277

    Article  Google Scholar 

  51. Huang Y, Qiao X, Ren P, Liu L, Pu C, Dustdar S, Chen J (2022) A lightweight collaborative deep neural network for the mobile web in edge cloud. IEEE Trans Mobile Comput 21(7):2289–2305

    Article  Google Scholar 

  52. Stoica I, Song D, Popa RA, Patterson D, Mahoney MW, Katz R, Joseph AD, Jordan M, Hellerstein JM, Gonzalez J, Goldberg K, Ghodsi A, Culler D, Abbeel P (2017) A Berkeley view of systems challenges for ai. Technical Report UCB/EECS-2017-159, EECS Department, University of California, Berkeley

  53. Cai H, Lin J, Lin Y, Liu Z, Tang H, Wang H, Zhu L, Han S (2022) Enable deep learning on mobile devices: methods, systems, and applications. ACM Trans Des Autom Electron Syst 27(3):1–50

    Article  Google Scholar 

  54. Finley DR (2006) HSP color model—alternative to HSV (HSB) and HSL. https://alienryderflex.com/hsp.html

  55. Lin M, Chen Q, Yan S (2014) Network in network. In: Proceedings of 2nd international conference on learning representations (ICLR)

  56. Goodfellow IJ, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  57. Documentation P (2022) socket—low-level networking interface. https://docs.python.org/3/library/socket.html

  58. Russell B, Torralba A, Murphy K, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173

    Article  Google Scholar 

  59. Rother C, Kolmogorov V, Blake A (2004) “GrabCut’’: interactive foreground extraction using iterated graph cuts. ACM Trans Graphics 23(3):309–314

    Article  Google Scholar 

  60. Clark A (2015) Pillow (PIL Fork) documentation. https://pillow.readthedocs.io/en/stable/

  61. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Proceedings of 3rd international conference on learning representations (ICLR)

  62. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9

  63. Weng W, Deaton J, Natarajan V, Elsayed GF, Liu Y (2020) Addressing the real-world class imbalance problem in dermatology. In: Machine learning for health workshop, ML4H@NeurIPS 2020, Virtual Event, 11 December 2020. Proceedings of Machine Learning Research, vol 136, pp 415–429

  64. Khushi M, Shaukat K, Alam TM, Hameed IA, Uddin S, Luo S, Yang X, Reyes MC (2021) A comparative performance analysis of data resampling methods on imbalance medical data. IEEE Access 9:109960–109975

    Article  Google Scholar 

  65. Ghorbani A, Natarajan V, Coz D, Liu Y (2020) DermGAN: synthetic generation of clinical skin images with pathology. In: Proceedings of the machine learning for health NeurIPS workshop, vol 116, pp 155–170

Download references

Author information

Author notes

  1. Zih-Hao Huang and Wei-Cheng Huang have contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Chiayi University, No. 300, Syuefu Rd., Chiayi City, 600355, Taiwan

    Zih-Hao Huang & Wei-Cheng Huang

  2. School of Post-Baccalaureate Chinese Medicine, Tzu Chi University, No. 701, Sec. 3, Zhongyang Rd., Hualien City, Hualien County, 97004, Taiwan

    Hsien-Chang Wu

  3. Department of Chinese Medicine, Taipei Tzu Chi Hospital, The Buddhist Tzu Chi Medical Foundation, No. 289, Jianguo Rd., Xindian Dist, New Taipei City, 23142, Taiwan

    Hsien-Chang Wu

  4. Department of Computer Science and Information Engineering, National Dong Hwa University, No. 1, Sec. 2, University Rd., Shoufeng Township, Hualien County, 974301, Taiwan

    Wen-Chieh Fang

Authors
  1. Zih-Hao Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Wei-Cheng Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Hsien-Chang Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Wen-Chieh Fang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Wen-Chieh Fang.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A

Appendix A

To conduct an initial investigation into the class imbalance problem in this study, we ensured that the number of data instances for each of the other three colors matched that of the gray-black category, which was set at 435. This was achieved by randomly selecting instances from the original sample.

To assess the pixel accuracy of various models, as presented in Sect. 4.4.2, we performed evaluations both with and without prior color correction for the input images. The results, as shown in Table 5, indicate that the introduction of color correction led to an improvement in classification accuracy ranging from approximately 4–\(9\%\) across these models.

Specifically, ResNeXt101, when coupled with image color correction, outperformed the other models in terms of accuracy. However, it is worth noting that the overall accuracy of all models remained lower compared to the results obtained from models trained on imbalanced class datasets, as demonstrated in Table 4. We attribute this disparity to the fact that the balanced class dataset used in this evaluation is smaller in size, consisting of 1740 instances, whereas the imbalanced class dataset contained 4048 instances.

Table 5 Accuracy comparison \((\%)\) of different tongue-coating color classification models with and without color correction, where bold numbers indicate the highest values
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, ZH., Huang, WC., Wu, HC. et al. TongueMobile: automated tongue segmentation and diagnosis on smartphones. Neural Comput & Applic 35, 21259–21274 (2023). https://doi.org/10.1007/s00521-023-08902-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08902-5

Keywords