A novel tongue feature extraction method on mobile devices

Highlights

• Construct a dataset containing lingual and sublingual images taken by smartphones.

• Propose a tongue segmentation framework called DAMI.

• Propose a lightweight tongue feature classification network based on ECA Attention.

• Develop a tongue feature extraction WeChat Mini Program for mobile devices.

Abstract

In Traditional Chinese Medicine (TCM), tongue diagnosis is essential for symptom differentiation and treatment selection. Compared with traditional tongue diagnostic instruments, deploying a tongue diagnosis system on mobile devices is more convenient to monitor general health and facilitates the development of telemedicine. However, limited by both the quality and quantity of tongue images taken by mobile devices, extracting tongue features of the images maintains a great challenge. In this paper, we present a tongue feature extraction method on mobile devices, containing a high-accuracy tongue segmentation method based on Moment Invariants with Data Augmentation (DAMI) and an efficient and lightweight feature classification model with an attention mechanism. Meanwhile, we construct a novel tongue image dataset from mobile devices for extracting tongue features, significantly, first including sublingual images which are beneficial to extracting sublingual vein features. Extensive experiments on two datasets demonstrate the effectiveness and robustness of our method. Furthermore, our method greatly reduces the computing and storage demands compared with other current methods, providing a good prerequisite for deployment on mobile devices. Finally, to demonstrate the potential application of our proposed method, we develop a TCM intelligence tongue diagnosis application, which can be accessed through the WeChat Mini Program or web version, exhibiting its great potential in clinical diagnosis and health monitoring.

Introduction

As the central organ of the human body, the tongue is an essential part of Traditional Chinese Medicine (TCM) diagnosis. Tongue diagnosis can reflect a patient’s physical condition by observing the state of the tongue, including the texture of the tongue, the shape of the tongue, and the state of the tongue coating, which can serve as important markers for exploring physiological transformations and pathological changes in the human body [1]. However, the traditional tongue diagnosis is susceptible to bias due to the personal level of physicians. Therefore, it is urgent to conduct objective and quantitative studies on tongue diagnosis to help physicians with the adjunctive therapy and improve the accuracy of TCM tongue diagnosis.

Tongue features are essential for symptom differentiation and treatment selection in TCM. The tongue includes the tongue body, the tongue coating, and the sublingual veins. And analysis of these tongue features can help the physician detect the patient’s physical condition and thus suggest effective treatment or disease control measures. For example, the features of the tongue coating can reflect changes in the body’s spleen and stomach. Among them, greasy tongue coating can reflect the degree of dampness and heat in the body. Studies have shown that the degree of greasy tongue coating is more correlated with chronic gastritis [2]. Moreover, the thickness of tongue coating reflects the changes in the functions of the body’s internal organs [3]. Both idiopathic membranous nephropathy and type 2 diabetes are closely related to the thinness of the tongue coating [4], [5]. In addition, the sublingual veins provide an insight into the health of the body, especially the condition of the heart and liver [6]. Analysis of the features of the sublingual veins has a great role in the diagnosis and prognosis of portal hypertension syndrome (PHS) and hepatocellular carcinoma (HCC) [6], [7]. However, there is a wide variety of these tongue features, and it is challenging to identify them one by one.

At present, the standardized and objective study of TCM tongue diagnosis is mainly done by using professional tongue diagnostic instruments to extract the tongue features. However, tongue diagnostic instruments are expensive and can only be used by patients in hospitals, limiting the use of tongue diagnostics in healthcare. At the same time, with the development of technology, mobile devices such as smartphones are becoming more and more convenient to use, and their built-in cameras are becoming more and more pixelated, which can fully satisfy our needs of taking tongue images for analysis. Therefore, the deployment of a tongue diagnosis system on mobile devices can facilitate the development of telemedicine, allowing people to interact with physicians through a tongue diagnosis system even if they do not go to the hospital, which dramatically improves the efficiency of medical diagnosis.

In order to quantify tongue feature extraction, many pieces of research based on traditional image processing techniques have been proposed in recent years. Shao et al. applied a preprocessed tongue image combined with the length–width, depth, and luminance variations of the concave surface of the tongue border. They used a threshold-based method to determine whether it was a dentate tongue [8]. Bai et al. combined the Otsu model with the split–merge method in RGB space and used a threshold to extract the thin or thick coating [9]. Deep learning has been adopted for tongue feature extraction with the rapidly increasing tongue image data and computational capability. For example, Hou et al. first extracted tongue contours using a contour extraction algorithm. They then applied a convolutional neural network (CNN) to tongue color classification to classify tongue color by deep learning [10]. Wang et al. proposed a method manually to segment the tongue image to remove the background noise, then used the ResNet34 network [11] to classify the tooth-marked tongue, and used an external dataset to verify the method’s robustness, which achieved excellent results [12]. And then, based on this, Wang et al. used ResNet34 to form the GreasyCoatNet34 network to classify the greasy tongue coating [13]. Xu et al. [14] obtained excellent results using a fine-grained classification network called DFL-CNN to classify tongue coating features after the segmentation of tongue images using neural networks. Tang et al. [15] used a multi-instance learning based approach in order to classify tongue coating features, first using prior knowledge of greasy coating to obtain suspicious patches, then using CNN to extract tongue features, and finally using multi-instance support vector machine (MI-SVM) to classify tongue coating features. We can find that all these methods are designed for a single feature such as cracked tongue, tooth-marked tongue or tongue coating features alone and cannot be widely used in all features of tongue images. In addition, the methods mentioned in [8], [9], [12], [13], [14], [15] use datasets taken in a standard environment (e.g., tongue diagnostic instrument) and may not be suitable for using on mobile devices, which take tongue images of different shapes and different lighting and angles. Also none of these deep learning-based methods are devised for mobile devices, not considering storage and computational cost on mobile devices.

In general, current methods have achieved specific results in extracting various tongue features, but there are still some problems to be solved. Firstly, most research work for extracting tongue features separate the tongue from the original image before identifying features to refrain from the background noise. However, these methods rely on manual operation [12], [13] or image processing algorithms [16] to segment tongue images, which will introduce considerable noise in the process of tongue image segmentation and affect the accuracy of tongue feature extraction. Secondly, deep learning methods can improve the accuracy of tongue feature classification, but they require high computational cost, which affects the running speed and experience on mobile devices. Finally, previous studies usually focused on extracting only one specific tongue feature, insufficient for TCM tongue diagnosis. Moreover, none of these studies paid attention to sublingual vein features, which are essential for the symptoms differentiation and treatment selection in TCM tongue diagnosis, especially for heart and liver-related diseases [6].

To address the above issues, we propose a deep learning-based tongue feature extraction framework for mobile devices that can adapt to various types of tongue feature extraction and strike a balance between accuracy and efficiency. Specifically, our contributions are summarized as follows:

(1)

We constructed a large dataset of tongue images taken by mobile devices. Most of the tongue image datasets currently used for research are captured by standard devices (e.g., tongue diagnostic instruments), while our work mainly targets applications on mobile devices, so we need to use the dataset captured by mobile devices for training. In particular, our dataset also includes sublingual vein features, which has never been done in previous work.

(2)

We propose a tongue segmentation framework called Moment Invariants with Data Augmentation (DAMI). This approach substantially improves the accuracy of tongue segmentation and can be flexibly applied to almost all segmentation networks with portability.

(3)

We propose an efficient attention mechanism-based tongue feature classification method, which can be applied to almost all tongue features and obtain higher classification accuracy. In addition, this approach considers the balance between accuracy and speed to obtain the best accuracy with minimum computational cost, making the model deployable to mobile devices such as smartphones.

(4)

We propose a feature extraction framework combining tongue segmentation and classification designed for mobile devices. In order to exploit the potential usefulness of this framework, we developed a tongue feature extraction system based on WeChat Mini Program. The development of this applet facilitates the development of telemedicine and also assists doctors to a certain extent, which is significant.