A novel weight pruning strategy for light weight neural networks with application to the diagnosis of skin disease

Highlights

• Utilize a variety of lightweight CNNs for automated skin cancer diagnosis.

• Propose a weight pruning strategy for accuracy improvement in lightweight CNNs.

• The relationship between weight value and classification accuracy was studied.

• Collect 11 different skin disease types over the past ten years as our dataset.

Abstract

In recent years, deep learning-based models have achieved significant advances in manifold computer vision problems, but tedious parameter tuning has complicated their application to computer-aided diagnostic (CAD) systems. As such, this study introduces a novel pruning strategy to improve the accuracy of five lightweight deep convolutional neural network (DCNN) architectures applied to the classification of skin disease. Unlike conventional pruning methods (such as optimal brain surgeon), the proposed technique does not change the model size yet improves performance after fine tuning. This training approach, intended to improve accuracy without increasing model complexity, is experimentally verified using 1167 pathological images. The clinical data included 11 different skin disease types collected over the past ten years, with varying image quantities in each category. A novel hierarchical pruning method, based on standard deviation, is then developed and used to prune parameters in each convolution layer according to the different weight distributions. This training strategy achieves an 83.5% Top-1 accuracy using a pruned MnasNet (12.5 MB), which is 1.8% higher than that of unpruned InceptionV3 (256 MB). Comparative experiments using other networks (MobileNetV2, SqueezeNet, ShuffleNetV2, Xception, ResNet50, DenseNet121) and dataset (HAM10000) also demonstrate consistent improvements when adopting the proposed model training technique. This distinctive robustness across various network types and simple deployment demonstrates the potential of this methodology for generalization to other computer vision tasks.

Introduction

Although Asians exhibit relatively low occurrences of non-melanoma skin cancers (NMSCs), basal cell carcinoma (BCC) remains one of the most common cancers in those of Chinese descent. As the largest organ in the human body, skin protects several other vital systems, which increases its risk for certain diseases. Skin cancer is a group of diseases that collectively makes up the most common malignancy in the United States. Due in part to increased occupational and recreational ultraviolet (UV) exposure, incidence of both malignant and benign skin tumors is increasing, especially among those with light-pigmented skin [1], [2]. Fortunately, early screenings for skin cancer can significantly improve the prognosis. During the diagnosis of skin cancer, dermatologists typically make a preliminary judgment based on the location of the disease and the appearance of the lesion, using non-invasive inspection methods such as dermoscopy. However, false-positive and false-negative diagnoses occur often in the detection of tumors [3]. The large variety of skin diseases necessitates pathological examination still as the ‘gold standard’ for determining whether a patient is diagnosed with cancer or an inflammatory skin disease.

Although pathological diagnosis offers higher accuracy, it is challenging to acquire valid information from histopathological images [4]. A routine examination requires physicians to combine a patient’s physical characteristics (such as irregular markings on the skin or hard lumps on the body) with high-resolution histopathological images for final diagnosis. In addition, government statistics indicate a severe shortage of dermatologists in underdeveloped regions of China as many smaller hospitals lack a professional dermatologist or pathologist. A recent study estimated the proportion of professional dermatologists to patients is as large as 1:60000 [5]. For this reason, the cost and treatment time for pathological diagnosis has increased significantly. How to shorten the diagnosis cycle and reduce human resource costs has become an urgent problem in many countries’ medical systems.

In recent years, the development of deep learning and the successful application of artificial convolutional neural networks (CNNs) to various computer vision tasks [6], [7] had increased interest in medical image classification [8]. A reliable computer-aided diagnostic (CAD) system, based on deep learning, would be highly beneficial for histopathological image analysis [9]. A system of this type would not only reduce workloads for doctors, mitigating the lack of medical resources in rural areas, but can also be used as a tool to train novice dermatologists and pathologists [10]. As a result, it could help to improve the accessibility of social medical care and the quality of life for skin cancer patients.

Multiple studies have investigated the use of machine learning for automated skin disease diagnosis in recent years. For example, in 2014, a multilayer feed-forward artificial neural network was developed and trained to diagnose erythemato-squamous diseases with an accuracy of 93.7% [11]. In addition to binary classification, deep neural networks have also been used for multi-classification tasks. Liao et al. introduced lesion-targeted CNNs for skin disease classification in 2016 [12]. Sun et al. classified 128 skin diseases using CaffeNet and VGGNet [13]. Haenssle et al. demonstrated that a CNN provided with level-I or level-II dermoscopy images (embedded with clinical information) outperformed a control group of physicians (75.7% compared with 82.5%, P $<$ 0.01) [14]. In 2019, the MobileNet model was used for classifying seven classes of skin lesion image and achieved an overall accuracy of 83.1%, which was comparable to skin pathologist [15]. In the same year, Dawid Połap et al. introduced a set of medical diagnosis model based on artificial intelligence [16], including a series of image preprocessing methods, key-point search algorithm and modified CNNs, and this research set a benchmark for the development of CAD. After that, based on genetic algorithm and a cascade of binary convolutional neural networks, Połap proposed a more efficient solution for the analysis of microscopy images and explored a new idea to train large networks [17].

The implementation of deep neural networks can be challenging for image processing tasks as CNN-based models typically include a complex structure with potentially hundreds of layers and large quantities of weight parameters. In addition, the number of calculations involved requires large amounts of memory and powerful processors, which can be cost-prohibitive. These limitations make it difficult to deploy a CNN framework for CAD in smaller clinics. As such, researchers have developed a variety of lightweight CNNs for mobile and embedded devices, which can significantly reduce the size of the model and improve computational efficiency [18]. However, this streamlined architecture necessitates that smaller CNNs sacrifice accuracy and require increased manual tuning during training, which increases model implementation costs. Thus, the development of an efficient training strategy could be a promising solution to compensate for the loss of accuracy, allowing CAD systems to operate with high reliability and low inference.

Recently, the problem of network performance improvement becomes a hot research topic, and there exist a series of approaches trying to explore new training strategies toward this task. In 2018, a born-again network was proposed for training a student CNN with a same or similar architecture to teacher model [19]. Different from Hinton’s research, this strategy cannot slim the heavy CNNs but make the student-BAN more effective than the teacher model. In CIFAR-10 and CIFAR-100 dataset, BAN-DenseNet reaches error rates of 3.5% and 15.5% respectively. Furlanello et al. show that small student network has untapped learning potential, which open a new door to improve the efficiency of lightweight CNNs. Based on this idea, a study in 2020 embeds pre-trained CNNs into a BAN framework and achieves the state-of-the-art accuracy [20]. In that paper, a series of theoretical analysis and experiments were developed to show that in meta-learning and few-shot image classification, self-distillation training with self-supervised method achieved a similar performance as supervised learning. However, this self-distillation method [19], [20] requires a lot of computing resources and it is hard to apply same configuration of hyperparameters to different datasets (it will be described in detail in Section 4). Thus, designing a training strategy that is easy to deploy and meanwhile has strong generalization ability is a challenging issue especially for deep model training with light-weight model parameters or small-scale dataset.

The objective of this study is to utilize a variety of lightweight CNNs for automated skin cancer diagnosis. A novel weight- pruning training strategy is developed by us to compensate for a loss of accuracy and further improve both model performance and reliability. Conventional lightweight CNNs are first introduced, which exhibit good performance for computer vision tasks, including SqueezeNet [21], MnasNet [22], MobileNetV2 [23], ShuffleNetV2 [24] and Xception [25]. To compensate for the under-fitting and high bias exhibited by small CNNs, we have included a series of improvements based on the dense–sparse–dense (DSD) training strategy proposed by Han [26]. Finally, a comprehensive analysis of weight parameter variations during training was used to develop a new pruning strategy by not only pruning connections with smaller or larger correlations but also investigating a pruning mechanism based on the standard deviation of the weights. This novel pruning mechanism can adaptively remove larger or smaller weights according to the distribution in each layer. Results show that this new training strategy further optimizes the weight parameters and allows lightweight CNNs to achieve a performance comparable to CNNs with larger layer quantities. The primary contributions of this study can be summarized as follows:

• Data used in the comparative experiments have been collected by our group during clinical diagnostic tests performed over the last ten years, in cooperation with a dermatosis hospital in Dalian, China. A series of 1167 pathological images are selected for the study, which included 11 different skin disease types. Each image is verified pathologically by a professional dermatologist, to ensure all labels are accurate and reliable. In order to verify the trainability of the models on our dataset, we also introduce a publicly available HAM10000 dataset, ISIC 2018 Lesion Diagnosis Challenge [27] to test the model inference on a larger dataset.

• Five popular lightweight CNNs are used for the automated classification of multi-type skin tumors based on pathological images. The inclusion of linear bottlenecks (in the case of MobileNetV2 and MnasNet) and the replacement of 3x3 filters with 1 × 1 filters (SqueezeNet) significantly improve the accuracy-latency tradeoff observed in conventional CNNs. Besides, ShuffleNetV2 [24] changes the structure of shortcut from “Add” to “Concat” on the basis of DenseNet [28] and introduce an operator called “channel split”. These results indicate that parameter quantities can be reduced significantly with little loss of accuracy. In addition, Dr. Dong et al. use InceptionV3 for feature extracting in cervical cell dataset and achieve an accuracy of more than 98% [29]. In this structure, convolutions are replaced with larger spatial filters in two smaller layers, thereby avoiding bottlenecks and extreme compression. InceptionV3 [30], ResNet50 [31] and DenseNet121 [28] respectively represent three directions to improve the accuracy: widening layers, deepening network, and multi-scale feature combination. These characteristics are comparable to a conventional CNN so that we employ them as baseline models as well as five lightweight networks.

• A novel weight parameter pruning strategy is then proposed to compensate for the loss of accuracy commonly found in lightweight CNNs. This approach involves not only pruning the unimportant CAD weights, but also retraining the model to achieve improved performance. It is observed in this study that both the smallest and largest partial weights had a negative effect on classification results, which suggests it is beneficial to simultaneously remove the maximum and minimum weights in the model. Compared with the DSD strategy [26], the primary advantage of this technique is its ability to adopt an appropriate pruning threshold for each layer based on the weight distribution.

• The relationship between the relative size of weight values and the classification accuracy is also investigated. Experiments show that excessively small or large weights can affect model performance. The mean and standard deviation of weights are monitored and used to adjust the pruning threshold for connections in different layers. When this distribution is more concentrated and closer to 0, smaller weights were removed and vice versa. This novel approach is then implemented with four lightweight models and used to train a series of high-performance classifiers. Comparisons with deep CNNs suggest the performance of this lightweight model (83.3% Top-1 accuracy with a pruned MnasNet) approach or even surpass that of common deep CNNs (81.5% Top-1 accuracy with unpruned InceptionV3). It is proved that this strategy has highly robust for different hyper-parameter configurations and exhibits positive effects for both lightweight and heavier models.

The remainder of this paper is organized as follows. Related works are briefly reviewed in Section 2. Details concerning the dataset and formulation of our methodology are presented in Section 3. Section 4 describes the testing configuration and results of a comparative experiment. A comprehensive discussion of these results is provided in Section 6, to demonstrate the effectiveness of the proposed technique. Finally, conclusions are summarized in Section 7.