Abstract
The number of deep learning (DL) layers increases, and following the performance of computing nodes improvement, the output accuracy of deep neural networks (DNN) faces a bottleneck problem. The resident network (RN) based DNN model was applied to address these issues recently. This paper improved the RN and developed a rectified linear unit (ReLU) based conditional generative adversarial nets (cGAN) to classify plantar pressure images. A foot scan system collected the plantar pressure images, in which normal (N), planus (PL), and talipes equinovarus feet (TE) data-sets were acquired subsequently. The 9-foot types named N, PL, TE, N-PL, N-TE, PL-N, PL-TE, TE-N, and TE-PL were classified using the proposed DNN models, named resident network-based conditional generative adversarial nets (RNcGAN). It improved the RN structure firstly and the cGAN system hereafter. In the classification of plantar pressure images, the pixel-level state matrix can be direct as an input, different from the previous image classification task with image reduction and feature extraction. cGAN can directly output the pixels of the image without any simplification. Finally, the model achieved better results in the evaluation indicators of accuracy (AC), sensitivity (SE), and F1-measurement (F1) by comparing to artificial neural networks (ANN), k-nearest neighbor (kNN), Fast Region-based Convolution Neural Network (Fast R-CNN), visual geometry group (VGG16), scaled-conjugate-gradient convolution neural networks (SCG-CNN), GoogleNet, AlexNet, ResNet-50–177, and Inception-v3. The final prediction of class accuracy is 95.17%. Foot type classification is vital for producing comfortable shoes in the industry.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahmed H, La HM, Tran K (2020) Rebar detection and localization for bridge deck inspection and evaluation using deep residual networks. Autom Constr 120:103393. https://doi.org/10.1016/j.autcon.2020.103393
Aldebeyan S, Sinno H, Alotaibi M, Makhdom AM, Hamdy RC (2018) Utility outcome assessment of pes planus deformity. Foot Ankle Surg 24(2):119–123. https://doi.org/10.1016/j.fas.2016.12.005
Angin S, Mickle KJ, Nester CJ (2018) Contributions of foot muscles and plantar fascia morphology to foot posture. Gait Post 61:238–242. https://doi.org/10.1016/j.gaitpost.2018.01.022
Azarmdel H, Jahanbakhshi A, Mohtasebi SS, Muñoz AR (2020) Evaluation of image processing technique as an expert system in mulberry fruit grading based on ripeness level using artificial neural networks (ANNs) and support vector machine (SVM). Postharvest Biol Technol 166:111201. https://doi.org/10.1016/j.postharvbio.2020.111201
Barker SL, Downing M, Chesney DJ, Maffulli N (2012) Assessment of calf volume in congenital talipes equinovarus by computer analysed digital photography. The Surgeon 10(2):84–89. https://doi.org/10.1016/j.surge.2011.01.002
Bello M, Nápoles G, Sánchez R, Bello R, Vanhoof K (2020) Deep neural network to extract high-level features and labels in multi-label classification problems. Neurocomputing 413:259–270. https://doi.org/10.1016/j.neucom.2020.06.117
Boob D, Dey SS, Lan G (2020) Complexity of training ReLU neural network. Discret Optim 2020:100620. https://doi.org/10.1016/j.disopt.2020.100620
Boxun Fu, Li Fu, Niu Yi, Hao Wu, Li Y, Shi G (2021) Conditional generative adversarial network for EEG-based emotion fine-grained estimation and visualization. J vis Commun Image Represent 74:102982. https://doi.org/10.1016/j.jvcir.2020.102982
Buldt AK, Allan JJ, Landorf KB, Menz HB (2018) The relationship between foot posture and plantar pressure during walking in adults: a systematic review. Gait Post 62:56–67. https://doi.org/10.1016/j.gaitpost.2018.02.026
Chicoine D, Bouchard M, Laurendeau S, Moisan G, Belzile EL, Corbeil P (2021) Biomechanical effects of three types of foot orthoses in individuals with posterior tibial tendon dysfunction. Gait Posture 83:237–244. https://doi.org/10.1016/j.gaitpost.2020.11.001
Ciocca G, Napoletano P, Schettini R (2018) CNN-based features for retrieval and classification of food images. Comput vis Image Underst 176–177:70–77. https://doi.org/10.1016/j.cviu.2018.09.001
Deeba K, Amutha B (2020) ResNet - deep neural network architecture for leaf disease classification. Microprocess Microsyst. https://doi.org/10.1016/j.micpro.2020.103364
Deforth M, Zwicky L, Horn T, Hintermann B (2019) The effect of foot type on the Achilles tendon moment arm and biomechanics. Foot 38:91–94. https://doi.org/10.1016/j.foot.2018.10.003
Dufour AB, Losina E, Menz HB, LaValley MP, Hannan MT (2017) Obesity, foot pain and foot disorders in older men and women. Obes Res Clin Pract 11(4):445–453. https://doi.org/10.1016/j.orcp.2016.11.001
Ferrando A, Salom M, Page A, Perez-Girbes A, Carlos Atienza M, Minguez Fe, Prat J (2020) Talipes equinovarus treatment in infants treated by the ponseti method compared with posterior-only release: a mid-childhood comparison of results. J Foot Ankle Surg 59(5):919–926. https://doi.org/10.1053/j.jfas.2018.12.046
Guo Y, Han S, Li Y, Zhang C (2018) Yu Bai (2018) K-Nearest Neighbor combined with guided filter for hyperspectral image classification. Procedia Comput Sci 129:159–165. https://doi.org/10.1016/j.procs.2018.03.066
Hanif MS, Bilal M (2020) Competitive residual neural network for image classification. ICT Express 6(1):28–37. https://doi.org/10.1016/j.icte.2019.06.001
Igarashi S, Sasaki Y, Mikami T, Sakuraba H, Fukuda S (2020) Anatomical classification of upper gastrointestinal organs under various image capture conditions using AlexNet. Comput Biol Med 124:103950. https://doi.org/10.1016/j.compbiomed.2020.103950
Janke J, Castelli M, Popovič A (2019) Analysis of the proficiency of fully connected neural networks in the process of classifying digital images. Benchmark of different classification algorithms on high-level image features from convolutional layers. Expert Syst Appl 135:12–38. https://doi.org/10.1016/j.eswa.2019.05.058
Kamenaga T, Nakano N, Takayama K, Tsubosaka M, Takashima Y, Kikuchi K, Fujita M, Kuroda Y, Hashimoto S, Hayashi S, Niikura T, Kuroda R, Matsumoto T (2021) Comparison of plantar pressure distribution during walking and lower limb alignment between modified kinematically and mechanically aligned total knee arthroplasty. J Biomech 120:110369. https://doi.org/10.1016/j.jbiomech.2021.110379
Kaymak S, Helwan A, Uzun D (2017) Breast cancer image classification using artificial neural networks. Procedia Comput Sci 120:126–131. https://doi.org/10.1016/j.procs.2017.11.219
Korkmaz SA, Binol H (2018) Classification of molecular structure images by using ANN, RF, LBP, HOG, and size reduction methods for early stomach cancer detection. J Mol Struct 1156:255–263. https://doi.org/10.1016/j.molstruc.2017.11.093
Kruger KM, Graf A, Flanagan A, McHenry BD, Altiok H, Smith PA, Harris GF, Krzak JJ (2019) Segmental foot and ankle kinematic differences between rectus, planus, and cavus foot types. J Biomech 94:180–186. https://doi.org/10.1016/j.jbiomech.2019.07.032
Lei H, Han T, Zhou F, Yu Z, Qin J, Elazab A, Lei B (2018) A deeply supervised residual network for HEp-2 cell classification via cross-modal transfer learning. Pattern Recogn 79:290–302. https://doi.org/10.1016/j.patcog.2018.02.006
Li Z, Dey N, Ashour AS, Luying Cao Yu, Wang DW, McCauley P, Balas VE, Shi K (2017) Fuqian Shi (2017) Convolutional neural network based clustering and manifold learning method for diabetic plantar pressure imaging dataset. Journal of Medical Imaging and Health Informatics 7(3):639–652. https://doi.org/10.1166/jmihi.2017.2082
Li Z, Wang D, Dey N, Ashour AS, Sherratt RS, Shi F (2019) Plantar pressure image fusion for comfort fusion in diabetes mellitus using an improved fuzzy hidden Markov model. Biocybernetics Biomed Eng 39(3):742–752. https://doi.org/10.1016/j.bbe.2019.06.007
Liang P, Deng C, Jun Wu, Yang Z (2020) Intelligent fault diagnosis of rotating machinery via wavelet transform, generative adversarial nets and convolutional neural network. Measurement 159:107768. https://doi.org/10.1016/j.measurement.2020.107768
Liming Xu, Zeng X, Huang Z, Li W, Zhang He (2020) Low-dose chest X-ray image super-resolution using generative adversarial nets with spectral normalization. Biomed Signal Process Control 55:101600. https://doi.org/10.1016/j.bspc.2019.101600
Liu Z, Yang C, Huang J, Liu S, Yumin Zhuo XuLu (2021) Deep learning framework based on integration of S-Mask R-CNN and Inception-v3 for ultrasound image-aided diagnosis of prostate cancer. Futur Gener Comput Syst 114:358–367. https://doi.org/10.1016/j.future.2020.08.015
McKinney J, Rac MWF, Gandhi M (2019) Congenital talipes equinovarus (clubfoot). Am J Obstet Gynecol 221(6):B10–B12. https://doi.org/10.1046/j.1469-7580.2003.00147.x
Mei Z, Ivanov K, Zhao G, Yuanyuan Wu, Liu M, Wang L (2020) Foot type classification using sensor-enabled footwear and 1D-CNN. Measurement 165:108–184. https://doi.org/10.1016/j.measurement.2020.108184
Montanelli H, Yang H (2020) Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem. Neural Netw 129:1–6. https://doi.org/10.1016/j.neunet.2019.12.013
Peng Y, Zhang L, Liu S, Wu X, Zhang Y, Wang X (2019) Dilated residual networks with symmetric skip connection for image denoising. Neurocomputing 345:67–76. https://doi.org/10.1016/j.neucom.2018.12.075
Shen Z-Y, Han S-Y, Li-Chen Fu, Hsiao P-Y, Lau Y-C (2019) Sheng-Jen Chang (2019) Deep convolution neural network with scene-centric and object-centric information for object detection. Image vis Comput 85:14–25. https://doi.org/10.1016/j.imavis.2019.03.004
Shia W-C, Chen D-R (2021) Classification of malignant tumors in breast ultrasound using a pretrained deep residual network model and support vector machine. Comput Med Imaging Graph 87:101829. https://doi.org/10.1016/j.compmedimag.2020.101829
Silpaja Chandrasekar K, Geetha P (2020) Multiple objects tracking by a highly decisive three-frame differencing-combined-background subtraction method with GMPFM-GMPHD filters and VGG16-LSTM classifier. J vis Commun Image Represent 72:102905. https://doi.org/10.1016/j.jvcir.2020.102905
Sultana F, Sufian A, Dutta P (2020) Evolution of image segmentation using deep convolutional neural network: a survey. Knowl-Based Syst 201:1062. https://doi.org/10.1016/j.knosys.2020.106062
Tang P, Wang H, Kwong S (2017) G-MS2F: GoogLeNet based multi-stage feature fusion of deep CNN for scene recognition. Neurocomputing 225:188–197. https://doi.org/10.1016/j.neucom.2016.11.023
Valentín MB, Bom CR, Coelho JM, Correia MD, de Albuquerque MP, de Albuquerque MP, Faria EL (2019) A deep residual convolutional neural network for automatic lithological facies identification in Brazilian pre-salt oilfield wellbore image logs. J Petrol Sci Eng 179:474–503. https://doi.org/10.1016/j.petrol.2019.04.030
Wang D, Li Z, Dey N, Ashour AS, Luminita Moraru R, Sherratt S, Shi F (2020) Deep- segmentation of plantar pressure images incorporating fully convolutional neural networks. Biocybernetics and Biomedical Engineering 40(1):546–558. https://doi.org/10.1016/j.bbe.2020.01.004
Wang D, Li Z, Dey N, Ashour AS, Moraru L, Biswas A, Shi F (2019b) Optical pressure sensors based plantar image segmenting using an improved fully convolutional network. Optik 179:99–114. https://doi.org/10.1016/j.ijleo.2018.10.155
Wang C, Li D, Li Z, Wang D, Dey N, Biswas A, Moraru L, Sherratt RS, Shi F (2019a) An efficient local binary pattern-based plantar pressure optical sensor image classification using convolutional neural networks. Optik 185:543–557. https://doi.org/10.1016/j.ijleo.2019.02.109
Xia Y, Zhang Le, Ravikumar N, Attar R, Piechnik SK, Neubauer S, Petersen SE, Frangi AF (2021) Recovering from missing data in population imaging - Cardiac MR image imputation via conditional generative adversarial nets. Med Image Anal 67:101812. https://doi.org/10.1016/j.media.2020.101812
Yalman Y (2014) Histogram based perceptual quality assessment method for color images. Comput Stand Interfaces 36(6):899–908. https://doi.org/10.1016/j.csi.2014.04.002
Yang W, Li Z, Wang C, Li J (2020) A multi-task Faster R-CNN method for 3D vehicle detection based on a single image. Appl Soft Comput 95:106533. https://doi.org/10.1016/j.asoc.2020.106533
Yao H, Dai F, Zhang S, Zhang Y, Tian Qi, Changsheng Xu (2019) DR2-Net: deep residual reconstruction network for image compressive sensing. Neurocomputing 359:483–493. https://doi.org/10.1016/j.neucom.2019.05.006
Funding
This work is supported by Huidong County Footwear Technology Innovation Center Construction Project of Guangdong Provincial Department of Science and Technology under No. 2017B090922003. The sponsor provided experimental design method, foot scan system and device, data collect and analysis.
Author information
Authors and Affiliations
Contributions
J. H.: Conceptualization, Methodology, Data curation, Writing—original draft preparation. D. W., Visualization, Validation; Z. L.: Writing—reviewing & editing, Supervision. N. D.: Visualization, Validation. R. G. C.: Validation, and Investigation F. S.: Software, Validation, and Investigation.
Corresponding author
Ethics declarations
Conflicts of interest
Jianlin Han, Dan Wang, Zairan Li, Nilanjan Dey, and Fuqian Shi declare that they have no conflict of interest.
Ethics approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants involved in the study.
Consent to participate
The foot scan data-set was collected in accordance with the code of conduct of research with human material in China. This study was approved by the ethical committee of the Huizhou University. All subjects gave written informed consent.
Consent for publication
All presentations of case reports have consent to publish.
Additional information
Communicated by Mu-Yen Chen.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Han, J., Wang, D., Li, Z. et al. Plantar pressure image classification employing residual-network model-based conditional generative adversarial networks: a comparison of normal, planus, and talipes equinovarus feet. Soft Comput 27, 1763–1782 (2023). https://doi.org/10.1007/s00500-021-06073-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-021-06073-w