Abstract
Physicians spend much time observing the facial symmetry of patients and collecting various data to arrive at an accurate clinical judgment. This study presents a transfer learning method for evaluating the degree of facial symmetry. The contour map of a face is used as training data, and the training module then classifies and scores the degree of facial symmetry. Our method enables rapid and accurate clinical assessments. In the experiments, we divided 195 contour maps of patients’ faces provided by physicians and then classified the data into four fractional levels based on the average scores of facial symmetry provided by doctors. Subsequently, the facial data were trimmed, ipped, and superimposed. After being processed, the extent of the contour overlap was used as the basis for learning. We used data augmentation to increase the amount of data. Finally, we applied fine-tuning and transfer learning to obtain prediction models, which showed excellent performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Fourcade A, Khonsari RH (2019) Deep learning in medical image analysis: a third eye for doctors. J Stomatol Oral Maxillofac Surg 120:279–288
Lin HH, Chiang WC, Yang CT, Cheng CT, Zhang T, Lo LJ (2021) On construction of transfer learning for facial symmetry assessment before and after orthognathic surgery. Comput Methods Programs Biomed 200:105928
Gonzalez-Franco Mar, Steed Anthony, Hoogendyk Steve, Ofek Eyal (2020) Using facial animation to increase the enfacement illusion and avatar self-identification. IEEE Trans Visual Comput Graphics 26(5):2023–2029
Bria A, Marrocco C, Tortorella F (2020) Addressing class imbalance in deep learning for small lesion detection on medical images. Comput Biol Med 120:103735
Rahman MM, Tan Y, Xue J, Shao L, Lu K (2019) 3D object detection: learning 3D bounding boxes from scaled down 2D bounding boxes in RGB-D images. Inf Sci 476:147–158
Yang C-T, Zhang T, Kristiani E, Cheng C-T, The Implementation of Objects Detection and Analysis Using Deep Learning with GPU, Published in The 9th International Conference on Frontier Computing (FC2019)
Ardakani AA, Kanafi AR, Acharya UR, Khadem N, Mohammadi A (2020) Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: results of 10 convolutional neural networks. Comput Biol Med 121:103795
Haas T, Schubert C, Eickhoff M, Pfeifer H (2020) BubCNN: Bubble detection using Faster RCNN and shape regression network. Chem Eng Sci, 21628
Zhang W, Du Y, Yoshida T, Wang Q (2018) DRI-RCNN: an approach to deceptive review identification using recurrent convolutional neural network. Inf Process Manag 54(4):576–592
Zhang T, Yang C-T, Huang H-N, Kristiani E, Sheen Y-J, Chen H-M (2019) The implementation of image-assisted diagnosis on dabetic foot wounds using machine learning, Published in Future ICT
Huang D-S, Jo K-H, Lee H-H, Kang H-J, Bevilacqua V (2009) Emerging intelligent computing technology and application. Springer, South Korea
Pathak AR, Pandey M, Rautaray S (2018) Application of deep learning for object detection. Procedia Comput Sci 132(1706–1717):2018
Ma X, Niu Y, Gu L, Wang Y, Zhao Y, Bailey J, Lu F (2020) Understanding adversarial attacks on deep learning based medical image analysis systems, Pattern Recognition In press, Journal pre-proof Available online 1
Zahia S, Zapirain MBG, Sevillano X, González A, Kim PJ, Elmaghraby A (2020) Pressure injury image analysis with machine learning techniques: a systematic review on previous and possible future methods. Artif Intell Med 102:101742
Hiroharu K, Harada T (2014) Image Reconstruction from Bag-of-Visual-Words. In: Conference on Computer Vision and Pattern Recognition (CVPR)
Koniusz P, Yan F, Gosselin P-H, Mikolajczyk K (2017) IEEE Xplore, IEEE, 1 February (2017). [Online]. Available: https://ieeexplore.ieee.org/abstract/document/7439823
Guyon I, Elisseeff A (2008) Feature extraction: foundations and applications, vol 207. Springer, Berlin
Anas IY, Bamgbose BO, Nuhu S (2019) A comparison between 2D and 3D methods of quantifying facial morphology. Heliyon 5(6):e01880
Akhil G, Senthil Kumar KP, Raja S, Janardhanan K (2015) Three-dimensional assessment of facial asymmetry: a systematic review. J Pharm Bioall Sci 7(Suppl S2):433–7
Berssenbrügge P, Berlin NF, Kebeck G, Runte C, Jung S, Kleinheinz J, Dirksen D (2014) 2D and 3D analysis methods of facial asymmetry in comparison. J Craniomaxillofac Surg 42(6):e327-34
Zou F, Shen L, Jie Z, Zhang W, Liu W, (2019) A Sufficient Condition for Convergences of Adam and RMSProp. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 11127–11135
Reddy RVK, Rao BS, Raju KP, (2019) Handwritten hindi digits recognition using convolutional neural network with RMSprop optimization. In: IEEE 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS),
Jahandad, Sam SM, Kamardin K, Sjarif NNA, Mohamed N (2019) Offline signature verification using deep learning convolutional neural network (CNN) architectures GoogLeNet inception-v1 and inception-v3. Procedia Comput Sci 161:475–483
Lo WW, Yang X (2019) An xception convolutional neural network for malware classification with transfer learning. IEEE 2019 10th IFIP International Conference on New Technologies, Mobility and Security(NTMS)
Blanco G, Traina AJM, Traina C Jr, Azevedo-Marques PM, Jorge AES, Oliveira D, Bedo MVN (2020) A superpixel-driven deep learning approach for the analysis of dermatological wounds. Comput Methods Programs Biomed 183:105079
Radenovic F, Tolias G, Chum O (2019) Fine-Tuning CNN Image Retrieval with No Human Annotation, IEEE Trans Pattern Anal Mach Intell Vol. 41(7)
Gulli A, Pal S (2017) Deep Learning with Keras. Packt Publishing Ltd, Birmingham
Zeng Dan, Liu Han, Zhao Fan, Ge Shiming, Shen Wei, Zhang Zhijiang (2019) Proposal pyramid networks for fast face detection. Inf Sci 495:136–149
Kristiani E, Chen YA, Yang CT, Huang CY, Tsan YT, Chan WC (2021) Using deep ensemble for influenza-like illness consultation rate prediction. Futur Gener Comput Syst 117:369–386
Zhang T, Yang C-T, Lin H-H, Lo L-J, Chan Y-W, The Implementation of Facial Symmetry Assessment before and after Orthognathic Surgery Using Transfer Learning, Published in The 3rd International Conference on Innovative Computing (IC2020)
Qiao Sibo, Pang Shanchen, Zhai Xue, Wang Min, Shihang Yu, Ding Tong, Cheng Xiaochun (2020) Human body multiple parts parsing for person reidentification based on Xception. Int J Comput Intell Syst 14(1):482–490
Meshach WT, Hemajothi S, Mary Anita EA (2021) Real-time facial expression recognition for affect identification using multi-dimensional SVM. J Ambient Intell Humaniz Comput 12(6):6355–6365
Sikander Gulbadan, Anwar Shahzad (2020) A novel machine vision-based 3D facial action unit identification for fatigue detection. IEEE Trans Intell Transp Syst 22(5):2730–2740
Acknowledgements
This work was supported by grants from the Ministry of Science and Technology, Taiwan (MOST 110-2314-B-182-057- ) as well as by Chang Gung Memorial Hospital Grant (CMRPG5L0041 and CMRPG5K0201).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lin, HH., Zhang, T., Wang, YC. et al. A system for quantifying facial symmetry from 3D contour maps based on transfer learning and fast R-CNN. J Supercomput 78, 15953–15973 (2022). https://doi.org/10.1007/s11227-022-04502-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-022-04502-7