The effect of augmentation and transfer learning on the modelling of lower-limb sockets using 3D adversarial autoencoders

Highlights

• Applying generative modelling in the field of Prosthetics & Orthotics is possible.

• Generative modelling improves socket 3D representation for manipulation/fabrication.

• Transfer-learning improves the reconstructive/generative potential of the model.

• A repository (with software, data, pre-trained models and documentation) is available.

Abstract

Lower limb amputation is a condition affecting millions of people worldwide. Patients are often prescribed with lower limb prostheses to aid their mobility, but these prostheses require frequent adjustments through an iterative and manual process, which heavily depends on patient feedback and on the prosthetist’s experience. New computer-aided design and manufacturing technologies have been emerging as ways to improve the fitting process by creating virtual models of the prosthesis’ interface component with the limb, the socket. Using Adversarial Autoencoders, a generative model describing both transtibial and transfemoral sockets was created. Two strategies were tested to counteract the small size of the dataset: transfer learning using the ModelNet dataset and data augmentation through a previously validated socket statistical shape model. The minimum reconstruction error was 0.00124 mm and was obtained for the model which combined the two approaches. A single-blind assessment conducted with prosthetists showed that, while generated and real shapes are distinguishable, most generated ones assume plausible shapes. Our results show that the use of transfer learning allowed for a correct training and regularization of the latent space, inducing in the model generative abilities with potential clinical applications.

Introduction

Lower limb loss has been defined as a complete loss of a part of the lower limb in the transverse anatomical plane. The two most common levels of amputation are transtibial (below the knee, TT) and transfemoral (above the knee, TF). The incidence of these amputations is expected to increase in the coming years, reaching 3.6M in the United States by 2050 [1]. Amputation can have a devastating effect on both physical and mental health [2]. To improve patient mobility, prostheses are commonly prescribed. There are several components in a prosthesis: the socket itself, the suspension, the knee (in the case of transfemoral prostheses) and the foot. The adjustment of the prosthesis to the patient includes a prolonged, iterative process, known as socket fitting, which consists of adapting the socket shape to the residual limb. This process heavily depends on the skill and experience of the prosthetist, as well as on patient feedback, which can sometimes be unreliable, with no quantitative information involved [3].

There have been several attempts to improve this process, both in industry and academia, by applying digital technologies to the manufacturing and design of sockets. Computer-Aided Design and Manufacturing (CAD/CAM) systems allow for digitization of sockets, creating a virtual representation that can be archived, replicated and corrected with more accuracy and precision through digital tools [4]. In comparison with the traditional process, this is a faster and cheaper method of socket adjustment. Most importantly, this method provides significant improvements in the quality of life of amputees when compared to traditional fitting techniques [5]. The application of generative models in this field can further contribute to the adaptability and use-cases of this virtualization process, by allowing for easier, domain-based manipulation of socket models.

Despite the wide range of anatomical variability in residual limbs, the prosthesis industry has converged to a small set of base designs for TT sockets and another restricted set for TF sockets. This limited variability implies that there are restrictions on what can be considered a valid shape. However, 3D scanners used in the prosthetic field nowadays are blind to this prior knowledge, which could improve the quality of their 3D representations. From this observation, we hypothesize that, with a diverse enough dataset, a generative mathematical model capable of encoding the variability present in the shapes can be built. In previous work, we have explored this concept through the use of statistical shape modelling on a dataset of point clouds obtained from 3D scans of lower limb sockets [6]. The present work explores the same hypothesis through the use of deep generative models combined with data augmentation and transfer learning, with validation in shape reconstruction and generation applications through quantitative and qualitative (evaluation with domain experts) methods. One of the main advantages of this method when compared to the previous approach is that the input data requires less pre-processing, thus maintaining data integrity. Another advantage is the ability to manipulate desired characteristics through latent space manipulation, therefore creating more personalized, adjustable models.

The first deep generative model for point clouds was proposed in [7], which presented several different approaches. One such approach used a Generative Adversarial Network (GAN) operating directly on the raw point clouds; a different one used a GAN that operates on a latent space created by a pre-trained Autoencoder (AE). As a way to optimize different prior distributions for the latent space, a 3D Adversarial Auto-Encoder (AAE) has been designed in [8]. The authors compare the generalization abilities of a 3D Variational Auto-Encoder (VAE), a 3D AAE (with a normal distribution) and 3D AAE-G (AAE with a mixture of Gaussians distribution) with the results obtained in [7], showing improved results for the 3D AAE-based architectures. Both these works were conducted on large scale 3D point cloud datasets, such as ModelNet [9]. To the best of the author’s knowledge, there have been no similar studies on datasets related to the Orthotics and Prosthetics field. When moving towards more specific applications such as this one, the vast amount of data required by Deep Learning (DL) architectures may not exist. Therefore, strategies such as transfer learning are often used as a way to use knowledge from analogous problems to ease the training process of a new network [10]. Data augmentation is also widely used as a method to surpass this issue. In contrast with the previously mentioned technique, augmentation approaches the problem from its root, the training dataset [11]. Both of these methods have been shown to improve the results of convolutional neural networks in small datasets [12].

The remainder of this document is organized as follows: in Section 2, the methods for data processing are described, along with an explanation of the architectures used, their training schemes, techniques for data augmentation and transfer learning and the methods for model evaluation. Section 3 presents the outcomes of the several training schemes used and the final evaluation performed on the best obtained model. Section 4 presents a critical analysis of the results, with some final remarks presented in Section 5.