Voxel-Based Three-Dimensional Neural Style Transfer

Abstract

Neural Style Transfer has been successfully applied in the creative process for generating novel artistic 2D images by transferring the style of a painting to an existing content image. These techniques which rely on deep neural networks have been extended to further computational creativity tasks like video, motion and animation stylization. However, only few research has been conducted to utilize Neural Style Transfer in the spatially three-dimensional space. Existing 2D/3D hybrid approaches avoid the extra dimension during the stylization process and add postprocessing or differentiable rendering to transform the results to 3D. In this paper, we propose for the first time a complete three-dimensional Neural Style Transfer pipeline based on a high-resolution voxel representation. Following our previous research, our architecture includes the standardized gram matrix style loss for noise reduction and visual improvement, the bipolar exponential activation function for symmetric feature distributions and best practices for the underlying classification network. In addition, we propose regularization terms for voxel-based 3D Neural Style Transfer optimization and demonstrate their capability to significantly reduce noise and undesired artefacts. We apply our 3D Neural Style Transfer pipeline on a set of style targets. The style transfer results are evaluated using 3D shape descriptors which confirm the subjective visual improvements.