Free-hand gesture recognition is a challenging task that has applications in various domains, including human-computer interaction, virtual reality, and gaming. In this paper, we propose a method for free-hand gesture recognition using Convolutional 3D networks with Cross Stitch Units for multimodal data. Our approach combines the intermediate representations learned by Convolutional 3D networks from the RGB and depth modalities using Cross Stitch Units, which allows the network to learn more robust and effective features for the task. We evaluate our approach on the publicly available Multi-Modal Hand Gesture Dataset (MMHGD). The MMHGD consists of a high number of samples of six free-hand gesture classes. It offers four modalities: RGB images, 3D point clouds, acceleration data recorded by a device on the users wrist, and audio data. First, we set a baseline by performing uni-modal gesture recognition on each of the two modalities (RGB and 3D) using a state-of-the-art Convolutional 3D network architecture. Then we show that free-hand gesture recognition can be further improved by using Cross Stitch Units to fuse both modalities. We show that uni-modal gesture recognition using Convolutional 3D networks already offers an acceptable gesture recognition accuracy. Using Cross Stitch Units for multimodal fusion can further improve the achieved gesture recognition accuracy and thus lead to even more reliable results.