Object tracking is one of the challenging problems in the field of computer vision. Affected by the unstructured environments, for example, the occlusion, noise, and light, These factors can affect the appearance of the specific object and result in failures when tracking specific objects. To address this issue, we propose a novel visual tracking method based on multimodal convolutional network learning. Our framework adopts a parallel structure, which consists of two shallow convolutional neural networks. First, the parallel network is used to draw the different features of the RGB-T (RGB and thermal) data separately. Second, this two kind of features are mixed together and finally the mixed feature is sent to domain-specific layers for binary classification and identification of the targets. We perform comprehensive experiments on RGBT234 visual data and the results prove that the proposed visual tracking method improves the effects significantly through the use of multi-modal features, which illustrates that our method is competitive in performances against with the state-of-the-art tracking algorithms.