In this paper, we present a novel visual tracker based on visible and infrared sequences. The extended region proposal network helps to automatically generate `object-like' proposals and `distance-based' proposals. In contrast to traditional tracking approaches that exploit the same or similar structural features for template matching, this approach dynamically manages the new compressive layers to refine the target-recognition performance. This paper presents an attractive multi-sensor fusion method which demonstrates the ability to enhance tracking precision, robustness, and reliability compared with that of single sensor. The integration of multiple features from different sensors with distinct characteristics resolves incorrect merge events caused by the inappropriate feature extracting and classifier for a frame. Long-term trajectories for object tracking are calculated using online support vector machines classifier. This algorithm illustrates favorable performance compared to the state-of-the-art methods on challenging videos.