Human action recognition is important in image and video processing with many applications. With the development of sensor technology, different cameras can be used for action acquisition, e.g., infrared cameras. Is it possible to adapt the visible light action recognizers to a new modality or domain? In this paper, we study the feasibility to adapt the action recognizer learned from visible light spectrum to infrared. A preliminary result is obtained on a large database based on an adaptive learning method, demonstrating the potential to perform cross-spectral action recognition.