We propose a convolutional three-stream network architecture for driver fatigue detection from infrared videos that are available both in the daytime and in the night time. Specifically, the convolutional three-stream network architecture incorporates current-infrared-frame-based spatial information, optical-flows-based short-term temporal information of two consecutive infrared frames and optical flow-motion history image-based (OF-MHI-based) temporal information within the infrared video sequence. And then these three networks are fused at the last convolutional layer by 3D CNN. Besides, an estimation method to evaluate the current driver fatigue level is proposed based on the fatigue detection results from previous frames, which helps to generate alerts properly in real-life driving applications. We show that the proposed method achieves state-of-the-art performance, 94.68% accuracy, in our driver behavior dataset using the infrared data.