We present a novel object detection method called the multi-domain attentive detection network (MDADN). For robust object detection, we do not only use red, green, and blue (RGB) image data but also infrared data. The MDADN adds attention modules to each layer to weigh multi-domains of data differently in a channel-wise and spatial-wise manner, which yields channel- and spatial-aware networks. The MDADN accurately detects objects (e.g., vehicles) in challenging environments, including nighttime, rainy and foggy days, and environments where even humans find it difficult to detect objects. Experimental results on the FLIR dataset demonstrate that the MDADN outperforms state-of-the-art methods in terms of accuracy and speed. The ablation study shows that the attention module and the fusion of multiple data sources help to considerably improve the accuracy.