There has been a breakthrough in cloud detection by using convolutional neural networks (CNNs) during these years. However, there are still weaknesses among current cloud detection algorithms because only cloud mask information is used. As clouds represent differently in different scenes, the scene information may give hints to improve cloud detection performance. Therefore, different from the previous cloud detection literature, in this letter, we propose an end-to-end new deep learning network named scene aggregation network (SAN), which aggregates the scene information in the framework. Specifically, basic features are first extracted by utilizing all levels of network features. Then, the aggregated features used to produce the final cloud masks are created by fusing the basic features and the specially introduced scene information. Experimental results have demonstrated that with scene information aggregated, our proposed method can be robust on images with different scenes. Additionally, as SAN outperforms other state-of-the-art methods, our proposed method suits for cloud detection and can achieve improvement on this task.