Sea fog detection is a challenging and significant task in the field of remote sensing. Deep learning-based methods have shown promising potential, but require a large amount of pixel-level labeled data that are time-consuming and labor-intensive to acquire. To scale up the dataset and overcome the limitations of pixel-level annotation, we attempt to explore the existing knowledge from historical statistics for label-efficient sea fog detection. In this article, we propose an image-level weakly supervised sea fog detection dataset (WS-SFDD) and a novel weakly supervised sea fog detection framework via prototype learning, named ProCAM. According to the sea fog events recorded by the Marine Weather Review published quarterly by the National Meteorological Center of China, we collect the sea fog images from Himawari-8 satellite data and obtain free image-level labels to construct the dataset. However, with image-level annotations, the existing weakly supervised semantic segmentation (WSSS) methods mainly rely on class activation maps (CAMs) and have limitations when applied to such a specific scenario: 1) the pseudo-labels (PLs) mainly cover the most discriminative part of object regions that are incomplete; 2) the background is complex with varying atmospheric conditions, and it is difficult to distinguish sea fog from low clouds due to their high similarity in spectral characteristics; and 3) the co-occurring context, such as “sea,” distracts the model and thus degrades the performance. To address the above issues, in our proposed ProCAM, we first design a prototype reactivation (PRA) module that reactivates self-similar sea fog regions by pixel-to-prototype feature matching to improve the robustness and completeness of CAMs. Then, we develop a pixel-to-prototype contrastive (PPC) learning method to increase the distance between sea fog and background in the embedding space for learning more discriminative dense features. Finally, a self-augmented regularization (SAR...