Utility pole detection and localization is the most fundamental application in aerial-optic cables using distributed acoustic sensing (DAS). The existing pole localization method recognizes the hammer knock signal on DAS traces by learning from knocking vibration patterns. However, it requires many efforts for data collection such as knocking every pole and manually labeling the poles’ locations, making this labor-intensive solution expensive, inefficient, and highly error-prone. In this paper, we propose a pole localization solution by learning the ambient data collected from a DAS system, which are vibration patterns excited by random ambient events, such as wind and nearby traffic. In detail, we investigate a universal framework for learning representations of ambient data in the frequency domain by contrastive learning of the similarity of low and high-frequency series. A Gaussian-based data reweighting kernel is employed for eliminating the effect of the label noise. Experimental results demonstrate the proposed methods outperform the existing contrastive learning methods on the real-world DAS ambient dataset.