Event camera calibration plays an irreplaceable role in event camera-based visual applications. Event cameras report per-pixel brightness changes as a stream of asynchronous events, making them insensitive to the spatial distribution of feature points. Consequently, traditional methods such as checkerboard and circle calibration algorithms, which rely on spatial intensity distribution for feature point extraction, are ill-suited for event camera calibration. To address this issue, a flexible and accurate event camera calibration method is presented in this work. The method analyzes the frequency characteristics of event data streams, introducing the groundbreaking use of Fourier transform in event camera calibration. Instead of relying on intensity, we extract feature points from the robust phase domain, which mitigates the impact of event camera noise. Event frames are derived from the event data streams, and we recover two crossed-phase maps using the Fourier transform. Feature points are then detected using the law of phase distribution and refined with sub-pixel accuracy. Subsequently, camera parameters are estimated. The performance of our method was rigorously validated from different perspectives, and the results show that the accuracy of the proposed method is substantially improved compared to the conventional methods. The root mean square errors (RMSEs) of the proposed method are consistently below 0.06 pixels, with some cases achieving an impressive 0.03 pixels. The accuracy of binocular reconstruction reaches 0.394 mm.