In the field of optical interferometry, phase retrieval is a critical step in acquiring the phase information of fringes. Various deep-learning-based methods have been proposed for phase retrieval from a single-frame interferogram. However, the existing methods still cannot obtain the unwrapped phase directly without the aid of extra steps. To truly fulfill end-to-end phase retrieval for various fringe patterns, we propose a novel method with carefully crafted network architecture and training methodology. Experimental results on simulated and actual interferograms show excellent accuracy, noise robustness, and demodulation efficiency without any further phase unwrapping or polynomial fitting required by the existing methods. Furthermore, the proposed method is compatible with non-Zernike-polynomial-phase interferograms containing phase discontinuities. These properties have qualified the proposed method for high-standard interferometric measurement for optical fabrication.