Magnetic flux leakage (MFL) testing is an effective non-destructive testing (NDT) method for pipeline safety operation, and defect detection is one of the core issues in MFL signal processing. Currently, MFL defect detection is tough due to insufficient defect samples. To obtain more serviceable information, heterogeneous signals are collected in MFL process, while taking full advantage of heterogeneous signals is still a hard problem. In this paper, an end-to-end MFL defect detection architecture with insufficient training samples called heterogeneous signal fusion method (IHSF) is proposed. Firstly, heterogeneous signals from the same pipeline are collected by axial and radial sensors. Secondly, the features from fine-tuned and non-fine-tuned pre-trained models are fused, which increases the generality and adaptability of the features. Moreover, fine-tuning of a few parameters reduces the number of parameters during model training, which is more suitable for the insufficient sample training. Thirdly, a mutual supervision training strategy based on the features fusion of general and adaptive features is proposed to update the parameters of the proposed network, which establishes the latent relationship of heterogeneous signals. Finally, experiments on MFL defect detection are conducted, and IHSF is compared to the state-of-the-art methods. The results validate that the proposed method is effective. Note to Practitioners—The motivation of this paper is a hot signal processing issue on pipeline safety operation called defect detection with insufficient samples. Traditionally, the defect is usually detected by training a large number of defect samples, while defect samples are usually insufficient in practical MFL measurement. Moreover, conventional deep networks only use a single signal, which ignores information from multiple signals. Regarding the problems above, a heterogeneous signal mutual supervision method is proposed to replace the traditional fixed label mec...