This paper proposes a novel scheme called as“frequency domain imbalance estimation” that estimates the imbalance of the Hilbert transformer in heterodyne multimode/band receivers with baseband automatic gain control (AGC). The proposed scheme uses correlation matrices in the frequency domain. This enables the receivers to keep high transmission performance in spite of the imbalance of the analog Hilbert transformer, by offsetting the imbalance. Moreover, the baseband AGC relaxes the requirement of the baseband A/D converter. The performance of imbalance estimation and imbalance cancellation is verified by computer simulation. As a result, it is shown that the proposed scheme not only estimates the imbalance of Hilbert transformer with extremely high precision, but also cancels the image-band interference such that it achieves the theoretical performance.