This article presents a fully integrated IEEE 802.11aj direct-conversion transceiver system in a 120-nm SiGe:C BiCMOS technology. The system includes a transmitter, a receiver, and two onboard substrates integrated waveguide-fed Yagi-Uda antennas. In the millimeter-wave frequency band, the common-mode (CM) signal in a transformer-based balun is a limiting factor for the linearity of differential devices. In this work, we analyze the cause of the CM effect and provide fast impedance equalization with a passive component compensation method to resolve this problem, which improves the linearity of the transmitter. A $T$ -type second-order harmonic termination method is applied at the power amplifier (PA) output to further improve the linearity of the transmitter. The self-mixing method-based power detector is adopted at the output of the transmitter to have an accurate image signal and local oscillator (LO) leakage calibration. The transmitter reaches an output 1-dB compression point (OP_1dB) of 14.6–16.4 dBm and a conversion gain (CG) higher than 24 dB in the IEEE 802.11aj frequency band, operating from 42.3 to 48.4 GHz. The receiver achieves a 3.8–4.1-dB noise figure, −18.7- to −22-dBm input 1-dB compression point, and a CG better than 43.8 dB in the IEEE 802.11aj frequency band. In the system wireless data transmission test, the transceiver is fully compliant with the error vector magnitude (EVM) requirement of the IEEE 802.11aj standard up to the 64-quadrature amplitude modulation (QAM) operating mode, and the measured transmitter-to-receiver EVM is better than −28.5 dB at a 1-m distance over the IEEE 802.11aj frequency band. Compared to other states of the art, the transmitter’s OP_1dB is the highest, and the demonstrated transceiver system delivers the highest performance in the IEEE 802.11aj wireless link.