In this paper, self-shielded quad-mode substrate integrated waveguide (SIW) bandpass filters (BPFs) are proposed by using a novel method of mixed-mode cavity, which is achieved by adding a metal patch to the middle substrate layer in a SIW cavity. The mixed-mode cavity can support two pairs of degenerate dual modes, i.e., SIW-cavity modes and strip-patch modes. Characteristics of the proposed mixed-mode SIW cavity are analyzed in detail for filters’ design. For the demonstration, an unbalanced quad-mode filter structure is proposed and analyzed, which can realize single- and dual-band filtering responses. Then, a balanced quad-mode filter structure is proposed and analyzed on the basis of unbalanced quad-mode filter and its odd symmetry of the four resonant modes. The balanced filter can also realize single- and dual-band filtering responses with common-mode inherently at passbands under differential-mode operation. Finally, two unbalanced and two balanced quad-mode filters with single- and dual-band responses are designed, fabricated and measured. The measured results agree well with the designed ones, which verifies the proposed method. The proposed mixed-mode SIW filters have the merits of high selectivity, compact size and high shielding from nearby circuits.