High-capacity TDM-PONs, TWDM PONs, and WDM PONs have great potential to be used as next-generation PONs in emerging broadband services for various applications. As PONs typically use direct detection techniques and low-cost components, transmission impairments become one of the main limiting factors. Various transmission impairment compensation techniques, including both optical and digital signal processing (DSP) based techniques, are proposed and demonstrated for high-speed TDM-, TWDM-, and WDM-PONs. In this article, we describe the next-generation PON schemes and principles of different compensation techniques. We propose filter type optical compensation and Volterra type electrical compensation techniques to compensate nonlinear inter-symbol interference caused by vestigial sideband, limited modulation bandwidth, in-phase and quadrature imbalance, dispersion, and square-law detection. In addition, analyses and comparison of the equilibrium performances, features, characteristics, and applicability of the various optical- and electrical-domain compensation schemes are presented.