Compensation for the nonlinear systems represented by polynomials involves polynomial inverse. In this paper, a new algorithm is proposed that gives the baseband polynomial inverse with a limited order. The algorithm employs orthogonal basis that is predetermined from the distribution of input signal and finds the coefficients of the inverse polynomial to minimize the mean square error. Compared with the well established p-th order inverse method, the proposed method can suppress the distortions better including higher order distortions. It is also extended to obtain memory polynomial inverse through a feedback-configured structure. Both numerical simulations and experimental results demonstrate that the proposed algorithm can provide good performance for compensating the nonlinear systems represented by baseband polynomials.