Using the matching pursuit principle we present recursive adaptive matching pursuit (RAMP), a new fast converging adaptive filter having a complexity that is linear in the number of filter coefficients. We show that the algorithm provides for an explicit tradeoff between complexity and performance. The algorithm's relation to a Gauss-Seidel iterative scheme applied to the normal equations is uncovered. This is useful in establishing convergence results. Finally, before presenting an example simulation showing the performance of the algorithm, we point out that the presented algorithm can be viewed as a generalization of the recently introduced fast Euclidian direction search (FEDS) algorithm.