In this paper, a fast leave-one-out (LOO) evaluation formula is introduced for least squares support vector machine (LS-SVM) classifiers. The computation cost can be reduced to approximately 1/N when compared to normal LOO procedure (N is the number of training samples). Inspired by its fast speed, we are able to use it to replace the original level 3 posterior probability approximation formula of the Bayesian framework for LS-SVM classifiers. The improved inference framework shows higher generalization performance and faster computation speed.