Optimally regularised kernel Fisher discriminant classification

Abstract

Mika, Rätsch, Weston, Schölkopf and Müller [Mika, S., Rätsch, G., Weston, J., Schölkopf, B., & Müller, K.-R. (1999). Fisher discriminant analysis with kernels. In Neural networks for signal processing: Vol. IX (pp. 41–48). New York: IEEE Press] introduce a non-linear formulation of Fisher’s linear discriminant, based on the now familiar “kernel trick”, demonstrating state-of-the-art performance on a wide range of real-world benchmark datasets. In this paper, we extend an existing analytical expression for the leave-one-out cross-validation error [Cawley, G. C., & Talbot, N. L. C. (2003b). Efficient leave-one-out cross-validation of kernel Fisher discriminant classifiers. Pattern Recognition, 36(11), 2585–2592] such that the leave-one-out error can be re-estimated following a change in the value of the regularisation parameter with a computational complexity of only $\mathcal{O}\left({\ell }^2\right)$ operations, which is substantially less than the $\mathcal{O}\left({\ell }^3\right)$ operations required for the basic training algorithm. This allows the regularisation parameter to be tuned at an essentially negligible computational cost. This is achieved by performing the discriminant analysis in canonical form. The proposed method is therefore a useful component of a model selection strategy for this class of kernel machines that alternates between updates of the kernel and regularisation parameters. Results obtained on real-world and synthetic benchmark datasets indicate that the proposed method is competitive with model selection based on $k$-fold cross-validation in terms of generalisation, whilst being considerably faster.

Introduction

In recent years the “kernel trick” has been applied to constructing non-linear equivalents of a wide range of classical linear statistical models, for instance ridge regression (Hoerl and Kennard, 1970, Saunders et al., 1998), principal component analysis (Jolliffe, 2002, Schölkopf et al., 1999) and Fisher’s linear discriminant (Fisher, 1936, Mika et al., 1999), in addition to more modern techniques, such as the maximal margin classifier (Boser et al., 1992, Cortes and Vapnik, 1995) (for an introduction to kernel learning methods, see Schölkopf and Smola (2002) or Shawe-Taylor and Cristianini (2004)). An important advantage of kernel models is that the parameters of the model are typically given by the solution of a convex optimisation problem, with a single, global optimum (Boyd & Vandenberghe, 2004). The generalisation properties of kernel models are however typically governed by a small number of regularisation and kernel parameters. Good values for these parameters must be determined during the model selection process. There is generally no guarantee that the model selection criterion is unimodal, and so simple grid-based search procedures are often employed in practical applications. In this paper, we propose a simple and computationally efficient method for choosing the regularisation parameter in kernel Fisher discriminant analysis so as to minimise an approximation to the leave-one-out cross-validation error. The resulting optimally regularised kernel Fisher discriminant (ORKFD) analysis algorithm then becomes attractive for small to medium-scale applications (currently anything less than a few thousand training patterns) as the algorithm is easily implemented (only 15 lines of code in the MATLAB programming environment) and inherently resistant to over-fitting.

The remainder of this paper is structured as follows: Section 2 reviews the kernel Fisher discriminant classifier and introduces the notation used throughout. Section 3 then proposes an efficient algorithm for selecting the regularisation for a KFD classifier, so as to minimise the leave-one-out cross-validation error, with a computational complexity of only $\mathcal{O}\left({\ell }^2\right)$ operations instead of the $\mathcal{O}\left({\ell }^3\right)$ operations of direct methods¹ (Cawley & Talbot, 2003b). Section 4 presents results obtained on a range of real-world benchmark datasets. The extension of this approach to closely related forms of least-squares kernel learning is discussed in Section 5. Finally, the work is summarised in Section 6.