Reciprocal curves

Abstract

Partial least squares (PLS) is a modeling technique that has met with considerable success, particularly in the fields of chemometrics and psychometrics. In this paper, we extend linear PLS to a nonlinear version called “reciprocal curves.” Reciprocal curves are smooth one-dimensional curves that pass through the center of the data, are co-consistent and share several optimality properties originally identified with PLS.

Introduction

Linear data analysis techniques are important tools to use when exploring data, but methods that allow for nonlinear structures should provide more relevant or parsimonious models resulting in a more fruitful exploratory analysis. In this paper we extend the notion of partial least squares (PLS) into a nonlinear domain using techniques developed by Hastie and Stuetzle (1989) to extend principal components analysis (PCA) to “principal curves.”

The origins of PLS can be traced to Herman Wold's original nonlinear iterative partial least squares (NIPALS) algorithm (Wold, 1966, Wold, 1982) that was developed to linearize models which were nonlinear in their parameters. The associated ad hoc estimation of parameters was accomplished through an iterative method involving linear least squares modeling and model relaxation (e.g. Varga, 1962). The NIPALS method was adapted for over-determined multiple regression problems (Wold et al., 1983) and that extension of NIPALS became known as PLS, and it is still implemented in various software packages based on the original NIPALS algorithm. Statisticians have adopted different perspectives on the same paradigm, such as discussed in de Jong (1993), Stone and Brooks (1990), Helland (1988), Frank and Friedman (1993), Hinkle (1995), Hinkle and Rayens, 1995, Hinkle and Rayens, 1999, Rayens and Andersen, 2003, Rayens and Andersen, 2004, and Rayens (2000). These views are more consistent with classical multivariate statistical theory and allow for the implementation of PLS by way of solutions to well-posed eigenstructure problems that clearly exhibit the compromise PLS strikes between variance summary and correlation. Unfortunately, this eigenstructure perspective is still not universally known within the community of users; and the effect on the eigenvalue problem of different constraints on the structure is less well-known. We will avoid peripheral discussions of these matters here by suppressing the constraints and defining PLS as follows.

Definition 1

Suppose the random vector $(\mathbf{X},\mathbf{Y})$ has $p+q>2$ elements, where $\mathbf{X}$ is $p$-dimensional, and associated positive definite covariance matrix

$$\mathbf{\Sigma }=\left(\begin{array}{cc}\hfill {\mathbf{\Sigma }}_{\mathbf{XX}}\hfill & \hfill {\mathbf{\Sigma }}_{\mathbf{XY}}\hfill \\ \hfill {\mathbf{\Sigma }}_{\mathbf{XY}}^{\prime }\hfill & \hfill {\mathbf{\Sigma }}_{\mathbf{YY}}\hfill \end{array}\right)\text{.}$$

The $r$th pair of PLS component scores, where $1⩽r⩽\min (p,q)$, is the pair of linear combinations, $u_r={\mathit{\alpha }}_r^{\mathrm{t}}\mathbf{X}$ and $v_r={\mathit{\beta }}_r^{\mathrm{t}}\mathbf{Y}$ such that the vectors ${\mathit{\alpha }}_r$ and ${\mathit{\beta }}_r$ are unit length and maximize $\mathrm{Cov}\left(u_r,v_r\right)$. The weight vectors ${\mathit{\alpha }}_r$ and ${\mathit{\beta }}_r$ are the $r$th PLS weight vectors.

As Frank and Friedman (1993) noted, the PLS paradigm is simply a penalized version of canonical covariates analysis (CCA) in the sense that

$$\mathrm{argmax}\left[\mathrm{Cov}\left({\mathit{\alpha }}_r^{\mathrm{t}}\mathbf{X},{\mathit{\beta }}_r^{\mathrm{t}}\mathbf{Y}\right)\right]=\mathrm{argmax}\left[\mathrm{Var}\left({\mathit{\alpha }}_r^{\mathrm{t}}\mathbf{X}\right)\right]\left[{\mathrm{Corr}}^2\left({\mathit{\alpha }}_r^{\mathrm{t}}\mathbf{X},{\mathit{\beta }}_r^{\mathrm{t}}\mathbf{Y}\right)\right]\left[\mathrm{Var}\left({\mathit{\beta }}_r^{\mathrm{t}}\mathbf{Y}\right)\right]\text{.}$$

From this we see that PLS seeks to summarize the variability in two separate spaces, subject to a penalty that requires that these two resulting summaries, ${\mathit{\alpha }}_r^{\mathrm{t}}\mathbf{X}$ and ${\mathit{\beta }}_r^{\mathrm{t}}\mathbf{Y}$, are also relatively well correlated. It is not hard to show that the first set of PLS weight vectors is determined by the eigenstructure

$${\mathbf{\Sigma }}_{\mathbf{XY}}{\mathit{\beta }}_1\propto {\mathit{\alpha }}_1\text{,}{\mathbf{\Sigma }}_{\mathbf{XY}}^{\mathrm{t}}{\mathit{\alpha }}_1\propto {\mathit{\beta }}_1\text{,}$$

and that subsequent solutions depend on the particular within-space constraints on the structure (see Rayens, 2000, for specific solutions obtained without the use of Lagrange multipliers). Thus the resulting PLS structures are vectors; or linear approximations of geometrical structures underlying the random vector spaces being modeled or described.

A slightly different perspective on PLS, provided by the following proposition, sets up the framework for extending PLS to nonlinear functions:

Proposition 1

The first pair of PLS components, for the random vector $(\mathbf{X},\mathbf{Y})$ minimize the expression

$$E\left[{∥\mathbf{X}-\mathit{\alpha }{\mathit{\alpha }}^{\mathrm{t}}\mathbf{X}∥}^2+{∥{\mathit{\alpha }}^{\mathrm{t}}\mathbf{X}-{\mathit{\beta }}^{\mathrm{t}}\mathbf{Y}∥}^2+{∥\mathbf{Y}-\mathit{\beta }{\mathit{\beta }}^{\mathrm{t}}\mathbf{Y}∥}^2\right]\text{.}$$

Hinkle (1995) shows that each subsequent pair of weight vectors from Definition 1 minimize the above-mentioned expectation provided these additional directions are constrained to be orthogonal in each space to those already extracted. Regardless of the constraint set, however, the first pair will always minimize the given expectation.

The form of Eq. (3) suggests that any underlying errors-in-variables model corresponding to the PLS paradigm, at least as far as the initial set of weight vectors are concerned, will correspond to three separate models:

• two PCA-type “outer relations”

  $$\mathbf{X}=\mathit{\alpha }u+\mathit{\varepsilon }\mathrm{and}\mathbf{Y}=\mathit{\beta }v+\mathit{\zeta }$$

• and one CCA-type “inner relation”

  $$v=\mathit{\delta }·u+\mathit{\xi }\text{,}$$

  where the “component scores” are defined as $u={\mathit{\alpha }}^{\mathrm{t}}\mathbf{X}$ and $v={\mathit{\beta }}^{\mathrm{t}}\mathbf{Y}$; and $\mathit{\varepsilon },\mathit{\zeta }$ and $\mathit{\xi }$ are independent error terms. Of course, this set of errors-in-variables models is “linear” in several senses. First, and most importantly, the “component structures” defined by the weight vectors $\mathit{\alpha }$ and $\mathit{\beta }$ are linear, and the relationship between scores $(u,v)$ is assumed linear as well.

In the following sections, a theory for “reciprocal curves” is developed which combines this theoretical perspective on PLS with Hastie's (1984) novel extension of principal components to “principal curves.” Note that in comparison to principal curves, which model a single random vector space, the method developed below is structurally different. In particular, variability in two spaces, each generated by different probability distributions and nonlinear structures, needs to be accounted for with an additional restriction that the resulting structures be optimally related.

Before continuing, we give two brief motivational examples. We will use both to illustrate the PLS approach to modeling, and also to suggest why the nonlinear generalization we are going to develop in Section 3 will be useful.