Discarding Variables in a Principal Component Analysis: Algorithms for All-Subsets Comparisons

Summary

The traditional approach to the interpretation of the results from a Principal Component Analysis implicitly discards variables that are weakly correlated with the most important and/or most interesting Principal Components. Some authors argue that this practice is potentially misleading and that it is preferable to take a variable selection approach, comparing variable subsets according to appropriate approximation criteria. In this paper, we propose algorithms for the comparison of all possible subsets according to some of the most important comparison criteria proposed to date. The computational effort of the proposed algorithms is studied and it is shown that, given current computer technology, they are feasible for problems involving up to thirty variables. A free-domain software implementation can be downloaded from the Internet.

Appendix: Counting Operations

Formulae (30) – (35) show the number of floating point operations per pivot required to update the comparison criteria (from a k-dimensional subset to a k + 1 dimensional set) and the matrix and/or vector elements associated with t given variables that will be needed in future steps.

Criteria Floating Point Operations per Pivot

$$\left| {\,{{\bf{S}}_{11}}} \right|\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,{1 \over 2}{t^2} + {3 \over 2}t + 1$$

(30)

$$tr\,{{\bf{S}}_{22\left| 1 \right.}}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\left( {2 + t} \right)\left( {p - k - 1} \right) - {1 \over 2}{t^2} + - {1 \over 2}t$$

(31)

$${\left\| {{{\bf{S}}_{22\left| 1 \right.}}} \right\|^2}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,{\left( {p - k - 1} \right)^2} + 2\left( {p - k - 1} \right)$$

(32)

$$tr\left( {{\bf{S}}_{22}^{ - 1}\,{{\bf{S}}_{22\left| 1 \right.}}} \right)\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,{3 \over 2}{\left( {p - k - 1} \right)^2} + {7 \over 2}\left( {p - k - 1} \right)$$

(33)

$$tr\left( {{{\left( {{\bf{S}}_{11}^{ - 1}\,{{\left( {{{\bf{S}}^2}} \right)}_{11}}} \right)}^2}} \right)\,\,\,\,\,\,\,{5 \over 2}{k^2} + ({{11} \over 2} + 3t)k + {1 \over 2}{t^2} + {7 \over 2}t + 2$$

(34)

$$tr\left( {{\bf{S}}_{11}^{ - 1}\,{\bf{S}}{{\bf{G}}_{11}}} \right)\,\,\,\,\,\,\,{1 \over 2}{t^2} + ({3 \over 2} + q)t + 2q$$

(35)

The results in (30), (31) and (35) follow from a direct count of the number of operations required to implement (9), (13) and (16) and update the different elements of S_22|1 and (v_a)_2|1 that will be needed later. The results in (32) and (33) follow from the fact that ${{\bf{S}}_{{2^\prime }{2^\prime }\left| {{1^\prime }} \right.}}$ and ${\bf{S}}_{{2^\prime }{2^\prime }}^{ - 1}$ have each ${{\left( {p - k - 1} \right)\,\left( {p - k} \right)} \over 2} = {1 \over 2}\left( {{{\left( {p - k - 1} \right)}^2} + \left( {p - k - 1} \right)} \right)$ different elements that need to be updated and squared or cross-multiplied. For these two criteria any sequencing of the F-MC algorithm leads to same effort. To compute the effort required to update the $tr\,\left( {{{\left( {{\bf{S}}_{11}^{ - 1}{{\left( {{{\bf{S}}^2}} \right)}_{11}}} \right)}^2}} \right)$ criterion we first note that the matrix $\left( {{\bf{S}}_{{1^\prime }{1^\prime }}^{ - 1}{{\left( {{{\bf{S}}^2}} \right)}_{{1^\prime }{1^\prime }}}} \right)$, not being in general symmetric, has (k + 1)² different elements. Therefore the number of operations required to update this criterion using (6), (14) and (15) equals $\left( {k + 1} \right)k + k + {\left( {k + 1} \right)^2} + {1 \over 2}\left( {k + 1} \right)\left( {k + 2} \right) = {5 \over 2}{k^2} + {{11} \over 2}k + 2$. In order to update the additional elements of ${\left( {{\bf{S}}_{11}^{ - 1}{{\left( {{{\bf{S}}^2}} \right)}_{1 \bullet }}} \right)^2},\left( {{\bf{S}}_{11}^{ - 1}\,{{\bf{S}}_{12}}} \right)$ and S_22|1 that will be needed in future steps ${1 \over 2}{t^2} + \left( {{7 \over 2} + 3k} \right)t$ more operations are required. The result in (34) is the sum of these two quantities.

Formulae (36) – (41) show the total number of floating point operations required by the different criteria.

Criteria Total Number of Floating Point Operations

$$\left| {{{\bf{S}}_{11}}} \right|\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,4\left( {{2^p}} \right) - {1 \over 2}{p^2} - {5 \over 2}p - 4$$

(36)

$$tr\,{{\bf{S}}_{22\left| 1 \right.}}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\left( {{3 \over 2}p - 1} \right)\left( {{2^p}} \right) - {1 \over 2}{p^2} - {3 \over 2}p + 1$$

(37)

$${\left\| {{{\bf{S}}_{22\left| 1 \right.}}} \right\|^2}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\left( {{1 \over 4}{p^2} + {5 \over 4}p} \right)\left( {{2^p}} \right) - {p^2} - 2p$$

(38)

$$tr\left( {{\bf{S}}_{22}^{ - 1}\,{{\bf{S}}_{22\left| 1 \right.}}} \right)\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\left( {{3 \over {16}}{p^2} + {{11} \over {16}}p - {7 \over 8}} \right)\left( {{2^p}} \right) + {1 \over 2}{p^3} - {p^2} - {1 \over 2}p + 2$$

(39)

$$tr\left( {{{\left( {{\bf{S}}_{11}^{ - 1}{{\left( {{{\bf{S}}^2}} \right)}_{11}}} \right)}^2}} \right)\quad \left( {{5 \over 8}{p^2} + {{19} \over 8}p - 2} \right)\left( {{2^p}} \right) + {1 \over 2}{p^3} - {1 \over 2}{p^2} - p + 2$$

(40)

$$tr\left( {{\bf{S}}_{11}^{ - 1}\,{\bf{S}}{{\bf{G}}_{11}}} \right)\quad \left( {3 + 3q} \right)\left( {{2^p}} \right) - {1 \over 2}{p^2} - \left( {{5 \over 2} + q} \right)p - \left( {3 + 3q} \right)$$

(41)

These results were derived with the help of the known equalities

$$\sum\limits_{a = 1}^{p - 1} {\left( {\begin{array}{*{20}c} p \\ a \\ \end{array} } \right)a = \left( {{2^{p - 1}} - 1} \right)p} $$

(42)

$$\sum\limits_{a = 1}^{p - 1} {\left( {\begin{array}{*{20}c} p \\ a \\ \end{array} } \right){a^2} = \left( {{p^2} + p} \right)\left( {{2^{p - 2}}} \right) - {p^2}} $$

(43)

and the result

$$\sum\limits_{t = 0}^{p - 1} {{2^{p - t - 1}}} \left( {a{t^2} + bt + c} \right) = (3a + b + c)\left( {{2^p}} \right) - a{p^2} - (2a + b)p - (3a + b + c)$$

(44)

which can be easily proved by induction.

In particular, the results in (36) and (41) follow directly from (30), (35) and (44). The result (37) follows from (31), (42), (44) and noticing that from a k-dimensional subset there are $\left( {\begin{array}{*{20}c} {p - t} \\ k \\ \end{array} } \right)$ different pivots on the variable X_u = X_t+1. Result (38) can be derived from (32), (42), (43) and (44). Result (39) follows from (33), (42), (43), (44), the addition of the ${{{P^2}\left( {P + 1} \right)} \over 2}$ operations required for the initial inversion of S, and noticing that for this criterion only $\left( {\begin{array}{*{20}c} {P - 1} \\ {P - K - 1} \\ \end{array} } \right)$ pivots to k-dimensional subsets (k = 1, 2, …, p − 2) are necessary. This latter remark follows from the fact that all subsets including one given variable can be evaluated simultaneously with the complementary subsets, using the relation $tr\left( {{\bf{S}}_{11}^{ - 1}\,{{\bf{S}}_{11\left| 2 \right.}}} \right) = 2k - p + tr\left( {{\bf{S}}_{22}^{ - 1}\,{{\bf{S}}_{22\left| 1 \right.}}} \right)$ which follows from (4). Finally, (40) was derived from (34), (42), (43), (44), and adding the ${{{p^3} + p} \over 2}$ operations needed for the initial computation of S².