Classifier selection using geometry preserving feature

Abstract

The selection of proper classifiers for a given data set is full of challenges. The critical problem of classifier selection is how to extract feature from data sets. This paper proposes a new method for feature extraction of a data set. Our method not only preserves the geometrical structure of a data set, but also characterizes the decision boundary of classification problems. Specifically speaking, the extracted feature can recover a data set that has the same Euclidean geometrical structure as the original data set. We present an efficient algorithm to compute the similarity between data set features. We theoretically analyze how the similarity between our features affects the performance of the support vector machine, a well-known classifier. The empirical results show that our method is effective in finding suitable classifiers.

Appendices

Appendix A: Proof of Theorem 2

Proof

Let $h(\textbf{x}) = \textbf{w}^\top \textbf{x}$, $h^\prime (\textbf{x}^\prime ) = {\textbf{w}^\prime }^\top \textbf{x}^\prime$ and $\Delta \textbf{w} = \textbf{w} - \textbf{w}^\prime$. Then we have

$$\begin{aligned} \begin{aligned}&\ |h(\textbf{x})-h^\prime (\textbf{x}^\prime ) |= |\textbf{w}^\top \textbf{x} - {\textbf{w}^\prime }^\top \textbf{x}^\prime |\\ &=\ |(\textbf{w}^\top \textbf{x} - {\textbf{w}^\prime }^\top \textbf{x}) + ({\textbf{w}^\prime }^\top \textbf{x} - {\textbf{w}^\prime }^\top \textbf{x}^\prime )|\\ \leqslant&\ |\textbf{w}^\top \textbf{x} - {\textbf{w}^\prime }^\top \textbf{x} |_2 + |{\textbf{w}^\prime }^\top \textbf{x} - {\textbf{w}^\prime }^\top \textbf{x}^\prime |_2\\ \leqslant&\ \Vert \textbf{w} - {\textbf{w}^\prime }\Vert _2\Vert \textbf{x}\Vert _2 + \Vert \textbf{w}^\prime \Vert _2\Vert \textbf{x} - \textbf{x}^\prime \Vert _2. \end{aligned} \end{aligned}$$

(A1)

Since $\textbf{w}$ and $\textbf{w}^\prime$ are the minimizers of the associated SVM problems (12), for all $t \in [0,1]$, we have

$$\begin{aligned} \begin{aligned}&\frac{1}{2}\Vert \textbf{w}\Vert ^2_2 + C \sum _{i=1}^n L(y_i\textbf{w}^\top \textbf{x}_i) \\ \leqslant&\ \frac{1}{2}\Vert \textbf{w} + t \Delta \textbf{w} \Vert ^2_2 + C \sum _{i=1}^n L(y_i(\textbf{w} + t \Delta \textbf{w})^\top \textbf{x}_i), \end{aligned} \end{aligned}$$

(A2)

and

$$\begin{aligned} \begin{aligned}&\frac{1}{2}\Vert \textbf{w}^\prime \Vert ^2_2 + C \sum _{i=1}^n L(y_i^\prime {\textbf{w}^\prime }^\top \textbf{x}_i^\prime ) \\ \leqslant&\ \frac{1}{2}\Vert \textbf{w}^\prime - t \Delta \textbf{w} \Vert ^2_2 + C \sum _{i=1}^n L(y_i^\prime (\textbf{w}^\prime - t \Delta \textbf{w})^\top \textbf{x}_i^\prime ). \end{aligned} \end{aligned}$$

(A3)

Summing (A2) and (A3), we obtain

$$\begin{aligned} \begin{aligned} t(1-t) \Vert \Delta \textbf{w} \Vert ^2_2&\leqslant C \sum _{i=1}^n \left[ \left( L(y_i(\textbf{w} + t \Delta \textbf{w})^\top \textbf{x}_i) - L(y_i\textbf{w}^\top \textbf{x}_i) \right) \right. \\&\ + \left. \left( L(y_i^\prime (\textbf{w}^\prime - t \Delta \textbf{w})^\top \textbf{x}_i^\prime - L(y_i^\prime {\textbf{w}^\prime }^\top \textbf{x}_i^\prime ) \right) \right] . \end{aligned} \end{aligned}$$

(A4)

Using the convexity of hinge loss, we have

$$\begin{aligned} \begin{aligned}&L(y_i(\textbf{w} + t \Delta \textbf{w})^\top \textbf{x}_i) - L(y_i\textbf{w}^\top \textbf{x}_i) \\ \leqslant&\ t \left( L(y_i {\textbf{w}^\prime }^\top \textbf{x}_i) - L(y_i\textbf{w}^\top \textbf{x}_i) \right) \end{aligned} \end{aligned}$$

(A5)

and

$$\begin{aligned} \begin{aligned}&L(y_i^\prime (\textbf{w}^\prime - t \Delta \textbf{w})^\top \textbf{x}_i^\prime ) - L(y_i^\prime {\textbf{w}^\prime }^\top \textbf{x}_i^\prime ) \\ \leqslant&\ -t \left( L(y_i^\prime {\textbf{w}^\prime }^\top \textbf{x}_i^\prime ) - L(y_i^\prime \textbf{w}^\top \textbf{x}_i^\prime ) \right) . \end{aligned} \end{aligned}$$

(A6)

Combing (A4), (A5) and (A6) and taking the limit $t \rightarrow 0$ leads to

$$\begin{aligned} \begin{aligned}&\Vert \textbf{w}^\prime -\textbf{w}\Vert ^2_2 \\ \leqslant&\ C\sum _{i=1}^n \left[ \left( L(y_i{\textbf{w}^\prime }^\top \textbf{x}_i)- L(y^\prime _i{\textbf{w}^\prime }^\top \textbf{x}^\prime _i) \right) \right. \\&\ + \left. \left( L(y^\prime _i \textbf{w}^\top \textbf{x}^\prime _i)- L(y_i \textbf{w}^\top \textbf{x}_i) \right) \right] \\ \leqslant&\ C\sum _{i=1}^n \Bigl [ \Vert \textbf{w}^\prime \Vert _2 \cdot \Vert y_i\phi (\textbf{x}_i) - y^\prime _i\phi (\textbf{x}^\prime _i)\Vert _2 \Bigr . \\&\ + \Bigl . \Vert \textbf{w}\Vert _2 \cdot \Vert y_i\phi (\textbf{x}_i) - y^\prime _i\phi (\textbf{x}^\prime _i)\Vert _2 \Bigr ]. \end{aligned} \end{aligned}$$

(A7)

The second inequality follows from the Lipschitz continuity of hinge loss. We write $\textbf{w}$ in terms of the dual variables $\alpha _i$, namely $\textbf{w} = \sum _{i=1}^n \alpha _i \textbf{x}_i$. Note that $0 \leqslant \alpha _i \leqslant C$ and $\textbf{x}_i = \textbf{G}^{1/2} \textbf{e}_i$. Then we obtain

$$\begin{aligned} \begin{aligned} \Vert \textbf{w} \Vert _2&= \Vert \sum _{i=1}^n \alpha _i \textbf{x}_i\Vert _2 \leqslant \sum _{i=1}^n |\alpha _i |\cdot \Vert \textbf{x}_i \Vert _2 \\&\leqslant C \sum _{i=1}^n \Vert \textbf{G}^{1/2} \textbf{e}_i \Vert _2 \leqslant n C \Vert \textbf{G}^{1/2} \Vert _2. \end{aligned} \end{aligned}$$

(A8)

Analogously, we have

$$\begin{aligned} \Vert \textbf{w}^\prime \Vert _2 \leqslant n C \Vert {\textbf{G}^\prime }^{1/2} \Vert _2. \end{aligned}$$

(A9)

Thus, (A7) can be rewritten as

$$\begin{aligned} \begin{aligned}&\Vert \textbf{w}^\prime -\textbf{w}\Vert ^2_2 \\ \leqslant&\ C_0^2 \sum _{i=1}^n\Vert y_i \textbf{x}_i -y^\prime _i \textbf{x}^\prime _i\Vert _2\\ &=\ C_0^2 \sum _{i=1}^n\Vert y_i\textbf{G}^{1/2}\textbf{e}_i - y^\prime _i{\textbf{G}^\prime }^{1/2}\textbf{e}_i\Vert _2\\ &=\ C_0^2 \sum _{i=1}^n\Vert (y_i\textbf{G}^{1/2}\textbf{e}_i - y^\prime _i\textbf{G}^{1/2}\textbf{e}_i) + (y^\prime _i\textbf{G}^{1/2}\textbf{e}_i - y^\prime _i{\textbf{G}^\prime }^{1/2}\textbf{e}_i)\Vert _2\\ \leqslant&\ C_0^2 \Bigl ( \Vert \textbf{G}^{1/2}\Vert _2 \sum _{i=1}^n |y_i - y^\prime _i|+ \sum _{i=1}^n \Vert \textbf{G}^{1/2}-{\textbf{G}^\prime }^{1/2}\Vert _2 \Bigr ) \\ \leqslant&\ C_0^2 \Vert \textbf{G}^{1/2}\Vert _2 \Vert \textbf{y} - \textbf{y}^\prime \Vert _1 + nC_0^2 \Vert \textbf{G} - \textbf{G}^\prime \Vert ^{1/2}_2 \end{aligned} \end{aligned}$$

(A10)

Here we use the inequality $\Vert \textbf{G}^{1/2}-{\textbf{G}^\prime }^{1/2}\Vert _2 \leqslant \Vert \textbf{G}-\textbf{G}^\prime \Vert ^{1/2}_2$.

We also have

$$\begin{aligned} \begin{aligned} \Vert \textbf{x} - \textbf{x}^\prime \Vert _2 &=\ \Vert \sum _{i=1}^n \beta _i \textbf{x}_i - \sum _{i=1}^n \beta _i \textbf{x}_i^\prime \Vert _2\\ &=\ \Vert \sum _{i=1}^n \beta _i (\textbf{G}^{1/2} - {\textbf{G}^\prime }^{1/2}) \textbf{e}_i\Vert _2 \\ \leqslant&\ \Vert \textbf{G}^{1/2} - {\textbf{G}^\prime }^{1/2}\Vert _2\Vert \varvec{\beta }\Vert _2 \\ \leqslant&\ \Vert \textbf{G} - {\textbf{G}^\prime }\Vert ^{1/2}_2\Vert \varvec{\beta }\Vert _2 \end{aligned} \end{aligned}$$

(A11)

and

$$\begin{aligned} \Vert \textbf{x} \Vert _2 = \Vert \sum _{i=1}^n \beta _i \textbf{x}_i \Vert _2 \leqslant \Vert \textbf{G}^{1/2}\Vert _2\Vert \varvec{\beta }\Vert _2. \end{aligned}$$

(A12)

Substituting (A9) to (A12) into (A1) yields

$$\begin{aligned} \begin{aligned}&|h(\textbf{x}) - h^\prime (\textbf{x}^\prime )|\\ \leqslant&\ C_0 (\Vert \textbf{G}^{1/2}\Vert _2\Vert \textbf{y} - \textbf{y}^\prime \Vert _1 + n \Vert \textbf{G} - \textbf{G}^\prime \Vert ^{1/2}_2)^{1/2} \Vert \textbf{G}^{1/2} \Vert _2 \Vert \varvec{\beta } \Vert _2 \\&+ nC \Vert {\textbf{G}^\prime }^{1/2} \Vert _2 \Vert \varvec{\beta }\Vert _2\Vert \textbf{G} - \textbf{G}^\prime \Vert ^{1/2}_2\\ \leqslant&\ C_0 (\Vert \textbf{G}^{1/2}\Vert _2^{1/2} \Vert \textbf{y} - \textbf{y}^\prime \Vert _1^{1/2} + \sqrt{n} \Vert \textbf{G} - \textbf{G}^\prime \Vert ^{1/4}_2) \Vert \textbf{G}^{1/2} \Vert _2 \Vert \varvec{\beta } \Vert _2 \\&+ nC \Vert {\textbf{G}^\prime }^{1/2} \Vert _2 \Vert \varvec{\beta }\Vert _2\Vert \textbf{G} - \textbf{G}^\prime \Vert ^{1/2}_2. \end{aligned} \end{aligned}$$

(A13)

The last inequality results from $(a+b)^{1/2}\leqslant \sqrt{a}+\sqrt{b}$. $\square$

Appendix B: Incomplete Cholesky decomposition (ICD) algorithm

The algorithm of ICD is shown in Algorithm 2.

Appendix C: Proof of Theorem 3

Proof

The second and the last statements can be inferred from the first and the third ones. Thus, we only need to proof (i) and (iii). Since $\textbf{G}$ and $\textbf{G}^\prime$ are isomorphic, there exists a bijection $\pi _0: \{1,2,\cdots ,n\}\rightarrow \{1,2,\cdots ,n\}$ such that

$$\begin{aligned} \textbf{G}(i,j) = \textbf{G}^\prime (\pi _0(i),\pi _0(j)),\ 1\leqslant i,j\leqslant n. \end{aligned}$$

We will argue by mathematical induction.

1) For $s = 1$, let $p_1 = \arg \max _{1 \leqslant i \leqslant n} \textbf{G}(i,i)$ and $q_1 = \arg \max _{1 \leqslant i \leqslant n} \textbf{G}^\prime (i,i)$. Since the largest diagonal elements of $\textbf{G}$ and $\textbf{G}^\prime$ are unique, we have $q_1 = \pi _0(p_1)$. We construct bijections

$$\begin{aligned} \omega _1(i) = \left\{ \begin{array}{ll} i, &{} i\ne 1, p_1;\\ p_1, &{} i = 1;\\ 1, &{} i = p_1. \end{array}\right. \end{aligned}$$

and

$$\begin{aligned} \omega ^\prime _1(i) = \left\{ \begin{array}{ll} i, &{} i\ne 1, q_1;\\ q_1, &{} i = 1;\\ 1, &{} i = q_1. \end{array}\right. \end{aligned}$$

If $p_1 = 1$ ($q_1 = 1$), then $\omega _1$ ($\omega ^\prime _1$) is an identity mapping. Define a composite function $\pi _1: \{1,2,\cdots ,n\}\rightarrow \{1,2,\cdots ,n\}$:

$$\begin{aligned} \pi _1(i) = \omega ^\prime _1\circ \pi _0\circ \omega _1(i). \end{aligned}$$

(C14)

Then

$$\begin{aligned} \pi _1(1) = \omega ^\prime _1\circ \pi _0\circ \omega _1(1) = \omega ^\prime _1\circ \pi _0(p_1) = \omega ^\prime _1(q_1) = 1. \end{aligned}$$

The first diagonal element of $\textbf{A}^{(1)}$ is

$$\begin{aligned} \begin{aligned} \textbf{A}^{(1)}(1,1)&= \sqrt{\textbf{G}(p_1, p_1)} = \sqrt{\textbf{G}^\prime (\pi _0(p_1),\pi _0( p_1))} \\&= \sqrt{\textbf{G}^\prime (q_1, q_1)} = \textbf{B}^{(1)}(1,1) = \textbf{B}^{(1)}(\pi _1(1),1). \end{aligned} \end{aligned}$$

The rest elements of the first column of $\textbf{A}^{(1)}$ are

$$\begin{aligned} \begin{aligned} \textbf{A}^{(1)}(i,1)&= \left\{ \begin{array}{ll} \frac{\textbf{G}(i, p_1)}{\textbf{A}^{(1)}(1,1)}, &{} i\ne p_1;\\ \frac{\textbf{G}(1, p_1)}{\textbf{A}^{(1)}(1,1)}, &{} i = p_1.\\ \end{array}\right. \\&= \frac{\textbf{G}(\omega _1(i), p_1)}{\textbf{A}^{(1)}(1,1)}, \ 2\leqslant i \leqslant n. \end{aligned} \end{aligned}$$

In a similar way,

$$\begin{aligned} \textbf{B}^{(1)}(i,1) = \frac{\textbf{G}^\prime (\omega _1^\prime (i), q_1)}{\textbf{B}^{(1)}(1,1)}, \ 2\leqslant i \leqslant n. \end{aligned}$$

Note that $\textbf{A}^{(1)}(1,1) = \textbf{B}^{(1)}(1,1)$, $\omega ^\prime _1\circ \omega ^\prime _1(i) = i$ and $\textbf{G}(i,p_1) = \textbf{G}^\prime (\pi _0(i),\pi _0(p_1))$. Then we obtain

$$\begin{aligned} \begin{aligned} \textbf{A}^{(1)}(i,1) = \ {}&\frac{\textbf{G}(\omega _1(i), p_1)}{\textbf{A}^{(1)}(1,1)} = \frac{\textbf{G}^\prime (\pi _0\circ \omega _1(i), \pi _0(p_1))}{\textbf{B}^{(1)}(1,1)}\\ = \ {}&\frac{\textbf{G}^\prime (\omega ^\prime _1\circ \omega ^\prime _1\circ \pi _0\circ \omega _1(i), q_1)}{\textbf{B}^{(1)}(1,1)} \\ = \ {}&\frac{\textbf{G}^\prime (\omega ^\prime _1\circ \pi _1(i), q_1)}{\textbf{B}^{(1)}(1,1)}\\ = \ {}&\textbf{B}^{(1)}(\pi _1(i),1), \ 2\leqslant i \leqslant n. \end{aligned} \end{aligned}$$

Let $\textbf{P}^{(1)} = \textbf{P}[1,p_1]$ and $\textbf{Q}^{(1)} = \textbf{P}[1,q_1]$. Denote $\textbf{G}^{(1)} = \textbf{P}^{(1)}\textbf{G}\textbf{P}^{(1)}$ and ${\textbf{G}^\prime }^{(1)} = \textbf{Q}^{(1)}\textbf{G}^\prime \textbf{Q}^{(1)}$, then

$$\begin{aligned} \textbf{G}^{(1)}(i,j) = (\textbf{P}^{(1)}\textbf{G}\textbf{P}^{(1)})(i,j) = \textbf{G}(\omega _1(i), \omega _1(j)) \end{aligned}$$

(C15)

and

$$\begin{aligned} \begin{aligned} {\textbf{G}^\prime }^{(1)}(\pi _1(i),\pi _1(j)) = \ {}&(\textbf{Q}^{(1)} \textbf{G}^\prime \textbf{Q}^{(1)})(\pi _1(i),\pi _1(j))\\ = \ {}&\textbf{G}^\prime (\omega ^\prime _1\circ \pi _1(i),\omega ^\prime _1\circ \pi _1(j))\\ = \ {}&\textbf{G}^\prime (\pi _0\circ \omega _1(i),\pi _0\circ \omega _1(j))\\ = \ {}&\textbf{G}(\omega _1(i), \omega _1(j)), \end{aligned} \end{aligned}$$

(C16)

where the third equality results from (C14). Combining (C15) and (C16) yields $\textbf{G}^{(1)}(i,j) = {\textbf{G}^\prime }^{(1)}(\pi _1(i),\pi _1(j))$. Therefore, the theorem holds for $s=1$.

2) Assume that the theorem holds for $s = m$. There exists a bijection $\pi _m:\{1,2,\cdots ,n\}\rightarrow \{1,2,\cdots ,n\}$ such that $\pi _m(i) = i$ ($1\leqslant i\leqslant m$) and for every i,

$$\begin{aligned} \textbf{A}^{(m)}(i,j) = \textbf{B}^{(m)}(\pi _m(i),j), \ 1\leqslant i\leqslant m. \end{aligned}$$

(C17)

There also exist permutation matrices $\textbf{P}^{(m)}$ and $\textbf{Q}^{(m)}$ such that

$$\begin{aligned} \textbf{G}^{(m)}(i,j) = {\textbf{G}^\prime }^{(m)}(\pi _m(i),\pi _m(j)), \end{aligned}$$

(C18)

where $\textbf{G}^{(m)} = \textbf{P}^{(m)} \textbf{G} \textbf{P}^{(m)}$ and ${\textbf{G}^\prime }^{(m)} = \textbf{Q}^{(m)} \textbf{G}^\prime \textbf{Q}^{(m)}$.

3) We will show that the theorem holds for $s = m+1$. The $(m+1)$th to the nth diagonal elements of $\textbf{A}^{(m)}$ are

$$\begin{aligned} \begin{aligned} \textbf{A}^{(m)}&(i,i) = \ \textbf{G}^{(m)}(i,i) - \sum _{j=1}^m\textbf{A}^{(m)}(i,j)^2\\ &=\ {\textbf{G}^\prime }^{(m)}(\pi _m(i),\pi _m(i)) - \sum _{j=1}^m\textbf{B}^{(m)}(\pi _m(i),j)^2\\ &=\ \textbf{B}^{(m)}(\pi _m(i),\pi _m(i)), \ m+1\leqslant i \leqslant n. \end{aligned} \end{aligned}$$

Note that $\pi _m(i) = i$ ($1\leqslant i\leqslant m$) and $\pi _m$ is a bijection. Let $p_{m+1} = \arg \max _{m+1\leqslant i\leqslant n}\textbf{A}^{(m)}(i,i)$ and $q_{m+1} = \arg \max _{m+1\leqslant i\leqslant n} \textbf{B}^{(m)}(i,i)$. We have $q_{m+1} = \pi _m(p_{m+1})$ due to the uniqueness of the largest diagonal elements of symmetric pivoting. We construct bijections

$$\begin{aligned} \omega _{m+1}(i) = \left\{ \begin{array}{ll} i, &{} i\ne m+1, p_{m+1};\\ p_{m+1}, &{} i = m+1;\\ m+1, &{} i = p_{m+1}. \end{array}\right. \end{aligned}$$

and

$$\begin{aligned} \omega ^\prime _{m+1}(i) = \left\{ \begin{array}{ll} i, &{} i\ne m+1, q_{m+1};\\ q_{m+1}, &{} i = m+1;\\ m+1, &{} i = q_{m+1}. \end{array}\right. \end{aligned}$$

Define a composite function $\pi _{m+1}:\{1,2,\cdots ,n\}\rightarrow \{1,2,\cdots ,n\}$:

$$\begin{aligned} \pi _{m+1}(i) = \omega ^\prime _{m+1}\circ \pi _m\circ \omega _{m+1}(i). \end{aligned}$$

(C19)

For $1 \leqslant i \leqslant m$,

$$\begin{aligned} \begin{aligned}&\pi _{m+1}(i) = \omega ^\prime _{m+1}\circ \pi _m\circ \omega _{m+1}(i)\\ &=\omega ^\prime _{m+1}\circ \pi _m(i) = \omega ^\prime _{m+1}(i) = i. \end{aligned} \end{aligned}$$

We also have

$$\begin{aligned} \begin{aligned}&\pi _{m+1}(m+1) = \omega ^\prime _{m+1}\circ \pi _m\circ \omega _{m+1}(m+1) \\ &=\omega ^\prime _{m+1}\circ \pi _m(p_{m+1}) = \omega ^\prime _{m+1}(q_{m+1}) = m+1. \end{aligned} \end{aligned}$$

Thus, $\pi _{m+1}(i) = i \ (1\leqslant i\leqslant m+1)$.

Let $\textbf{P}^{(m+1)} = \textbf{P}[m+1,p_{m+1}] \textbf{P}^{(m)}$ and $\textbf{Q}^{(m+1)} = \textbf{P}[m+1,q_{m+1}] \textbf{Q}^{(m)}$. Denote $\textbf{G}^{(m+1)} = \textbf{P}^{(m+1)} \textbf{G} \textbf{P}^{(m+1)}$ and ${\textbf{G}^\prime }^{(m+1)} = \textbf{Q}^{(m+1)} {\textbf{G}^\prime } \textbf{Q}^{(m+1)}$, then

$$\begin{aligned} \begin{aligned} \textbf{G}^{(m+1)}(i,j) &=\ (\textbf{P}^{(m+1)} \textbf{G} \textbf{P}^{(m+1)})(i,j) \\ &=\ \textbf{G}^{(m)}(\omega _{m+1}(i), \omega _{m+1}(j)), \end{aligned} \end{aligned}$$

(C20)

and

$$\begin{aligned} \begin{aligned}&\ {\textbf{G}^\prime }^{(m+1)}(\pi _{m+1}(i),\pi _{m+1}(j)) \\ &=\ (\textbf{Q}^{(m+1)} {\textbf{G}^\prime } \textbf{Q}^{(m+1)})(\pi _{m+1}(i),\pi _{m+1}(j))\\ &=\ {\textbf{G}^\prime }^{(m)}(\omega ^\prime _{m+1}\circ \pi _{m+1}(i),\omega ^\prime _{m+1}\circ \pi _{m+1}(j))\\ &=\ {\textbf{G}^\prime }^{(m)}(\pi _m\circ \omega _{m+1}(i),\pi _m\circ \omega _{m+1}(j))\\ &=\ \textbf{G}^{(m)}(\omega _{m+1}(i), \omega _{m+1}(j)), \end{aligned} \end{aligned}$$

(C21)

where the last two equations follows from (C19) and (C18), respectively. Combining (C20) and (C21) yields

$$\begin{aligned} \textbf{G}^{(m+1)}(i,j) = {\textbf{G}^\prime }^{(m+1)}(\pi _{m+1}(i), \pi _{m+1}(j)). \end{aligned}$$

For $1 \leqslant i \leqslant n$ and $1 \leqslant j \leqslant m$,

$$\begin{aligned} \begin{aligned} \textbf{A}^{(m+1)}(i,j) &=\left\{ \begin{array}{ll} \textbf{A}^{(m)}(i,j), &{} i\ne m+1, p_{m+1};\\ \textbf{A}^{(m)}(p_{m+1},j), &{} i = m+1;\\ \textbf{A}^{(m)}(m+1,j), &{} i = p_{m+1}.\\ \end{array}\right. \\ = \ {}&\textbf{A}^{(m)}(\omega _{m+1}(i),j) \\ = \ {}&\textbf{B}^{(m)}(\pi _m\circ \omega _{m+1}(i),j)\\ = \ {}&\textbf{B}^{(m)}(\omega ^\prime _{m+1}\circ \pi _{m+1}(i),j)\\ = \ {}&\textbf{B}^{(m+1)}(\pi _{m+1}(i),j), \end{aligned} \end{aligned}$$

where the third and the fourth equations results from (C17) and (C19), respectively. For the $(m+1)$th diagonal element of $\textbf{A}^{(m+1)}$,

$$\begin{aligned} \begin{aligned}&\textbf{A}^{(m+1)}(m+1,m+1) = \sqrt{\textbf{G}^{(m)}(p_{m+1}, p_{m+1})} \\ = \ {}&\sqrt{{\textbf{G}^\prime }^{(m)}(\pi _m(p_{m+1}),\pi _m( p_{m+1}))} \\ = \ {}&\sqrt{{\textbf{G}^\prime }^{(m)}(q_{m+1}, q_{m+1})} \\ = \ {}&\textbf{B}^{(m+1)}(m+1,m+1) \end{aligned} \end{aligned}$$

and for $m + 2 \leqslant i \leqslant n$,

$$\begin{aligned} \begin{aligned}&\ \textbf{A}^{(m + 1)}(i,m + 1) \\ &=\ \frac{{\textbf{G}^{(m + 1)}(i,m + 1)}}{{\textbf{A}^{(m + 1)}(m + 1,m + 1)}} -\\&\ \frac{\sum \limits _{l = 1}^m {\textbf{A}^{(m + 1)}(i,l)} \textbf{A}^{(m + 1)}(m + 1,l)}{\textbf{A}^{(m + 1)}(m + 1,m + 1)}\\ &=\ \frac{{\textbf{G}^\prime }^{(m + 1)}({\pi _{m + 1}}(i),{\pi _{m + 1}}(m + 1))}{\textbf{B}^{(m + 1)}(m + 1,m + 1)} -\\&\ \frac{\sum \limits _{l = 1}^m {\textbf{B}^{(m + 1)}({\pi _{m + 1}}(i),l)} \textbf{B}^{(m + 1)}({\pi _{m + 1}}(m + 1),l)}{\textbf{B}^{(m + 1)}(m + 1,m + 1)}\\ &=\ \frac{{\textbf{G}^\prime }^{(m+1)}({\pi _{m + 1}}(i),m + 1)}{\textbf{B}^{(m + 1)}(m + 1,m + 1)} -\\&\ \frac{\sum \limits _{l = 1}^m {\textbf{B}^{(m + 1)}({\pi _{m + 1}}(i),l)} \textbf{B}^{(m + 1)}(m + 1,l)}{\textbf{B}^{(m + 1)}(m + 1,m + 1)}\\ &=\ \textbf{B}^{(m + 1)}({\pi _{m + 1}}(i),m + 1). \end{aligned} \end{aligned}$$

Thus, for every i we obtain

$$\begin{aligned} \textbf{A}^{(m+1)}(i,j) = \textbf{B}^{(m+1)}(\pi _{m+1}(i),j),\ 1\leqslant j\leqslant m+1. \end{aligned}$$

The proof is complete by mathematical induction.

$\square$