Condorcet’s Jury Theorem for Consensus Clustering

Abstract

Condorcet’s Jury Theorem has been invoked for ensemble classifiers to indicate that the combination of many classifiers can have better predictive performance than a single classifier. Such a theoretical underpinning is unknown for consensus clustering. This article extends Condorcet’s Jury Theorem to the mean partition approach under the additional assumptions that a unique but unknown ground-truth partition exists and sample partitions are drawn from a sufficiently small ball containing the ground-truth.

A Proof of Theorem 2

To prove Theorem 2, it is helpful to use a suitable representation of partitions. We suggest to represent partitions as points of some geometric space, called orbit space [20]. Orbit spaces are well explored, possess a rich geometrical structure and have a natural connection to Euclidean spaces [3, 19, 30].

1.1 A.1 Partition Spaces

We denote the natural projection that sends matrices to the partitions they represent by

$$ \pi : {\mathcal {X}} \rightarrow {\mathcal {P}}, \quad {\varvec{X}} \mapsto \pi ({\varvec{X}}) = X. $$

The group $\varPi = \varPi ^\ell$ of all ($\ell \times \ell$)-of all ($\ell \times \ell$)-permutation matrices is a discontinuous group that acts on ${\mathcal {X}}$ by matrix multiplication, that is

$$ \cdot : \varPi \times {\mathcal {X}} \rightarrow {\mathcal {X}}, \quad ({\varvec{P}}, {\varvec{X}}) \mapsto {\varvec{PX}}. $$

The orbit of ${\varvec{X}} \in {\mathcal {X}}$ is the set $\mathop {\left[ {\varvec{X}} \right] } = \mathop {\left\{ {\varvec{PX}} \,:\, {\varvec{P}} \in \varPi \right\} }$. The orbit space of partitions is the quotient space ${\mathcal {X}}/\varPi = \mathop {\left\{ \mathop {\left[ {\varvec{X}} \right] } \,:\, {\varvec{X}} \in {\mathcal {X}} \right\} }$ obtained by the action of the permutation group $\varPi$ on the set ${\mathcal {X}}$. We write ${\mathcal {P}} = {\mathcal {X}}/\varPi$ to denote the partition space and $X \in {\mathcal {P}}$ to denote an orbit $[{\varvec{X}}] \in {\mathcal {X}}/\varPi$. The natural projection $\pi : {\mathcal {X}} \rightarrow {\mathcal {P}}$ sends matrices ${\varvec{X}}$ to the partitions $\pi ({\varvec{X}}) = \mathop {\left[ {\varvec{X}} \right] }$ they represent. The partition space ${\mathcal {P}}$ is endowed with the intrinsic metric $\delta$ defined by $\delta (X, Y) = \min \mathop {\left\{ \Vert {{\varvec{X}} - {\varvec{Y}}}\Vert \,:\, {\varvec{X}} \in X, {\varvec{Y}} \in Y \right\} }$.

1.2 A.2 Dirichlet Fundamental Domains

We use the following notations: By $\overline{{\mathcal {U}}}$ we denote the closure of a subset ${\mathcal {U}} \subseteq {\mathcal {X}}$, by $\partial {\mathcal {U}}$ the boundary of ${\mathcal {U}}$, and by ${\mathcal {U}}^\circ$ the open subset $\overline{{\mathcal {U}}} \setminus \partial {\mathcal {U}}$. The action of permutation ${\varvec{P}} \in \varPi$ on the subset ${\mathcal {U}}\subseteq {\mathcal {X}}$ is the set defined by ${\varvec{P}}\,{\mathcal {U}} = \mathop {\left\{ {\varvec{PX}} \, :\, {\varvec{X}} \in {\mathcal {U}} \right\} }$. By $\varPi ^* = \varPi \setminus \mathop {\left\{ {\varvec{I}} \right\} }$ we denote the subset of ($\ell \times \ell$)-permutation matrices without identity matrix ${\varvec{I}}$.

A subset ${\mathcal {F}}$ of ${\mathcal {X}}$ is a fundamental set for $\varPi$ if and only if ${\mathcal {F}}$ contains exactly one representation ${\varvec{X}}$ from each orbit $\mathop {\left[ {\varvec{X}} \right] } \in {\mathcal {X}}/\varPi$. A fundamental domain of $\varPi$ in ${\mathcal {X}}$ is a closed connected set ${\mathcal {F}} \subseteq {\mathcal {X}}$ that satisfies

• 1. $\displaystyle {\mathcal {X}} = \bigcup _{{\varvec{P}} \in \varPi } {\varvec{P}}{\mathcal {F}}$

• 2. ${\varvec{P}} {\mathcal {F}}^\circ \cap {\mathcal {F}}^\circ = \emptyset$ for all ${\varvec{P}} \in \varPi ^*$.

Proposition 1

Let ${\varvec{Z}}$ be a representation of an asymmetric partition $Z \in {\mathcal {P}}$. Then

$$ {\mathcal {D}}_{{\varvec{Z}}} = \mathop {\left\{ {\varvec{X}} \in {\mathcal {X}} \,:\, \Vert {{\varvec{X}} - {\varvec{Z}}}\Vert \le \Vert {{\varvec{X}} - {\varvec{PZ}}}\Vert \text { for all }{\varvec{P}} \in \varPi \right\} } $$

is a fundamental domain, called Dirichlet fundamental domain of ${\varvec{Z}}$.

Proof

[30], Theorem 6.6.13.    $\square$

Lemma 1

Let ${\mathcal {D}}_{{\varvec{Z}}}$ be a Dirichlet fundamental domain of representation ${\varvec{Z}}$ of an asymmetric partition $Z \in {\mathcal {P}}$. Suppose that ${\varvec{X}}$ and ${\varvec{X}}'$ are two different representations of a partition X such that ${\varvec{X}}, {\varvec{X}}' \in {\mathcal {D}}_{{\varvec{Z}}}$. Then ${\varvec{X}}, {\varvec{X}}' \in \partial {\mathcal {D}}_{{\varvec{Z}}}$.

Proof

[19], Prop. 3.13 and [22], Prop. A.2.    $\square$

1.3 A.3 Multiple Alignments

Let ${\mathcal {S}}_n = \mathop {\left( X_1, \ldots , X_n \right) }$ be a sample of n partitions $X_i \in {\mathcal {P}}$. A multiple alignment of ${\mathcal {S}}_n$ is an n-tuple consisting of representations ${\varvec{X}}_{i}\in X_{i}$. By

we denote the set of all multiple alignments of ${\mathcal {S}}_n$. A multiple alignment is said to be in optimal position with representation ${\varvec{Z}}$ of a partition Z, if all representations ${\varvec{X}}_{i}$ of are in optimal position with ${\varvec{Z}}$. The mean of a multiple alignment is denoted by

$$\begin{aligned} {\varvec{M}}_{\mathfrak {X}} = \frac{1}{n} \sum _{i=1}^n {\varvec{X}}_{i}. \end{aligned}$$

An optimal multiple alignment is a multiple alignment that minimizes the function

$$\begin{aligned} f_n\!\mathop {\left( \mathfrak {X} \right) } = \frac{1}{n^2}\sum _{i=1}^n \sum _{j=1}^n \Vert {{\varvec{X}}_{i} - {\varvec{X}}_{j}}\Vert {^2}. \end{aligned}$$

The problem of finding an optimal multiple alignment is that of finding a multiple alignment with smallest average pairwise squared distances in ${\mathcal {X}}$. To show equivalence between mean partitions and an optimal multiple alignments, we introduce the sets of minimizers of the respective functions $F_n$ and $f_n$:

For a given sample ${\mathcal {S}}_n$, the set ${\mathcal {M}}(F_n)$ is the mean partition set and ${\mathcal {M}}(f_n)$ is the set of all optimal multiple alignments. The next result shows that any solution of $F_n$ is also a solution of $f_n$ and vice versa.

Theorem 3

For any sample ${\mathcal {S}}_n \in {\mathcal {P}}^n$, the map

is surjective.

Proof

[23], Theorem 4.1.    $\square$

1.4 A.4 Proof of Theorem 2

Parts 1–8 show the assertion of Eq. (2) and Part 9 shows the assertion of Eq. (3).

1 Without loss of generality, we pick a representation ${\varvec{X}}_{*}$ of the ground-truth partition $X_*$. Let ${\varvec{Z}}$ be a representation of Z in optimal position with ${\varvec{X}}_{*}$. By

$$ {\mathcal {A}}_{{\varvec{Z}}} = \mathop {\left\{ {\varvec{X}} \in {\mathcal {X}} \,:\, \Vert {{\varvec{X}}-{\varvec{Z}}}\Vert \le \alpha _Z/4 \right\} } $$

we denote the asymmetry ball of representation ${\varvec{Z}}$. By construction, we have ${\varvec{X}}_{*} \in {\mathcal {A}}_{{\varvec{Z}}}$.

2 Since $\varPi$ acts discontinuously on ${\mathcal {X}}$, there is a bijective isometry

$$ \phi :{\mathcal {A}}_{{\varvec{Z}}} \rightarrow {\mathcal {A}}_Z, \quad {\varvec{X}} \mapsto \pi ({\varvec{X}}) $$

according to [30], Theorem 13.1.1.

3 From [22], Theorem 3.1 follows that the mean partition M of ${\mathcal {S}}_n$ is unique. We show that $M \in {\mathcal {A}}_Z$. Suppose that is a multiple alignment in optimal position with ${\varvec{Z}}$. Since $\phi :{\mathcal {A}}_{{\varvec{Z}}} \rightarrow {\mathcal {A}}_Z$ is a bijective isometry, we have

showing that the multiple alignment is optimal. From Theorem 3 follows that

is a representation of a mean partition M of ${\mathcal {S}}_n$. Since ${\mathcal {A}}_{{\varvec{Z}}}$ is convex, we find that ${\varvec{M}} \in {\mathcal {A}}_{{\varvec{Z}}}$ and therefore $M \in {\mathcal {A}}_Z$.

4 From Part 1–3 of this proof follows that the multiple alignment is in optimal position with ${\varvec{X}}_{*}$. We show that there is no other multiple alignment of ${\mathcal {S}}_n$ with this property. Observe that ${\mathcal {A}}_{{\varvec{Z}}}$ is contained in the Dirichlet fundamental domain ${\mathcal {D}}_{{\varvec{Z}}}$ of representation ${\varvec{Z}}$. Let ${\mathcal {S}}_{{\varvec{Z}}} = \phi ({\mathcal {S}}_Q)$ be a representation of the support in ${\mathcal {A}}_{{\varvec{Z}}}^\circ$. Then by assumption, we have ${\mathcal {S}}_{{\varvec{Z}}} \subseteq {\mathcal {A}}_{{\varvec{Z}}}^\circ \subset {\mathcal {D}}_{{\varvec{Z}}}$ showing that ${\mathcal {S}}_{{\varvec{Z}}}$ lies in the interior of ${\mathcal {D}}_{{\varvec{Z}}}$. From the definition of a fundamental domain together with Lemma 1 follows that is the unique optimal alignment in optimal position with ${\varvec{X}}_{*}$.

5 With the same argumentation as in the previous part of this proof, we find that ${\varvec{M}}$ is the unique representation of M in optimal position with ${\varvec{X}}_{*}$.

6 Let $z \in {\mathcal {Z}}$ be a data point. Since ${\varvec{X}}_{i} \in X_i$ is the unique representation in optimal position with ${\varvec{X}}_{*}$, the vote of $X_i$ on data point z is of the form $V_{X_i}(z) = V_{{\varvec{X}}_{i}}(z)$ for all $i \in \mathop {\left\{ 1, \ldots , n \right\} }$. With the same argument, we have $V_n(z) = V_{M}(z) = V_{{\varvec{M}}}(z)$.

7 By ${\varvec{x}}^{(i)}(z)$ we denote the column of ${\varvec{X}}_{i}$ that represents z. By definition, we have

$$ p_z = \mathbb {P}\mathop {\left( V_{X_i}(z) = 1 \right) } = \mathbb {P}\mathop {\left( \mathop {\left\langle {\varvec{x}}^{(i)}(z), {\varvec{x}}^*(z) \right\rangle } > 0.5 \right) } $$

for all $i \in \mathop {\left\{ 1, \ldots , n \right\} }$. Since $X_i$ and $X_*$ are both hard partitions, we find that

$$ \mathop {\left\langle {\varvec{x}}^{(i)}(z), {\varvec{x}}^*(z) \right\rangle } = \mathbb {I}\mathop {\left\{ {\varvec{x}}^{(i)}(z) = {\varvec{x}}^*(z) \right\} }, $$

where $\mathbb {I}$ denotes the indicator function.

8 From the Mean Partition Theorem follows that

$$ {\varvec{m}}(z) = \frac{1}{n} \sum _{i=1}^n {\varvec{x}}^{(i)}(z) $$

is the column of ${\varvec{M}}$ that represents z. Then the agreement of ${\varvec{M}}$ on z is given by

$$\begin{aligned} k_{{\varvec{M}}}(z)&= \mathop {\left\langle {\varvec{m}}(z), {\varvec{x}}^*(z) \right\rangle }\\&= \frac{1}{n}\sum _{i=1}^n \mathop {\left\langle {\varvec{x}}^{(i)}(z), {\varvec{x}}^*(z) \right\rangle }\\&= \frac{1}{n}\sum _{i=1}^n \mathbb {I}\mathop {\left\{ {\varvec{x}}^{(i)}(z) = {\varvec{x}}^*(z) \right\} }. \end{aligned}$$

Thus, the agreement $k_{{\varvec{M}}}(z)$ counts the fraction of sample partitions $X_i$ that correctly classify z. Let

$$ p_n = \mathbb {P}\mathop {\left( h_n(z) = 1 \right) } = \mathbb {P}\mathop {\left( k_{{\varvec{M}}}(z) > 0.5 \right) } $$

denote the probability that the majority of the sample partitions $X_i$ correctly classifies z. Since the votes of the sample partitions are assumed to be independent, we can compute $p_n$ using the binomial distribution

$$ p_n = \sum _{i=r}^n \left( {\begin{array}{c}n\\ i\end{array}}\right) p^i (1-p)^{n-i}, $$

where $r = \lfloor n/2 \rfloor + 1$ and $\lfloor a \rfloor$ is the largest integer b with $b \le a$. Then the assertion of Eq. (2) follows from [16], Theorem 1.

9 We show the assertion of Eq. (3). By assumption, the support ${\mathcal {S}}_Q$ is contained in an open subset of the asymmetry ball ${\mathcal {A}}_Z$. From [22], Theorem 3.1 follows that the expected partition $M_Q$ of Q is unique. Then the sequence $(M_n)_{n \in \mathbb {N}}$ converges almost surely to the expected partition $M_Q$ according to [20], Theorem 3.1 and Theorem 3.3. From the first eight parts of the proof follows that the limit partition $M_Q$ agrees on any data point z almost surely with the ground-truth partition $X_*$. This shows the assertion.