Model-free global likelihood subsampling for massive data

Abstract

Most existing studies for subsampling heavily depend on a specified model. If the assumed model is not correct, the performance of the subsample may be poor. This paper focuses on a model-free subsampling method, called global likelihood subsampling, such that the subsample is robust to different model choices. It leverages the idea of the global likelihood sampler, which is an effective and robust sampling method from a given continuous distribution. Furthermore, we accelerate the algorithm for large-scale datasets and extend it to deal with high-dimensional data with relatively low computational complexity. Simulations and real data studies are conducted to apply the proposed method to regression and classification problems. It illustrates that this method is robust against different modeling methods and has promising performance compared with some existing model-free subsampling methods for data compression.

Appendix

Proof of Lemma 1

By Theorem 2 in Jiang (2017), we directly have

$$\begin{aligned} \sup _{\varvec{\theta }\in \mathbb {R}^d} \vert \hat{f}_\textrm{K}(\varvec{\theta })-f(\varvec{\theta })\vert = O_\textrm{P}(\Delta ). \end{aligned}$$

Let the support of f be ${\mathcal T}$ and the volume of ${\mathcal T}$ is $v({\mathcal T})=V_T<\infty$. Then,

$$\begin{aligned} \sup _{\varvec{\theta }\in \mathbb {R}^d} \vert \hat{F}_\textrm{K}(\varvec{\theta })-F(\varvec{\theta })\vert =&\sup _{\varvec{\theta }\in \mathbb {R}^d} \left| \int _{(-\infty ,\varvec{\theta }]\cap {\mathcal T}} (\hat{f}_\textrm{K}(\varvec{\vartheta })-f(\varvec{\vartheta })) \textrm{d} \varvec{\vartheta }\right| \\ \le&\sup _{\varvec{\theta }\in \mathbb {R}^d} \int _{(-\infty ,\varvec{\theta }]\cap {\mathcal T}}\vert \hat{f}_\textrm{K}(\varvec{\vartheta })-f(\varvec{\vartheta })\vert \textrm{d} \varvec{\vartheta }\\ \le&V_T \cdot \sup _{\varvec{\theta }\in \mathbb {R}^d} \vert \hat{f}_\textrm{K}(\varvec{\theta })-f(\varvec{\theta })\vert = O_\textrm{P}(\Delta ), \end{aligned}$$

which completes the proof. $\square$

Proof of Lemma 2

Let $\tilde{F}(\varvec{\theta })$ be the multinomial distribution placing mass $\hat{f}_\textrm{K}({\varvec{s}}_k)/\sum _{i=1}^M \hat{f}_\textrm{K}({\varvec{s}}_i)$ at ${\varvec{s}}_k, k=1,\ldots ,M$. Similar to the proof of Lemma 1(1) in Yi et al. (2022), if ${\mathcal S}$ is the low-discrepancy point set and $\hat{f}_\textrm{K}({\varvec{x}})I_{[-\infty ,\varvec{\theta }]}({\varvec{x}})$ is of uniformly bounded variation for any $\varvec{\theta }\in \mathbb {R}^d$, then for any $\varvec{\theta }\in \mathbb {R}^d$, we have

$$\begin{aligned} \vert \tilde{F}(\varvec{\theta })-\hat{F}_\textrm{K}(\varvec{\theta })\vert =O(\kappa (M)). \end{aligned}$$

Let $D_\textrm{m}=\{(2j-1)/2n: j=1,\ldots ,n\}$. Then, by the proof of Theorem 2 in Yi et al. (2022), we can obtain that

$$\begin{aligned} F_{{\mathcal Z}_\textrm{s}}(\varvec{\theta })=\frac{1}{n}\sum _{u\in D_\textrm{m}}\varrho _{\varvec{\theta }}(u), ~~ \tilde{F}(\varvec{\theta })=\int _{[0,1]}\varrho _{\varvec{\theta }}(u) \textrm{d}u. \end{aligned}$$

Thus, $\vert F_{{\mathcal Z}_\textrm{s}}(\varvec{\theta })-\tilde{F}(\varvec{\theta })\vert$ can be viewed as the quasi-Monte Carlo error based on $D_\textrm{m}$. The star discrepancy of $D_\textrm{m}$ is always no more than 1/n. Hence, by the Koksma-Hlawka inequality (Hlawka 1961), if $\varrho _{\varvec{\theta }}(u)$ is of uniformly bounded variation for any $\varvec{\theta }\in \mathbb {R}^d$ and run size M of the space-filling design ${\mathcal S}$, then for any $\varvec{\theta }\in \mathbb {R}^d$, we have

$$\begin{aligned} \vert F_{{\mathcal Z}_\textrm{s}}(\varvec{\theta })-\tilde{F}(\varvec{\theta })\vert =O\left( \frac{1}{n}\right) . \end{aligned}$$

Since the above formula is valid for any $\varvec{\theta }\in \mathbb {R}^d$, it is also true for the supremum value. Then, by the triangle inequality,

$$\begin{aligned}&\sup _{\varvec{\theta }\in \mathbb {R}^d}\vert F_{\tilde{{\mathcal Z}}_\textrm{s}}(\varvec{\theta }) - \hat{F}_\textrm{K}(\varvec{\theta })\vert \\&\quad \le \sup _{\varvec{\theta }\in \mathbb {R}^d} \vert \tilde{F}(\varvec{\theta })-\hat{F}_\textrm{K}(\varvec{\theta })\vert + \sup _{\varvec{\theta }\in \mathbb {R}^d} \vert F_{{\mathcal Z}_\textrm{s}}(\varvec{\theta })-\tilde{F}(\varvec{\theta })\vert \\&\quad = O\left( \max \left\{ \kappa (M), \frac{1}{n}\right\} \right) , \end{aligned}$$

which completes the proof.

Proof of Theorem 3

Based on Lemmas 1 and 2, by the triangle inequality, we directly have

$$\begin{aligned} \sup _{\varvec{\theta }\in \mathbb {R}^d}\vert F_{\tilde{{\mathcal Z}}_\textrm{s}}(\varvec{\theta }) - F(\varvec{\theta })\vert = O_{\textrm{P}}\left( \max \left\{ \Delta ,\kappa (M), \frac{1}{n}\right\} \right) . \end{aligned}$$

Let ${\mathcal D}= \otimes _{i=1}^d [a_i,b_i]$ and $\delta$ be the volume of the union of the balls ${\mathcal B}({\varvec{s}}_i, r),i=1,\ldots , M$, divided by the volume of $\otimes _{i=1}^d [a_i-r,b_i+r]$. Since r is set as half of the separation distance of ${\mathcal S}$, we have $\delta \prod _{i=1}^d (2r+b_i-a_i)=M v_d r^d$. Without loss of generality, let $a_i=0$ and $b_i=1$ for $i=1,\ldots ,d$, which leads to $r=1/[(Mv_d\delta ^{-1})^{1/d}-2]$. Then, we have $r=O(M^{-1/d})$, which is also true when $a_i$ and $b_i (a_i<b_i)$ take any other bounded values for $i=1,\ldots ,d$.

Based on the result, we have $\Delta =O(M^{-\alpha /d}+(M\log N/N)^{1/2})$. In addition, we can easily obtain that $M^{-\alpha /d}>1/M$ if $d=1$, and $M^{-\alpha /d} >(\log M)^d/M$ as $M \rightarrow \infty$ if $d>1$. Thus, $O_{\textrm{P}}(\max \{\Delta , \kappa (M)\})=O_{\textrm{P}}(M^{-\alpha /d}+(M\log N/N)^{1/2})$ as $M \rightarrow \infty$, which completes the proof.

Proof of Lemma 4

For any ${\varvec{z}}_\textrm{s} \in {\mathcal Z}_\textrm{s}$ and $\varvec{\theta }\in \mathbb {R}^d$, we have

$$\begin{aligned} \textrm{P}({\varvec{z}}_\textrm{s} \in (-\infty , \varvec{\theta }]){} & {} = \textrm{E}(\textrm{E}(I_{(-\infty , \varvec{\theta }]}({\varvec{z}}_\textrm{s}\mid {\mathcal Z})))=\textrm{E}(F_{{\mathcal Z}}(\varvec{\theta }))\\ {}{} & {} =F(\varvec{\theta }), \end{aligned}$$

where $F_{{\mathcal Z}}$ is the ECDF of ${\mathcal Z}$. Hence, any element in ${\mathcal Z}_\textrm{s}$ follows the CDF F. By the multivariate Dvoretzky-Kiefer-Wolfowitz inequality (Naaman 2021), we have

$$\begin{aligned} \sup _{\varvec{\theta }\in \mathbb {R}^d}\vert F(\varvec{\theta })-F_{{\mathcal Z}_\textrm{s}}(\varvec{\theta })\vert =O_\textrm{P}\left( \frac{1}{\sqrt{N_\textrm{s}}}\right) , \end{aligned}$$

which completes the proof.