Predicting missing values: a comparative study on non-parametric approaches for imputation

Abstract

Missing data is an expected issue when large amounts of data is collected, and several imputation techniques have been proposed to tackle this problem. Beneath classical approaches such as MICE, the application of Machine Learning techniques is tempting. Here, the recently proposed missForest imputation method has shown high imputation accuracy under the Missing (Completely) at Random scheme with various missing rates. In its core, it is based on a random forest for classification and regression, respectively. In this paper we study whether this approach can even be enhanced by other methods such as the stochastic gradient tree boosting method, the C5.0 algorithm, BART or modified random forest procedures. In particular, other resampling strategies within the random forest protocol are suggested. In an extensive simulation study, we analyze their performances for continuous, categorical as well as mixed-type data. An empirical analysis focusing on credit information and Facebook data complements our investigations.

Appendix

Missing values have been inserted artificially under consideration of the various missing mechanisms. For the MCAR and MAR condition, a special kind of mechanism has been implemented, which will be described in the following:

• 1. Missing completely at random We replace values randomly with missing values. For every variable $\mathbf {X}_{j}$, $j = 1,\ldots ,p$, we assumed that $R_{ij} {\mathop {\sim }\limits ^{iid}} Bernoulli(1-r)$, $i = 1,\ldots ,n$ where $r \in \{0.1, 0.2, 0.3\}$ is the overall missing rate based on $n \cdot p$.

• 2. Missing at random We implement this mechanism by building dependency structures across missing values of subsequent variables using the logistic regression. First, randomly select $j^{* }\in \{1,\ldots ,p\}$ as the initial index and assume that $R_{ij^{*}} {\mathop {\sim }\limits ^{iid}} Bernoulli(1 - r)$, where $r \in \{0.1, 0.2,$$0.3\}$ is the overall missing rate. The missing values for the subsequent variable $\mathbf {X}_{j_{s}^{*}}$ are inserted using the observed components of $\mathbf {X}_{j^{*}}$ as covariate values within a logistic regression model. The response variables are randomly generated in an upstream step to estimate model parameters. Therefore, let $\mathbf {X}_{j^{*}}^{obs}$ be the sub-vector of observed components of $\mathbf {X}_{j^{*}}$. We construct a training response by generating $\tilde{R}_{ij_{s}^{*}} {\mathop {\sim }\limits ^{iid}} Bernoulli(1 - r)$ for all $i \in \mathbf {i}_{j^{*}}^{obs}$ and set $\{ \tilde{\mathbf {R}}_{j_{s}^{*}}, \mathbf {X}_{j^{*}}^{obs} \}$ as the training sample on which the logistic regression will be conducted. If $\hat{p}_{i j_{s}^{*}}$ is the predicted probability of $(\tilde{R}_{ij_{s}^{*}} = 1 | X_{ij^{*}}),$$i \in \mathbf {i}_{j^{*}}^{obs}$, then for the observations $k \in \mathbf {i}_{j^{*}}^{obs}$ with the $\lfloor r \cdot n \rfloor$ - smallest values of $\hat{p}_{i j_{s}^{*}}$, we set $R_{kj_{s}^{*}} = 0$. The process is continued in a pairwise fashion until all variables have been treated.