Abstract
A common assumption in supervised learning is that the training and test input points follow the same probability distribution. However, this assumption is not fulfilled, e.g., in interpolation, extrapolation, or active learning scenarios. The violation of this assumption—known as the covariate shift—causes a heavy bias in standard generalization error estimation schemes such as cross-validation and thus they result in poor model selection. In this paper, we therefore propose an alternative estimator of the generalization error. Under covariate shift, the proposed generalization error estimator is unbiased if the learning target function is included in the model at hand and it is asymptotically unbiased in general. Experimental results show that model selection with the proposed generalization error estimator is compared favorably to cross-validation in extrapolation.
The authors would like to thank Dr. Motoaki Kawanabe and Dr. Gilles Blanchard for their valuable comments. We acknowledge the Alexander von Humboldt Foundation and from the PASCAL Network of Excellence (EU #506778) for financial support.
An erratum to this chapter can be found at http://dx.doi.org/10.1007/11550907_163 .
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Fedorov, V.V.: Theory of Optimal Experiments. Academic Press, New York (1972)
Heckman, J.J.: Sample selection bias as a specification error. Econometrica 47(1), 153–162 (1979)
Shimodaira, H.: Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90(2), 227–244 (2000)
Vapnik, V.N.: Statistical Learning Theory. John Wiley & Sons, Inc., New York (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sugiyama, M., Müller, KR. (2005). Model Selection Under Covariate Shift. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds) Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005. ICANN 2005. Lecture Notes in Computer Science, vol 3697. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11550907_37
Download citation
DOI: https://doi.org/10.1007/11550907_37
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-28755-1
Online ISBN: 978-3-540-28756-8
eBook Packages: Computer ScienceComputer Science (R0)