skip to main content
10.1145/1273496.1273632acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlConference Proceedingsconference-collections
Article

Asymptotic Bayesian generalization error when training and test distributions are different

Published: 20 June 2007 Publication History

Abstract

In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian generalization error and establish its upper bound under a very general setting. Our important finding is that lower order terms---which can be ignored in the absence of the distribution change---play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.

References

[1]
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, AC-19, 716--723.
[2]
Baldi, P., Brunak, S., & Stolovitzky, G. A. (1998). Bioinformatics: The machine learning approach. Cambridge: MIT Press.
[3]
Fedorov, V. V. (1972). Theory of optimal experiments. New York: Academic Press.
[4]
Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153--162.
[5]
Huang, J., Smola, A., Gretton, A., Borgwardt, K. M., & Schöölkopf, B. (2007). Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. Platt and T. Hoffman (Eds.), Advances in neural information processing systems 19. Cambridge, MA: MIT Press.
[6]
Kanamori, T., & Shimodaira, H. (2003). Active learning algorithm using the maximum weighted log-likelihood estimator. Journal of Statistical Planning and Inference, 116, 149--162.
[7]
Lin, Y., Lee, Y., & Wahba, G. (2002). Support vector machines for classification in nonstandard situations. Machine Learning, 46, 191--202.
[8]
Rissanen, J. (1986). Stochastic complexity and modeling. Annals of Statistics, 14, 1080--1100.
[9]
Schölkopf, B., & Smola, A. J. (2002). Learning with kernels. Cambridge, MA: MIT Press.
[10]
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6 (2), 461--464.
[11]
Shelton, C. R. (2001). Importance sampling for reinforcement learning with multiple objectives. PhD thesis. Massachusetts Institute of Technology.
[12]
Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90, 227--244.
[13]
Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, Series B, 36, 111--147.
[14]
Sugiyama, M. (2006). Active learning in approximately linear regression based on conditional expectation of generalization error. Journal of Machine Learning Research, 7, 141--166.
[15]
Sugiyama, M., Krauledat, M., & Müüller, K.-R. (2007). Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8.
[16]
Sugiyama, M., & Müüller, K.-R. (2005). Input-dependent estimation of generalization error under covariate shift. Statistics & Decisions, 23, 249--279.
[17]
Watanabe, S. (1999). Algebraic analysis for singular statistical estimation. Lecture Notes on Computer Science Springer, 1720, 39--50.
[18]
Watanabe, S. (2001a). Algebraic analysis for nonidentifiable learning machines. Neural Computation, 13 (4), 899--933.
[19]
Watanabe, S. (2001b). Algebraic information geometry for learning machines with singularities. Advances in Neural Information Processing Systems, 14, 329--336.
[20]
White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1--25.
[21]
Wiens, D. P. (2000). Robust weights and designs for biased regression models: Least squares and generalized M-estimation. Journal of Statistical Planning and Inference, 83, 395--412.
[22]
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002). Braincomputer interfaces for communication and control. Clinical Neurophysiology, 113, 767--791.

Cited By

View all
  • (2024)Supervised algorithmic fairness in distribution shiftsProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/909(8225-8233)Online publication date: 3-Aug-2024
  • (2024)Single-trial detection and session detection in a rapid serial visual presentation task with covariate, target probability, and concept shifts2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650085(1-7)Online publication date: 30-Jun-2024
  • (2023)Posterior Covariance Information Criterion for Weighted InferenceNeural Computation10.1162/neco_a_0159235:7(1340-1361)Online publication date: 12-Jun-2023
  • Show More Cited By
  1. Asymptotic Bayesian generalization error when training and test distributions are different

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICML '07: Proceedings of the 24th international conference on Machine learning
    June 2007
    1233 pages
    ISBN:9781595937933
    DOI:10.1145/1273496
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    • Machine Learning Journal

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 June 2007

    Permissions

    Request permissions for this article.

    Check for updates

    Qualifiers

    • Article

    Conference

    ICML '07 & ILP '07
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 140 of 548 submissions, 26%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)23
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Supervised algorithmic fairness in distribution shiftsProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/909(8225-8233)Online publication date: 3-Aug-2024
    • (2024)Single-trial detection and session detection in a rapid serial visual presentation task with covariate, target probability, and concept shifts2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650085(1-7)Online publication date: 30-Jun-2024
    • (2023)Posterior Covariance Information Criterion for Weighted InferenceNeural Computation10.1162/neco_a_0159235:7(1340-1361)Online publication date: 12-Jun-2023
    • (2023)Witnessing Erosion of Membership Inference Defenses: Understanding Effects of Data Drift in Membership PrivacyProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3607199.3607224(250-263)Online publication date: 16-Oct-2023
    • (2022)Recent advances in algebraic geometry and Bayesian statisticsInformation Geometry10.1007/s41884-022-00083-97:S1(187-209)Online publication date: 6-Dec-2022
    • (2020)Prognosticating the Shade Change after Softener Application using Artificial Neural NetworksAutex Research Journal10.2478/aut-2020-001921:1(79-84)Online publication date: 6-Jul-2020
    • (2020)Targeting Prospective CustomersManagement Science10.1287/mnsc.2019.330866:6(2495-2522)Online publication date: 1-Jun-2020
    • (2019)Learning under p-tampering poisoning attacksAnnals of Mathematics and Artificial Intelligence10.1007/s10472-019-09675-1Online publication date: 3-Dec-2019
    • (2018)Credit Card Fraud Detection: A Realistic Modeling and a Novel Learning StrategyIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2017.273664329:8(3784-3797)Online publication date: Aug-2018
    • (2015)On semi-supervised linear regression in covariate shift problemsThe Journal of Machine Learning Research10.5555/2789272.291210116:1(3183-3217)Online publication date: 1-Jan-2015
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media