Skip to main content
SpringerLink
Log in

A computational framework for empirical Bayes inference

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

In empirical Bayes inference one is typically interested in sampling from the posterior distribution of a parameter with a hyper-parameter set to its maximum likelihood estimate. This is often problematic particularly when the likelihood function of the hyper-parameter is not available in closed form and the posterior distribution is intractable. Previous works have dealt with this problem using a multi-step approach based on the EM algorithm and Markov Chain Monte Carlo (MCMC). We propose a framework based on recent developments in adaptive MCMC, where this problem is addressed more efficiently using a single Monte Carlo run. We discuss the convergence of the algorithm and its connection with the EM algorithm. We apply our algorithm to the Bayesian Lasso of Park and Casella (J. Am. Stat. Assoc. 103:681–686, 2008) and on the empirical Bayes variable selection of George and Foster (J. Am. Stat. Assoc. 87:731–747, 2000).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Andrieu, C., Moulines, E., Priouret, P.: Stability of stochastic approximation under verifiable conditions. SIAM J. Control Optim. 44, 283–312 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  • Andrieu, C., Thoms, J.: A tutorial on adaptive MCMC. Stat. Comput. 18, 343–373 (2008)

    Article  MathSciNet  Google Scholar 

  • Atchade, Y., Fort, G.: Limit theorems for some adaptive MCMC algorithms with sub-geometric kernels. Bernoulli 16, 116–154 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  • Atchade, Y., Fort, G., Moulines, E., Priouret, P.: Adaptive Markov chain Monte Carlo: Theory and methods. Tech. rep. (2009)

  • Benveniste, A., Métivier, M., Priouret, P.: Adaptive Algorithms and Stochastic Approximations. Applications of Mathematics. Springer, Paris (1990)

    MATH  Google Scholar 

  • Cappe, O.E.M., Ryden, T.: Inference in Hidden Markov Models. Springer Series in Statistics. Springer, New York (2005)

    MATH  Google Scholar 

  • Carlin, B.P., Gelfand, A.E.: Approaches for empirical Bayes confidence intervals. J. Am. Stat. Assoc. 85, 105–114 (1990)

    Article  MATH  MathSciNet  Google Scholar 

  • Carlin, B.P., Louis, T.A.: Empirical Bayes: Past, present and future. J. Am. Stat. Assoc. 95, 1286–1289 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  • Casella, G.: Empirical Bayes Gibbs sampling. Biostatistics 2, 485–500 (2001)

    Article  MATH  Google Scholar 

  • Chen, H., Zhu, Y.-M.: Stochastic approximation procedures with randomly varying truncations. Sci. Sin. 1, 914–926 (1986)

    MathSciNet  Google Scholar 

  • Delyon, B., Lavielle, M., Moulines, E.: Convergence of a stochastic approximation version of the em algorithm. Ann. Stat. 27, 94–128 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  • Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. J. R. Stat. Soc. Ser. B 39, 1–38 (1977) (with discussion)

    MATH  MathSciNet  Google Scholar 

  • George, E.I., Foster, D.P.: Calibration and empirical Bayes variable selection. J. Am. Stat. Assoc. 87, 731–747 (2000)

    MATH  MathSciNet  Google Scholar 

  • Gu, M.G., Kong, F.H.: A stochastic Approximation Algorithm with Markov Chain Monte Carlo method for incomplete data estimation problems. Proc. Natl. Acad. Sci. USA 95, 7270–7274 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  • Kushner, K., Yin, Y.: Stochastic Approximation and Recursive Algorithms and Applications. Springer, New York (2003)

    MATH  Google Scholar 

  • Laird, N.M., Louis, T.A.: Empirical Bayes confidence intervals based on Bootstrap samples. J. Am. Stat. Assoc. 82, 739–750 (1987) (with discussion)

    Article  MATH  MathSciNet  Google Scholar 

  • Lange, K.: A gradient algorithm locally equivalent to the em algorithm. J. R. Stat. Soc. Ser. B 57, 425–437 (1995)

    MATH  Google Scholar 

  • Morris, C.N.: Parametric empirical Bayes inference: Theory and applications. J. Am. Stat. Assoc. 78, 47–65 (1983)

    Article  MATH  Google Scholar 

  • Park, T., Casella, G.: The Bayesian LASSO. J. Am. Stat. Assoc. 103, 681–686 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  • Roberts, G.O., Rosenthal, J.S.: Geometric ergodicity and hybrid Markov chains. Electron. Commun. Probab. 2(2), 13–25 (1997) (electronic)

    MATH  MathSciNet  Google Scholar 

  • Snijders, T.A.B.: Markov chain Monte Carlo estimation of exponential random graph models. J. Soc. Struct. 3, 47–65 (2002) Web journal available from http://www.cmu.edu/joss/content/articles/volindex.html

    Google Scholar 

  • Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996)

    MATH  MathSciNet  Google Scholar 

  • Wei, G.C.G., Tanner, M.A.: A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. J. Am. Stat. Assoc. 85, 699–704 (1990)

    Article  Google Scholar 

  • West, M.: On scale mixtures of normal distributions. Biometrika 74, 446–448 (1987)

    Article  Google Scholar 

  • Younes, L.: Estimation and annealing for Gibbsian fields. Ann. Inst. Henri Poincaré, Probab. Stat. 24, 269–294 (1988)

    MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Statistics, The University of Michigan, Ann Arbor, 48109, MI, USA

    Yves F. Atchadé

Authors
  1. Yves F. Atchadé
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yves F. Atchadé.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Atchadé, Y.F. A computational framework for empirical Bayes inference. Stat Comput 21, 463–473 (2011). https://doi.org/10.1007/s11222-010-9182-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-010-9182-3

Keywords

Search

Navigation