Skip to main content
Springer Nature Link
Log in

Smooth functional tempering for nonlinear differential equation models

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

Differential equations are used in modeling diverse system behaviors in a wide variety of sciences. Methods for estimating the differential equation parameters traditionally depend on the inclusion of initial system states and numerically solving the equations. This paper presents Smooth Functional Tempering, a new population Markov Chain Monte Carlo approach for posterior estimation of parameters. The proposed method borrows insights from parallel tempering and model based smoothing to define a sequence of approximations to the posterior. The tempered approximations depend on relaxations of the solution to the differential equation model, reducing the need for estimating the initial system states and obtaining a numerical differential equation solution. Rather than tempering via approximations to the posterior that are more heavily rooted in the prior, this new method tempers towards data features. Using our proposed approach, we observed faster convergence and robustness to both initial values and prior distributions that do not reflect the features of the data. Two variations of the method are proposed and their performance is examined through simulation studies and a real application to the chemical reaction dynamics of producing nylon.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Atchadé, Y., Liu, J.: The Wang-Landau algorithm in general state spaces: applications and convergence analysis. Stat. Sin. 20, 209–233 (2010)

    MATH  Google Scholar 

  • Barenco, M., Tomescu, D., Brewer, D., Callard, R., Stark, J., Hubank, M.: Ranked prediction of p53 targets using hidden variable dynamic modeling. Genome Biol. 7(3), R25 (2006)

    Article  Google Scholar 

  • Bates, D.M., Watts, D.B.: Nonlinear Regression Analysis and Its Applications. Wiley, New York (1988)

    Book  MATH  Google Scholar 

  • Bois, F.Y.: GNU MCSim: Bayesian statistical inference for SBML-coded systems biology models. Bioinformatics 25(11), 1453–1454 (2009)

    Article  Google Scholar 

  • Brunel, N.J.B.: Parameter estimation of ODE’s via nonparametric estimators. Electron. J. Stat. 2, 1242–1267 (2008)

    Article  MathSciNet  Google Scholar 

  • Calderhead, B., Girolami, M.: Estimating Bayes factors via thermodynamic integration and population MCMC. Comput. Stat. Data Anal. 53(12), 4028–4045 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Calderhead, B., Girolami, M., Lawrence, N.D.: Accelerating Bayesian inference over nonlinear differential equations with Gaussian processes. In: Advances in Neural Information Processing Systems, pp. 217–224 (2009)

    Google Scholar 

  • Chou, I.C., Voit, E.O.: Recent developments in parameter estimation and structure identification of biochemical and genomic systems. Math. Biosci. 219, 57–83 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Deuflhard, P., Bornemann, F.: Scientific Computing with Ordinary Differential Equations. Springer, New York (2000)

    Google Scholar 

  • Eilers, P.: A perfect smoother. Anal. Chem. 75, 3631–3636 (2003)

    Article  Google Scholar 

  • Eilers, P.H.C., Marx, B.D.: Flexible smoothing with B-splines and penalties (with discussion). Stat. Sci. 11, 89–102 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  • Esposito, W.R., Floudas, C.: Deterministic global optimization in nonlinear optimal control problems. J. Glob. Optim. 17, 97–126 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  • FitzHugh, R.: Impulses and physiological states in models of nerve membrane. Biophys. J. 1, 445–466 (1961)

    Article  Google Scholar 

  • Friel, N., Pettitt, A.N.: Marginal likelihood estimation via power posteriors. J. R. Stat. Soc., Ser. B, Stat. Methodol. 70(3), 589–607 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  • Gao, P., Honkela, A., Rattray, M., Lawrence, N.D.: Gaussian process modelling of latent chemical species: applications to inferring transcription factor activities. Bioinformatics 24, 70–75 (2008)

    Article  Google Scholar 

  • Gelman, A., Bois, F.Y., Jiang, J.: Physiological pharmacokinetic analysis using population modeling and informative prior distributions. J. Am. Stat. Assoc. 91, 1400–1412 (1996)

    Article  MATH  Google Scholar 

  • Geweke, J.: Evaluating the accuracy of sampling-based approaches to the calculation of posterior moments. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics. Proceedings of the Fourth Valencia International Meeting, vol. 4, pp. 169–193. Clarendon Press, Oxford (1992)

    Google Scholar 

  • Geyer, C.J.: Markov chain Monte Carlo maximum likelihood. In: Computing Science and Statistics: Proceedings of the 23rd Symposium on the Interface, pp. 156–163 (1991)

    Google Scholar 

  • Geyer, C.J., Thompson, E.A.: Annealing Markov Chain Monte Carlo with applications to ancestral inference. J. Am. Stat. Assoc. 90, 909–920 (1995)

    Article  MATH  Google Scholar 

  • Gonzalez, O., Küper, C., Jung, K., Naval Jr. P., Mendoza, E.: Parameter estimation using simulated annealing for S-system models of biochemical networks. Bioinformatics 23, 480–486 (2007)

    Article  Google Scholar 

  • Gramacy, R., Samworth, R., King, R.: Importance tempering. Stat. Comput. 20, 1–7 (2010)

    Article  MathSciNet  Google Scholar 

  • Gutenkunst, R.N., Casey, F.P., Waterfall, J.J., Myers, C.R., Sethna, J.P.: Extracting falsifiable predictions from sloppy models. Ann. N.Y. Acad. Sci. 1115, 203–211 (2007a)

    Article  Google Scholar 

  • Gutenkunst, R.N., Waterfall, J.J., Casey, F.P., Brown, K.S., Myers, C.R., Sethna, J.P.: Universally sloppy parameter sensitivities in systems biology models. PLoS Comput. Biol. 3, 1871–1878 (2007b)

    Article  MathSciNet  Google Scholar 

  • Huang, Y., Liu, D., Wu, H.: Hierarchical Bayesian methods for estimation of parameters in a longitudinal HIV dynamic system. Biometrics 62, 413–423 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Huang, Y., Wu, H.: A bayesian approach for estimating antiviral efficacy in HIV dynamic models. J. Appl. Stat. 33, 155–174 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Jasra, A., Stephens, D.A., Holmes, C.C.: On population-based simulation for static inference. Stat. Comput. 17, 263–279 (2007)

    Article  MathSciNet  Google Scholar 

  • Kass, R.E., Raftery, A.: Bayes factors. J. Am. Stat. Assoc. 90(430), 773–795 (1995)

    Article  MATH  Google Scholar 

  • Klinke, D.J.: An empirical Bayesian approach for model-based inference of cellular signaling networks. BMC Bioinform. 10, 371 (2009)

    Article  Google Scholar 

  • Le Novère, N., Bornstein, B., Broicher, A., Courtot, M., Donizelli, M., Dharuri, H., Li, L., Sauro, H., Schilstra, M., Shapiro, B., Snoep, J., Hucka, M.: BioModels database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res. 34(Suppl 1), D689–D691 (2006)

    Article  Google Scholar 

  • Li, L., Brown, M.B., Lee, K.H., Gupta, S.: Estimation and inference for a spline-enhanced population pharmacokinetic model. Biometrics 58, 601–611 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  • Liang, F., Wong, W.: Evolutionary Monte Carlo sampling: applications to Cp model sampling and change-point problem. Stat. Sin. 10, 317–342 (2000)

    MATH  Google Scholar 

  • Liang, F., Wong, W.H.: Real-parameter evolutionary Monte Carlo with applications to Bayesian mixture models. J. Am. Stat. Assoc. 96, 653–666 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  • Liang, H., Miao, H., Wu, H.: Estimation of constant and time-varying dynamic parameters of HIV infection in a nonlinear differential equation model. Ann. Appl. Stat. 4, 460–483 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  • Liang, H., Wu, H.: Parameter estimation for differential equation models using a framework of measurement error in regression models. J. Am. Stat. Assoc. 103, 1570–1583 (2008)

    Article  MathSciNet  Google Scholar 

  • Liu, Jun S.: Monte Carlo strategies in Scientific Computing. Springer, New York (2001)

    MATH  Google Scholar 

  • Marinari, E., Parisi, G.: Simulated tempering: a new Monte Carlo scheme. Europhys. Lett. 19, 451–458 (1992)

    Article  Google Scholar 

  • Marlin, T.E.: Process Control. McGraw-Hill, New York (2000)

    Google Scholar 

  • The MathWorks: Matlab ®7 Mathematics. The Mathworks, Inc. Natick, MA (2010)

  • Miao, H., Dykes, C., Demeter, L.M., Wu, H., Avenue, E., York, N., York, N.: Differential equation modeling of HIV viral fitness experiments: model identification, model selection, and multimodel inference. Biometrics 65, 292–300 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Nagumo, J.S., Arimoto, S., Yoshizawa, S.: An active pulse transmission line simulating a nerve axon. Proc. Inst. Radio Eng. 50, 2061–2070 (1962)

    Google Scholar 

  • Neal, R.M.: Sampling from multimodal distributions using tempered transitions. Stat. Comput. 4, 353–366 (1996)

    Article  Google Scholar 

  • Olhede, S.: Discussion on the paper by Ramsay, Hooker, Campbell and Cao. J. R. Stat. Soc. B 69, 772–779 (2008)

    Google Scholar 

  • Poyton, A., Varziri, M., McAuley, K., McLellan, P., Ramsay, J.: Parameter estimation in continuous-time dynamic models using principal differential analysis. Comput. Chem. Eng. 30, 698–708 (2006)

    Article  Google Scholar 

  • Qi, X., Zhao, H.: Asymptotic efficiency and finite-sample properties of the generalized profiling estimation of parameters in ordinary differential equations. Ann. Stat. 38(1), 435–481 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  • Raftery, A., Lewis, S.: How many iterations in the Gibbs sampler. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics. Proceedings of the Fourth Valencia International Meeting, vol. 4, pp. 763–773. Clarendon Press, Oxford (1992)

    Google Scholar 

  • Ramsay, J.O., Hooker, G., Campbell, D., Cao, J.: Parameter estimation for differential equations: a generalized smoothing approach (with discussion). J. R. Stat. Soc. B 69, 741–796 (2007)

    Article  MathSciNet  Google Scholar 

  • Ramsay, J.O., Silverman, B.W.: Functional Data Analysis. Springer, New York (2005)

    Google Scholar 

  • Raue, A., Kreutz, C., Maiwald, T., Bachmann, J., Schilling, M., Klingmüller, U., Timmer, J.: Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics 25, 1923–1929 (2009)

    Article  Google Scholar 

  • Rodriguez-Fernandez, M., Mendes, P., Banga, J.R.: A hybrid approach for efficient and robust parameter estimation in biochemical pathways. Biosystems 83, 248–65 (2006)

    Article  Google Scholar 

  • Rogers, S., Khanin, R., Girolami, M.: Bayesian model-based inference of transcription factor activity. BMC Bioinform. 8, S2 (2007)

    Article  Google Scholar 

  • Salway, R., Wakefield, J.: Gamma generalized linear models for pharmacokinetic data. Biometrics 64, 620–626 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  • Varah, J.: A spline least squares method for numerical parameter estimation in differential equations. SIAM J. Sci. Stat. Comput. 3, 28–46 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  • Vilela, M., Borges, C.C.H., Vinga, S., Vasconcelos, A.T.R., Santos, H., Voit, E.O., Almeida, J.S.: Automated smoother for the numerical decoupling of dynamics models. BMC Bioinform. 8, 305 (2007)

    Article  Google Scholar 

  • Voit, E.O., Almeida, J.: Decoupling dynamical systems for pathway identification from metabolic profiles. Bioinformatics 20, 1670–1681 (2004)

    Article  Google Scholar 

  • Voit, E.O., Sauvegeau, M.: Power-law approach to modeling biological systems; III. Methods of analysis. J. Ferment. Technol. 60, 233–241 (1982)

    Google Scholar 

  • Walley, P., Moral, S.: Upper probabilities based only on the likelihood function. J. R. Stat. Soc. B 61, 831–847 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  • Wakefield, J.: The Bayesian analysis of population pharmacokinetic models. J. Am. Stat. Assoc. 91, 62–75 (1996)

    Article  MATH  Google Scholar 

  • Wakefield, J., Bennett, J.: The Bayesian modeling of covariates for population pharmacokinetic models. J. Am. Stat. Assoc. 91, 917–927 (1996)

    Article  MATH  Google Scholar 

  • Wu, H., Zhu, H., Miao, H., Perelson, A.S.: Parameter identifiability and estimation of HIV/AIDS dynamic models. Bull. Math. Biol. 70, 785–799 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  • Zheng, W., McAuley, K.B., Marchildon, E.K., Zhen Yao, K.: Effects of end-group balance on melt-phase nylon 612 polycondensation: experimental study and mathematical model. Ind. Eng. Chem. Res. 44, 2675–2686 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Statistics and Actuarial Science, Simon Fraser University, 13450 102nd avenue, Surrey, BC, Canada, V3T 0A3

    David Campbell

  2. Department of Mathematics and Statistics, McGill University, 805 Sherbrooke Ouest, Montreal, QC, Canada, H3A 2K6

    Russell J. Steele

Authors
  1. David Campbell
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Russell J. Steele
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Russell J. Steele.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Campbell, D., Steele, R.J. Smooth functional tempering for nonlinear differential equation models. Stat Comput 22, 429–443 (2012). https://doi.org/10.1007/s11222-011-9234-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-011-9234-3

Keywords