research-article

Optimal parameter trajectory estimation in parameterized SDEs: An algorithmic procedure

Authors:

Shalabh Bhatnagar,

Vivek Kumar MishraAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 19, Issue 2

Article No.: 8, Pages 1 - 27

https://doi.org/10.1145/1502787.1502791

Published: 23 March 2009 Publication History

Abstract

We consider the problem of estimating the optimal parameter trajectory over a finite time interval in a parameterized stochastic differential equation (SDE), and propose a simulation-based algorithm for this purpose. Towards this end, we consider a discretization of the SDE over finite time instants and reformulate the problem as one of finding an optimal parameter at each of these instants. A stochastic approximation algorithm based on the smoothed functional technique is adapted to this setting for finding the optimal parameter trajectory. A proof of convergence of the algorithm is presented and results of numerical experiments over two different settings are shown. The algorithm is seen to exhibit good performance. We also present extensions of our framework to the case of finding optimal parameterized feedback policies for controlled SDE and present numerical results in this scenario as well.

References

[1]

Abdulla, M. S. and Bhatnagar, S. 2007. Reinforcement learning based algorithms for average cost Markov decision processes. Discrete Event Dynam. Syst. 17, 1, 23--52.

Digital Library

[2]

Bertsekas, D. P. 1995. Dynamic Programming and Optimal Control. Athena Scientific, Belmont, MA.

Digital Library

[3]

Bertsekas, D. P. and Gallager, R. G. 1991. Data Networks. Prentice-Hall, New York.

Digital Library

[4]

Bertsekas, D. P. and Tsitsiklis, J. N. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.

Digital Library

[5]

Bhatnagar, S. 2005. Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization. ACM Trans. Modeling Comput. Simul. 15, 1, 74--107.

Digital Library

[6]

Bhatnagar, S. 2007. Adaptive Newton-based smoothed functional algorithms for simulation optimization. ACM Trans. Model. Comput. Simul. 18, 1, 1--35.

Digital Library

[7]

Bhatnagar, S. and Borkar, V. S. 1998. A two time scale stochastic approximation scheme for simulation based parametric optimization. Prob. Eng. Inf. Sci. 12, 519--531.

[8]

Bhatnagar, S. and Borkar, V. S. 2003. Multiscale chaotic SPSA and smoothed functional algorithms for simulation optimization. Simul. 79, 10, 568--580.

[9]

Bhatnagar, S., Fu, M. C., Marcus, S. I., and Bhatnagar, S. 2001. Two timescale algorithms for simulation optimization of hidden Markov models. IIE Trans. 33, 3, 245--258.

[10]

Bhatnagar, S. and Karmeshu. 2007. Monte-Carlo estimation of time-dependent statistical characteristics of a process governed by a random differential equation. Submitted.

[11]

Bhatnagar, S. and Kumar, S. 2004. A simultaneous perturbation stochastic approximation based actor-critic algorithm for Markov decision processes. IEEE Trans. Autom. Control 49, 4, 592--598.

[12]

Campillo, F. and Traore, A. 1994. Lyapunov exponents of controlled SDEs and stabilizability property: Some examples. Rapport de Recherche 2397, INRIA.

[13]

Campillo, F. and Traore, A. 1995. A stabilization algorithm for linear controlled SDEs. In Proceedings of IEEE Conference on Decision and Control, 1034--1035.

[14]

Charalambos, C. D., Djouadi, S. M., and Denic, S. Z. 2005. Stochastic power control for wireless networks via SDEs: Probabilistic qos measures. IEEE Trans. Inf. Theory 51, 12, 4396--4401.

Digital Library

[15]

Glasserman, P. 2005. Monte Carlo Methods in Financial Engineering. Springer, New York.

[16]

Glynn, P. W. 1990. Likelihood ratio gradient estimation for stochastic systems. Commun. ACM 33, 10, 75--84.

Digital Library

[17]

Hirsch, M. W. 1989. Convergent activation dynamics in continuous time networks. Neural Netw. 2, 331--349.

Digital Library

[18]

Ho, Y. C. and Cao, X. R. 1991. Perturbation Analysis of Discrete Event Dynamical Systems. Kluwer, Boston.

[19]

Konda, V. R. and Tsitsiklis, J. N. 2003. Actor-Critic algorithms. SIAM J. Control Optimiz. 42, 4, 1143--1166.

Digital Library

[20]

Korn, R. and Kraft, H. 2002. A stochastic control approach to portfolio problems with stochastic interest rates. SIAM J. Control Optimiz. 40, 4, 1250--1269.

Digital Library

[21]

Kushner, H. J. and Clark, D. S. 1978. Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer, New York.

[22]

Kushner, H. J. and Dupuis, P. G. 2001. Numerical Methods for Stochastic Control Problems in Continuous Time. Springer, New York.

Digital Library

[23]

Lim, A. E. B., Zhou, X. Y., and Moore, J. B. 2003. Multiple-Objective risk-sensitive control and its small noise limit. Automatica 39, 533--541.

Digital Library

[24]

Liu, T., Bahl, P., and Chlamtac, I. 1998. Mobility modeling, location tracking, and trajectory prediction in wireless atm networks. IEEE J. Selected Areas Commun. 16, 6, 922--936.

Digital Library

[25]

Marbach, P. and Tsitsiklis, J. N. 2001. Simulation-based optimization of Markov reward processes. IEEE Trans. Autom. Control 46, 2, 191--209.

[26]

Moose, R. L., Vanlandingham, H. F., and McCabe, D. H. 1979. Modeling and estimation for tracking maneuvering targets. IEEE Trans. Aerospace Electron. Syst. AES-15, 3, 448--456.

[27]

Nelson, R. 1987. Stochastic catastrophe theory in computer performance modeling. J. Assoc. Comput. Mach. 34, 3, 661--685.

Digital Library

[28]

Primak, S., Kontorovich, V., and Lyandres, V. 2004. Stochastic Methods and Their Applications to Communications: Stochastic Differential Equations Approach. Wiley, West Sussex, UK.

[29]

Rubinstein, R. Y. 1981. Simulation and the Monte Carlo Method. Wiley, New York.

Digital Library

[30]

Singer, R. A. 1970. Estimating optical tracking filter performance for manned maneuvering targets. IEEE Trans. Aerospace Electron. Syst. AES-6, 4, 473--483.

[31]

Spall, J. C. 1992. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans. Autom. Control. 37, 3, 332--341.

[32]

Styblinski, M. A. and Tang, T.-S. 1990. Experiments in nonconvex optimization: Stochastic approximation with function smoothing and simulated annealing. Neural Netw. 3, 467--483.

Digital Library

[33]

Vazquez-Abad, F. J. and Kushner, H. J. 1992. Estimation of the derivative of a stationary measure with respect to a control parameter. J. Appl. Probability 29, 343--352.

Cited By

Chakravarty SPadakandla SBhatnagar S(2013)A simulation‐based algorithm for optimal pricing policy under demand uncertaintyInternational Transactions in Operational Research10.1111/itor.1206421:5(737-760)Online publication date: 30-Dec-2013
https://doi.org/10.1111/itor.12064
Bhatnagar SPrasad HPrashanth LBhatnagar SPrasad HPrashanth L(2013)Communication NetworksStochastic Recursive Algorithms for Optimization10.1007/978-1-4471-4285-0_14(257-280)Online publication date: 2013
https://doi.org/10.1007/978-1-4471-4285-0_14
Bhatnagar SPrasad HPrashanth LBhatnagar SPrasad HPrashanth L(2013)IntroductionStochastic Recursive Algorithms for Optimization10.1007/978-1-4471-4285-0_1(3-12)Online publication date: 2013
https://doi.org/10.1007/978-1-4471-4285-0_1
Show More Cited By

Index Terms

Optimal parameter trajectory estimation in parameterized SDEs: An algorithmic procedure
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation theory
2. Mathematics of computing
  1. Probability and statistics
    1. Probabilistic algorithms
    2. Probabilistic reasoning algorithms
      1. Markov-chain Monte Carlo methods
      2. Sequential Monte Carlo methods

Recommendations

Optimal Stochastic Parameter Design for Estimation Problems

In this study, the aim is to perform optimal stochastic parameter design in order to minimize the cost of a given estimator. Optimal probability distributions of signals corresponding to different parameters are obtained in the presence and absence of ...
Infinite Horizon Forward-Backward SDEs and Open-Loop Optimal Controls for Stochastic Linear-Quadratic Problems with Random Coefficients

In this paper, we introduce a new infinite horizon domination-monotonicity framework. In this framework, by the method of continuation and some subtle techniques, we obtain an existence and uniqueness result and a pair of estimates for the solutions to a ...
Optimal pointwise approximation of SDEs from inexact information

We study a pointwise approximation of solutions of systems of stochastic differential equations. We assume that an approximation method can use values of the drift and diffusion coefficients which are perturbed by some deterministic noise. Let 1,20 be ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 19, Issue 2

March 2009

142 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/1502787

Issue’s Table of Contents

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 March 2009

Accepted: 01 June 2008

Revised: 01 January 2008

Received: 01 May 2007

Published in TOMACS Volume 19, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
379
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chakravarty SPadakandla SBhatnagar S(2013)A simulation‐based algorithm for optimal pricing policy under demand uncertaintyInternational Transactions in Operational Research10.1111/itor.1206421:5(737-760)Online publication date: 30-Dec-2013
https://doi.org/10.1111/itor.12064
Bhatnagar SPrasad HPrashanth LBhatnagar SPrasad HPrashanth L(2013)Communication NetworksStochastic Recursive Algorithms for Optimization10.1007/978-1-4471-4285-0_14(257-280)Online publication date: 2013
https://doi.org/10.1007/978-1-4471-4285-0_14
Bhatnagar SPrasad HPrashanth LBhatnagar SPrasad HPrashanth L(2013)IntroductionStochastic Recursive Algorithms for Optimization10.1007/978-1-4471-4285-0_1(3-12)Online publication date: 2013
https://doi.org/10.1007/978-1-4471-4285-0_1
Karmeshu Bhatnagar SMishra V(2011)An Optimized SDE Model for Slotted AlohaIEEE Transactions on Communications10.1109/TCOMM.2011.041111.09011359:6(1502-1508)Online publication date: Jun-2011
https://doi.org/10.1109/TCOMM.2011.041111.090113
Bhatnagar SKarmeshu (2011)Monte-Carlo estimation of time-dependent statistical characteristics of random dynamical systemsApplied Mathematical Modelling10.1016/j.apm.2010.12.02435:6(3063-3079)Online publication date: Jun-2011
https://doi.org/10.1016/j.apm.2010.12.024
Bhatnagar SCochran JCox LKeskinocak PKharoufeh JSmith J(2011)Simultaneous Perturbation and Finite Difference MethodsWiley Encyclopedia of Operations Research and Management Science10.1002/9780470400531.eorms0784Online publication date: 15-Feb-2011
https://doi.org/10.1002/9780470400531.eorms0784

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents