article

Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization

Author:

Shalabh BhatnagarAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 15, Issue 1

Pages 74 - 107

https://doi.org/10.1145/1044322.1044326

Published: 01 January 2005 Publication History

Abstract

We develop in this article, four adaptive three-timescale stochastic approximation algorithms for simulation optimization that estimate both the gradient and Hessian of average cost at each update epoch. These algorithms use four, three, two, and one simulation(s), respectively, and update the values of the decision variable and Hessian matrix components simultaneously, with estimates based on the simultaneous perturbation methodology. Our algorithms use coupled stochastic recursions that proceed using three different timescales or step-size schedules. We present a detailed convergence analysis of the algorithms and show numerical experiments using all the developed algorithms on a two-node network of M/G/1 queues with feedback for a 50-dimensional parameter vector. We provide comparisons of the performance of these algorithms with two recently developed two-timescale steepest descent simultaneous perturbation analogs that use randomized and deterministic perturbation sequences, respectively. We also present experiments to explore the sensitivity of the algorithms to their associated parameters. The algorithms that use four and three simulations, respectively, perform significantly better than the rest of the algorithms.

References

[1]

Andradóttir, S. 1996. Optimization of the transient and steady-state behavior of discrete event systems. Manag. Sci. 42, 5, 717--737.

[2]

Bertsekas, D. P. 1999. Nonlinear Programming. Athena Scientific, Belmont.

[3]

Bertsekas, D. P. and Tsitsiklis, J. N. 1989. Parallel and Distributed Computation. Prentice Hall, New Jersey.

[4]

Bhatnagar, S. 1997. Multiscale Stochastic Approximation Algorithms with Applications to ABR Service in ATM Networks. Ph. D. thesis, Department of Electrical Engineering, Indian Institute of Science, Bangalore, India.

[5]

Bhatnagar, S. and Borkar, V. S. 1997. Multiscale stochastic approximation for parametric optimization of hidden Markov models. Prob. Eng. and Info. Sci. 11, 509--522.

[6]

Bhatnagar, S. and Borkar, V. S. 1998. A two time scale stochastic approximation scheme for simulation based parametric optimization. Prob. Eng. and Info. Sci. 12, 519--531.

[7]

Bhatnagar, S. and Borkar, V. S. 2003. Multiscale chaotic SPSA and smoothed functional algorithms for simulation optimization. Simulation 79, 10, 568--580.

[8]

Bhatnagar, S., Fu, M. C., Marcus, S. I., and Bhatnagar, S. 2001a. Two timescale algorithms for simulation optimization of hidden Markov models. IIE Trans. 33, 3, 245--258.

[9]

Bhatnagar, S., Fu, M. C., Marcus, S. I., and Fard, P. J. 2001b. Optimal structured feedback policies for ABR flow control using two-timescale SPSA. IEEE/ACM Trans. Network. 9, 4, 479--491.

[10]

Bhatnagar, S., Fu, M. C., Marcus, S. I., and Wang, I.-J. 2003. Two-timescale simultaneous perturbation stochastic approximation using deterministic perturbation sequences. ACM Trans. Modell. Comput. Simul. 13, 2, 180--209.

[11]

Brandiere, O. 1998. Some pathological traps for stochastic approximation. SIAM J. Contr. Optim. 36, 1293--1314.

[12]

Chen, H. F., Duncan, T. E., and Pasik-Duncan, B. 1999. A Kiefer-Wolfowitz algorithm with randomized differences. IEEE Trans. Auto. Cont. 44, 3, 442--453.

[13]

Chong, E. K. P. and Ramadge, P. J. 1993. Optimization of queues using an infinitesimal perturbation analysis-based stochastic algorithm with general update times. SIAM J. Cont. Optim. 31, 3, 698--732.

[14]

Chong, E. K. P. and Ramadge, P. J. 1994. Stochastic optimization of regenerative systems using infinitesimal perturbation analysis. IEEE Trans. Auto. Cont. 39, 7, 1400--1410.

[15]

Dippon, J. and Renz, J. 1997. Weighted means in stochastic approximation of minima. SIAM J. Contr. Optim. 35, 1811--1827.

[16]

Fabian, V. 1971. Stochastic approximation. In Optimizing Methods in Statistics J. J. Rustagi, Ed. Academic Press, New York, NY, 439--470.

[17]

Fu, M. C. 1990. Convergence of a stochastic approximation algorithm for the GI/G/1 queue using infinitesimal perturbation analysis. J. Optim. Theo. Appl. 65, 149--160.

[18]

Fu, M. C. and Hill, S. D. 1997. Optimization of discrete event systems via simultaneous perturbation stochastic approximation. IIE Trans. 29, 3, 233--243.

[19]

Gelfand, S. B. and Mitter, S. K. 1991. Recursive stochastic algorithms for global optimization in Rd*. SIAM J. Cont. Optim. 29, 5, 999--1018.

[20]

Hirsch, M. W. 1989. Convergent activation dynamics in continuous time networks. Neural Networks 2, 331--349.

[21]

Ho, Y. C. and Cao, X. R. 1991. Perturbation Analysis of Discrete Event Dynamical Systems. Kluwer, Boston, MA.

[22]

Kiefer, E. and Wolfowitz, J. 1952. Stochastic estimation of the maximum of a regression function. Ann. Math. Statist. 23, 462--466.

[23]

Kleinman, N. L., Spall, J. C., and Naiman, D. Q. 1999. Simulation-based optimization with stochastic approximation using common random numbers. Manag. Sci. 45, 1570--1578.

[24]

Kushner, H. J. and Clark, D. S. 1978. Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer Verlag, New York, NY.

[25]

Kushner, H. J. and Yin, G. G. 1997. Stochastic Approximation Algorithms and Applications. Springer Verlag, New York, NY.

[26]

Lasalle, J. P. and Lefschetz, S. 1961. Stability by Liapunov's Direct Method with Applications. Academic Press, New York, NY.

[27]

L'Ecuyer, P. and Glynn, P. W. 1994. Stochastic optimization by simulation: convergence proofs for the GI/G/1 queue in steady-state. Manag. Sci. 40, 11, 1562--1578.

[28]

Luman, R. R. 2000. Upgrading complex systems of systems: a CAIV methodology for warfare area requirements allocation. Military Operations Research 5, 2, 53--75.

[29]

Pemantle, R. 1990. Nonconvergence to unstable points in urn models and stochastic approximations. Annals of Prob. 18, 698--712.

[30]

Polyak, B. T. and Juditsky, A. B. 1992. Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30, 4, 838--855.

[31]

Robbins, H. and Monro, S. 1951. A stochastic approximation method. Ann. Math. Statist. 22, 400--407.

[32]

Ruppert, D. 1985. A Newton-Raphson version of the multivariate Robbins-Monro procedure. Annals Statist. 13, 236--245.

[33]

Spall, J. C. 1992. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans. Auto. Cont. 37, 3, 332--341.

[34]

Spall, J. C. 1997. A one-measurement form of simultaneous perturbation stochastic approximation. Automatica 33, 109--112.

[35]

Spall, J. C. 2000. Adaptive stochastic approximation by the simultaneous perturbation method. IEEE Trans. Autom. Contr. 45, 1839--1853.

[36]

Zhu, X. and Spall, J. C. 2002. A modified second-order SPSA optimization algorithm for finite samples. Int. J. Adapt. Contr. Sign. Proce. 16, 397--409.

Cited By

Hu JSong MFu M(2024)Quantile Optimization via Multiple-Timescale Local Search for Black-Box FunctionsOperations Research10.1287/opre.2022.0534Online publication date: 12-Mar-2024
https://doi.org/10.1287/opre.2022.0534
Cao HHu JLian T(2024)Blackbox Simulation OptimizationJournal of the Operations Research Society of China10.1007/s40305-024-00549-wOnline publication date: 30-Jul-2024
https://doi.org/10.1007/s40305-024-00549-w
Hu JPeng YZhang GZhang Q(2022)A Stochastic Approximation Method for Simulation-Based Quantile OptimizationINFORMS Journal on Computing10.1287/ijoc.2022.121434:6(2889-2907)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1287/ijoc.2022.1214
Show More Cited By

Index Terms

Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation theory
2. Mathematics of computing
  1. Probability and statistics

Recommendations

Adaptive Newton-based multivariate smoothed functional algorithms for simulation optimization

In this article, we present three smoothed functional (SF) algorithms for simulation optimization. While one of these estimates only the gradient by using a finite difference approximation with two parallel simulations, the other two are adaptive Newton-...
Stochastic approximation algorithms for constrained optimization via simulation

We develop four algorithms for simulation-based optimization under multiple inequality constraints. Both the cost and the constraint functions are considered to be long-run averages of certain state-dependent single-stage functions. We pose the problem ...
Two-timescale simultaneous perturbation stochastic approximation using deterministic perturbation sequences

Simultaneous perturbation stochastic approximation (SPSA) algorithms have been found to be very effective for high-dimensional simulation optimization problems. The main idea is to estimate the gradient using simulation output performance measures at ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 15, Issue 1

January 2005

107 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/1044322

Issue’s Table of Contents

Copyright © 2005 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2005

Published in TOMACS Volume 15, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
781
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)4

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hu JSong MFu M(2024)Quantile Optimization via Multiple-Timescale Local Search for Black-Box FunctionsOperations Research10.1287/opre.2022.0534Online publication date: 12-Mar-2024
https://doi.org/10.1287/opre.2022.0534
Cao HHu JLian T(2024)Blackbox Simulation OptimizationJournal of the Operations Research Society of China10.1007/s40305-024-00549-wOnline publication date: 30-Jul-2024
https://doi.org/10.1007/s40305-024-00549-w
Hu JPeng YZhang GZhang Q(2022)A Stochastic Approximation Method for Simulation-Based Quantile OptimizationINFORMS Journal on Computing10.1287/ijoc.2022.121434:6(2889-2907)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1287/ijoc.2022.1214
Mok RAhmad M(2022)Smoothed Functional Algorithm with Norm-limited Update Vector for Identification of Continuous-time Fractional-order Hammerstein ModelsIETE Journal of Research10.1080/03772063.2022.215287970:2(1814-1832)Online publication date: 28-Dec-2022
https://doi.org/10.1080/03772063.2022.2152879
Mok RAhmad M(2022)Power Production Optimization of Model-Free Wind Farm Using Smoothed Functional AlgorithmProceedings of the 6th International Conference on Electrical, Control and Computer Engineering10.1007/978-981-16-8690-0_60(679-689)Online publication date: 9-Mar-2022
https://doi.org/10.1007/978-981-16-8690-0_60
Choudhary K(2021)Quantum computation for predicting electron and phonon properties of solidsJournal of Physics: Condensed Matter10.1088/1361-648X/ac115433:38(385501)Online publication date: 20-Jul-2021
https://doi.org/10.1088/1361-648X/ac1154
L. A. PBhatnagar SBhavsar NFu MMarcus S(2020)Random Directions Stochastic Approximation With Deterministic PerturbationsIEEE Transactions on Automatic Control10.1109/TAC.2019.293082165:6(2450-2465)Online publication date: Jun-2020
https://doi.org/10.1109/TAC.2019.2930821
Bhatnagar SPatel SKarmeshu (2018)A stochastic approximation approach to active queue managementTelecommunications Systems10.1007/s11235-017-0377-168:1(89-104)Online publication date: 1-May-2018
https://dl.acm.org/doi/10.1007/s11235-017-0377-1
L. A. PBhatnagar SFu MMarcus S(2017)Adaptive System Optimization Using Random Directions Stochastic ApproximationIEEE Transactions on Automatic Control10.1109/TAC.2016.260064362:5(2223-2238)Online publication date: May-2017
https://doi.org/10.1109/TAC.2016.2600643
Lakshmanan KBhatnagar S(2017)Quasi-Newton smoothed functional algorithms for unconstrained and constrained simulation optimizationComputational Optimization and Applications10.1007/s10589-016-9875-466:3(533-556)Online publication date: 1-Apr-2017
https://dl.acm.org/doi/10.1007/s10589-016-9875-4
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents