research-article

Stochastic approximation algorithms for constrained optimization via simulation

Authors:

Shalabh Bhatnagar,

N. Hemachandra,

Vivek Kumar MishraAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 21, Issue 3

Article No.: 15, Pages 1 - 22

https://doi.org/10.1145/1921598.1921599

Published: 04 February 2011 Publication History

Abstract

We develop four algorithms for simulation-based optimization under multiple inequality constraints. Both the cost and the constraint functions are considered to be long-run averages of certain state-dependent single-stage functions. We pose the problem in the simulation optimization framework by using the Lagrange multiplier method. Two of our algorithms estimate only the gradient of the Lagrangian, while the other two estimate both the gradient and the Hessian of it. In the process, we also develop various new estimators for the gradient and Hessian. All our algorithms use two simulations each. Two of these algorithms are based on the smoothed functional (SF) technique, while the other two are based on the simultaneous perturbation stochastic approximation (SPSA) method. We prove the convergence of our algorithms and show numerical experiments on a setting involving an open Jackson network. The Newton-based SF algorithm is seen to show the best overall performance.

References

[1]

Bertsekas, D. P. and Tsitsiklis, J. N. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.

Digital Library

[2]

Bhatnagar, S. 2005. Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization. ACM Trans. Model. Comput. Simul. 15, 1, 74--107.

Digital Library

[3]

Bhatnagar, S. 2007. Adaptive Newton-based smoothed functional algorithms for simulation optimization. ACM Trans. Model. Comput. Simul. 18, 1, 27--62.

Digital Library

[4]

Bhatnagar, S. 2010. An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes. Syst. Control Lett. 59, 760--766.

[5]

Bhatnagar, S. and Borkar, V. S. 1998. A two time scale stochastic approximation scheme for simulation based parametric optimization. Prob. Eng. Inf. Sci. 12, 519--531.

[6]

Bhatnagar, S. and Borkar, V. S. 2003. Multiscale chaotic SPSA and smoothed functional algorithms for simulation optimization. Simulation 79, 10, 568--580.

[7]

Bhatnagar, S., Fu, M. C., Marcus, S. I., and Bhatnagar, S. 2001. Two timescale algorithms for simulation optimization of hidden Markov models. IIE Trans. 33, 3, 245--258.

[8]

Borkar, V. S. 1997. Stochastic approximation with two timescales. Syst. Control Lett. 29, 291--294.

Digital Library

[9]

Borkar, V. S. 2005. An actor-critic algorithm for constrained Markov decision processes. Syst. Control Lett. 54, 207--213.

[10]

Brandiere, O. 1998. Some pathological traps for stochastic approximation. SIAM J. Control Optim. 36, 1293--1314.

Digital Library

[11]

Cassandras, C. G. 1993. Discrete Event Systems: Modeling and Performance Analysis. Aksen Associates, Boston, MA.

[12]

Chen, H. F., Duncan, T. E., and Pasik-Duncan, B. 1999. A Kiefer-Wolfowitz algorithm with randomized differences. IEEE Trans. Auto. Cont. 44, 3, 442--453.

[13]

Hirsch, M. W. 1989. Convergent activation dynamics in continuous time networks. Neural Netw. 2, 331--349.

Digital Library

[14]

Ho, Y. C. and Cao, X. R. 1991. Perturbation Analysis of Discrete Event Dynamical Systems. Kluwer, Boston.

[15]

Katkovnik, V. Y. and Kulchitsky, Y. 1972. Convergence of a class of random search algorithms. Autom. Remote Control 8, 1321--1326.

[16]

Kushner, H. J. and Yin, G. G. 1997. Stochastic Approximation Algorithms and Applications. Springer, Berlin.

[17]

Mas-Colell, A., Whinston, M. D., and Green, J. R. 1995. Microeconomic Theory. Oxford University Press, Oxford, UK.

[18]

Meyn, S. P. and Tweedie, R. L. 1993. Markov Chains and Stochastic Stability. (2nd ed.), Springer, Berlin.

[19]

Sadegh, P. 1997. Constrained optimization via stochastic approximation with a simultaneous perturbation gradient approximation. Automatica 33, 889--892.

Digital Library

[20]

Spall, J. C. 1992. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans. Autom. Control 37, 3, 332--341.

[21]

Spall, J. C. 2000. Adaptive stochastic approximation by the simultaneous perturbation method. IEEE Trans. Autom. Control 45, 1839--1853.

[22]

Styblinski, M. A. and Tang, T.-S. 1990. Experiments in nonconvex optimization: Stochastic approximation with function smoothing and simulated annealing. Neural Netw. 3, 467--483.

Digital Library

[23]

Tadic, V. B., Doucet, A., and Singh, S. 2003. Two time-scale stochastic approximation for constrained stochastic optimization and constrained Markov decision problems. In Proceedings of the American Control Conference, 4736--4741.

[24]

Wang, I.-J. and Spall, J. C. 2008. Stochastic optimization with inequality constraints using simultaneous perturbations and penalty functions. Int. J. Control 81, 8, 1232--1238.

[25]

Whitney, J. E. and Hill, S. D. 2001. Constrained optimization over discrete sets via SPSA with application to non-separable resource allocation. In Proceedings of the Winter Simulation Conference, 313--317.

Digital Library

Cited By

Horng SLin S(2020)Coupling Elephant Herding with Ordinal Optimization for Solving the Stochastic Inequality Constrained Optimization ProblemsApplied Sciences10.3390/app1006207510:6(2075)Online publication date: 19-Mar-2020
https://doi.org/10.3390/app10062075
Ciupageanu DBarelli LLazaroiu G(2020)Real-time stochastic power management strategies in hybrid renewable energy systems: A review of key applications and perspectivesElectric Power Systems Research10.1016/j.epsr.2020.106497187(106497)Online publication date: Oct-2020
https://doi.org/10.1016/j.epsr.2020.106497
Barelli LBidini GCiupăgeanu DPianese CPolverino PSorrentino M(2020)Stochastic power management approach for a hybrid solid oxide fuel cell/battery auxiliary power unit for heavy duty vehicle applicationsEnergy Conversion and Management10.1016/j.enconman.2020.113197221(113197)Online publication date: Oct-2020
https://doi.org/10.1016/j.enconman.2020.113197
Show More Cited By

Index Terms

Stochastic approximation algorithms for constrained optimization via simulation
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation theory
2. Mathematics of computing
  1. Probability and statistics

Recommendations

Rate of Convergence for Constrained Stochastic Approximation Algorithms

There is a large literature on the rate of convergence problem for general unconstrained stochastic approximations. Typically, one centers the iterate $\theta_n$ about the limit point $\bar\theta$ and then normalizes by dividing by the square root of ...
A strongly sub-feasible primal-dual quasi interior-point algorithm for nonlinear inequality constrained optimization

In this paper, a primal-dual quasi interior-point algorithm for inequality constrained optimization problems is presented. At each iteration, the algorithm solves only two or three reduced systems of linear equations with the same coefficient matrix. The ...
A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

In this work, we present a globalized stochastic semismooth Newton method for solving stochastic optimization problems involving smooth nonconvex and nonsmooth convex terms in the objective function. We assume that only noisy gradient and Hessian ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 21, Issue 3

March 2011

141 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/1921598

Issue’s Table of Contents

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 February 2011

Accepted: 01 August 2010

Revised: 01 August 2010

Received: 01 May 2009

Published in TOMACS Volume 21, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Department of Science and Technology, Ministry of Science and Technology

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
798
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)3

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Horng SLin S(2020)Coupling Elephant Herding with Ordinal Optimization for Solving the Stochastic Inequality Constrained Optimization ProblemsApplied Sciences10.3390/app1006207510:6(2075)Online publication date: 19-Mar-2020
https://doi.org/10.3390/app10062075
Ciupageanu DBarelli LLazaroiu G(2020)Real-time stochastic power management strategies in hybrid renewable energy systems: A review of key applications and perspectivesElectric Power Systems Research10.1016/j.epsr.2020.106497187(106497)Online publication date: Oct-2020
https://doi.org/10.1016/j.epsr.2020.106497
Barelli LBidini GCiupăgeanu DPianese CPolverino PSorrentino M(2020)Stochastic power management approach for a hybrid solid oxide fuel cell/battery auxiliary power unit for heavy duty vehicle applicationsEnergy Conversion and Management10.1016/j.enconman.2020.113197221(113197)Online publication date: Oct-2020
https://doi.org/10.1016/j.enconman.2020.113197
Ciupageanu DBarelli LOttaviano APelosi DLazaroiu G(2019)Innovative power management of hybrid energy storage systems coupled to RES plants: The Simultaneous Perturbation Stochastic Approximation approach2019 IEEE PES Innovative Smart Grid Technologies Europe (ISGT-Europe)10.1109/ISGTEurope.2019.8905775(1-5)Online publication date: Sep-2019
https://doi.org/10.1109/ISGTEurope.2019.8905775
Wang ZIerapetritou M(2018)Constrained optimization of black-box stochastic systems using a novel feasibility enhanced Kriging-based methodComputers & Chemical Engineering10.1016/j.compchemeng.2018.07.016118(210-223)Online publication date: Oct-2018
https://doi.org/10.1016/j.compchemeng.2018.07.016
Pujowidianto N(2017)Constrained optimization via simulation models for new product innovationIOP Conference Series: Materials Science and Engineering10.1088/1757-899X/245/1/012024273(012024)Online publication date: 28-Nov-2017
https://doi.org/10.1088/1757-899X/245/1/012024
Lakshmanan KBhatnagar S(2017)Quasi-Newton smoothed functional algorithms for unconstrained and constrained simulation optimizationComputational Optimization and Applications10.1007/s10589-016-9875-466:3(533-556)Online publication date: 1-Apr-2017
https://dl.acm.org/doi/10.1007/s10589-016-9875-4
Chang KLu H(2016)Quantile-Based Simulation Optimization With Inequality Constraints: Methodology and ApplicationsIEEE Transactions on Automation Science and Engineering10.1109/TASE.2015.240673613:2(701-708)Online publication date: Apr-2016
https://doi.org/10.1109/TASE.2015.2406736
Chang H(2016)Sleeping experts and bandits approach to constrained Markov decision processesAutomatica (Journal of IFAC)10.1016/j.automatica.2015.10.01563:C(182-186)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1016/j.automatica.2015.10.015
Prashanth LGhavamzadeh M(2016)Variance-constrained actor-critic algorithms for discounted and average reward MDPsMachine Language10.1007/s10994-016-5569-5105:3(367-417)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1007/s10994-016-5569-5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents