David J. Eckman
Shane G. Henderson
Abstract
It is tempting to reuse replications taken during a simulation optimization search as input to a ranking-and-selection procedure. However, even when the random inputs used to generate replications are identically distributed and independent within and across systems, we show that for searches that use the observed performance of explored systems to identify new systems, the replications are conditionally dependent given the sequence of returned systems. Through simulation experiments, we demonstrate that reusing the replications taken during search in selection and subset-selection procedures can result in probabilities of correct and good selection well below the guaranteed levels. Based on these negative findings, we call into question the guarantees of established ranking-and-selection procedures that reuse search data. We also rigorously define guarantees for ranking-and-selection procedures after search and discuss how procedures that only provide guarantees in the preference zone are ill-suited to this setting.
- Mohamed A. Ahmed and Talal M. Alkhamis. 2002. Simulation-based optimization using simulated annealing with ranking and selection. Computers and Operations Research 29, 4 (2002), 387--402.Google Scholar
Cross Ref
- Robert E. Bechhofer. 1954. A single-sample multiple decision procedure for ranking means of normal populations with known variances. Annals of Mathematical Statistics 25, 1 (1954), 16--39.Google ScholarCross Ref
- Robert E. Bechhofer, Thomas J. Santner, and David M. Goldsman. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. Wiley, New York.Google Scholar
- Justin Boesel, Barry L. Nelson, and Nobuaki Ishii. 2003a. A framework for simulation-optimization software. IIE Transactions 35, 3 (2003), 221--229.Google ScholarCross Ref
- Justin Boesel, Barry L. Nelson, and Seong-Hee Kim. 2003b. Using ranking and selection to “clean up” after simulation optimization. Operations Research 51, 5 (2003), 814--825. Google ScholarDigital Library
- Edward J. Dudewicz and Siddhartha R. Dalal. 1975. Allocation of observations in ranking and selection with unequal variances. The Indian Journal of Statistics 37, 1 (1975), 28--78.Google Scholar
- Eyal Even-Dar, Shie Mannor, and Yishay Mansour. 2006. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of Machine Learning Research 7, June (2006), 1079--1105. Google ScholarDigital Library
- Mingbin Feng and Jeremy Staum. 2017. Green simulation: Reusing the output of repeated experiments. ACM Transactions on Modeling and Computer Simulation 27, 4 (2017), 23:1--23:28. Google ScholarDigital Library
- Peter Frazier. 2014. A fully sequential elimination procedure for indifference-zone ranking and selection with tight bounds on probability of correct selection. Operations Research 62, 4 (2014), 926--942. Google ScholarDigital Library
- Michael Fu. 2015. Handbook of Simulation Optimization, Vol. 216. Springer, New York. Google ScholarDigital Library
- Peter Glynn and Sandeep Juneja. 2015. Selecting the best system, large deviations, and multi-armed bandits. Preprint arXiv:1507.04565v2. http://arxiv.org/abs/1507.04564v2.Google Scholar
- Shanti S. Gupta. 1965. On some multiple decision (selection and ranking) rules. Technometrics 7, 2 (1965), 225--245.Google ScholarCross Ref
- Shane G. Henderson and Barry L. Nelson (Eds.). 2006. Handbooks in Operations Research and Management Science: Simulation, Vol. 13. Elsevier, Amsterdam.Google Scholar
- L. Jeff Hong and Barry L. Nelson. 2007. Selecting the best system when systems are revealed sequentially. IIE Transactions 39, 7 (2007), 723--734.Google ScholarCross Ref
- Susan R. Hunter and Raghu Pasupathy. 2010. Large-deviation sampling laws for constrained simulation optimization on finite sets. In Proceedings of the 2010 Winter Simulation Conference. 995--1002. Google ScholarDigital Library
- Seong-Hee Kim and Barry L. Nelson. 2001. A fully sequential procedure for indifference-zone selection in simulation. ACM Transactions on Modeling and Computer Simulation (TOMACS) 11, 3 (2001), 251--273. Google ScholarDigital Library
- Seong-Hee Kim and Barry L. Nelson. 2006a. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research 54, 3 (2006), 475--488. Google ScholarDigital Library
- Seong-Hee Kim and Barry L. Nelson. 2006b. Selecting the best system. In Simulation, edited by Shane G. Henderson and Barry L. Nelson, Handbooks in Operations Research and Management Sscience, Vol. 13. Elsevier, Amsterdam.Google Scholar
- Barry L. Nelson and Frank Matejcik. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Management Science 41, 12 (1995), 1935--1945.Google ScholarCross Ref
- Barry L. Nelson, Julie Swann, David Goldsman, and Wheyming Song. 2001. Simple procedures for selecting the best simulated system when the number of alternatives is large. Operations Research 49, 6 (2001), 950--963. Google ScholarDigital Library
- Sigurdur Olafsson. 1999. Iterative ranking-and-selection for large-scale optimization. In Proceedings of the 1999 Winter Simulation Conference. 479--485. Google ScholarDigital Library
- Juta Pichitlamken and Barry L. Nelson. 2001. Selection-of-the-best procedures for optimization via simulation. In Proceedings of the 2001 Winter Simulation Conference. 401--407. Google ScholarDigital Library
- Juta Pichitlamken and Barry L. Nelson. 2003. A combined procedure for optimization via simulation. ACM Transactions on Modeling and Computer Simulation 13, 2 (2003), 155--179. Google ScholarDigital Library
- Juta Pichitlamken, Barry L. Nelson, and L. Jeff Hong. 2006. A sequential procedure for neighborhood selection-of-the-best in optimization via simulation. European Journal of Operational Research 173, 1 (2006), 283--298.Google ScholarCross Ref
- Yosef Rinott. 1978. On two-stage selection procedures and related probability inequalities. Communications in Statistics—Theory and Methods 7, 8 (1978), 799--811.Google Scholar
- Todd A. Sriver, James W. Chrissis, and Mark A. Abramson. 2009. Pattern search ranking and selection algorithms for mixed variable simulation-based optimization. European Journal of Operations Research 198, 3 (2009), 878--890.Google ScholarCross Ref
- Paul van der Laan. 1992. Subset selection of an almost best treatment. Biometrical Journal 34, 6 (1992), 647--656.Google ScholarCross Ref
Index Terms
- Reusing Search Data in Ranking and Selection: What Could Possibly Go Wrong?
Recommendations
Replicated Computations Results (RCR) Report for “Reusing Search Data in Ranking and Selection: What Could Possibly Go Wrong?”
“Reusing Search Data in Ranking and Selection: What Could Possibly Go Wrong?” [2] by Eckman and Henderson rigorously defines the statistical guarantees for ranking-and-selection (R8S) procedures after random search, and points out that the simulation ...
Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey
An important use for discrete-event simulation models lies in comparing and contrasting competing design alternatives without incurring any physical costs. This article presents a survey of the literature for two widely used classes of statistical ...
Selecting the best stochastic system for large scale problems in DEDS
We consider the problem of selecting the stochastic system with the best expected performance measure, when the number of alternative systems is large. We consider the case of discrete event dynamic systems (DEDS) where the standard clock simulation ...
Comments