Skip to main content
SpringerLink
Log in

Modeling uncertainty of expert elicitation for use in risk-based optimization

  • Original Research
  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

Capital budgeting optimization models, used in a broad number of fields, require certain and uncertain parameters. Often times, elicited subject matter expert (SME) opinion is used as a parameter estimate, which does not always yield perfect information or correspond to a single value. Because of the uncertainty of the elicitation, the unknown true value of a parameter can be modeled as a random variable from a to-be-determined distribution. We estimate a univariate distribution using four different approaches, the Beta and Gaussian distributions, a standard Gaussian Kernel estimate, and an exponential epi-spline. We also capture dependencies within the parameters through three multivariate approaches: the multivariate Gaussian distribution, the multivariate Kernel and the multivariate exponential epi-spline. This is the first three-dimensional application of the latter. Sampling from the densities, we generate scenarios and implement a superquantile risk-based, capital budgeting optimization model. Numerical experiments contrast the differences between estimators, as well as their effects on an optimal solution. Our findings demonstrate that naively averaging the SME observations for use in optimization, rather than incorporating uncertainty, results in an overly optimistic portfolio. The flexibility of the exponential epi-spline estimator to fuse soft information with observed data produces reasonable density functions for univariate and multivariate random variables. Including a decision-maker’s risk-averseness through risk-based optimization delivers conservative results while incorporating the uncertainty of unknown parameters. We demonstrate a 20% improvement for this specific case when using our approach as opposed to the naive method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Arigoni, A. (2016). Optimization techniques in coal markets: A global cost minimization and a multi-stage procurement strategy. Dissertation, Colorado School of Mines.

  • Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79. https://projecteuclid.org/download/pdfview_1/euclid.ssu/1268143839.

  • Assistant Secretary of the Army (Financial Management and Comptroller). (1994). Planning, programing, budgeting, and execution system. Headquarters, Department of the Army.

  • Brown, G. G., Carlyle, M., Salmerón, J., & Wood, K. (2006). Defending critical infrastructure. Interfaces, 36(6), 530–544.

    Article  Google Scholar 

  • Brown, G. G., Dell, R. F., & Newman, A. M. (2004). Optimizing military capital planning. Interfaces, 34(6), 415–425.

    Article  Google Scholar 

  • Buttrey, S, Royset, J. O., & Wets, R. J. B. (2014). episplineDensity: Density estimation with soft information by exponential epi-splines. http://CRAN.R-project.org/package=episplineDensity, R package version 0.0-1.

  • Casella, G., Robert, C. P., & Wells, M. T. (2004). Generalized accept-reject sampling schemes. Lecture Notes-Monograph Series, pp. 342–347.

  • Desouza, K. C., & Awazu, Y. (2006). Knowledge management at SMEs: Five peculiarities. Journal of Knowledge Management, 10(1), 32–43.

    Article  Google Scholar 

  • Deveans, T., Jacobs, P., Gaver, D., & Teter, M. (2014). Analysis of the capability portfolio review (cpr). Tech. rep., DTIC Document.

  • Drud, A. S. (1994). CONOPT—A large-scale GRG code. ORSA Journal on Computing, 6(2), 207–216.

    Article  Google Scholar 

  • Dunlop, D. D., & Tamhane, A. C. (2000). Statistics and data analysis: From elementary to intermediate. New York: Prentice Hall.

    Google Scholar 

  • Duong, T. (2015). ks: Kernel smoothing. http://CRAN.R-project.org/package=ks, R package version 1.10.0.

  • Efron, B. (1981). Nonparametric estimates of standard error: The jackknife, the bootstrap and other methods. Biometrika, 68(3), 589–599.

    Article  Google Scholar 

  • Evers, L., Dollevoet, T., Barros, A. I., & Monsuur, H. (2014). Robust UAV mission planning. Annals of Operations Research, 222(1), 293–315.

    Article  Google Scholar 

  • Ewing, P. L, Jr., Tarantino, W., & Parnell, G. S. (2006). Use of decision analysis in the Army Base Realignment And Closure (BRAC) 2005 military value analysis. Decision Analysis, 3(1), 33–49.

    Article  Google Scholar 

  • Feng, Y., Rios, I., Ryan, S. M., Spürkel, K., Watson, J. P., Wets, R. J. B., et al. (2015). Toward scalable stochastic unit commitment. Part 1: Load scenario generation. Energy Systems, 6(3), 309–329. https://doi.org/10.1007/s12667-015-0146-8.

    Article  Google Scholar 

  • GAMS. (2015). GAMS distribution 24.4.5. Washington, DC.

  • Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701.

    Article  Google Scholar 

  • Gelfand, A. E., Mallick, B. K., & Dey, D. K. (1995). Modeling expert opinion arising as a partial probabilistic specification. Journal of the American Statistical Association, 90(430), 598–604.

    Article  Google Scholar 

  • Genz, A., Bretz, F. (2009). Computation of multivariate normal and t probabilities. Lecture Notes in Statistics, Vol. 195.

  • Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646–675.

    Article  Google Scholar 

  • IBM. (2015). ILOG CPLEX, IBM—International Business Machines Corporation, Incline Village, NV.

  • Koç, A., Morton, D. P., Popova, E., Hess, S. M., Kee, E., & Richards, D. (2009). Prioritizing project selection. The Engineering Economist, 54(4), 267–297.

    Article  Google Scholar 

  • Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In International Joint Conference on Artificial Intelligence (IJCAI) (Vol. 14, pp. 1137–1145). https://www.researchgate.net/profile/Ron_Kohavi/publication/2352264_A_Study_of_Cross-Validation_and_Bootstrap_for_Accuracy_Estimation_and_Model_Selection/links/02e7e51bcc14c5e91c000000.pdf.

  • Little, R. J. (1988). Missing-data adjustments in large surveys. Journal of Business & Economic Statistics, 6(3), 287–296.

    Google Scholar 

  • Marks, C., Smead, K., & Alt, J. (2013). Enhancing subject matter expert elicitation techniques. Tech. Rep. TRAC-M-TR-13-048, TRADOC Analysis Center - Monterey, 700 Dyer Road Monterey, California 93943.

  • Moynihan, R. A. (2005). Investment analysis using the Portfolio Analysis Machine (PALMA) tool. Tech. rep., Technical Report MITRE Case.

  • Oakley, J. E., & O’Hagan, A. (2007). Uncertainty in prior elicitations: A nonparametric approach. Biometrika, 94(2), 427–441.

    Article  Google Scholar 

  • Odierno, R. T. (2015). Force 2025 and beyond: How the Army is preparing for its future. Defense One.

  • O’Hagan, A. (1998). Eliciting expert beliefs in substantial practical applications. Journal of the Royal Statistical Society: Series D (The Statistician), 47(1), 21–35.

    Google Scholar 

  • O’Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., et al. (2006). Uncertain judgements: Eliciting experts’ probabilities. Hoboken: Wiley.

    Book  Google Scholar 

  • Papp, D., & Alizadeh, F. (2014). Shape-constrained estimation using nonnegative splines. Journal of Computational and Graphical Statistics, 23(1), 211–231.

    Article  Google Scholar 

  • Pfeffermann, D., Krieger, A. M., & Rinott, Y. (1998). Parametric distributions of complex survey data under informative probability sampling. Statistica Sinica, 8, 1087–1114.

    Google Scholar 

  • R Core Team. (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.

  • Rios, I., Wets, R. J. B., & Woodruff, D. L. (2015). Multi-period forecasting and scenario generation with limited data. Computational Management Science, 12(2), 267–295.

    Article  Google Scholar 

  • Rockafellar, R. T., & Royset, J. O. (2015). Engineering decisions under risk averseness. ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part A: Civil Engineering, 1(2), 04015003.

    Article  Google Scholar 

  • Rockafellar, R. T., & Royset, J. O. (2018). Superquantile/CVaR risk measures: Second-order theory. Annals of Operations Research, 262(1), 3–28.

    Article  Google Scholar 

  • Rockafellar, R. T., & Uryasev, S. (2000). Optimization of conditional value-at-risk. Journal of Risk, 2, 21–42.

    Article  Google Scholar 

  • Rockafellar, R. T., & Uryasev, S. (2002). Conditional value-at-risk for general loss distributions. Journal of Banking & Finance, 26(7), 1443–1471.

    Article  Google Scholar 

  • Rodríguez, J. D., Perez, A., & Lozano, J. A. (2010). Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3), 569–575.

    Article  Google Scholar 

  • Royset, J. O., Sukumar, N., & Wets, R. J. B. (2013). Uncertainty quantification using exponential epi-splines. In Proceedings of the international conference on structural safety and reliability.

  • Royset, J. O., & Wets, R. J. B. (2014). From data to assessments and decisions: Epi-spline technology. In Bridging data and decisions, chap. 3, pp. 27–53.

  • Royset, J. O., & Wets, R. J. B. (2015). Fusion of hard and soft information in nonparametric density estimation. European Journal of Operational Research, 247(2), 532–547.

    Article  Google Scholar 

  • Royset, J. O., & Wets, R. J. B. (2016). Multivariate epi-splines and evolving function identification problems. Set-Valued and Variational Analysis, 24(4), 517–545. Erratum, pp. 547–549.

  • Safaei, N., Banjevic, D., & Jardine, A. K. (2011). Workforce-constrained maintenance scheduling for military aircraft fleet: A case study. Annals of Operations Research, 186(1), 295–316.

    Article  Google Scholar 

  • Shao, J. (1993). Linear model selection by cross-validation. Journal of the American Statistical Association, 88(422), 486–494.

    Article  Google Scholar 

  • Shapiro, A., & Philpott, A. (2007). A tutorial on stochastic programming. http://www2.isye.gatech.edu/ashapiro/publications.html.

  • Singham, D. I., Royset, J. O., & Wets, R. J. B. (2013). Density estimation of simulation output using exponential epi-splines. In Proceedings of the 2013 winter simulation conference: simulation: making decisions in a complex world (pp. 755–765). IEEE Press.

  • Tastle, W. J., & Wierman, M. J. (2007). Consensus and dissension: A measure of ordinal dispersion. International Journal of Approximate Reasoning, 45(3), 531–545.

    Article  Google Scholar 

  • Teter, M., Brickey, A., Newman, A., & Royset, J. (2018). Optimizing underground mine scheduling under production rate uncertainty. Working paper.

  • Teter, M. D. (2014). Applying subject matter expertise (SME) elicitation techniques to TRAC studies. Tech. rep., DTIC Document.

  • Teter, M. D., Newman, A. M., & Weiss, M. (2016). Consistent notation for presenting complex optimization models in technical writing. Surveys in Operations Research and Management Science, 21(1), 1–17.

    Article  Google Scholar 

  • Under Secretary of Defense (Acquisition, Technology, and Logistics). (2002). DOD deskbook 5000.2-R Appendix 6: Technology readiness levels and their definitions. Department of Defense.

  • Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1–67. http://www.jstatsoft.org/v45/i03/.

  • Wets, R. J. B., & Rios, I. (2015). Modeling and estimating commodity prices: Copper prices. Mathematics and Financial Economics, 9(4), 247–270.

    Article  Google Scholar 

Download references

Acknowledgements

This material is based upon work supported in part by the U.S. Army Research Laboratory and the U.S. Army Research Office under Grant Numbers MIPR0010347614 and MIPR0010504121, respectively. The authors would like to acknowledge Professor Amanda Hering, Department of Statistical Science, Baylor University, for her invaluable advice in reviewing portions of the paper and Professor Samuel Buttrey, Operations Research Department, Naval Postgraduate School, for his assistance in review of and collaboration on portions of the R-code.

Author information

Authors and Affiliations

  1. Operations Research with Engineering Program, Colorado School of Mines, 1500 Illinois St., Golden, CO, 80401, USA

    Michael D. Teter

  2. Department of Operations Research, Naval Postgraduate School, 1 University Circle, Monterey, CA, 93943, USA

    Johannes O. Royset

  3. Mechanical Engineering Department, Colorado School of Mines, 1500 Illinois St., Golden, CO, 80401, USA

    Alexandra M. Newman

Authors
  1. Michael D. Teter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Johannes O. Royset
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexandra M. Newman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexandra M. Newman.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Teter, M.D., Royset, J.O. & Newman, A.M. Modeling uncertainty of expert elicitation for use in risk-based optimization. Ann Oper Res 280, 189–210 (2019). https://doi.org/10.1007/s10479-018-3011-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10479-018-3011-z

Keywords

Search

Navigation