Skip to main content
SpringerLink
Log in

Efficient algorithms of pathwise dynamic programming for decision optimization in mining operations

  • S.I.: CLAIO 2016
  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

Complexity and uncertainty associated with commodity resource valuation and extraction requires stochastic control methods suitable for high dimensional states. Recent progress in duality and trajectory-wise techniques has introduced a variety of fresh ideas to this field with surprising results. This paper presents a concept which implements this promising development and illustrates it on a selection of traditional commodity extraction problems. We describe efficient algorithms for obtaining approximate solutions along with a diagnostic technique, which provides a quantitative measure for solution performance in terms of the distance between the approximate and the optimal control policy. All quantitative tools are efficiently implemented and are publicly available within a user friendly package in the statistical language R, which can help practitioners in a broad range of decision optimization problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Andersen, L., & Broadie, M. (2004). Primal-dual simulation algorithm for pricing multidimensional american options. Management Science, 50(9), 1222–1234.

    Google Scholar 

  • Bauerle, N., & Rieder, U. (2011). Markov decision processes with applications to finance. Heidelberg: Springer.

    Google Scholar 

  • Bayraktar, E., & Egami, M. (2010). On the one-dimensional optimal switching problem. Mathematics of Operations Research, 35(1), 140–159.

    Google Scholar 

  • Belomestny, A., Kolodko, N., & Schoenmakers, J. (2010). Regression methods for stochastic control problems and their convergence analysis. SIAM Journal on Control and Optimization, 48(5), 3562–3588.

    Google Scholar 

  • Bender, C., & Dokuchaev, N. (2017). A first order bspde for swing option pricing. Mathematical Finance, 27(3), 902–925.

    Google Scholar 

  • Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Belmont: Athena Scientific.

    Google Scholar 

  • Bhattacharya, S. (1978). Project valuation with mean-reverting cash flow streams. The Journal of Finance, 33(5), 1317–1331.

    Google Scholar 

  • Bjork, T. (2005). Arbitrage theory in continuous time. Oxford: Oxford University Press.

    Google Scholar 

  • Boomsma, T., Meade, N., & Fleten, S. (2012). Renewable energy inestments under different support schemes: A real options approach. European Journal of Operational Research, 220(1), 225–237.

    Google Scholar 

  • Brandao, L. E., & Dyer, J. S. (2005). Decision analysis and real options: A discrete time approach to real option valuation. Annals of Operations Research, 135(1), 21–39.

    Google Scholar 

  • Brennan, M., & Schwartz, E. (1985). Evaluating natural resource investments. The Journal of Business, 58(2), 135–157.

    Google Scholar 

  • Broadie, M., & Glasserman, P. (1997). Pricing American-style securities using simulation. Journal of Economic Dynamics and Control, 21(8), 1323–1352.

    Google Scholar 

  • Brown, D., Smith, J., & Sun, P. (2010). Information relaxations and duality in stochastic dynamic programs. Operations Research, 58(4), 785–801.

    Google Scholar 

  • Carmona, R., & Ludkovski, M. (2010). Valuation of energy storage: An optimal switching approach. Quantitative Finance, 10(4), 359–374.

    Google Scholar 

  • Carmona, R., & Touzi, N. (2008). Optimal multiple stopping and valuation of swing options. Mathematical Finance, 18, 239–268.

    Google Scholar 

  • Carriere, J. F. (1996). Valuation of the early-exercise price for options using simulations and nonparametric regression. Insurance: Mathematics and Economics, 19, 19–30.

    Google Scholar 

  • Cartea, A., Jaimungal, S., & Qin, Z. (2016). Model uncertainty in commodity markets. SIAM Journal on Financial Mathematics, 7(1), 1–33.

    Google Scholar 

  • Casassus, J., & Collin-Dufresne, P. (2005). Stochastic convenience yield implied from commodity futures and interest rates. The Journal of Finance, 60(5), 2283–2331.

    Google Scholar 

  • Clement, E., Lamberton, D., & Protter, P. (2002). An analysis of the Longstaff–Schwartz algorithm for American option pricing. Finance and Stochastics, 6(4), 449–471.

    Google Scholar 

  • Cortazar, G., Schwartz, E. S., & Casassus, J. (2001). Optimal exploration investments under price and geological-technical uncertainty: A real options model. R&D Management, 31(2), 181–189.

    Google Scholar 

  • Devalkar, S. K., Anupindi, R., & Sinha, A. (2011). Integrated optimization of procurement, processing, and trade of commodities. Operations Research, 59(6), 1369–1381.

    Google Scholar 

  • Dias, M. (2004). Valuation of exploration and production assets: An overview of real options models. Journal of Petroleum Science and Engineering, 44, 93–114.

    Google Scholar 

  • Dixit, A. K., & Pindyck, R. S. (1994). Investment under uncertainty. Princeton: Princeton University Press.

    Google Scholar 

  • Egloff, D. (2005). Monte Carlo algorithms for optimal stopping and statistical learning. The Annals of Applied Probability, 15, 1396–1432.

    Google Scholar 

  • Egloff, D., Kohler, M., & Todorovic, N. (2007). A dynamic look-ahead Monte Carlo algorithm. The Annals of Applied Probability, 17, 1138–1171.

    Google Scholar 

  • Fan, J., & Gijbels, I. (1996). Local polynomial modelling and its applications. London: Chapman & Hall.

    Google Scholar 

  • Gamba, A. (2002). Real options valuation: A Monte Carlo simulation approach. In Working paper.

  • Geman, H. (2005). Commodities and commodity derivatives, modeling and pricing for agriculturals, metals and energy. Hoboken: Wiley.

    Google Scholar 

  • Gibson, R., & Schwartz, E. S. (1990). Stochastic convenience yield and the pricing of oil contingent claims. The Journal of Finance, 45(3), 959–976.

    Google Scholar 

  • Glasserman, P. (2003). Monte Carlo methods in financial engineering. Berlin: Springer.

    Google Scholar 

  • Göbel, J., Keeler, H. P., Krzesinski, A. E., & Taylor, P. G. (2016). Bitcoin blockchain-dynamics: The selfish-mine strategy in the presence of propagation delay. Performance Evaluation, 104(Supplement C), 23–41.

    Google Scholar 

  • Gobet, E., Lemor, J.-P. (2008). Numerical simulation of bsdes using empirical regression methods: Theory and practice. arXiv:0806.4447.

  • Gobet, E., Lemor, J.-P., & Warin, X. (2005). A regression-based Monte Carlo method to solve backward stochastic differential equations. The Annals of Applied Probability, 114(3), 2172–2202.

    Google Scholar 

  • Gruenspan, C., Perez-Marco, R. (2017). Double spend races. In Working paper.

  • Hauskrecht, M. (2000). Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research, 13, 33–94.

    Google Scholar 

  • Hilliard, J. E., & Reis, J. (1998). Valuation of commodity futures and options under stochastic convenience yields, interest rates, and jump diffusions in the spot. Journal of Financial and Quantitative Analysis, 33(1), 61–86.

    Google Scholar 

  • Hinz, J. (2014). Optimal stochastic switching under convexity assumptions. SIAM Journal on Control and Optimization, 52(1), 164–188.

    Google Scholar 

  • Hinz, J. (2015). Using convex switching techniques for partially observable decision processes. IEEE Transactions on Automatic Control, PP(99), 1–1.

    Google Scholar 

  • Hinz, J., & Fehr, M. (2010). Storage costs in commodity option pricing. SIAM Journal on Financial Mathematics, 1, 729–751.

    Google Scholar 

  • Hinz, J., von Grafenstein, L., Verschuere, M., & Wilhelm, M. (2005). Pricing electricity risk by interest rate methods. Quantitative Finance, 5(1), 49–60.

    Google Scholar 

  • Hinz, J., & Yap, N. (2015). Algorithms for optimal control of stochastic switching systems. Theory of Probability and its Applications, 60(4), 770–800.

    Google Scholar 

  • Hinz, J., & Yee, J. (2017a). Optimal forward trading and battery control under renewable electricity generation. Journal of Banking and Finance. https://doi.org/10.1016/j.jbankfin.2017.06.006.

    Google Scholar 

  • Hinz, J., Yee, J. (2017b). Convex switching systems. https://cran.r-project.org/web/packages/rcss/.

  • Hinz, J., & Yee, J. (2017c). An algorithmic approach to optimal asset liquidation problems. Asia-Pac Financ Markets, 24(2), 109–129.

    Google Scholar 

  • Hinz, J., & Yee, J. (2017d). Stochastic switching for partially observable dynamics and optimal assetallocation. International Journal of Control, 90(3), 553–565.

    Google Scholar 

  • Jaimungal, S., de Souza, M. O., & Zubelli, J. P. (2013). Real option pricing with mean-reverting investment and project value. The European Journal of Finance, 19(7–8), 625–644.

    Google Scholar 

  • Jaimungal, S., & Surkov, V. (2009). Stepping through fourier space. Risk, 22(7), 78–83.

    Google Scholar 

  • Jaimungal, S., & Surkov, V. (2011). Levy-based cross-commodity models and derivative valuation. SIAM Journal on Financial Mathematics, 2(1), 464–487.

    Google Scholar 

  • Longstaff, F., & Schwartz, E. (2001). Valuing American options by simulation: A simple least-squares approach. Review of Financial Studies, 14(1), 113–147.

    Google Scholar 

  • McDonald, R. L., & Siegel, D. R. (1985). Investment and the valuation of firms when there is an option to shut down. International Economic Review, 26(2), 331–349.

    Google Scholar 

  • Meinshausen, N., & Hambly, B. (2004). Monte Carlo methods for the valuation of multiple-exercise options. Mathematical Finance, 14(4), 557–583.

    Google Scholar 

  • Nadarajah, S., Margot, F., & Secomandi, N. (2017). Comparison of least squares Monte Carlo methods with applications to energy real options. European Journal of Operational Research, 256(1), 196–204.

    Google Scholar 

  • Nakomoto, S. (2008). A peer-to-peer electronic cash system. In: Working paper.

  • Oeksendal, B., & Sulem, A. (2005). Applied stochastic control of jump diffusions. Berlin: Springer.

    Google Scholar 

  • Ormoneit, D., & Glynn, P. (2002). Kernel-based reinforcement learning. Machine Learning, 49, 161–178.

    Google Scholar 

  • Ormoneit, D., & Glynn, P. (2002). Kernel-based reinforcement learning in average-cost problems. IEEE Transactions in Automatic Control, 47, 1624–1636.

    Google Scholar 

  • Paschke, R., & Propkopczuk, M. (2010). Commodity derivatives valuation with autoregressive and moving average components in the price dynamics. Journal of Banking and Finance, 34(1), 2742–2752.

    Google Scholar 

  • Pham, H. (2009). Continuous-time stochastic control and optimization with financial applications (1st ed.). Berlin: Springer.

    Google Scholar 

  • Powell, W. B. (2007). Approximate dynamic programming: Solving the curses of dimensionality. Hoboken: Wiley.

    Google Scholar 

  • Puterman, M. (1994). Markov decision processes: Discrete stochastic dynamic programming. New York: Wiley.

    Google Scholar 

  • Rogers, L. (2007). Pathwise stochastic optimal control. SIAM Journal on Control and Optimization, 46(3), 1116–1132.

    Google Scholar 

  • Rosenfeld, M. (2014). Analysis of hashrate-based double spending. In: Working paper.

  • Ross, S. A. (1978). A simple approach to the valuation of risky streams. The Journal of Business, 51(3), 453–475.

    Google Scholar 

  • Sarkar, S. (2003). The effect of mean reversion on investment under uncertainty. Journal of Economic Dynamics and Control, 28(2), 377–396.

    Google Scholar 

  • Schwartz, E. (1997). The stochastic behavior of commodity prices: Implications for valuation and hedging. The Journal of Finance, 52(3), 923–973.

    Google Scholar 

  • Schwartz, E. S., & Smith, J. E. (2000). Short-term variations and long-term dynamics in commodity prices. Management Science, 46(7), 893–911.

    Google Scholar 

  • Stentoft, L. (2013). American option pricing using simulation with an application to the garch model. In: Handbook of Research Methods and Applications in Empirical Finance (pp. 114–147).

  • Stentoft, L. (2004). Convergence of the least squares Monte Carlo approach to American option valuation. Management Science, 50(9), 576–611.

    Google Scholar 

  • Tan, K. S., & Boyle, P. P. (2000). Applications of randomized low discrepancy sequences to the valuation of complex securities. Journal of Economic Dynamics and Control, 24(11), 1747–1782.

    Google Scholar 

  • Tan, K. S., & Boyle, P. P. (2001). Pricing american options: A comparison of Monte Carlo simulation approaches. Journal of Computational Finance, 4(3), 39–88.

    Google Scholar 

  • Trigeorgis, L. (1993). The nature of option interactions and the valuation of investments with multiple real options. Journal of Financial and Quantitative Analysis, 28(1), 1–20.

    Google Scholar 

  • Trigeorgis, L. (1996). Real options: Managerial flexibility and strategy in resource allocation. Cambridge: MIT Press.

    Google Scholar 

  • Tsekrekos, A. (2010). The effect of mean reversion on entry and exit decisions under uncertainty. Journal of Economic Dynamics and Control, 34(4), 725–742.

    Google Scholar 

  • Tsekrekos, A., Shackleton, M., & Wojakowski, R. (2012). Evaluating natural resource investments under different model dynamics: Managerial insights. European Financial Management, 18(4), 543–577.

    Google Scholar 

  • Tsekrekos, A. E., Shackleton, M. B., & Wojakowski, R. (2012). Evaluating natural resource investments under different model dynamics: Managerial insights. European Financial Management, 18(4), 543–575.

    Google Scholar 

  • Tsekrekos, A., & Yannacopoulos, A. (2016). Optimal switching decisions under stochastic volatility with fast mean reversion. European Journal of Operational Research, 251(1), 148–157.

    Google Scholar 

  • Tsitsiklis, J. N., & Van Roy, B. (1999). Optimal stopping of Markov processes: Hilbert space, theory, approximation algorithms, and an application to pricing high-dimensional financial derivatives. IEEE Transactions on Automatic Control, 44(10), 1840–1851.

    Google Scholar 

  • Tsitsiklis, J. N., & Van Roy, B. (2001). Regression methods for pricing complex American-style options. IEEE Transactions on Neural Networks, 12(4), 694–703.

    Google Scholar 

  • Zhu, J. (2017). Advanced Monte-Carlo methods for pricing Bermudan options and their applications in real option analysis. Master Thesis, MacQuarie University.

Download references

Author information

Authors and Affiliations

  1. University of Technology Sydney, Sydney, Australia

    Juri Hinz & Jeremy Yee

  2. CSIRO, Canberra, Australia

    Tanya Tarnopolskaya & Jeremy Yee

Authors
  1. Juri Hinz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tanya Tarnopolskaya
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeremy Yee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juri Hinz.

Appendix: Source code listings

Appendix: Source code listings

Having removed all variables and loaded our package

figure c

the parameters of state evolution are defined

figure d

to introduce the grid

figure e

and disturbance sampling

figure f

The position control is determined by

figure g

Introduce the reward functions in terms of sub-gradient representations

figure h

Next, call the Bellman recursion measuring the required time

figure i

The solution diagnostics requires disturbance simulations

figure j

which are used to generate Monte-Carlo paths

figure k

Furthermore, a vectorized version of the reward functions must be supplied (for details, see package documentation)

figure l

Having obtained the nearly-optimal policy

figure m

the diagnostics method is applied. First, we generate disturbance sub-simulations

figure n

and determine the martingale increments

figure o

to assess the quality of the control policy

figure p

The results can be obtained by inspecting appropriate fields of the output.

figure q

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hinz, J., Tarnopolskaya, T. & Yee, J. Efficient algorithms of pathwise dynamic programming for decision optimization in mining operations. Ann Oper Res 286, 583–615 (2020). https://doi.org/10.1007/s10479-018-2910-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10479-018-2910-3

Keywords

Search

Navigation