Skip to main content
SpringerLink
Log in

Cosine Policy Iteration for Solving Infinite-Horizon Markov Decision Processes

  • Conference paper
MICAI 2009: Advances in Artificial Intelligence (MICAI 2009)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 5845))

Included in the following conference series:

Abstract

Police Iteration (PI) is a widely used traditional method for solving Markov Decision Processes (MDPs). In this paper, the cosine policy iteration (CPI) method for solving complex problems formulated as infinite-horizon MDPs is proposed. CPI combines the advantages of two methods: i) Cosine Simplex Method (CSM) which is based on the Karush, Kuhn, and Tucker (KKT) optimality conditions and finds rapidly an initial policy close to the optimal solution and ii) PI which is able to achieve the global optimum. In order to apply CSM to this kind of problems, a well- known LP formulation is applied and particular features are derived in this paper. Obtained results show that the application of CPI solves MDPs in a lower number of iterations that the traditional PI.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Putterman, L.: Markov Decision Processes Discrete Stochastic Dynamic Programming. Wiley-Interscience, Hoboken (2005)

    Google Scholar 

  2. Dai, P., Goldsmith, J.: Topological Value Iteration Algorithm for Markov Decision Processes. International Join Conference on Artificial Intelligence (2007)

    Google Scholar 

  3. Bernstein, D.S., Amato, C., Hanse, E.A., Zilberstein, S.: Policy Iteration for Decentralized Control of Markov Decision Processes. Journal of Artificial Intelligence Research, 89–132 (2009)

    Google Scholar 

  4. Kim, K.-E., Dean, T.L., Meuleau, N.: Approximate solutions to Factored Markov Decision Processes via Greedy Search in the Space of Finite State Controllers. American Association for Artificial Intelligence (2000)

    Google Scholar 

  5. Howard, R.: Dynamic Programming and Markov processes. Wiley, New York (1960)

    MATH  Google Scholar 

  6. Santos, M.S., Rust, J.: Convergence properties of policy iteration. SIAM Journal on Control and Optimization 42(6), 2094–2115 (2003)

    Article  MathSciNet  Google Scholar 

  7. Dantzig, G.B., Thapa, M.N.: Linear Programming. 1: Introduction. Springer Series in Operations Research and Financial Engineering, 1st edn. (1997)

    Google Scholar 

  8. Chvátal, V.: Linear Programming. W. H. Freeman and Company, New York (1983)

    MATH  Google Scholar 

  9. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (2003) Republished 2003: Dover; ISBN 0486428095

    MATH  Google Scholar 

  10. Bertsekas, D.P.: Network Optimization continuous and discrete models. Athena Scientific (1998)

    Google Scholar 

  11. Bello, D., Riano, G.: Linear Programming solvers for Markov Decision Processes. Systems and Information Engineering Design Symposium, IEEE 28(28), 90–95 (2006)

    Article  Google Scholar 

  12. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, Heidelberg (1999)

    MATH  Google Scholar 

  13. Trigos, F., Frausto-Solis, J., Rivera-Lopez, R.: A Simplex Cosine Method for Solving the Klee-Minty Cube. In: Advances in Simulation System Theory and Systems Engineering, vol. 70X, pp. 27–32. WSEAS Press (2002) ISBN 960852

    Google Scholar 

  14. Trigos, F., Frausto-Solis, J.: Experimental Evaluation of the Theta Heuristic for Starting the Cosine Simplex Method. In: International Conference in Computational Science and its Applications, ICCSA, Proceedings Part IV (2005)

    Google Scholar 

  15. Corley, H.W., Rosenberger, J., Wei-Chang, Y., Sung, T.K.: The Cosine Simplex Algorithm. The International Journal of Advanced Manufacturing Technology 27(9-10), 1047–1050 (2006)

    Article  Google Scholar 

  16. Wei-Chang, Y., Corley, H.W.: A simple direct cosine simplex algorithm. Applied Mathematics and Computation 214(1), 178–186 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  17. Netlib Repository at UTK and ORNL, http://www.netlib.org/lp/data/

Download references

Author information

Authors and Affiliations

  1. Tecnológico de Monterrey Campus Cuernavaca, Autopista del Sol Km 104+06, Colonia Real del Puente, 62790, Xochitepec, Morelos, México

    Juan Frausto-Solis & Elizabeth Santiago

  2. Tecnológico de Monterrey Campus Estado de, México

    Jaime Mora-Vargas

Authors
  1. Juan Frausto-Solis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elizabeth Santiago
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jaime Mora-Vargas
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Centro de Investigación en Matemáticas, AC. (CIMAT), Area de Computación, Calle Jalisco S/N, Mineral de Valenciana, Guanajuato, CP 36250, Guanajuato, México

    Arturo Hernández Aguirre

  2. Ciencias de la Computación, Tecnológico de Monterrey, Campus Estado de México, Carretera al lago de Guadalupe Km 3,5, Col. Margarita Maza de Juárez, Atizapán de Zaragoza,, CP 52926, Estado de México, México

    Raúl Monroy Borja

  3. Instituto Nacional de Astrofísica, Optica y Electrónica (INAOE), Ciencias Computacionales, Luis Enrique Erro No. 1, Santa María Tonantzintla, CP 72840, Puebla, México

    Carlos Alberto Reyes Garciá

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Frausto-Solis, J., Santiago, E., Mora-Vargas, J. (2009). Cosine Policy Iteration for Solving Infinite-Horizon Markov Decision Processes. In: Aguirre, A.H., Borja, R.M., Garciá, C.A.R. (eds) MICAI 2009: Advances in Artificial Intelligence. MICAI 2009. Lecture Notes in Computer Science(), vol 5845. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-05258-3_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-05258-3_7

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-05257-6

  • Online ISBN: 978-3-642-05258-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation