Skip to main content
SpringerLink
Log in

The Value of Knowledge: Joining Reward and Epistemic Certainty Optimisation for Anxiety-Sensitive Planning

  • Conference paper
  • First Online:
Autonomous Agents and Multiagent Systems. Best and Visionary Papers (AAMAS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14456))

Included in the following conference series:

  • 64 Accesses

Abstract

Anxiety is one of the most basic emotional states and also the most common disorder. AI agents however are typically focused on maximising performance, concentrating on expected values and disregarding the degree of exposure to uncertainty. This paper introduces a formalism derived from Partially Observable Markov Decision Processes (POMDPs) to give the first model based on cognitive psychology of the anxiety induced by epistemic uncertainty (i.e. the lack of precision of knowledge about the current state of the world). An algorithm to generate policies balancing reward maximisation and anxiety reduction is given. It is then used on a classical example to demonstrate how this can lead in some cases to a dramatic reduction of epistemic uncertainty for nearly no cost and thus a more human-friendly reward optimisation. The empirical validation shows results reminiscent of behaviours that cognitive psychology identifies as coping mechanisms to anxiety.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.pomdp.org/examples/ [24].

References

  1. Araya, M., Buffet, O., Thomas, V., Charpillet, F.: A pomdp extension with belief-dependent rewards. In: Advances in Neural Information Processing Systems, vol. 23 (2010)

    Google Scholar 

  2. Ayer, T., Alagoz, O., Stout, N.K.: Or forum-a POMDP approach to personalize mammography screening decisions. Oper. Res. 60(5), 1019–1034 (2012)

    Article  MathSciNet  Google Scholar 

  3. Bach, D.R., Dolan, R.J.: Knowing how much you don’t know: a neural organization of uncertainty estimates. Nat. Rev. Neurosci. 13, 572–586 (2012)

    Article  Google Scholar 

  4. Balaban, E., Arnon, T., Shirley, M.H., Brisson, S.F., Gao, A.: A system health aware pomdp framework for planetary rover traverse evaluation and refinement. In: 2018 AIAA Information Systems-AIAA Infotech @ Aerospace (2018)

    Google Scholar 

  5. Barlow, D.: Anxiety and Its Disorders: The Nature and Treatment of Anxiety and Panic. Anxiety and Its Disorders: The Nature and Treatment of Anxiety and Panic, Guilford Publications (2004)

    Google Scholar 

  6. Beck, A.T., Clark, D.A.: An information processing model of anxiety: automatic and strategic processes. Behav. Res. Ther. 35(1), 49–58 (1997)

    Article  Google Scholar 

  7. Bravo, R.Z.B., Leiras, A., Cyrino Oliveira, F.L.: The use of UAVs in humanitarian relief: an application of POMDP-based methodology for finding victims. Prod. Oper. Manage. 28(2), 421–440 (2019)

    Google Scholar 

  8. Broekens, J., Jacobs, E., Jonker, C.M.: A reinforcement learning model of joy, distress, hope and fear. Connect. Sci. 27(3), 215–233 (2015)

    Article  Google Scholar 

  9. Cassandra, A.R.: A survey of POMDP applications. In: Working Notes of AAAI Fall Symposium on Planning with Partially Observable Markov Decision Processes (1998)

    Google Scholar 

  10. Cassandra, A.R., Kaelbling, L.P., Littman, M.L.: Acting optimally in partially observable stochastic domains. In: Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence, AAAI 1994, pp. 1023–1028. AAAI Press (1994)

    Google Scholar 

  11. Çelik, M., Ergun, O., Keskinocak, P.: The post-disaster debris clearance problem under incomplete information. Oper. Res. 63(1), 65–85 (2015)

    Google Scholar 

  12. Corotis, R., Ellis, H., Jiang, M.: Modeling of risk-based inspection, maintenance and life-cycle cost with partially observable markov decision processes. Struct. Infrastruct. Eng. 1, 75–84 (2005)

    Google Scholar 

  13. Duke, K.E., Goldsmith, K., Amir, O.: Is the preference for certainty always so certain? J. Assoc. Consum. Res. 3(1), 63–80 (2018)

    Google Scholar 

  14. Institute of Health Metrics and Evaluation, Global Health Data Exchange. www.vizhub.healthdata.org/gbd-results/, Accessed 14 Mai 2022

  15. Ibekwe, H., Kamrani, A.: Navigation for autonomous robots in partially observable facilities. In: 2014 World Automation Congress, pp. 1–5 (2014)

    Google Scholar 

  16. Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artif. Intell. 101(1–2), 99–134 (1998)

    Article  MathSciNet  Google Scholar 

  17. Mafi, N., Abtahi, F., Fasel, I.: Information theoretic reward shaping for curiosity driven learning in POMDPs. In: 2011 IEEE International Conference on Development and Learning (ICDL), vol. 2, pp. 1–7 (2011)

    Google Scholar 

  18. Marecki, J., Varakantham, P.: Risk-sensitive planning in partially observable environments. In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, AAMAS. 2, pp. 1357–1368 (2010)

    Google Scholar 

  19. Miceli, M., Castelfranchi, C.: Anxiety as an “epistemic” emotion: an uncertainty theory of anxiety. Anxiety Stress Coping 18, 291–319 (2005)

    Google Scholar 

  20. Niroui, F., Sprenger, B., Nejat, G.: Robot exploration in unknown cluttered environments when dealing with uncertainty. In: 2017 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS), pp. 224–229. IEEE (2017)

    Google Scholar 

  21. Geneva: World Health Organization: Mental health and covid-19: Early evidence of the pandemic’s impact (2022)

    Google Scholar 

  22. Osmanağaoğlu, N., Creswell, C., Dodd, H.F.: Intolerance of uncertainty, anxiety, and worry in children and adolescents: a meta-analysis. J. Affect. Disord. 225, 80–90 (2018)

    Article  Google Scholar 

  23. Papakonstantinou, K., Shinozuka, M.: Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part II: POMDP implementation. Reliab. Eng. Syst. Saf. 130, 214–224 (2014)

    Google Scholar 

  24. pomdp.org: Examples. www.pomdp.org/examples/, Accessed 26 Jun 2022

  25. Roy, N., Gordon, G., Thrun, S.: Finding approximate POMDP solutions through belief compression. J. Artif. Intell. Res. 23, 1–40 (2005)

    Article  Google Scholar 

  26. Silver, D., Veness, J.: Monte-carlo planning in large pomdps. In: Lafferty, J., Williams, C., Shawe-Taylor, J., Zemel, R., Culotta, A. (eds.) Advances in Neural Information Processing Systems, vol. 23. Curran Associates, Inc. (2010)

    Google Scholar 

  27. Thomas, V., Hutin, G., Buffet, O.: Monte Carlo information- oriented planning (2021). www.arxiv.org/abs/2103.11345

  28. Vanhée, L., Jeanpierre, L., Mouaddib, A.I.: Anxiety-sensitive planning: from formal foundations to algorithms and applications. In: Proceedings of the International Conference on Automated Planning and Scheduling, vol. 32, pp. 730–740 (2022)

    Google Scholar 

  29. Wang, Q.A.: Probability distribution and entropy as a measure of uncertainty. J. Phys. A: Math. Theor. 41(6), 065004 (2008)

    Article  MathSciNet  Google Scholar 

  30. Wu, F., Ramchurn, S.D., Chen, X.: Coordinating human-UAV teams in disaster response. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI), pp. 524–530 (2016)

    Google Scholar 

  31. Zhang, W., Wang, H.: Diagnostic policies optimization for chronic diseases based on POMDP model. In: Healthcare (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DI ENS, École normale supérieure, Université PSL, CNRS, 75005, Paris, France

    Linda Gutsche

  2. Department of Computing Science, Umeå University, 907 36, Umeå, Sweden

    Loïs Vanhée

Authors
  1. Linda Gutsche
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Loïs Vanhée
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linda Gutsche .

Editor information

Editors and Affiliations

  1. Politecnico di Milano, Milan, Italy

    Francesco Amigoni

  2. Rutgers University, Piscataway, NJ, USA

    Arunesh Sinha

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gutsche, L., Vanhée, L. (2024). The Value of Knowledge: Joining Reward and Epistemic Certainty Optimisation for Anxiety-Sensitive Planning. In: Amigoni, F., Sinha, A. (eds) Autonomous Agents and Multiagent Systems. Best and Visionary Papers. AAMAS 2023. Lecture Notes in Computer Science(), vol 14456. Springer, Cham. https://doi.org/10.1007/978-3-031-56255-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56255-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56254-9

  • Online ISBN: 978-3-031-56255-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation