Skip to main content

Conceptualizing Multi-party AI Reliance for Design Research

  • Conference paper
  • First Online:
Design Science Research for a Resilient Future (DESRIST 2024)

Abstract

Appropriate reliance on artificial intelligence (AI)-based systems is paramount to leverage increasing AI performance. However, multi-party settings, where multiple parties with diverging interests interact with the support of AI systems, are currently neglected. In this study, we use Heider’s balance theory to derive a framework that allows us to conceptualize and analyze reliance on AI in multi-party settings. We then use this framework to analyze two large design science research projects. First, we analyze financial advisory service encounters, where the role inequality of advisor and client can lead to a dominance of the advisor. Second, we analyze used car market negotiations, where the problem of partial reliance on AI systems creates a misalignment between the two parties, ultimately failing the negotiations. Finally, we discuss implications and future research on AI reliance in multi-party settings and highlight that this study should serve as a starting point in investigating AI reliance in multi-party settings.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. OpenAI: Introducing ChatGPT (2023). https://openai.com/blog/chatgpt

  2. Faisal, A., Kamruzzaman, M., Yigitcanlar, T., Currie, G.: Understanding autonomous vehicles. J. Transp. Land Use 12, 45–72 (2019)

    Article  Google Scholar 

  3. Rädsch, T., Eckhardt, S., Leiser, F., Pandl, K.D., Thiebes, S., Sunyaev, A.: What your radiologist might be missing: using machine learning to identify mislabeled instances of X-ray images (2021)

    Google Scholar 

  4. EU: Ethics guidelines for trustworthy AI | Shaping Europe’s digital future. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai. Accessed 15 Jan 2024

  5. Chatterjee, S., Sarker, S., Lee, M.J., Xiao, X., Elbanna, A.: A possible conceptualization of the information systems (IS) artifact: a general systems theory perspective 1. Inf. Syst. J. 31, 550–578 (2021)

    Article  Google Scholar 

  6. Sutton, S.G., Arnold, V., Holt, M.: An extension of the theory of technology dominance: capturing the underlying causal complexity. Int. J. Account. Inf. Syst. 50, 100626 (2023)

    Article  Google Scholar 

  7. Lee, J.D., See, K.A.: Trust in automation: designing for appropriate reliance. Hum. Factors 31 (2004)

    Google Scholar 

  8. Schemmer, M., Kuehl, N., Benz, C., Bartos, A., Satzger, G.: Appropriate reliance on ai advice: conceptualization and the effect of explanations. In: Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 410–422. ACM, Sydney (2023)

    Google Scholar 

  9. Passi, S., Vorvoreanu, M.: Overreliance on AI literature review. Microsoft Research (2022)

    Google Scholar 

  10. Wang, L., Jamieson, G.A., Hollands, J.G.: Selecting methods for the analysis of reliance on automation. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 52, pp. 287–291 (2008)

    Google Scholar 

  11. Dikmen, M., Burns, C.: The effects of domain knowledge on trust in explainable AI and task performance: a case of peer-to-peer lending. Int. J. Hum. Comput. Stud. 162 (2022)

    Google Scholar 

  12. Glick, A., Clayton, M., Angelov, N., Chang, J.: Impact of explainable artificial intelligence assistance on clinical decision-making of novice dental clinicians. JAMIA Open 5 (2022)

    Google Scholar 

  13. Eisenhardt, K.M.: Agency theory: an assessment and review. Acad. Manag. Rev. 14, 57–74 (1989)

    Article  Google Scholar 

  14. Scott, A., Vick, S.: Patients, doctors and contracts: an application of principal-agent theory to the doctor-patient relationship. Scott. J. Polit. Econ. 46, 111–134 (1999)

    Article  Google Scholar 

  15. Golec, J.H.: Empirical tests of a principal-agent model of the investor-investment advisor relationship. J. Financ. Quant. Anal. 27, 81–95 (1992)

    Article  Google Scholar 

  16. Miller, G.J., Whitford, A.B.: Trust and incentives in principal-agent negotiations: the ‘insurance/incentive trade-off.’ J. Theor. Polit. 14, 231–267 (2002)

    Article  Google Scholar 

  17. Chiang, C.-W., Lu, Z., Li, Z., Yin, M.: Are two heads better than one in AI-assisted decision making? Comparing the behavior and performance of groups and individuals in human-AI collaborative recidivism risk assessment. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–18. ACM, Hamburg (2023)

    Google Scholar 

  18. Heider, F.: Attitudes and cognitive organization. J. Psychol. 21, 107–112 (1946)

    Article  Google Scholar 

  19. Dzindolet, M.T., Pierce, L.G., Beck, H.P., Dawe, L.A.: Misuse and disuse of automated aids. Presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting (1999)

    Google Scholar 

  20. Parasuraman, R., Riley, V.: Humans and automation: use, misuse, disuse, abuse. Hum. Factors 39, 230–253 (1997)

    Article  Google Scholar 

  21. Chiang, C.-W., Yin, M.: Exploring the effects of machine learning literacy interventions on laypeople’s reliance on machine learning models. In: International Conference on Intelligent User Interfaces Proceedings, IUI, pp. 148–161. Association for Computing Machinery (2022)

    Google Scholar 

  22. Haight, J.M., Kecojevic, V.: Automation vs. human intervention: what is the best fit for the best performance? Process Saf. Progr. 24, 45–51 (2005)

    Google Scholar 

  23. Nourani, M., et al.: Anchoring bias affects mental model formation and user reliance in explainable AI systems. In: 26th International Conference on Intelligent User Interfaces, pp. 340–350. ACM, College Station (2021)

    Google Scholar 

  24. Schmitt, A., Wambsganss, T., Söllner, M., Janson, A.: Towards a trust reliance paradox? Exploring the gap between perceived trust in and reliance on algorithmic advice. In: International Conference on Information Systems (ICIS), Austin, Texas (2021)

    Google Scholar 

  25. Lai, V., Chen, C., Liao, Q.V., Smith-Renner, A., Tan, C.: Towards a science of human-ai decision making: a survey of empirical studies. arXiv preprint arXiv:2112.11471 (2021)

  26. Guggenberger, T., Lämmermann, L., Urbach, N., Walter, A., Hofmann, P.: Task delegation from AI to humans: a principal-agent perspective. In: ICIS 2023 Proceedings (2023)

    Google Scholar 

  27. Vössing, M., Kühl, N., Lind, M., Satzger, G.: Designing transparency for effective human-ai collaboration. Inf. Syst. Front. 24, 877–895 (2022)

    Article  Google Scholar 

  28. Heider, F.: The Psychology of Interpersonal Relations. Psychology Press (2013)

    Google Scholar 

  29. Aronson, E.: The theory of cognitive dissonance: a current perspective. In: Berkowitz, L. (ed.) Advances in Experimental Social Psychology, pp. 1–34. Academic Press (1969)

    Google Scholar 

  30. Cartwright, D., Harary, F.: Structural balance: a generalization of Heider’s theory. Psychol. Rev. 63, 277–293 (1956)

    Article  Google Scholar 

  31. Roy, S., Gammoh, B.S., Koh, A.C.: Predicting the effectiveness of celebrity endorsements using the balance theory. J. Cust. Behav. 11, 33–52 (2012)

    Article  Google Scholar 

  32. Fink, J.S., Parker, H.M., Brett, M., Higgins, J.: Off-field behavior of athletes and team identification: using social identity theory and balance theory to explain fan reactions. J. Sport Manag. 23, 142–155 (2009)

    Article  Google Scholar 

  33. Min, J.H.J., Chang, H.J.J., Jai, T.-M.C., Ziegler, M.: The effects of celebrity-brand congruence and publicity on consumer attitudes and buying behavior. Fash Text. 6, 10 (2019)

    Article  Google Scholar 

  34. Qi, L., et al.: Structural balance theory-based E-commerce recommendation over big rating data. IEEE Trans. Big Data 4, 301–312 (2018)

    Article  Google Scholar 

  35. Nakanishi, H., Nakazawa, S., Ishida, T., Takanashi, K., Isbister, K.: Can software agents influence human relations? Balance theory in agent-mediated communities. In: Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 717–724. Association for Computing Machinery, New York (2003)

    Google Scholar 

  36. Bucher, A., Dolata, M., Eckhardt, S., Staehelin, D., Schwabe, G.: Talking to multi-party conversational agents in advisory services: command-based vs. conversational interactions. Proc. ACM Hum.-Comput. Interact. 8, 1–25 (2024)

    Google Scholar 

  37. Eckhardt, S., et al.: “Garbage in, garbage out”: mitigating human biases in data entry by means of artificial intelligence. Presented at the IFIP Conference on Human-Computer Interaction (2023)

    Google Scholar 

  38. Eckhardt, S., Bucher, A., Kalunder, M., Dolata, M., Agotai, D., Schwabe, G.: Secondary mental models: introducing conversational agents in financial advisory service encounters. In: ICIS 2023 Proceedings (2023)

    Google Scholar 

  39. Gregory, R., Muntermann, J.: Theorizing in design science research: inductive versus deductive approaches (2011)

    Google Scholar 

  40. Van de Ven, A.H.: Engaged Scholarship: A Guide for Organizational and Social Research. Oxford University Press (2007)

    Google Scholar 

  41. Eckhardt, S., Sprenkamp, K., Zavolokina, L., Bauer, I., Schwabe, G.: Can artificial intelligence help used-car dealers survive in a data-driven used-car market? In: Drechsler, A., Gerber, A., Hevner, A. (eds.) DESRIST 2022. LNCS, vol. 13229, pp. 115–127. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-06516-3_9

    Chapter  Google Scholar 

  42. Sonnenberg, C., vom Brocke, J.: Evaluations in the science of the artificial – reconsidering the build-evaluate pattern in design science research. In: Peffers, K., Rothenberger, M., Kuechler, B. (eds.) DESRIST 2012. LNCS, vol. 7286, pp. 381–397. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29863-9_28

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sven Eckhardt .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Eckhardt, S., Dolata, M., Bauer-Hänsel, I., Schwabe, G. (2024). Conceptualizing Multi-party AI Reliance for Design Research. In: Mandviwalla, M., Söllner, M., Tuunanen, T. (eds) Design Science Research for a Resilient Future. DESRIST 2024. Lecture Notes in Computer Science, vol 14621. Springer, Cham. https://doi.org/10.1007/978-3-031-61175-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-61175-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-61174-2

  • Online ISBN: 978-3-031-61175-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics