Skip to main content
Springer Nature Link
Log in

Ex-Post Evaluation of Data-Driven Decisions: Conceptualizing Design Objectives

  • Conference paper
  • First Online:
Perspectives in Business Informatics Research (BIR 2022)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 462))

Included in the following conference series:

  • 2124 Accesses

Abstract

This paper addresses a need for developing ex-post evaluation for data-driven decisions resulting from collaboration between humans and machines. As a first step of a design science project, we propose four design objectives for an ex-post evaluation solution, from the perspectives of both theory (concepts from the literature) and practice (through a case of industrial production planning): (1) incorporate multi-faceted decision evaluation criteria across the levels of environment, organization, and decision itself and (2) acknowledge temporal requirements of the decision contexts at hand, (3) define applicable mode(s) of collaboration between humans and machines to pursue collaborative rationality, and (4) enable a (potentially automated) feedback loop for learning from the (discrete or continuous) evaluations of past decisions. The design objectives contribute by supporting the development of solutions for the observed lack of ex-post methods for evaluating data-driven decisions to enhance human-machine collaboration in decision making. Our future research involves design and implementation efforts through on-going industry-academia cooperation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ajzen, I.: The social psychology of decision making. In: Social Psychology: handbook of basic principles, pp. 297–325 (1996)

    Google Scholar 

  2. Argyris, C., Schön, D.A.: Organizational Learning: A Theory of Action Perspective. Addison-Wesley. 77/78, 345 (1997). https://doi.org/10.2307/40183951

  3. Bailey, D.E., Barley, S.R.: Beyond design and use: How scholars should study intelligent technologies. Inf. Organ. 30(2), 100286 (2020). https://doi.org/10.1016/j.infoandorg.2019.100286

  4. Bouyssou, D. (ed.): Evaluation and Decision Models: A Critical Perspective. Kluwer Academic Publishers, Boston (2000)

    MATH  Google Scholar 

  5. vom Brocke, J., et al.: Process Science: The Interdisciplinary Study of Continuous Change. Social Science Research Network, Rochester, NY (2021). https://doi.org/10.2139/ssrn.3916817

  6. Chiang, L., et al.: Big data analytics in chemical engineering. Ann. Rev. Chem. Biomol. Eng. 8(1), 63–85 (2017). https://doi.org/10.1146/annurev-chembioeng-060816-101555

    Article  Google Scholar 

  7. Duan, Y., et al.: Artificial intelligence for decision making in the era of Big Data – evolution, challenges and research agenda. Int. J. Inf. Manage. 48, 63–71 (2019). https://doi.org/10.1016/j.ijinfomgt.2019.01.021

    Article  Google Scholar 

  8. Dwivedi, Y.K., et al.: Artificial Intelligence (AI): multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Int. J. Inf. Manage. 57, 101994 (2021). https://doi.org/10.1016/j.ijinfomgt.2019.08.002

    Article  Google Scholar 

  9. Elgendy, N., et al.: DECAS: a modern data-driven decision theory for big data and analytics. J. Decis. Syst. 31, 1–37 (2021). https://doi.org/10.1080/12460125.2021.1894674

  10. Gigerenzer, G., Gaissmaier, W.: Decision making: nonrational theories. In: International Encyclopedia of the Social & Behavioral Sciences, pp. 911–916. Elsevier (2015). https://doi.org/10.1016/B978-0-08-097086-8.26017-0

  11. Gigerenzer, G., Gaissmaier, W.: Heuristic decision making. Ann. Rev. Psychol. 62(1), 451–482 (2011). https://doi.org/10.1146/annurev-psych-120709-145346

    Article  Google Scholar 

  12. Grønsund, T., Aanestad, M.: Augmenting the algorithm: Emerging human-in-the-loop work configurations. J. Strat. Inf. Syst. 29(2), 101614 (2020). https://doi.org/10.1016/j.jsis.2020.101614

  13. Grover, V., et al.: The perils and promises of big data research in information systems. J. Assoc. Inf. Syst. 21(2), 9 (2020). https://doi.org/10.17705/1jais.00601

  14. Herm-Stapelberg, N., Rothlauf, F.: The crowd against the few: measuring the impact of expert recommendations. Decis. Support Syst. 138, 113345 (2020). https://doi.org/10.1016/j.dss.2020.113345

    Article  Google Scholar 

  15. Hevner, A.R., et al.: Design science in information systems research. MIS Q. 28(1), 75–105 (2004). https://doi.org/10.2307/25148625

    Article  Google Scholar 

  16. Ioannidis, J.P.A., et al.: Forecasting for COVID-19 has failed. Int J Forecast. 38, 423–438 (2020). https://doi.org/10.1016/j.ijforecast.2020.08.004

    Article  Google Scholar 

  17. Klecun, E., Cornford, T.: A critical approach to evaluation. Eur. J. Inf. Syst. 14(3), 229–243 (2005). https://doi.org/10.1057/palgrave.ejis.3000540

    Article  Google Scholar 

  18. Kotsiantis, S.B., et al.: Machine learning: a review of classification and combining techniques. Artif. Intell. Rev. 26(3), 159–190 (2007). https://doi.org/10.1007/s10462-007-9052-3

    Article  MathSciNet  MATH  Google Scholar 

  19. Lebovitz, S. et al.: Is AI ground truth really true? The dangers of training and evaluating AI tools based on experts’ know-what. MIS Q. 45(3), 1501–1526 (2021). https://doi.org/10.25300/MISQ/2021/16564

  20. Lyytinen, K., et al.: Metahuman systems = humans + machines that learn. J. Inf. Technol. 36(4), 427–445 (2020). https://doi.org/10.1177/0268396220915917

    Article  Google Scholar 

  21. Magrabi, F., et al.: Artificial intelligence in clinical decision support: challenges for evaluating AI and practical implications: a position paper from the IMIA technology assessment & quality development in health informatics working group and the EFMI working group for assessment of health information systems. Yearb Med. Inform. 28(01), 128–134 (2019). https://doi.org/10.1055/s-0039-1677903

    Article  Google Scholar 

  22. March, J.G.: Bounded Rationality, Ambiguity, and the Engineering of Choice. The Bell Journal of Economics. 9(2), 587–608 (1978). https://doi.org/10.2307/3003600

    Article  Google Scholar 

  23. March, J.G.: Primer on Decision Making: How Decisions Happen. Simon and Schuster (1994)

    Google Scholar 

  24. Masha, E.M.: The case for data driven strategic decision making. Eur. J. Bus. Manage. 6, 137–146 (2014)

    Google Scholar 

  25. Namvar, M., Intezari, A.: Wise data-driven decision-making. In: Dennehy, D., Griva, A., Pouloudi, N., Dwivedi, Y.K., Pappas, I., Mäntymäki, M. (eds.) Responsible AI and Analytics for an Ethical and Inclusive Digitized Society. LNCS, vol. 12896, pp. 109–119. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85447-8_10

    Chapter  Google Scholar 

  26. Nasir, M., et al.: Developing a decision support system to detect material weaknesses in internal control. Decis. Support Syst. 151, 113631 (2021). https://doi.org/10.1016/j.dss.2021.113631

    Article  Google Scholar 

  27. Nutt, P.C., Wilson, D.C.: Handbook of Decision Making. John Wiley, Chichester (2010)

    Google Scholar 

  28. Peffers, K., et al.: A design science research methodology for information systems research. J. Manag. Inf. Syst. 24(3), 45–77 (2007). https://doi.org/10.2753/MIS0742-1222240302

    Article  Google Scholar 

  29. Peffers, K., Rothenberger, M., Kuechler, B. (eds.): Design Science Research in Information Systems. Advances in Theory and Practice. LNCS, vol. 7286. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29863-9

    Book  Google Scholar 

  30. Pettigrew, A.M.: Contextualist research and the study of organizational change processes. Res. Inf. Syst. 1, 53–78 (1985)

    Google Scholar 

  31. Phillips-Wren, G., et al.: Cognitive bias, decision styles, and risk attitudes in decision making and DSS. J. Decis. Syst. 28(2), 63–66 (2019). https://doi.org/10.1080/12460125.2019.1646509

    Article  Google Scholar 

  32. Power, D.J., et al.: Analytics, bias, and evidence: the quest for rational decision making. J. Decis. Syst. 28(2), 120–137 (2019). https://doi.org/10.1080/12460125.2019.1623534

    Article  Google Scholar 

  33. Ransbotham, S. et al.: Expanding AI’s Impact With Organizational Learning. https://sloanreview.mit.edu/projects/expanding-ais-impact-with-organizational-learning/. Accessed 22 Dec 2021

  34. Raschka, S.: Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning. http://arxiv.org/abs/1811.12808 (2020)

  35. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson, Hoboken (2021)

    MATH  Google Scholar 

  36. Shrestha, Y.R., et al.: Organizational decision-making structures in the age of artificial intelligence. Calif. Manage. Rev. 61(4), 66–83 (2019). https://doi.org/10.1177/0008125619862257

    Article  Google Scholar 

  37. Simon, H.A.: A behavioral model of rational choice. Q. J. Econ. 69(1), 99 (1955). https://doi.org/10.2307/1884852

  38. Smith, J.A.: Qualitative Psychology: A Practical Guide to Research Methods. SAGE, Thousand Oaks (2015)

    Google Scholar 

  39. Snowden, D.J., Boone, M.E.: A leader’s framework for decision making. Harv. Bus. Rev. 85, 68 (2007)

    Google Scholar 

  40. Sonnenberg, C., vom Brocke, J.: Evaluations in the science of the artificial – reconsidering the build-evaluate pattern in design science research. In: Peffers, K., Rothenberger, M., Kuechler, B. (eds.) Design Science Research in Information Systems. Advances in Theory and Practice. LNCS, vol. 7286, pp. 381–397. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29863-9_28

    Chapter  Google Scholar 

  41. Stockdale, R., Standing, C.: An interpretive approach to evaluating information systems: a content, context, process framework. Eur. J. Oper. Res. 173(3), 1090–1102 (2006). https://doi.org/10.1016/j.ejor.2005.07.006

    Article  MATH  Google Scholar 

  42. Sturm, T., et al.: Coordinating human and machine learning for effective organization learning. MISQ. 45(3), 1581–1602 (2021). https://doi.org/10.25300/MISQ/2021/16543

  43. Sturm, T., et al.: The Case of Human-Machine Trading as Bilateral Organizational Learning. 18 (2021)

    Google Scholar 

  44. Tomperi, J., et al.: Mass-balance based soft sensor for monitoring ash content at two-ply paperboard manufacturing. Nord. Pulp Pap. Res. J. 37(1), 175–183 (2022). https://doi.org/10.1515/npprj-2021-0046

    Article  Google Scholar 

  45. Troisi, O., et al.: Growth hacking: Insights on data-driven decision-making from three firms. Ind. Mark. Manage. 90, 538–557 (2020). https://doi.org/10.1016/j.indmarman.2019.08.005

    Article  Google Scholar 

  46. Trunk, A., Birkel, H., Hartmann, E.: On the current state of combining human and artificial intelligence for strategic organizational decision making. Bus. Res. 13(3), 875–919 (2020). https://doi.org/10.1007/s40685-020-00133-x

    Article  Google Scholar 

  47. Tversky, A., Kahneman, D.: Rational choice and the framing of decisions. J. Business. 59(4), S251–S278 (1986)

    Article  Google Scholar 

  48. Tversky, A., Kahneman, D.: The framing of decisions and the psychology of choice. Science 211(4481), 453–458 (1981). https://doi.org/10.1126/science.7455683

    Article  MathSciNet  MATH  Google Scholar 

  49. Vo, N.N.Y., et al.: Deep learning for decision making and the optimization of socially responsible investments and portfolio. Decis. Support Syst. 124, 113097 (2019). https://doi.org/10.1016/j.dss.2019.113097

    Article  Google Scholar 

  50. van Voorst, S., Zwaan, P.: The (non-)use of ex post legislative evaluations by the European commission. J. Eur. Publ. Policy 26(3), 366–385 (2019). https://doi.org/10.1080/13501763.2018.1449235

    Article  Google Scholar 

  51. Weick, K.E.: Sensemaking in Organizations. Sage Publications, Thousand Oaks (1995)

    Google Scholar 

  52. Weirich, P.: Realistic Decision Theory: Rules for Nonideal Agents in Nonideal Circumstances. Oxford University Press, Oxford (2004)

    Book  Google Scholar 

Download references

Acknowledgment

This research has been partially funded by the ITEA3 project Oxilate (https://itea3.org/project/oxilate.html).

Author information

Authors and Affiliations

  1. M3S, Faculty of Information Technology and Electrical Engineering, University of Oulu, Oulu, Finland

    Nada Elgendy & Tero Päivärinta

  2. Department of Computer Science, Electrical and Space Engineering, Lulea University of Technology, Lulea, Sweden

    Ahmed Elragal

  3. Environmental and Chemical Engineering, Faculty of Technology, University of Oulu, Oulu, Finland

    Markku Ohenoja

Authors
  1. Nada Elgendy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmed Elragal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Markku Ohenoja
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tero Päivärinta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nada Elgendy .

Editor information

Editors and Affiliations

  1. Riga Technical University, Riga, Latvia

    Ērika Nazaruka

  2. Rostock University, Rostock, Germany

    Kurt Sandkuhl

  3. Jönköping University, Jönköping, Sweden

    Ulf Seigerroth

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Elgendy, N., Elragal, A., Ohenoja, M., Päivärinta, T. (2022). Ex-Post Evaluation of Data-Driven Decisions: Conceptualizing Design Objectives. In: Nazaruka, Ē., Sandkuhl, K., Seigerroth, U. (eds) Perspectives in Business Informatics Research. BIR 2022. Lecture Notes in Business Information Processing, vol 462. Springer, Cham. https://doi.org/10.1007/978-3-031-16947-2_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16947-2_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16946-5

  • Online ISBN: 978-3-031-16947-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics