Skip to main content
Springer Nature Link
Log in

Operational Collective Intelligence of Humans and Machines

  • Conference paper
  • First Online:
Human Interface and the Management of Information (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14691))

Included in the following conference series:

Abstract

We explore the use of aggregative crowdsourced forecasting (ACF) [2, 42] as a mechanism to help operationalize “collective intelligence” of human-machine teams for coordinated actions. We adopt the definition for Collective Intelligence as: “A property of groups that emerges from synergies among data-information-knowledge, software-hardware, and individuals (those with new insights as well as recognized authorities) that enables just-in-time knowledge for better decisions than these three elements acting alone.” [52] Collective Intelligence emerges from new ways of connecting humans and AI to enable decision-advantage, in part by creating and leveraging additional sources of information that might otherwise not be included. Aggregative crowdsourced forecasting (ACF) is a recent key advancement towards Collective Intelligence wherein predictions (X% probability that Y will happen) and rationales (why I believe it is this probability that X will happen) are elicited independently from a diverse crowd, aggregated, and then used to inform higher-level decision-making. This research asks whether ACF, as a key way to enable Operational Collective Intelligence, could be brought to bear on operational scenarios (i.e., sequences of events with defined agents, components, and interactions) and decision-making, and considers whether such a capability could provide novel operational capabilities to enable new forms of decision-advantage.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Atanasov, P., et al.: Distilling the wisdom of crowds: prediction markets vs. prediction polls. Manag. Sci. 63(3), 691–706 (2017)

    Google Scholar 

  2. Benjamin, D.M., et al.: Hybrid forecasting of geopolitical events. AI Mag. (2023)

    Google Scholar 

  3. Bollier, D., Firestone, C.M., et al.: The promise and peril of big data. Aspen Institute, Communications and Society Program Washington, DC (2010)

    Google Scholar 

  4. Brynjolfsson, E., Mitchell, T.: What can machine learning do? Workforce implications. Science 358(6370), 1530–1534 (2017)

    Article  Google Scholar 

  5. Budach, L., et al.: The effects of data quality on machine learning performance. arXiv preprint arXiv:2207.14529 (2022)

  6. Budescu, D.V., Chen, E.: Identifying expertise to extract the wisdom of crowds. Manage. Sci. 61(2), 267–280 (2015)

    Article  Google Scholar 

  7. Budescu, D.V., Fiedler, K., et al.: Confidence in aggregation of opinions from multiple sources. In: Information Sampling and Adaptive Cognition, pp. 327–352 (2006)

    Google Scholar 

  8. Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  9. Da, Z., Huang, X.: Harnessing the wisdom of crowds. Manage. Sci. 66(5), 1847–1867 (2020)

    Article  Google Scholar 

  10. Dawes, R.M., Faust, D., Meehl, P.E.: Clinical versus actuarial judgment. Science 243(4899), 1668–1674 (1989)

    Article  Google Scholar 

  11. Dellermann, D., Ebel, P., Söllner, M., Leimeister, J.M.: Hybrid intelligence. Bus. Inf. Syst. Eng. 61, 637–643 (2019)

    Article  Google Scholar 

  12. Dietvorst, B.J., Simmons, J.P., Massey, C.: Algorithm aversion: people erroneously avoid algorithms after seeing them err. J. Exp. Psychol. Gen. 144(1), 114 (2015)

    Article  Google Scholar 

  13. Dong, L., Zheng, H., Li, L., Hao, L.: Human-machine hybrid prediction market: a promising sales forecasting solution for e-commerce enterprises. Electron. Commer. Res. Appl. 56, 101216 (2022)

    Article  Google Scholar 

  14. Galán, J.J., Carrasco, R.A., LaTorre, A.: Military applications of machine learning: a bibliometric perspective. Mathematics 10(9), 1397 (2022)

    Article  Google Scholar 

  15. Galton, F.: Vox populi. Nature 75(1949), 450–451 (1907)

    Article  Google Scholar 

  16. Garcez, A.D., et al.: Neural-symbolic learning and reasoning: a survey and interpretation. In: Neuro-Symbolic Artificial Intelligence: The State of the Art, vol. 342, no. 1, p. 327 (2022)

    Google Scholar 

  17. Goldstein, S.: December 2015. https://www.iarpa.gov/research-programs/hfc

  18. Gurney, N., Pynadath, D.V., Wang, N.: Measuring and predicting human trust in recommendations from an AI teammate. In: Degen, H., Ntoa, S. (eds.) HCII 2022. LNCS, vol. 13336, pp. 22–34. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05643-7_2

  19. Gurney, N., Pynadath, D.V., Wang, N.: Comparing psychometric and behavioral predictors of compliance during human-AI interactions. In: Meschtscherjakov, A., Midden, C., Ham, J. (eds) PERSUASIVE 2023, vol. 13832, pp. 175–197. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30933-5_12

  20. Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., Bing, G.: Learning from class-imbalanced data: review of methods and applications. Expert Syst. Appl. 73, 220–239 (2017)

    Article  Google Scholar 

  21. Hassani, H., Silva, E.S.: Forecasting with big data: a review. Ann. Data Sci. 2, 5–19 (2015)

    Article  Google Scholar 

  22. Heinrich, B., Hristova, D., Klier, M., Schiller, A., Szubartowicz, M.: Requirements for data quality metrics. J. Data Inf. Qual. (JDIQ) 9(2), 1–32 (2018)

    Google Scholar 

  23. Huber, D.J., et al.: MATRICS: a system for human-machine hybrid forecasting of geopolitical events. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 2028–2032. IEEE (2019)

    Google Scholar 

  24. Iandoli, L., Klein, M., Zollo, G.: Enabling on-line deliberation and collective decision-making through large-scale argumentation: a new approach to the design of an internet-based mass collaboration platform. Int. J. Decis. Support Syst. Technol. (IJDSST) 1(1), 69–92 (2009)

    Article  Google Scholar 

  25. Jordan, M.I., Mitchell, T.M.: Machine learning: trends, perspectives, and prospects. Science 349(6245), 255–260 (2015)

    Article  MathSciNet  Google Scholar 

  26. Kamar, E., Hacker, S., Horvitz, E.: Combining human and machine intelligence in large-scale crowdsourcing. In: AAMAS, vol. 12, pp. 467–474 (2012)

    Google Scholar 

  27. Kameda, T., Toyokawa, W., Tindale, R.S.: Information aggregation and collective intelligence beyond the wisdom of crowds. Nat. Rev. Psychol. 1(6), 345–357 (2022)

    Article  Google Scholar 

  28. Kott, A., Ownby, M.: Toward a research agenda in adversarial reasoning: computational approaches to anticipating the opponent’s intent and actions. arXiv preprint arXiv:1512.07943 (2015)

  29. Kurvers, R.H., Nuzzolese, A.G., Russo, A., Barabucci, G., Herzog, S.M., Trianni, V.: Automating hybrid collective intelligence in open-ended medical diagnostics. Proc. Natl. Acad. Sci. 120(34), e2221473120 (2023)

    Article  Google Scholar 

  30. Landemore, H.: Collective wisdom: old and new. In: Collective Wisdom: Principles and Mechanisms, vol. 1, pp. 1–20 (2012)

    Google Scholar 

  31. Leigh, A., Wolfers, J.: Competing approaches to forecasting elections: economic models, opinion polling and prediction markets. Econ. Rec. 82(258), 325–340 (2006)

    Article  Google Scholar 

  32. Levy, P., Bononno, R.: Collective Intelligence: Mankind’s Emerging World in Cyberspace. Perseus Books, USA (1997)

    Google Scholar 

  33. Li, H., Liu, Q.: Cheaper and better: selecting good workers for crowdsourcing. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 3, pp. 20–21 (2015)

    Google Scholar 

  34. Li, H., Zhao, B., Fuxman, A.: The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing. In: Proceedings of the 23rd International Conference on World Wide Web, pp. 165–176 (2014)

    Google Scholar 

  35. Liu, J., et al.: Towards out-of-distribution generalization: a survey. arXiv preprint arXiv:2108.13624 (2021)

  36. Logg, J.M., Minson, J.A., Moore, D.A.: Algorithm appreciation: people prefer algorithmic to human judgment. Organ. Behav. Hum. Decis. Process. 151, 90–103 (2019)

    Article  Google Scholar 

  37. Lorenz, J., Rauhut, H., Schweitzer, F., Helbing, D.: How social influence can undermine the wisdom of crowd effect. Proc. Natl. Acad. Sci. 108(22), 9020–9025 (2011)

    Article  Google Scholar 

  38. Malone, T.W., Laubacher, R., Dellarocas, C.: The collective intelligence genome. MIT Sloan Manag. Rev. (2010)

    Google Scholar 

  39. Mannes, A.E., Soll, J.B., Larrick, R.P.: The wisdom of select crowds. J. Pers. Soc. Psychol. 107(2), 276 (2014)

    Article  Google Scholar 

  40. Mellers, B., et al.: Psychological strategies for winning a geopolitical forecasting tournament. Psychol. Sci. 25(5), 1106–1115 (2014)

    Article  Google Scholar 

  41. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Google Scholar 

  42. Morstatter, F., et al.: SAGE: a hybrid geopolitical event forecasting system. In: IJCAI, vol. 1, pp. 6557–6559 (2019)

    Google Scholar 

  43. Parasuraman, R., Riley, V.: Humans and automation: use, misuse, disuse, abuse. Hum. Fact. 39(2), 230–253 (1997)

    Article  Google Scholar 

  44. Peled, A.: The politics of big data: a three-level analysis. In: European Consortium of Political Research (ECPR) General Conference, Bordeaux, France (2013)

    Google Scholar 

  45. Pencheva, I., Esteve, M., Mikhaylov, S.J.: Big data and AI-a transformational shift for government: so, what next for research? Public Policy Adm. 35(1), 24–44 (2020)

    Google Scholar 

  46. Pynadath, D.V., Gurney, N., Wang, N.: Explainable reinforcement learning in human-robot teams: the impact of decision-tree explanations on transparency. In: 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 749–756. IEEE (2022)

    Google Scholar 

  47. Rafner, J., et al.: Revisiting citizen science through the lens of hybrid intelligence. arXiv preprint arXiv:2104.14961 (2021)

  48. Ratner, B.: Statistical and Machine-Learning Data Mining: Techniques for Better Predictive Modeling and Analysis of Big Data. CRC Press (2017)

    Google Scholar 

  49. Russakovsky, O., Li, L.J., Fei-Fei, L.: Best of both worlds: human-machine collaboration for object annotation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2121–2131 (2015)

    Google Scholar 

  50. Shoeibi, A., et al.: Automated detection and forecasting of covid-19 using deep learning techniques: a review. Neurocomputing, 127317 (2024)

    Google Scholar 

  51. Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: 2010 IEEE Symposium on Security and Privacy, pp. 305–316. IEEE (2010)

    Google Scholar 

  52. Suran, S., Pattanaik, V., Draheim, D.: Frameworks for collective intelligence: a systematic literature review. ACM Comput. Surv. (CSUR) 53(1), 1–36 (2020)

    Article  Google Scholar 

  53. Surowiecki, J.: The Wisdom of Crowds. Anchor (2005)

    Google Scholar 

  54. Svenmarck, P., Luotsinen, L., Nilsson, M., Schubert, J.: Possibilities and challenges for artificial intelligence in military applications. In: Proceedings of the NATO Big Data and Artificial Intelligence for Military Decision Making Specialists’ Meeting, pp. 1–16 (2018)

    Google Scholar 

  55. Wang, N., Pynadath, D.V., Hill, S.G.: Trust calibration within a human-robot team: comparing automatically generated explanations. In: 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 109–116. IEEE (2016)

    Google Scholar 

  56. Wang, X., Hyndman, R.J., Li, F., Kang, Y.: Forecast combinations: an over 50-year review. Int. J. Forecast. 39(4), 1518–1547 (2023)

    Article  Google Scholar 

  57. Welinder, P., Branson, S., Perona, P., Belongie, S.: The multidimensional wisdom of crowds. In: Advances in Neural Information Processing Systems, vol. 23 (2010)

    Google Scholar 

  58. Wu, Y., Ma, L., Yuan, X., Li, Q.: Human-machine hybrid intelligence for the generation of car frontal forms. Adv. Eng. Inform. 55, 101906 (2023)

    Article  Google Scholar 

  59. Zhang, Y., Liao, Q.V., Bellamy, R.K.: Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 295–305 (2020)

    Google Scholar 

Download references

Acknowledgments

The project or effort depicted was or is sponsored by the U.S. Government under contract number W911NF-14-D-0005. The content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Author information

Authors and Affiliations

  1. Institute for Creative Technologies, University of Southern California, Los Angeles, 90094, USA

    Nikolos Gurney & David V. Pynadath

  2. Computer Science Department, University of Southern California, Los Angeles, 90007, USA

    David V. Pynadath

  3. Information Sciences Institute, University of Southern California, Los Angeles, 90292, USA

    Fred Morstatter, Adam Russell & Gleb Satyukov

Authors
  1. Nikolos Gurney
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fred Morstatter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David V. Pynadath
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adam Russell
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gleb Satyukov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nikolos Gurney .

Editor information

Editors and Affiliations

  1. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  2. Tokyo University of Science, Tokyo, Japan

    Yumi Asahi

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gurney, N., Morstatter, F., Pynadath, D.V., Russell, A., Satyukov, G. (2024). Operational Collective Intelligence of Humans and Machines. In: Mori, H., Asahi, Y. (eds) Human Interface and the Management of Information. HCII 2024. Lecture Notes in Computer Science, vol 14691. Springer, Cham. https://doi.org/10.1007/978-3-031-60125-5_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-60125-5_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-60124-8

  • Online ISBN: 978-3-031-60125-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics