Skip to main content
SpringerLink
Log in

DecisionMind: revealing human cognition states in data analytics-driven decision making with a multimodal interface

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Despite the recognized value of machine learning (ML) techniques and high expectation of applying ML techniques within various applications, significant barriers to widespread adoption and local implementation of ML approaches still exist in the areas of trust (of ML results), comprehension (of ML processes) and related workload, as well as confidence (in decision making based on ML results) by users. This paper argues that the revealing of human cognition states with a multimodal interface during ML-based data analytics-driven decision making could provide a rich view for both ML researchers and domain experts to learn the effectiveness of ML technologies in applications. On the one hand, human cognition states could help understand to what degree users accept innovative technologies. On the other hand, through understanding human cognition states during data analytics-driven decision making, ML-based decision attributes and even ML models can be adaptively refined in order to make ML transparent. The paper also identifies examples of impact challenges and obstacles, as well as high-demand research directions in making ML transparent.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Zhou J, Khawaja MA, Li Z et al (2016) Making machine learning useable by revealing internal states update—a transparent approach. Int J Comput Sci Eng 13:378–389

    Google Scholar 

  2. Zhou J, Bridon C, Chen F et al (2015) Be informed and be involved: effects of uncertainty and correlation on user’s confidence in decision making. In: Proceedings of ACM SIGCHI conference on human factors in computing systems (CHI2015) works-in-progress

  3. Zhou J, Sun J, Chen F et al (2015) Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Trans Comput Hum Interact 21:33

    Article  Google Scholar 

  4. Helgee EA (2010) Improving drug discovery decision making using machine learning and graph theory in QSAR modeling. Ph.D. thesis, University of Gothenburg

  5. Wang W, Li Z, Wang Y, Chen F (2013) Indexing cognitive workload based on pupillary response under luminance and emotional changes. In: Proceedings of the 2013 international conference on intelligent user interfaces. ACM, New York, NY, USA, pp 247–256

  6. Holzinger A (2013) Human–computer interaction and knowledge discovery (HCI-KDD): What is the benefit of bringing those two fields to work together? In: Cuzzocrea A, Kittl C, Simos DE et al (eds) Availability, reliability, and security in information systems and HCI. Springer, Berlin, pp 319–328

    Chapter  Google Scholar 

  7. Khawaji A, Chen F, Marcus N, Zhou J (2013) Trust and cooperation in text-based computer-mediated communication. In: Proceedings of the 25th Australian computer–human interaction conference: augmentation, application, innovation, collaboration. ACM, New York, NY, USA, pp 37–40

  8. Zhou J, Chen F (2015) Making machine learning useable. Int J Intell Syst Technol Appl 14:91. doi:10.1504/IJISTA.2015.074069

    Google Scholar 

  9. Moll J, De Oliveira-Souza R, Zahn R (2008) The neural basis of moral cognition: sentiments, concepts, and values. Ann N Y Acad Sci 1124:161–180. doi:10.1196/annals.1440.005

    Article  Google Scholar 

  10. van Gog T, Paas F, Marcus N et al (2008) The mirror neuron system and observational learning: implications for the effectiveness of dynamic visualizations. Educ Psychol Rev 21:21–30. doi:10.1007/s10648-008-9094-3

    Google Scholar 

  11. Ayres P, Paas F (2009) Interdisciplinary perspectives inspiring a new generation of cognitive load research. Educ Psychol Rev 21:1–9. doi:10.1007/s10648-008-9090-7

    Article  Google Scholar 

  12. Chen F, Zhou J, Wang Y et al (2016) Robust multimodal cognitive load measurement. Springer, Berlin

    Book  Google Scholar 

  13. Krueger F, McCabe K, Moll J et al (2007) Neural correlates of trust. PNAS 104:20084–20089. doi:10.1073/pnas.0710103104

    Article  Google Scholar 

  14. Aimone JA, Houser D, Weber B (2014) Neural signatures of betrayal aversion: an fMRI study of trust. Proc Biol Sci 281:20132127. doi:10.1098/rspb.2013.2127

    Article  Google Scholar 

  15. Hahn T, Notebaert K, Anderl C et al (2015) How to trust a perfect stranger: predicting initial trust behavior from resting-state brain-electrical connectivity. Soc Cogn Affect Neurosci 10:809–813. doi:10.1093/scan/nsu122

    Article  Google Scholar 

  16. Heekeren HR, Marrett S, Ungerleider LG (2008) The neural systems that mediate human perceptual decision making. Nat Rev Neurosci 9:467–479. doi:10.1038/nrn2374

    Article  Google Scholar 

  17. Smith LJ, Anand P, Benattayallah A, Hodgson TL (2015) An fMRI investigation of moral cognition in healthcare decision making. J Neurosci Psychol Econ 8:116–133. doi:10.1037/npe0000038

    Article  Google Scholar 

  18. White TP, Engen NH, Sørensen S et al (2014) Uncertainty and confidence from the triple-network perspective: voxel-based meta-analyses. Brain Cogn 85:191–200. doi:10.1016/j.bandc.2013.12.002

  19. Franco-Watkins AM, Johnson JG (2011) Applying the decision moving window to risky choice: comparison of eye-tracking and mouse-tracing methods. Judgm Decis Mak 6:740–749

  20. Schall JD (2001) Neural basis of deciding, choosing and acting. Nat Rev Neurosci 2:33–42

    Article  Google Scholar 

  21. Guetl C, Pivec M, Trummer C et al (2005) ADELE (adaptive e-learning with eye-tracking): theoretical background, system architecture and application scenarios. Eur J Open Distance E Learn 2: 1–8

  22. Arshad S, Wang Y, Chen F (2015) Interactive mouse stream as real-time indicator of user’s cognitive load. In: Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems, pp 1025–1030

  23. Khawaja MA, Chen F, Marcus N (2014) Measuring cognitive load using linguistic features: implications for usability evaluation and adaptive interaction design. Int J Hum Comput Interact 30:343–368. doi:10.1080/10447318.2013.860579

    Article  Google Scholar 

  24. Khawaji A, Chen F, Zhou J, Marcus N (2014) Trust and cognitive load in the text-chat environment: the role of mouse movement. In: Proceedings of the 26th Australian computer–human interaction conference on designing futures: the future of design, pp 324–327

  25. Jenkins Q, Jiang X (2010) Measuring trust and application of eye tracking in human robotic interaction. In: Proceedings of the 2010 industrial engineering research conference

  26. Preuschoff K, Hart BM, Einhäuser W (2011) Pupil dilation signals surprise: evidence for noradrenaline’s role in decision making. Front Neurosci 5:115. doi:10.3389/fnins.2011.00115

    Article  Google Scholar 

  27. Fiedler S, Glockner A (2012) The dynamics of decision making in risky choice: an eye-tracking analysis. Front Psychol 3:1–18

    Article  Google Scholar 

  28. Li Z, Zhang B, Wang Y et al (2014) Water pipe condition assessment: a hierarchical beta process approach for sparse incident data. Mach Learn 95:11–26

    Article  MathSciNet  Google Scholar 

  29. Peper E, Harvey R, Lin I-M et al (2007) Is there more to blood volume pulse than heart rate variability respiratory sinus arrhythmia, and cardiorespiratory synchrony? Biofeedback 35:54–61

    Google Scholar 

  30. Oppenheim AV, Schafer RW (2010) Discrete-time signal processing. Pearson, Upper Saddle River

    MATH  Google Scholar 

  31. Healey J, Picard R (2000) SmartCar: detecting driver stress. In: Proceedings of 15th international conference on pattern recognition 2000. pp 218–221

  32. Zhai J, Barreto AB, Chin C, Li C (2005) Realization of stress detection using psychophysiological signals for improvement of human–computer interactions. Proc IEEE SoutheastCon 2005:415–420

    Google Scholar 

  33. Kristal-Boneh E, Raifel M, Froom P, Ribak J (1995) Heart rate variability in health and disease. Scand J Work Environ Health 21:85–95

    Article  Google Scholar 

  34. Karvonen MJ, Kentala E, Mustala O (1957) The effects of training on heart rate: a longitudinal study. Ann Med Exp Biol Fenn 35:307–315

    Google Scholar 

  35. Tanaka H, Monahan KD, Seals DR (2001) Age-predicted maximal heart rate revisited. J Am Coll Cardiol 37:153–156

    Article  Google Scholar 

Download references

Acknowledgements

Authors thank all volunteer participants for the experiment. This work was supported in part by the Asian Office of Aerospace Research & Development (AOARD) under grant No. FA2386-14-1-0022 AOARD 134131.

Author information

Authors and Affiliations

  1. DATA61, CSIRO, Australia, 13 Garden Street, Eveleigh, NSW, 2015, Australia

    Jianlong Zhou & Fang Chen

Authors
  1. Jianlong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianlong Zhou.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, J., Chen, F. DecisionMind: revealing human cognition states in data analytics-driven decision making with a multimodal interface. J Multimodal User Interfaces 12, 67–76 (2018). https://doi.org/10.1007/s12193-017-0249-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-017-0249-8

Keywords

Search

Navigation