Skip to main content
Springer Nature Link
Log in

Augmenting human senses to improve the user experience in cars: applying augmented reality and haptics approaches to reduce cognitive distances

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Augmenting people’s senses with computational support can improve the ability to perceive information and perform tasks. However, the impact of such augmentation may fluctuate according to user context, thereby impacting the quality of a user experience. In this paper, we present two systems that assess the in-situ effects of augmenting senses using Augmented Reality and Haptic technologies. We demonstrate that sensory augmentation systems can improve performance when users are multitasking; however, a hybrid assessment, including eye tracking and psycho-physiological measurement, reveals that the benefits and costs of such systems can differ depending on the demographics of a population with different cognitive capabilities. For elder adults, sensory augmentation improved perception for responding to local incidents in the physical space, but a richer intervention using sensory augmentation (visual, auditory, and haptic) strains cognitive load. For younger adults, additional modes for providing sensory information increased attentiveness for performing tasks, but can lead to overloading of already used sensory channels. Thus, sensory augmentation was more advantageous for improving global awareness for situated physical space, rather than responding to local incidents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Anderson JR (2002) Spanning seven orders of magnitude: a challenge for cognitive modeling. Cogn Sci 26(1):85–112

    Article  Google Scholar 

  2. Canham M, Hegarty M (2010) Effects of knowledge and display design on comprehension of complex graphics. Learn Instr 20(2):155–166

    Article  Google Scholar 

  3. Carton A, Dunne LE (2013) Tactile distance feedback for firefighters: design and preliminary evaluation of a sensory augmentation glove. In Proceedings of the 4th Augmented Human International Conference (AH’13). ACM, New York, pp 58–64

    Google Scholar 

  4. Cegarra J, Chevalier A (2008) The use of tholos software for combining measures of mental workload: toward theoretical and methodological improvements. Behav Res Methods 40(4):988–1000

    Article  Google Scholar 

  5. Chandler P, Sweller J (1991) Cognitive load theory and the format of instruction. Cogn Instr 8(4):293–332

    Article  Google Scholar 

  6. Chandler P, Sweller J (1992) The split-attention effect as a factor in the design of instruction. Br J Educ Psychol 62(2):233–246

    Article  Google Scholar 

  7. Chang W, Hwang W, Ji YG (2011) Haptic seat interfaces for driver information and warning systems. Int J Hum Comput Interact 27:1119–1132

    Article  Google Scholar 

  8. Clive-smith M (2013) Eidos: Audio/Visual Sensory Augmentation. Media Evolution, The Conference Wearable Technology

  9. Collins CC (1985) On mobility aids for the blind. In: Warren DH, Strelow ER (eds) Electronic spatial sensing for the blind. Martinus Nijhoff, Boston, pp 35–64

    Chapter  Google Scholar 

  10. Conati C, Maclaren H (2009) Empirically building and evaluating a probabilistic model of user affect. User Model User-Adap Inter 19:267–303

    Article  Google Scholar 

  11. Cook AM (1982) Sensory and communication aids. In: Cook AM, Webster JG (eds) Therapeutic Medical Devices: Application and Design. Prentice-Hall, Englewood Cliffs, pp 152–201

    Google Scholar 

  12. Coyne JT, Baldwin C, Cole A, Sibley C, Roberts DM (2009) Applying Real Time Physiological Measures of Cognitive Load to Improve Training. HCI 16, of Lecture Notes in Computer Science, vol 5638. Springer, 469–478

  13. D’Mello S, Olney A, Williams C, Hays P (2012) Gaze tutor: a gaze-reactive intelligent tutoring system. Int J Hum Comput Stud 70(5):370–389, Elsevier

    Google Scholar 

  14. De Waard D (1996) The measurement of drivers’ mental workload. PhD Thesis, University of Groningen

  15. Fisk AD, Derrick WL, Schneider W (1986–87). A methodological assessment and evaluation of dual-task paradigms. Curr Psychol 5(4): 315–327

  16. Goldberg JH, Stimson MJ, Lewenstein M, Scott N, Wichansky AM (2002) Eye tracking in web search tasks: design implications, In Proceedings of the 2002 symposium on Eye tracking research & applications (ETRA’02). ACM, New York, pp 51–58

    Google Scholar 

  17. Haapalainen E, Kim S, Forlizzi J, Dey A (2010) Psycho-physiological Measures for assessing Cognitive Load. In: Proc. Ubicomp. 301–310

  18. Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human Mental Workload. Amsterdam, North-Holland, pp 139–183

    Chapter  Google Scholar 

  19. Hwang S, Ryu J (2010) The Haptic steering Wheel: Vibro-tactile based navigation for the driving environment. In: IEEE Intl’ Conf. PERCOM Workshops. 660–665.

  20. Iqbal ST, Zheng XS, Bailey BP (2004) Task evoked pupillary response to mental workload in human-computer interaction. In: Proceedings of the ACM conference on human factors in computing systems. 1477–1480

  21. Javal E (1879) Essai sur la Physiologie de la Lecture Annales D’Oculistique

  22. Kaczmarek KA (1995) Sensory augmentation and substitution. In: Bronzino JD (ed) CRC handbook of biomedical engineering. CRC Press, Boca Raton, pp 2100–2109

    Google Scholar 

  23. Kahneman D (1973) Attention and Effort. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  24. Kalyuga S (2007) Expertise reversal effect and its implications for learner-tailored instruction. Educ Psychol Rev 19:509–539

    Article  Google Scholar 

  25. Kalyuga S, Rikers R, Pass F (2012) Educational implications of expertise reversal effect in learning and performance of complex cognitive and sensorimotor skills. Educ Psychol Rev 24:313–337

    Article  Google Scholar 

  26. Kern D, Marshall P, Hornecker E, Rogers Y, Schmidt A (2009) Enhancing Navigation Information with Tactile Output Embedded into the Steering Wheel. In: Tokuda H, Beigl M, Friday A, Brush AJB, Tobe Y (eds) Pervasive 2009. LNCS, vol 5538. Springer, Heidelberg, pp 42–58

    Google Scholar 

  27. Kim S, Dey AK (2009) Simulated Augmented Reality Windshield Display as a Cognitive Mapping Aid for Elder Driver Navigation. In: Proc. CHI. 133–142

  28. Kim S, Aleven V, Dey AK (2014) Understanding expert-novice differences in geometry problem-solving tasks: a sensor-based approach, In CHI’14 Extended Abstracts on Human Factors in Computing Systems (CHI EA’14). ACM, New York, pp 1867–1872

    Google Scholar 

  29. Kim S, Dey AK, Lee J, Forlizzi J (2011) Usability of car dashboard displays for elder drivers. In: Proc. CHI. 493–502

  30. Kim S, Hong J, Li KA, Forlizzi J, Dey AK (2012) Route Guidance Modality for Elder Driver Navigation. Pervasive 12:179–196

    Google Scholar 

  31. Klauer SG, Guo F, Simons-Morton BG, Ouimet MC, Lee SE, Dingus TA (2014) Distracted driving and risk of road crashes among novice and experienced drivers. N Engl J Med 370(1):54–59

    Article  Google Scholar 

  32. Kort B, Reilly R, Picard RW (2001) An Affective Model of Interplay Between Emotions and Learning: Reengineering educational Pedagogy-Building a Learning Companion. In: Proceedings of International Conference on Advanced Learning Technologies (ICALT 2001), August 2001, Madison, WI

  33. Kramer AF (1991) Physiological metrics of mental workload: a review of recent progress. In: Damos DL (ed) Multiple-task performance. Taylor & Francis, London, pp 279–328

    Google Scholar 

  34. Larson R, Csikszentmihalyi M (1983) The experience sampling method. New Dir Methodol Soc BehavSci 15:41–56

    Google Scholar 

  35. Mauri M, Magagnin V, Cipresso P, Mainardi L, Brown EN, Cerutti S, Villamira M, Barbieri R (2010) Psychophysiological signals associated with affective states. Conf Proc IEEE Eng Med Biol Soc 2010:3563–3566

    Google Scholar 

  36. Mercier J, Leger PM, Girard C, Dion JS (2012) Bridging the gap between cognitive neuroscience and education: psychophysiological and behavioral data collection in authentic contexts. Neuroeducation 1(1):5–28

    Google Scholar 

  37. Mital A, Govindaraju M (1999) Is it possible to have a single measure for all work? Int J Ind Eng Theory 6:190–195

    Google Scholar 

  38. Narzt W, Pomberger G, Ferscha A, Kolb D, Muller R, Wieghardt J, Hortner H, Lindinger C (2006) Augmented reality navigation systems. Univ Access Inf Soc 4(3):177–187

    Article  Google Scholar 

  39. Paas FGWC, van Merrienboer JJG (1994) Variability of worked examples and transfer of geometrical problem solving skills: a cognitive load approach. J Educ Psychol 86:122–133

    Article  Google Scholar 

  40. Paas F, Renkel A, Sweller J (2004) Cognitive load theory: instructional implications of the interaction between information structures and cognitive architecture. Instr Sci 32:1–8

    Article  Google Scholar 

  41. Poole A, Ball LJ, Phillips P (2004) In search of salience: A response time and eye movement analysis of bookmark recognition, In People and Computers XVIII (Proceedings of HCI 2004). Springer, London, pp 363–378

    Google Scholar 

  42. Posner MI, Boies SJ (1971) Components of attention. Psychol Rev 78(5):391–408

    Article  Google Scholar 

  43. Salvucci DD (1999) Mapping eye movements to cognitive processes (Tech. Rep. No. CMU-CS-99-131). Doctoral Dissertation, Department of Computer Science, Carnegie Mellon University.

  44. Salvucci DD (2001) Predicting the effects of in-car interface use on driver performance: an integrated model approach. Int J Hum Comput Stud 55(1):85–107

    Article  MATH  Google Scholar 

  45. Schiessl M, Duda S, Tholke A, Fischer R (2003) Eye tracking and its application in usability and media research. MMI interaktiv, Vol. 6, available at: useworld.net/ausgaben/3-2003/MMI-Interaktiv0303_SchiesslDudaThoelkeFischer.pdf

  46. Setz C, Arnrich B, Schumm J, La Marca R, Tröster G, Ehlert U (2010) Discriminating stress from cognitive load using a wearable EDA device. IEEE Trans Inf Technol Biomed 14(2):410–417

    Article  Google Scholar 

  47. Simon HA (1971) Designing organizations for an information rich world. In Computers, Communications, and the Public Interest. 37–72

  48. Sottilare R, Graesser A, Hu X, Holden H (2013) Design Recommendations for Intelligent Tutoring Systems: Learner Modeling, vol 1. Army Research Laboratory, Orlando

    Google Scholar 

  49. Wickens CD (1981) Processing Resources in Attention, Dual Task Performance, and Workload Assessment Technical Report EPL-81-3/ONR-81-3. University of Illinois, Engineering-Psychology Research Laboratory, Champaign

    Google Scholar 

  50. Wickens CD (1984) Processing resources in attention. In: Parasuraman R, Davies R (eds) Varieties of Attention. Academic, New York, pp 63–101

    Google Scholar 

  51. Wickens CD (1991) Processing resources and attention. In: Damos DL (ed) Multiple Task Performance. Taylor and Francis, Ltd., Bristol, pp 3–34

    Google Scholar 

  52. Yarbus AL (1967) Eye Movements and Vision. Plenum, New York, Originally published in Russian 1962

    Book  Google Scholar 

Download references

Acknowledgments

This project is funded in part by General Motors, Carnegie Mellon University’s Technologies for Safe and Efficient Transportation, The National USDOT University Transportation Center for Safety (T-SET UTC) which is sponsored by the US Department of Transportation.

Author information

Authors and Affiliations

  1. Human-Computer Interaction Institute (HCII), Carnegie Mellon University (CMU), 5000 Forbes Avenue, Pittsburgh, PA, USA

    SeungJun Kim & Anind K. Dey

Authors
  1. SeungJun Kim
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Anind K. Dey
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to SeungJun Kim.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, S., Dey, A.K. Augmenting human senses to improve the user experience in cars: applying augmented reality and haptics approaches to reduce cognitive distances. Multimed Tools Appl 75, 9587–9607 (2016). https://doi.org/10.1007/s11042-015-2712-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2712-4

Keywords