Abstract
Modern Companion-Technologies provide multimodal and adaptive interaction possibilities. However, it is still unclear which user characteristics should be used in which manner to optimally support the interaction. An important aspect is that users themselves learn and adapt their behavior and preferences based on their own experiences. In other words, certain characteristics of user behavior are slowly but continuously changed and updated by the users themselves over multiple encounters with the Companion-Technology. Thus, a biological adaptive multimodal system observes and interacts with an electronic one, and vice versa. Consequently, such a user-centered interaction history is essential and should be integrated in the prediction of user behavior. Doing so enables the Companion to achieve more robust predictions of user behavior, which in turn leads to better fusion decisions and more efficient customization of the UI. We present the development of an experimental paradigm based on visual search tasks. The setup allows the induction of various user experiences as well as the testing of their effects on user behavior and preferences during multimodal interaction.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ailomaa, M., Melichar, M., Rajman, M., Lisowska, A., Armstrong, S.: Archivus: a multimodal system for multimedia meeting browsing and retrieval. In: Proceedings of the COLING/ACL on Interactive Presentation Sessions, pp. 49–52. Association for Computational Linguistics, Stroudsburg, PA (2006)
Ajzen, I.: Attitudes, Personality, and Behavior. McGraw-Hill International, Maidenhead (2005)
Bellik, Y., Rebaï, I., Machrouh, E., Barzaj, Y., Jacquet, C., Pruvost, G., Sansonnet, J.P.: Multimodal interaction within ambient environments: an exploratory study. In: Human-Computer Interaction–INTERACT 2009, pp. 89–92. Springer, Berlin (2009)
Biundo, S., Wendemuth, A.: Companion-technology for cognitive technical systems. Künstl. Intell. 30(1), 71–75 (2016). Special issue on companion technologies
Bolt, R.A.: “Put-That-There”: Voice and Gesture at the Graphics Interface, vol. 14. ACM, New York (1980)
Camp, G., Paas, F., Rikers, R., van Merrienboer, J.: Dynamic problem selection in air traffic control training: a comparison between performance, mental effort and mental efficiency. Comput. Hum. Behav. 17(5), 575–595 (2001)
Carter, S., Mankoff, J., Klemmer, S.R., Matthews, T.: Exiting the cleanroom: on ecological validity and ubiquitous computing. Hum. Comput. Interact. 23(1), 47–99 (2008)
Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM International Conference on Multimedia, pp. 31–40. ACM, New York (1997)
De Angeli, A., Gerbino, W., Cassano, G., Petrelli, D.: Visual display, pointing, and natural language: the power of multimodal interaction. In: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 164–173. ACM, New York (1998)
Dey, P., Madhvanath, S., Ranjan, A., Das, S.: An exploration of gesture-speech multimodal patterns for touch interfaces. In: Proceedings of the 3rd International Conference on Human Computer Interaction, pp. 79–83. ACM, New York (2011)
Domjan, M.: The principles of learning and behavior. Cengage Learning, Stamford, CT (2014)
Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne, D., Kohlas, J. (eds.) Human Machine Interaction. Lecture Notes in Computer Science, vol. 5440, pp. 3–26. Springer, Berlin, Heidelberg (2009)
Haas, E.C., Pillalamarri, K.S., Stachowiak, C.C., McCullough, G.: Temporal binding of multimodal controls for dynamic map displays: a systems approach. In: Proceedings of the 13th International Conference on Multimodal Interfaces, pp. 409–416. ACM, New York (2011)
Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol. 52, 139–183 (1988)
Hollender, N., Hofmann, C., Deneke, M., Schmitz, B.: Integrating cognitive load theory and concepts of human–computer interaction. Comput. Hum. Behav. 26(6), 1278–1288 (2010)
Jaimes, A., Sebe, N.: Multimodal human–computer interaction: a survey. Comput. Vis. Image Underst. 108(1), 116–134 (2007)
Jöst, M., Häußler, J., Merdes, M., Malaka, R.: Multimodal interaction for pedestrians: an evaluation study. In: Proceedings of the 10th International Conference on Intelligent User Interfaces, pp. 59–66. ACM, New York (2005)
Käster, T., Pfeiffer, M., Bauckhage, C.: Combining speech and haptics for intuitive and efficient navigation through image databases. In: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 180–187. ACM, New York (2003)
Kieffer, S., Carbonell, N.: How really effective are multimodal hints in enhancing visual target spotting? Some evidence from a usability study. J. Multimodal User Interfaces 1(1), 1–5 (2007)
Koons, D.B., Sparrell, C.J., Thorisson, K.R.: Integrating Simultaneous Input from Speech, Gaze, and Hand Gestures. MIT, Menlo Park, CA, pp. 257–276 (1993)
Kruijff-Korbayová, I., Blaylock, N., Gerstenberger, C., Rieser, V., Becker, T., Kaisser, M., Poller, P., Schehl, J.: An experiment setup for collecting data for adaptive output planning in a multimodal dialogue system. In: Proceedings of ENLG (2005)
Lalanne, D., Nigay, L., Robinson, P., Vanderdonckt, J., Ladry, J.F., et al.: Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 153–160. ACM, New York (2009)
Lee, M., Billinghurst, M.: A wizard of oz study for an ar multimodal interface. In: Proceedings of the 10th International Conference on Multimodal Interfaces, pp. 249–256. ACM, New York (2008)
Lee, J.H., Poliakoff, E., Spence, C.: The effect of multimodal feedback presented via a touch screen on the performance of older adults. In: Haptic and Audio Interaction Design, pp. 128–135. Springer, Berlin (2009)
Mignot, C., Valot, C., Carbonell, N.: An experimental study of future “natural” multimodal human-computer interaction. In: INTERACT’93 and CHI’93 Conference Companion on Human Factors in Computing Systems, pp. 67–68. ACM, New York (1993)
Müller, H.J., Krummenacher, J.: Visual search and selective attention. Vis. Cogn. 14(4–8), 389–410 (2006)
Neal, J.G., Shapiro, S.C.: Intelligent multi-media interface technology. ACM SIGCHI Bull. 20(1), 75–76 (1988)
Ouellette, J.A., Wood, W.: Habit and intention in everyday life: the multiple processes by which past behavior predicts future behavior. Psychol. Bull. 124(1), 54 (1998)
Oviatt, S.: Multimodal interactive maps: designing for human performance. Hum. Comput. Interact. 12(1), 93–129 (1997)
Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions. Hum. Comput. Interact. 15(4), 263–322 (2000)
Oviatt, S., Coulston, R., Lunsford, R.: When do we interact multimodally?: Cognitive load and multimodal communication patterns. In: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 129–136. ACM, New York (2004)
Oviatt, S., Lunsford, R., Coulston, R.: Individual differences in multimodal integration patterns: what are they and why do they exist? In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 241–249. ACM, New York (2005)
Oviatt, S., Arthur, A., Cohen, J.: Quiet interfaces that help students think. In: Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology, pp. 191–200. ACM, New York (2006)
Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)
Paivio, A.: Mental Representations: A Dual Coding Approach. Oxford University Press, Oxford (1990)
Ratzka, A.: Explorative studies on multimodal interaction in a PDA- and desktop-based scenario. In: Proceedings of the 10th International Conference on Multimodal Interfaces, pp. 121–128. ACM, New York (2008)
Reeves, L.M., Lai, J., Larson, J.A., Oviatt, S., Balaji, T.S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J.C., McTear, M., Raman, T., Stanney, K.M., Su, H., Wang, Q.Y.: Guidelines for multimodal user interface design. Commun. ACM 47(1), 57–59 (2004)
Reis, T., de Sá, M., Carriço, L.: Multimodal interaction: real context studies on mobile digital artefacts. In: Haptic and Audio Interaction Design, pp. 60–69. Springer, Berlin (2008)
Ren, X., Zhang, G., Dai, G.: An experimental study of input modes for multimodal human-computer interaction. In: Advances in Multimodal Interfaces-ICMI 2000, pp. 49–56. Springer, Berlin (2000)
Ruiz, N., Chen, F., Oviatt, S.: Multimodal input. In: Multimodal Signal Processing: Theory and Applications for Human-Computer Interaction, p. 231. Academic, Boston (2009)
Savidis, A., Stephanidis, C.: Unified user interface design: designing universally accessible interactions. Interact. Comput. 16(2), 243–270 (2004)
Schüssel, F., Honold, F., Weber, M.: Influencing factors on multimodal interaction during selection tasks. J. Multimodal User Interfaces 7(4), 299–310 (2013)
Schüssel, F., Honold, F., Weber, M., Schmidt, M., Bubalo, N., Huckauf, A.: Multimodal interaction history and its use in error detection and recovery. In: Proceedings of the 16th ACM International Conference on Multimodal Interaction, ICMI ’14, pp. 164–171. ACM, New York (2014). doi:10.1145/2663204.2663255
Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Cogn. Psychol. 12(1), 97–136 (1980)
Van Merrienboer, J.J., Sweller, J.: Cognitive load theory and complex learning: recent developments and future directions. Educ. Psychol. Rev. 17(2), 147–177 (2005)
Wasinger, R., Krüger, A.: Modality preferences in mobile and instrumented environments. In: Proceedings of the 11th International Conference on Intelligent User Interfaces, pp. 336–338. ACM, New York (2006)
Wickens, C.D.: Multiple resources and mental workload. Hum. Factors J. Hum. Factors Ergon. Soc. 50(3), 449–455 (2008)
Acknowledgements
This work was done within the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Bubalo, N., Schüssel, F., Honold, F., Weber, M., Huckauf, A. (2017). Interaction History in Adaptive Multimodal Interaction. In: Biundo, S., Wendemuth, A. (eds) Companion Technology. Cognitive Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-43665-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-43665-4_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-43664-7
Online ISBN: 978-3-319-43665-4
eBook Packages: Computer ScienceComputer Science (R0)