skip to main content
10.1145/3173225.3173257acmconferencesArticle/Chapter ViewAbstractPublication PagesteiConference Proceedingsconference-collections
research-article

Multimodal Effects of Color and Haptics on Intuitive Interaction with Tangible User Interfaces

Published:18 March 2018Publication History

ABSTRACT

Tangible User Interfaces (TUIs) allow users to sense and manipulate digital information through physical objects. Although haptic properties are emphasized, TUIs are presented in and perceived through multiple modalities. Especially visual properties like color shape the users expectations about the relation between tangibles and the abstract data they represent and control. Viewing TUIs as multisensory percepts, we present an empirical study that quantifies benefits of an explicit design for color for intuitive interaction. In a cross-cultural experiment, 75 participants (Germans and Japanese) matched tangible objects of different colors, sizes, weights or temperatures with abstract words. The results indicate that multimodal representations increase the efficiency, effectiveness and user satisfaction, but only if colors and haptic properties evoke congruent associations. Incongruently designed tangibles led to a 17% increase in response speed, -28% lower accuracy and -61% lower preference ratings compared to congruently designed tangibles.

References

  1. Fabienne Baider and Sara Gesuato. 2002. Burning with desire in English and French: cross-linguistic lexical restrictions on the LOVE IS FIRE metaphor. In 8th International Pragmatics Conference.Google ScholarGoogle Scholar
  2. Carrie Anne Balcer. 2014. Visual Cues Effects on Temperature Perception. Northern Michigan University.Google ScholarGoogle Scholar
  3. Christa Baldauf. 1997. Metapher und Kognition. Grundlagen einer neuen Theorie der Alltagsmetapher. Peter Lang Verlag.Google ScholarGoogle Scholar
  4. Paul Bertelson and De Gelder. 2004. The psychology of multimodal perception. In Crossmodal space and crossmodal attention. 141--177.Google ScholarGoogle Scholar
  5. Jonathan S. Cant, Mary Ellen Large, Lindsay McCall, and Melvyn A. Goodale. 2008. Independent processing of form, colour, and texture in object perception. Perception 37, 1: 57--78.Google ScholarGoogle ScholarCross RefCross Ref
  6. David B Centerbar, Simone Schnall, Gerald L Clore, and Erika D Garvin. 2008. Affective incoherence: when affective concepts and embodied reactions clash. Journal of personality and social psychology 94, 4: 560--578.Google ScholarGoogle ScholarCross RefCross Ref
  7. Philip R. Cohen and David R. McGee. 2004. Tangible multimodal interfaces for safety-critical applications. Communications of the ACM 47, 1: 41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Jules B. Davidoff. 1991. Cognition through color. Mit Press Cambridge, MA.Google ScholarGoogle Scholar
  9. Ophelia Deroy and Charles Spence. 2013. Why we are not all synesthetes (not even weakly so). Psychonomic bulletin & review 20, 4: 643--64.Google ScholarGoogle Scholar
  10. Marc O Ernst and Martin S Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870: 429--433.Google ScholarGoogle Scholar
  11. JBF van Erp, CJAM Willemse, JB Janssen, and A Toet. 2014. Sensators: Active multisensory tangible user interfaces.Google ScholarGoogle Scholar
  12. Karla K Evans and Anne Treisman. 2010. Natural cross-modal mappings between visual and auditory features. Journal of Vision 10, 6.Google ScholarGoogle ScholarCross RefCross Ref
  13. Min Fan and Alissa N. Antle. 2015. Tactile Letters. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '14, 673--678. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Feng Feng and Tony Stockman. 2017. An investigation of dynamic crossmodal instantiation in TUIs. In Proceedings of the 19th ACM International Conference on Multimodal Interaction - ICMI 2017, 82--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Anna Fenko, Jacco J. Otten, and Hendrik N J Schifferstein. 2010. Describing product experience in different languages: The role of sensory modalities. Journal of Pragmatics 42, 12: 3314--3327.Google ScholarGoogle ScholarCross RefCross Ref
  16. Anna Fenko, Hendrik N J Schifferstein, and Paul Hekkert. 2009. Which senses dominate at different stages of product experience? In Undisciplined! Design Research Society Conference 2008, 289/1--289/12.Google ScholarGoogle Scholar
  17. Nina Gaißert, Christian Wallraven, and Heinrich H. Bülthoff. 2010. Visual and haptic perceptual spaces show high similarity in humans. Journal of Vision 10, 11: 2--2.Google ScholarGoogle ScholarCross RefCross Ref
  18. Raymond W. Gibbs, Paula Lenz Costa Lima, and Edson Francozo. 2004. Metaphor is grounded in embodied experience. Journal of Pragmatics 36, 7: 1189--1210.Google ScholarGoogle ScholarCross RefCross Ref
  19. David Hecht, Miriam Reiner, and Avi Karni. 2008. Multisensory enhancement: gains in choice and in simple response times. Experimental Brain Research 189, 2: 133--143.Google ScholarGoogle ScholarCross RefCross Ref
  20. Hsin-Ni Ho, Daisuke Iwai, Yuki Yoshikawa, Junji Watanabe, and Shinya Nishida. 2014. Combining colour and temperature: A blue object is more likely to be judged as warm than a red object. Scientific reports 4: 5527.Google ScholarGoogle Scholar
  21. Geert Hofstede. 2001. Culture's consequences: comparing values, behavior, institutions and organizations across nations. Sage Publications, Inc.Google ScholarGoogle Scholar
  22. Shinobu Ishihara. 1917. Tests for Colour Blindness. Handaya Hongo Harukich, Tokyo.Google ScholarGoogle Scholar
  23. Hiroshi Ishii. 2008. Tangible bits: beyond pixels. In Proceedings of the 2nd international conference on Tangible and Embedded Intreaction, xv--xxv. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. L Kim, H Cho, S Park, and M Han. 2007. A tangible user interface with multimodal feedback. conference on human-computer interaction. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. George Lakoff. 1994. Master Metaphor List. University of California.Google ScholarGoogle Scholar
  26. George Lakoff and Mark Johnson. 1997. Metaphors We Live By. The production of reality: essays and reading on social interaction: 124--134.Google ScholarGoogle Scholar
  27. Christophe Lalanne and Jean Lorenceau. 2004. Crossmodal integration for perception and action. Journal of Physiology Paris 98, 1: 265--279.Google ScholarGoogle ScholarCross RefCross Ref
  28. Sophie Lebrecht. 2012. 'Micro-valences': Affective valence in 'neutral' everyday objects. Brown University.Google ScholarGoogle Scholar
  29. Susan J. Lederman, Georgie Thorne, and Bill Jones. 1986. Perception of texture by vision and touch: multidimensionality and intersensory integration. Journal of Experimental Psychology. Human Perception and Performance 12, 2: 169--180.Google ScholarGoogle ScholarCross RefCross Ref
  30. Diana Löffler. 2017. Color, Metaphor and Culture. PhD dissertation. University of Würzburg. Retrieved from https://d-nb.info/1141054450Google ScholarGoogle Scholar
  31. Diana Löffler, L. Arlt, T. Toriizuka, Robert Tscharn, and J Hurtienne. 2016. Substituting Color for Haptic Attributes in Conceptual Metaphors for Tangible Interaction Design. In TEI'16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, 118--125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Geke Ludden, Hendrik Schifferstein, and Paul Hekkert. 2009. Visual-Tactual Incongruities in Products as Sources of Surprise. Empirical Studies of the Arts 27, 1: 61--87.Google ScholarGoogle ScholarCross RefCross Ref
  33. Geke Ludden, Hendrik Schifferstein, and Paul Hekkert. 2012. Beyond surprise: A longitudinal study on the experience of visual-tactual incongruities in products. International Journal of Design 6, 1: 1--10.Google ScholarGoogle Scholar
  34. Lawrence E Marks. 1990. Synaesthesia: Perception and Metaphor. In Aesthetic Illusion: Theoretical and Historical Approaches. Walter de Gruyter, 28--40.Google ScholarGoogle Scholar
  35. Shinichi Nakagawa. 2004. A farewell to Bonferroni: the problem of low statistical power and publication bias. Behavioral Ecology 15, 6: 1044--1045.Google ScholarGoogle ScholarCross RefCross Ref
  36. Anja Naumann, Jörn Hurtienne, Johann Habakuk Israel, Carsten Mohs, Martin Christof Kindsmüller, Herbert A. Meyer, Steffi Hußlein, and IUUI Research Group. 2007. Intuitive use of user interfaces: Defining a vague concept. In Engineering Psychology and Cognitive Ergonomics, 128--136. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Charles E Osgood, William H May, and M S Miron. 1975. Cross-Cultural Universals of Affective Meaning. University of Illinois Press, Urbana.Google ScholarGoogle Scholar
  38. Sharon Oviatt and Philip Cohen. 2000. Perceptual user interfaces: multimodal interfaces that process what comes naturally. Communications of the ACM 43, 3: 45--53. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Sharon Oviatt and Sharon. 1999. Ten myths of multimodal interaction. Communications of the ACM 42, 11: 74--81. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Sara Price and Carey Jewitt. 2013. A multimodal approach to examining 'embodiment' in tangible learning environments. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction - TEI '13, 43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Eckard Riedenklau, Thomas Hermann, and Helge Ritter. 2012. An integrated multi-modal actuated tangible user interface for distributed collaborative planning. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction - TEI '12, 169. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Irvin Rock and Jack Victor. 1964. Vision and touch: An experimentally created conflict between the two senses. Science 143, 3606: 594--596.Google ScholarGoogle Scholar
  43. Paul Sambre. 2000. Metaphor and Metonymy at the Crossroads: A Cognitive Perspective. Pragmatics 10, 3: 344--345.Google ScholarGoogle Scholar
  44. Hendrik Schifferstein. 2006. The perceived importance of sensory modalities in product usage: A study of self-reports. Acta Psychologica 121, 1: 41--64.Google ScholarGoogle ScholarCross RefCross Ref
  45. Hendrik Schifferstein and Marc P.H.D. Cleiren. 2005. Capturing Product Experience - A split modality approach. Acta Psychologica 118, 3: 293--318.Google ScholarGoogle ScholarCross RefCross Ref
  46. Hendrik Schifferstein, Anna Fenko, Pieter M A Desmet, David Labbe, and Nathalie Martin. 2013. Influence of package design on the dynamics of multisensory and emotional food experience. Food Quality and Preference 27, 1: 18--25.Google ScholarGoogle ScholarCross RefCross Ref
  47. Hendrik Schifferstein, Jacco J. Otten, Fien Thoolen, and Paul Hekkert. 2010. The experimental assessment of sensory dominance in a product development context. Journal of Design Research 8, 2: 119--144.Google ScholarGoogle ScholarCross RefCross Ref
  48. Charles Spence. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73, 4: 971--995.Google ScholarGoogle ScholarCross RefCross Ref
  49. Anatol Stefanowitsch. 2006. Words and their metaphors: A corpus-based approach. Trends in Linguistics Studies and Monographs 171, 63.Google ScholarGoogle Scholar
  50. Barry Stein and Alex Meredith. 1993. The Merging of the Sense. Mit Press Cambridge, MA.Google ScholarGoogle Scholar
  51. Jon Tolaas. 1991. Notes on the Origin of Some Spatialization Metaphors. Metaphor and Symbolic Activity 6, 3: 203--218.Google ScholarGoogle ScholarCross RefCross Ref
  52. W Valk, J Rypkema, and JBF van Erp. 2014. Facilitating planning: Tangible objects with multimodal feedback mitigate cognitive workload. Auvray, M. Duriez, C., 9th.Google ScholarGoogle Scholar
  53. Laura Walker, Peter Walker, and Brian Francis. 2012. A common scheme for cross-sensory correspondences across stimulus domains. Perception 41, 10: 1186--1192.Google ScholarGoogle ScholarCross RefCross Ref
  54. Lisa Wastiels, Hendrik N J Schifferstein, Ine Wouters, and Ann Heylighen. 2013. Touching materials visually: About the dominance of vision in building material assessment. International Journal of Design 7, 2: 31--41.Google ScholarGoogle Scholar
  55. Andy T. Woods, Charles Spence, Natalie Butcher, and Ophelia Deroy. 2013. Fast lemons and sour boulders: Testing crossmodal correspondences using an internet-based testing methodology. i-Perception 4, 6: 365--379.Google ScholarGoogle Scholar

Index Terms

  1. Multimodal Effects of Color and Haptics on Intuitive Interaction with Tangible User Interfaces

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      TEI '18: Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction
      March 2018
      763 pages
      ISBN:9781450355681
      DOI:10.1145/3173225

      Copyright © 2018 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 18 March 2018

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      TEI '18 Paper Acceptance Rate37of130submissions,28%Overall Acceptance Rate393of1,367submissions,29%

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader