ABSTRACT
Tangible User Interfaces (TUIs) allow users to sense and manipulate digital information through physical objects. Although haptic properties are emphasized, TUIs are presented in and perceived through multiple modalities. Especially visual properties like color shape the users expectations about the relation between tangibles and the abstract data they represent and control. Viewing TUIs as multisensory percepts, we present an empirical study that quantifies benefits of an explicit design for color for intuitive interaction. In a cross-cultural experiment, 75 participants (Germans and Japanese) matched tangible objects of different colors, sizes, weights or temperatures with abstract words. The results indicate that multimodal representations increase the efficiency, effectiveness and user satisfaction, but only if colors and haptic properties evoke congruent associations. Incongruently designed tangibles led to a 17% increase in response speed, -28% lower accuracy and -61% lower preference ratings compared to congruently designed tangibles.
- Fabienne Baider and Sara Gesuato. 2002. Burning with desire in English and French: cross-linguistic lexical restrictions on the LOVE IS FIRE metaphor. In 8th International Pragmatics Conference.Google Scholar
- Carrie Anne Balcer. 2014. Visual Cues Effects on Temperature Perception. Northern Michigan University.Google Scholar
- Christa Baldauf. 1997. Metapher und Kognition. Grundlagen einer neuen Theorie der Alltagsmetapher. Peter Lang Verlag.Google Scholar
- Paul Bertelson and De Gelder. 2004. The psychology of multimodal perception. In Crossmodal space and crossmodal attention. 141--177.Google Scholar
- Jonathan S. Cant, Mary Ellen Large, Lindsay McCall, and Melvyn A. Goodale. 2008. Independent processing of form, colour, and texture in object perception. Perception 37, 1: 57--78.Google ScholarCross Ref
- David B Centerbar, Simone Schnall, Gerald L Clore, and Erika D Garvin. 2008. Affective incoherence: when affective concepts and embodied reactions clash. Journal of personality and social psychology 94, 4: 560--578.Google ScholarCross Ref
- Philip R. Cohen and David R. McGee. 2004. Tangible multimodal interfaces for safety-critical applications. Communications of the ACM 47, 1: 41. Google ScholarDigital Library
- Jules B. Davidoff. 1991. Cognition through color. Mit Press Cambridge, MA.Google Scholar
- Ophelia Deroy and Charles Spence. 2013. Why we are not all synesthetes (not even weakly so). Psychonomic bulletin & review 20, 4: 643--64.Google Scholar
- Marc O Ernst and Martin S Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870: 429--433.Google Scholar
- JBF van Erp, CJAM Willemse, JB Janssen, and A Toet. 2014. Sensators: Active multisensory tangible user interfaces.Google Scholar
- Karla K Evans and Anne Treisman. 2010. Natural cross-modal mappings between visual and auditory features. Journal of Vision 10, 6.Google ScholarCross Ref
- Min Fan and Alissa N. Antle. 2015. Tactile Letters. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '14, 673--678. Google ScholarDigital Library
- Feng Feng and Tony Stockman. 2017. An investigation of dynamic crossmodal instantiation in TUIs. In Proceedings of the 19th ACM International Conference on Multimodal Interaction - ICMI 2017, 82--90. Google ScholarDigital Library
- Anna Fenko, Jacco J. Otten, and Hendrik N J Schifferstein. 2010. Describing product experience in different languages: The role of sensory modalities. Journal of Pragmatics 42, 12: 3314--3327.Google ScholarCross Ref
- Anna Fenko, Hendrik N J Schifferstein, and Paul Hekkert. 2009. Which senses dominate at different stages of product experience? In Undisciplined! Design Research Society Conference 2008, 289/1--289/12.Google Scholar
- Nina Gaißert, Christian Wallraven, and Heinrich H. Bülthoff. 2010. Visual and haptic perceptual spaces show high similarity in humans. Journal of Vision 10, 11: 2--2.Google ScholarCross Ref
- Raymond W. Gibbs, Paula Lenz Costa Lima, and Edson Francozo. 2004. Metaphor is grounded in embodied experience. Journal of Pragmatics 36, 7: 1189--1210.Google ScholarCross Ref
- David Hecht, Miriam Reiner, and Avi Karni. 2008. Multisensory enhancement: gains in choice and in simple response times. Experimental Brain Research 189, 2: 133--143.Google ScholarCross Ref
- Hsin-Ni Ho, Daisuke Iwai, Yuki Yoshikawa, Junji Watanabe, and Shinya Nishida. 2014. Combining colour and temperature: A blue object is more likely to be judged as warm than a red object. Scientific reports 4: 5527.Google Scholar
- Geert Hofstede. 2001. Culture's consequences: comparing values, behavior, institutions and organizations across nations. Sage Publications, Inc.Google Scholar
- Shinobu Ishihara. 1917. Tests for Colour Blindness. Handaya Hongo Harukich, Tokyo.Google Scholar
- Hiroshi Ishii. 2008. Tangible bits: beyond pixels. In Proceedings of the 2nd international conference on Tangible and Embedded Intreaction, xv--xxv. Google ScholarDigital Library
- L Kim, H Cho, S Park, and M Han. 2007. A tangible user interface with multimodal feedback. conference on human-computer interaction. Google ScholarDigital Library
- George Lakoff. 1994. Master Metaphor List. University of California.Google Scholar
- George Lakoff and Mark Johnson. 1997. Metaphors We Live By. The production of reality: essays and reading on social interaction: 124--134.Google Scholar
- Christophe Lalanne and Jean Lorenceau. 2004. Crossmodal integration for perception and action. Journal of Physiology Paris 98, 1: 265--279.Google ScholarCross Ref
- Sophie Lebrecht. 2012. 'Micro-valences': Affective valence in 'neutral' everyday objects. Brown University.Google Scholar
- Susan J. Lederman, Georgie Thorne, and Bill Jones. 1986. Perception of texture by vision and touch: multidimensionality and intersensory integration. Journal of Experimental Psychology. Human Perception and Performance 12, 2: 169--180.Google ScholarCross Ref
- Diana Löffler. 2017. Color, Metaphor and Culture. PhD dissertation. University of Würzburg. Retrieved from https://d-nb.info/1141054450Google Scholar
- Diana Löffler, L. Arlt, T. Toriizuka, Robert Tscharn, and J Hurtienne. 2016. Substituting Color for Haptic Attributes in Conceptual Metaphors for Tangible Interaction Design. In TEI'16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, 118--125. Google ScholarDigital Library
- Geke Ludden, Hendrik Schifferstein, and Paul Hekkert. 2009. Visual-Tactual Incongruities in Products as Sources of Surprise. Empirical Studies of the Arts 27, 1: 61--87.Google ScholarCross Ref
- Geke Ludden, Hendrik Schifferstein, and Paul Hekkert. 2012. Beyond surprise: A longitudinal study on the experience of visual-tactual incongruities in products. International Journal of Design 6, 1: 1--10.Google Scholar
- Lawrence E Marks. 1990. Synaesthesia: Perception and Metaphor. In Aesthetic Illusion: Theoretical and Historical Approaches. Walter de Gruyter, 28--40.Google Scholar
- Shinichi Nakagawa. 2004. A farewell to Bonferroni: the problem of low statistical power and publication bias. Behavioral Ecology 15, 6: 1044--1045.Google ScholarCross Ref
- Anja Naumann, Jörn Hurtienne, Johann Habakuk Israel, Carsten Mohs, Martin Christof Kindsmüller, Herbert A. Meyer, Steffi Hußlein, and IUUI Research Group. 2007. Intuitive use of user interfaces: Defining a vague concept. In Engineering Psychology and Cognitive Ergonomics, 128--136. Google ScholarDigital Library
- Charles E Osgood, William H May, and M S Miron. 1975. Cross-Cultural Universals of Affective Meaning. University of Illinois Press, Urbana.Google Scholar
- Sharon Oviatt and Philip Cohen. 2000. Perceptual user interfaces: multimodal interfaces that process what comes naturally. Communications of the ACM 43, 3: 45--53. Google ScholarDigital Library
- Sharon Oviatt and Sharon. 1999. Ten myths of multimodal interaction. Communications of the ACM 42, 11: 74--81. Google ScholarDigital Library
- Sara Price and Carey Jewitt. 2013. A multimodal approach to examining 'embodiment' in tangible learning environments. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction - TEI '13, 43. Google ScholarDigital Library
- Eckard Riedenklau, Thomas Hermann, and Helge Ritter. 2012. An integrated multi-modal actuated tangible user interface for distributed collaborative planning. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction - TEI '12, 169. Google ScholarDigital Library
- Irvin Rock and Jack Victor. 1964. Vision and touch: An experimentally created conflict between the two senses. Science 143, 3606: 594--596.Google Scholar
- Paul Sambre. 2000. Metaphor and Metonymy at the Crossroads: A Cognitive Perspective. Pragmatics 10, 3: 344--345.Google Scholar
- Hendrik Schifferstein. 2006. The perceived importance of sensory modalities in product usage: A study of self-reports. Acta Psychologica 121, 1: 41--64.Google ScholarCross Ref
- Hendrik Schifferstein and Marc P.H.D. Cleiren. 2005. Capturing Product Experience - A split modality approach. Acta Psychologica 118, 3: 293--318.Google ScholarCross Ref
- Hendrik Schifferstein, Anna Fenko, Pieter M A Desmet, David Labbe, and Nathalie Martin. 2013. Influence of package design on the dynamics of multisensory and emotional food experience. Food Quality and Preference 27, 1: 18--25.Google ScholarCross Ref
- Hendrik Schifferstein, Jacco J. Otten, Fien Thoolen, and Paul Hekkert. 2010. The experimental assessment of sensory dominance in a product development context. Journal of Design Research 8, 2: 119--144.Google ScholarCross Ref
- Charles Spence. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73, 4: 971--995.Google ScholarCross Ref
- Anatol Stefanowitsch. 2006. Words and their metaphors: A corpus-based approach. Trends in Linguistics Studies and Monographs 171, 63.Google Scholar
- Barry Stein and Alex Meredith. 1993. The Merging of the Sense. Mit Press Cambridge, MA.Google Scholar
- Jon Tolaas. 1991. Notes on the Origin of Some Spatialization Metaphors. Metaphor and Symbolic Activity 6, 3: 203--218.Google ScholarCross Ref
- W Valk, J Rypkema, and JBF van Erp. 2014. Facilitating planning: Tangible objects with multimodal feedback mitigate cognitive workload. Auvray, M. Duriez, C., 9th.Google Scholar
- Laura Walker, Peter Walker, and Brian Francis. 2012. A common scheme for cross-sensory correspondences across stimulus domains. Perception 41, 10: 1186--1192.Google ScholarCross Ref
- Lisa Wastiels, Hendrik N J Schifferstein, Ine Wouters, and Ann Heylighen. 2013. Touching materials visually: About the dominance of vision in building material assessment. International Journal of Design 7, 2: 31--41.Google Scholar
- Andy T. Woods, Charles Spence, Natalie Butcher, and Ophelia Deroy. 2013. Fast lemons and sour boulders: Testing crossmodal correspondences using an internet-based testing methodology. i-Perception 4, 6: 365--379.Google Scholar
Index Terms
Multimodal Effects of Color and Haptics on Intuitive Interaction with Tangible User Interfaces
Recommendations
Substituting Color for Haptic Attributes in Conceptual Metaphors for Tangible Interaction Design
TEI '16: Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied InteractionStudies in tangible interaction have investigated how physical object attributes can stand for abstract content (e.g. IMPORTANT IS HEAVY). A less expensive and more practical alternative to dynamically change, for example, the size, weight or ...
The Future of Tangible User Interfaces
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing SystemsTangible user interfaces have a rich history in HCI research ever since their introduction two decades ago. But what are the practical implications, the commercial potential, and the future of this influential paradigm? This panel starts by looking into ...
Population stereotypes of color attributes for tangible interaction design
TEI '14: Proceedings of the 8th International Conference on Tangible, Embedded and Embodied InteractionA promising approach to facilitate the design of intuitive interaction with tangible user interfaces (TUIs) is making use of image-schematic metaphors. Image-schematic metaphors function as population stereotypes that define the relations between ...
Comments