skip to main content
10.1145/1452392.1452423acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Crossmodal congruence: the look, feel and sound of touchscreen widgets

Published: 20 October 2008 Publication History

Abstract

Our research considers the following question: how can visual, audio and tactile feedback be combined in a congruent manner for use with touchscreen graphical widgets? For example, if a touchscreen display presents different styles of visual buttons, what should each of those buttons feel and sound like? This paper presents the results of an experiment conducted to investigate methods of congruently combining visual and combined audio/tactile feedback by manipulating the different parameters of each modality. The results indicate trends with individual visual parameters such as shape, size and height being combined congruently with audio/tactile parameters such as texture, duration and different actuator technologies. We draw further on the experiment results using individual quality ratings to evaluate the perceived quality of our touchscreen buttons then reveal a correlation between perceived quality and crossmodal congruence. The results of this research will enable mobile touchscreen UI designers to create realistic, congruent buttons by selecting the most appropriate audio and tactile counterparts of visual button styles.

References

[1]
Hall, M., Hoggan, E. and Brewster, S. A. T-Bars: Towards Tactile User Interfaces for Touchscreen Mobiles. In Proc MobileHCI' 08, ACM Press (2008).
[2]
Lee, J. C., Dietz, P. H., Leigh, D., Yerazunis, W. S. and Hudson, S. E. Haptic Pen: A Tactile Feedback Stylus for Touch Screens In Proc 17th annual ACM symposium on User interface software and technology ACM Press (2004), 291--294.
[3]
Kaaresoja, T., Brown, L. M. and Linjama, J. Snap-Crackle-Pop: Tactile Feedback for Mobile Touch Screens. In Proc Eurohaptics '06, (2006), 565--566.
[4]
Poupyrev, I., Maruyama, S. and Rekimoto, J. Ambient Touch: Designing Tactile Interfaces for Handheld Devices. In Proc 15th annual ACM symposium on User interface software and technology, ACM Press (2002), 51--60.
[5]
Fukumoto, M. and Sugimura, T. Active Click: Tactile Feedback for Touch Panels. In Proc CHI' 01, ACM Press (2001), 121--122.
[6]
Hoggan, E., Brewster, S. A. and Johnston, J. Investigating the Effectiveness of Tactile Feedback for Mobile Touchscreens. In Proc CHI' 08, ACM Press (2008), 1573--1582.
[7]
Brewster, S. A. Overcoming the Lack of Screen Space on Mobile Computers. Personal and Ubiquitous Computing 6, 3 (2002), 188--205.
[8]
McGee, M. R., Gray, P. and Brewster, S. A. Mixed Feelings: Multimodal Perception of Virtual Roughness. In Proc EuroHaptics, Springer LNCS (2002),
[9]
Hoggan, E. and Brewster, S. A. Designing Audio and Tactile Crossmodal Icons for Mobile Devices. In Proc Proceedings of the 9th international conference on Multimodal interfaces, ACM (2007), 162--169.
[10]
Lewkowicz, D. J. The Development of Intersensory Temporal Perception: An Epigenetic Systems/Limitations View. Psychological Bulletin 126, (2000), 281--308.
[11]
Shimojo, S. and Shams, L. Sensory Modalities Are Not Separate Modalities: Plasticity and Interaction. Current Opinion in Neurobiology 11, 4 (2001), 505--509.
[12]
Huang, G., Metaxas, D. and Govindaraj, M. Feel The "Fabric": An Audio-Haptic Interface. In Proc Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association (2003), 52--61.
[13]
Driver, J. and Spence, C. Multisensory Perception: Beyond Modularity and Convergence. Current Biology 10, 20 (2000), 731--735.
[14]
McGurk, H. and MacDonald, J. W. Hearing Lips and Seeing Voices. Nature 64, 2 (1976), 746--748.
[15]
Bermant, R. I. and Welch, R. B. The Effect of Degree of Visual-Auditory Stimulus Separation and Eye Position Upon the Spatial Interaction of Vision and Audition. Perceptual and Motor Skill 43, (1976), 487--493.
[16]
Jousmaki, V. and Hari, R. Parchment-Skin Illusion: Sound-Biased Touch. Current Biology 8, 6 (1998), 190--191.
[17]
Laitinen, P. and Mäenpää, J. Enabling Mobile Haptic Design: Piezoelectric Actuator Technology Properties in Hand Held Devices. In Proc HAVE'06 (Haptic Audio Virtual Environments), (2006).
[18]
Pesqueux, L. and Rouaud, M., Vibration Level of Mobile Phones' Silent Alerts, PhD Thesis - Department of Acoustics, Aalborg University, 2005
[19]
Apple Macintosh Human Interface Guidelines. Addison-Wesley, 1992.
[20]
Zhou, L., Raju, B. I. and Srinivasan, M. A. Relevant Stimuli and Their Relationships to Primate Sa-I Mechanoreceptive Responses under Static Sinusoidan Indentation. In Proc WORLDCOMP'06 / The 2006 International Conference on Modeling, Simulation & Visualization Methods, CSREA Press, USA (2006), 67--73.
[21]
Lawrence, D. A. and Chapel, J. D. Performance Trade-Offs for Hand Controlled Design. In Proc 1994 IEEE Conference on Robotics and Automation, (1994), 3211--3216.
[22]
Bolanowski, S. J., Jr., Gescheider, G. A., Verrillo, R. T. and Checkosky, C. M. Four Channels Mediate the Mechanical Aspects of Touch. Journal of the Acoustic Society of America 84, 5 (1988), 1680--1694.
[23]
Fletcher, H. and A., M. W. Loudness: Its Definition, Measurement, and Calculation. Journal of Acoustical Society of America 5, (1933), 82--108.

Cited By

View all
  • (2024)Sound-to-Touch Crossmodal Pitch Matching for Short SoundsIEEE Transactions on Haptics10.1109/TOH.2023.333822417:1(2-7)Online publication date: Jan-2024
  • (2024)In-the-Wild Vibrotactile Sensation: Perceptual Transformation of Vibrations from Smartphones2024 IEEE 12th Region 10 Humanitarian Technology Conference (R10-HTC)10.1109/R10-HTC59322.2024.10778694(1-8)Online publication date: 1-Oct-2024
  • (2023)Discrimination Accuracy of Sequential Versus Simultaneous Vibrotactile Stimulation on the ForearmApplied Sciences10.3390/app1401004314:1(43)Online publication date: 20-Dec-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces
October 2008
322 pages
ISBN:9781605581989
DOI:10.1145/1452392
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. auditory/tactile/visual congruence
  2. crossmodal interaction
  3. mobile touchscreen interaction
  4. touchscreen widgets

Qualifiers

  • Research-article

Conference

ICMI '08
Sponsor:
ICMI '08: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES
October 20 - 22, 2008
Crete, Chania, Greece

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)29
  • Downloads (Last 6 weeks)3
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Sound-to-Touch Crossmodal Pitch Matching for Short SoundsIEEE Transactions on Haptics10.1109/TOH.2023.333822417:1(2-7)Online publication date: Jan-2024
  • (2024)In-the-Wild Vibrotactile Sensation: Perceptual Transformation of Vibrations from Smartphones2024 IEEE 12th Region 10 Humanitarian Technology Conference (R10-HTC)10.1109/R10-HTC59322.2024.10778694(1-8)Online publication date: 1-Oct-2024
  • (2023)Discrimination Accuracy of Sequential Versus Simultaneous Vibrotactile Stimulation on the ForearmApplied Sciences10.3390/app1401004314:1(43)Online publication date: 20-Dec-2023
  • (2022)Birdbox: Exploring the User Experience of Crossmodal, Multisensory Data RepresentationsProceedings of the 21st International Conference on Mobile and Ubiquitous Multimedia10.1145/3568444.3568455(12-21)Online publication date: 27-Nov-2022
  • (2022)Texture Classification by Audio-Tactile Crossmodal Congruence2022 IEEE Haptics Symposium (HAPTICS)10.1109/HAPTICS52432.2022.9765614(1-7)Online publication date: 21-Mar-2022
  • (2020)It’s All in the TimingACM Transactions on Computer-Human Interaction10.1145/338635827:3(1-29)Online publication date: 31-May-2020
  • (2020)Defining Haptic Experience: Foundations for Understanding, Communicating, and Evaluating HXProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376280(1-13)Online publication date: 21-Apr-2020
  • (2020)A Review of Surface Haptics: Enabling Tactile Effects on Touch SurfacesIEEE Transactions on Haptics10.1109/TOH.2020.299071213:3(450-470)Online publication date: 1-Jul-2020
  • (2019)The Graphical Access Challenge for People with Visual Impairments: Positions and Pathways ForwardInteractive Multimedia - Multimedia Production and Digital Storytelling10.5772/intechopen.82289Online publication date: 25-Sep-2019
  • (2019)"Like Popcorn"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300689(1-13)Online publication date: 2-May-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media