research-article

HapticSeer: A Multi-channel, Black-box, Platform-agnostic Approach to Detecting Video Game Events for Real-time Haptic Feedback

Authors:

Yun-Ting Cheng,

Mike Y. ChenAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 418, Pages 1 - 14

https://doi.org/10.1145/3411764.3445254

Published: 07 May 2021 Publication History

Abstract

Haptic feedback significantly enhances virtual experiences. However, supporting haptics currently requires modifying the codebase, making it impractical to add haptics to popular, high-quality experiences such as best selling games, which are typically closed-source. We present HapticSeer, a multi-channel, black-box, platform-agnostic approach to detecting game events for real-time haptic feedback. The approach is based on two key insights: 1) all games have 3 types of data streams: video, audio, and controller I/O, that can be analyzed in real-time to detect game events, and 2) a small number of user interface design patterns are reused across most games, so that event detectors can be reused effectively. We developed an open-source HapticSeer framework and implemented several real-time event detectors for commercial PC and VR games. We validated system correctness and real-time performance, and discuss feedback from several haptics developers that used the HapticSeer framework to integrate research and commercial haptic devices.

References

[1]

Anonymous. 2020. DS4-BT - PS4 Developer Wiki. https://www.psdevwiki.com/ps4/DS4-BT#UART_HCI

[2]

Christiane Attig, Nadine Rauh, Thomas Franke, and Josef F. Krems. 2017. System Latency Guidelines Then and Now – Is Zero Latency Really Considered Necessary?. In Engineering Psychology and Cognitive Ergonomics: Cognition and Design, Don Harris(Ed.). Springer International Publishing, Cham, 3–14.

[3]

bHaptics. 2020. bHaptics Audio-to-Haptic. https://www.bhaptics.com/support/faq/#Audio-to-Haptic

[4]

bHaptics. 2020. bHaptics tactsuit. https://www.bhaptics.com/tactsuit/

[5]

Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. YOLOv4: Optimal Speed and Accuracy of Object Detection. arxiv:2004.10934 [cs.CV]

[6]

Mike Channell. 2020. The 20 best driving games of the last decade: 10-1. https://www.topgear.com/car-news/electric/20-best-driving-games-last-decade-10-1.

[7]

Mike Channell. 2020. The 20 best driving games of the last decade: 20-11. https://www.topgear.com/car-news/electric/20-best-driving-games-last-decade-20-11.

[8]

Daniel K.Y. Chen, Jean-Baptiste Chossat, and Peter B. Shull. 2019. HaptiVec: Presenting Haptic Feedback Vectors in Handheld Controllers Using Embedded Tactile Pin Arrays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300401

Digital Library

[9]

Alfredo Cuzzocrea. 2009. Synopsis data structures for representing, querying, and mining data streams. In Handbook of Research on Innovations in Database Technologies and Applications: Current and Future Trends. IGI Global, Pennsylvania, USA, 701–715.

[10]

Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui, and Anne-Marie Kermarrec. 2003. The Many Faces of Publish/Subscribe. ACM Comput. Surv. 35, 2 (June 2003), 114–131. https://doi.org/10.1145/857076.857078

Digital Library

[11]

ForceTubeVR. 2020. ForceTubeVR Documentation. https://www.protubevr.com/img/cms/ForceTubeVR_Documentation.pdf

[12]

Samuel Horti GamesRadar Staff. 2020. The 25 best FPS games of all time. https://www.gamesradar.com/best-fps-games/.

[13]

Topi Kaaresoja, Stephen Brewster, and Vuokko Lantz. 2014. Towards the Temporally Perfect Virtual Button: Touch-Feedback Simultaneity and Perceived Quality in Mobile Touchscreen Press Interactions. ACM Trans. Appl. Percept. 11, 2, Article 9 (June 2014), 25 pages. https://doi.org/10.1145/2611387

Digital Library

[14]

Hayato Kajiyama, Akifumi Inoue, and Tohru Hoshi. 2015. SHAPIO: Shape I/O Controller for Video Games. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (London, United Kingdom) (CHI PLAY ’15). Association for Computing Machinery, New York, NY, USA, 565–570. https://doi.org/10.1145/2793107.2810318

Digital Library

[15]

Shi-Hong Liu, Pai-Chien Yen, Yi-Hsuan Mao, Yu-Hsin Lin, Erick Chandra, and Mike Y. Chen. 2020. HeadBlaster: A Wearable Approach to Simulating Motion Perception Using Head-Mounted Air Propulsion Jets. ACM Trans. Graph. 39, 4, Article 84 (jul 2020), 12 pages. https://doi.org/10.1145/3386569.3392482

Digital Library

[16]

Logitech. 2015. Logitech G920 & G29 Driving Force Steering Wheels & Pedals. https://www.logitechg.com/en-us/products/driving/driving-force-racing-wheel.html

[17]

Kuba Łopatka, Józef Kotus, and Andrzej Czyżewski. 2015. Detection, classification and localization of acoustic events in the presence of background noise for acoustic surveillance of hazardous situations. Multimedia Tools and Applications 429 (12 2015). https://doi.org/10.1007/s11042-015-3105-4

Digital Library

[18]

G Mirone. 2010. Multi-body elastic simulation of a go-kart: Correlation between frame stiffness and dynamic performance. International Journal of Automotive Technology 11, 4(2010), 461–469.

[19]

Jakob Nielsen. 1993. Usability engineering. Morgan Kaufmann an imprint of Academic Press, a Harcourt Science and Technology Company, Burlington, Massachusetts.

[20]

Miller Puckette. 1996. Pure Data. In Proceedings of the second intercollege computer music concerts. Self-published, Tachikawa, Japan, 37–41.

[21]

Felix Richter. 2019. The Most Important Gaming Platforms in 2019. https://www.statista.com/chart/4527/game-developers-platform-preferences/

[22]

SimRacingStudio. 2020. Sim Racing Studio. https://www.simracingstudio.com/

[23]

Freepik Company S.L.2020. Flaticon, the largest database of free vector icons. https://www.flaticon.com/

[24]

R. Smith. 2007. An Overview of the Tesseract OCR Engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), Vol. 2. IEEE, Parana, Brazil, 629–633. https://doi.org/10.1109/ICDAR.2007.4376991

[25]

Monstrous Software. 2003. Car Physics for Games. https://asawicki.info/Mirror/Car Physics for Games/Car Physics for Games.html

[26]

Valve. 2020. Steam Hardware and Software Survey. https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam/

[27]

Xbox. 2016. Xbox Wireless Controller. https://www.xbox.com/en-US/accessories/controllers/xbox-wireless-controller

[28]

Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: Using GUI Screenshots for Search and Automation. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST ’09). Association for Computing Machinery, New York, NY, USA, 183–192. https://doi.org/10.1145/1622176.1622213

Digital Library

Cited By

Samanta SBanerjee SUpadhyay P(2025)How Does Games User Research Measure and Guide Haptic Experience Design? A Critical ReviewHuman-Computer Interaction. Design and Research10.1007/978-3-031-80832-6_2(23-38)Online publication date: 14-Feb-2025
https://doi.org/10.1007/978-3-031-80832-6_2
Crespo-Aguado MLozano RHernandez-Gobertti FMolner NGomez-Barquero D(2024)Flexible Hyper-Distributed IoT–Edge–Cloud Platform for Real-Time Digital Twin Applications on 6G-Intended Testbeds for Logistics and IndustryFuture Internet10.3390/fi1611043116:11(431)Online publication date: 20-Nov-2024
https://doi.org/10.3390/fi16110431
Cheng CChen YHandani SBalabantaray AChen M(2024)Paired-EMS: Enhancing Electrical Muscle Stimulation (EMS)-based Force Feedback Experience by Stimulating Both Muscles in Antagonistic PairsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642841(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642841
Show More Cited By

Index Terms

HapticSeer: A Multi-channel, Black-box, Platform-agnostic Approach to Detecting Video Game Events for Real-time Haptic Feedback

Index terms have been assigned to the content through auto-classification.

Recommendations

Haptic around: multiple tactile sensations for immersive environment and interaction in virtual reality
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

In this paper, we present Haptic Around, a hybrid-haptic feedback system, which utilizes fan, hot air blower, mist creator and heat light to recreate multiple tactile sensations in virtual reality for enhancing the immersive environment and interaction. ...
The Effect of Body-Based Haptic Feedback on Player Experience During VR Gaming
Virtual, Augmented and Mixed Reality
Abstract
As the interest in virtual reality (VR) technology as a game console has rapidly grown over the past few years, many new technologies are also being developed to further enhance the VR gaming experience. One of these technologies is haptic ...
Miniature Haptics: Experiencing Haptic Feedback through Hand-based and Embodied Avatars
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

We present Miniature Haptics, a new approach to providing realistic haptic experiences by applying miniaturized haptic feedback to hand-based, embodied avatars. By shrinking haptics to a much smaller scale, Miniature Haptics enables the exploration of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Ministry of Science and Technology of Taiwan

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
751
Total Downloads

Downloads (Last 12 months)106
Downloads (Last 6 weeks)24

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Samanta SBanerjee SUpadhyay P(2025)How Does Games User Research Measure and Guide Haptic Experience Design? A Critical ReviewHuman-Computer Interaction. Design and Research10.1007/978-3-031-80832-6_2(23-38)Online publication date: 14-Feb-2025
https://doi.org/10.1007/978-3-031-80832-6_2
Crespo-Aguado MLozano RHernandez-Gobertti FMolner NGomez-Barquero D(2024)Flexible Hyper-Distributed IoT–Edge–Cloud Platform for Real-Time Digital Twin Applications on 6G-Intended Testbeds for Logistics and IndustryFuture Internet10.3390/fi1611043116:11(431)Online publication date: 20-Nov-2024
https://doi.org/10.3390/fi16110431
Cheng CChen YHandani SBalabantaray AChen M(2024)Paired-EMS: Enhancing Electrical Muscle Stimulation (EMS)-based Force Feedback Experience by Stimulating Both Muscles in Antagonistic PairsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642841(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642841
Chen JWang YLuximon Y(2024)CamTroller: An Auxiliary Tool for Controlling Your Avatar in PC Games Using Natural Motion MappingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642511(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642511
Jung KKim SOh SYoon S(2024)HapMotion: motion-to-tactile framework with wearable haptic devices for immersive VR performance experienceVirtual Reality10.1007/s10055-023-00910-z28:1Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1007/s10055-023-00910-z
Yun GMun MLee JKim DTan HChoi S(2023)Generating Real-Time, Selective, and Multimodal Haptic Effects from Sound for Gaming Experience EnhancementProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580787(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580787
Mamoru DPanditha APerera WGanegoda G(2022)Conceptual Representation and Evaluation of an FPS Game Commentary Generator2022 2nd International Conference on Image Processing and Robotics (ICIPRob)10.1109/ICIPRob54042.2022.9798735(1-6)Online publication date: 12-Mar-2022
https://doi.org/10.1109/ICIPRob54042.2022.9798735

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten