KeyFlow: Acoustic Motion Sensing for Cursor Control on Any Keyboard
Authors: 
Yujia Liu, Qihang Shan, Zhihao Yao, Qiuyu LuAuthors Info & Claims
Yujia Liu, Qihang Shan, Zhihao Yao, Qiuyu LuAuthors Info & ClaimsArticle No.: 84, Pages 1 - 3
Abstract
Despite typing being a critical operation in the digital age, users still need to frequently switch between the mouse and keyboard while typing. We introduce KeyFlow, a tool that integrates mouse functionality into the keyboard through machine learning, allowing users to glide their fingers across the keyboard surface to move the cursor. The whole process does not press the keys down to differentiate from normal typing and avoid false touches. KeyFlow uses any computer-built-in microphones to capture the acoustic features of these gliding gestures, requiring no specialized equipment and can be set up and tested independently within 5 minutes. Our user research indicates that, compared to traditional keyboard and mouse methods, this system reduces hand movement distance by 78.3%, making the typing experience more focused.
Supplemental Material
MP4 File
Presentation video - short version
- Download
- 24.28 MB
PDF File
Poster
- Download
- 2.20 MB
References
[1]
Xiaojun Bi, Barton A. Smith, and Shumin Zhai. 2010. Quasi-Qwerty Soft Keyboard Optimization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 283–286. https://doi.org/10.1145/1753326.1753367
[2]
John Brooke. 1996. Sus: a “quick and dirty’usability. Usability evaluation in industry 189, 3 (1996), 189–194.
[3]
Stuart K Card, William K English, and Betty J Burr. 1978. Evaluation of mouse, rate-controlled isometric joystick, step keys, and text keys for text selection on a CRT. Ergonomics 21, 8 (1978), 601–613.
[4]
Wenqiang Chen, Yanming Lian, Lu Wang, Rukhsana Ruby, Wen Hu, and Kaishun Wu. 2017. Virtual Keyboard for Wearable Wristbands. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems (Delft, Netherlands) (SenSys ’17). Association for Computing Machinery, New York, NY, USA, Article 44, 2 pages. https://doi.org/10.1145/3131672.3136984
[5]
Wolfgang Fallot-Burghardt, Morten Fjeld, C Speirs, S Ziegenspeck, Helmut Krueger, and Thomas Läubli. 2006. Touch&Type: a novel pointing device for notebook computers. In Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles. 465–468.
[6]
Mayank Goel, Brendan Lee, Md. Tanvir Islam Aumi, Shwetak Patel, Gaetano Borriello, Stacie Hibino, and Bo Begole. 2014. SurfaceLink: Using Inertial and Acoustic Sensing to Enable Multi-Device Interaction on a Surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1387–1396. https://doi.org/10.1145/2556288.2557120
[7]
Teng Han, Khalad Hasan, Keisuke Nakamura, Randy Gomez, and Pourang Irani. 2017. SoundCraft: Enabling Spatial Interactions on Smartwatches Using Hand Generated Acoustics. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 579–591. https://doi.org/10.1145/3126594.3126612
[8]
Chris Harrison and Scott E. Hudson. 2008. Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST ’08). Association for Computing Machinery, New York, NY, USA, 205–208. https://doi.org/10.1145/1449715.1449747
[9]
Jinki Jung, Jinwoo Jeon, Hyeopwoo Lee, Kichan Kwon, Jamal Zemerly, and Hyun Seung Yang. 2014. Augmented Keyboard: A Virtual Keyboard Interface for Smart Glasses. In Proceedings of the 13th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (Shenzhen, China) (VRCAI ’14). Association for Computing Machinery, New York, NY, USA, 159–164. https://doi.org/10.1145/2670473.2670482
[10]
Ülkü Meteriz Yýldýran, Necip Fazýl Yýldýran, and David Mohaisen. 2022. AcousticType: Smartwatch-Enabled Cross-Device Text Entry Method Using Keyboard Acoustics. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 352, 7 pages. https://doi.org/10.1145/3491101.3519691
[11]
OpenAI. 2022. AI-powered language model developed by OpenAI. OpenAI. https://chat.openai.com/
[12]
Stuart Taylor, Cem Keskin, Otmar Hilliges, Shahram Izadi, and John Helmes. 2014. Type-Hover-Swipe in 96 Bytes: A Motion Sensing Mechanical Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1695–1704. https://doi.org/10.1145/2556288.2557030
[13]
Robert Wang, Sylvain Paris, and Jovan Popović. 2011. 6D hands: markerless hand-tracking for computer aided design. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 549–558.
[14]
Andrew D Wilson. 2006. Robust computer vision-based detection of pinching for one and two-handed gesture input. In Proceedings of the 19th annual ACM symposium on User interface software and technology. 255–258.
[15]
Naoki Yanagihara and Buntarou Shizuki. 2018. Cubic Keyboard for Virtual Reality. In Proceedings of the 2018 ACM Symposium on Spatial User Interaction (Berlin, Germany) (SUI ’18). Association for Computing Machinery, New York, NY, USA, 170. https://doi.org/10.1145/3267782.3274687
[16]
Mingrui Ray Zhang, He Wen, and Jacob O. Wobbrock. 2019. Type, Then Correct: Intelligent Text Correction Techniques for Mobile Text Entry Using Neural Networks. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 843–855. https://doi.org/10.1145/3332165.3347924
Index Terms
- KeyFlow: Acoustic Motion Sensing for Cursor Control on Any Keyboard
Recommendations
Gesture keyboard with a machine learning requiring only one camera
AH '12: Proceedings of the 3rd Augmented Human International ConferenceIn this paper, the authors propose a novel gesture-based virtual keyboard (Gesture Keyboard) that uses a standard QWERTY keyboard layout, and requires only one camera, and employs a machine learning technique. Gesture Keyboard tracks the user's fingers ...
Text Entry for Ultra-Small Touchscreens Using a Fixed Cursor and Movable Keyboard
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing SystemsEmerging ultra-small wearables like smartwatches pose a design challenge for touch-based text entry, due to the "fat finger" problem, wherein users struggle to select elements much smaller than their fingers. In order to address the challenge, we ...
Comments
Information & Contributors
Information
Published In
October 2024
394 pages
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 October 2024
Check for updates
Author Tags
Qualifiers
- Poster
- Research
- Refereed limited
Conference
UIST '24: The 37th Annual ACM Symposium on User Interface Software and Technology
October 13 - 16, 2024
PA, Pittsburgh, USA
Acceptance Rates
Overall Acceptance Rate 355 of 1,733 submissions, 20%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 81Total Downloads
- Downloads (Last 12 months)81
- Downloads (Last 6 weeks)19
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format