skip to main content
10.1145/3654777.3676328acmotherconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

HandPad: Make Your Hand an On-the-go Writing Pad via Human Capacitance

Published: 11 October 2024 Publication History

Abstract

The convenient text input system is a pain point for devices such as AR glasses, and it is difficult for existing solutions to balance portability and efficiency. This paper introduces HandPad, the system that turns the hand into an on-the-go touchscreen, which realizes interaction on the hand via human capacitance. HandPad achieves keystroke and handwriting inputs for letters, numbers, and Chinese characters, reducing the dependency on capacitive or pressure sensor arrays. Specifically, the system verifies the feasibility of touch point localization on the hand using the human capacitance model and proposes a handwriting recognition system based on Bi-LSTM and ResNet. The transfer learning-based system only needs a small amount of training data to build a handwriting recognition model for the target user. Experiments in real environments verify the feasibility of HandPad for keystroke (accuracy of 100%) and handwriting recognition for letters (accuracy of 99.1%), numbers (accuracy of 97.6%) and Chinese characters (accuracy of 97.9%).

References

[1]
Jiban Adhikary and Keith Vertanen. 2021. Text Entry in Virtual Environments using Speech and a Midair Keyboard. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2648–2658. https://doi.org/10.1109/TVCG.2021.3067776
[2]
A Ahlbom, U Bergqvist, JH Bernhardt, JP Cesarini, M Grandolfo, M Hietanen, AF Mckinlay, MH Repacholi, David H Sliney, J AJ Stolwijk, 1998. Guidelines for limiting exposure to time-varying electric, magnetic, and electromagnetic fields (up to 300 GHz). Health physics 74, 4 (1998), 494–521.
[3]
Karan Ahuja, Paul Streli, and Christian Holz. 2021. TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 997–1009. https://doi.org/10.1145/3472749.3474801
[4]
Tobias Boceck, Sascha Sprott, Huy Viet Le, and Sven Mayer. 2019. Force Touch Detection on Capacitive Sensors Using Deep Neural Networks(MobileHCI ’19). Article 42, 6 pages.
[5]
Jingye Chen, Bin Li, and Xiangyang Xue. 2021. Zero-Shot Chinese Character Recognition with Stroke-Level Decomposition. CoRR abs/2106.11613 (2021). arXiv:2106.11613https://arxiv.org/abs/2106.11613
[6]
Mingshi Chen, Panlong Yang, Jie Xiong, Maotian Zhang, Youngki Lee, Chaocan Xiang, and Chang Tian. 2019. Your Table Can Be an Input Panel: Acoustic-Based Device-Free Interaction Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 1, Article 3 (mar 2019), 21 pages. https://doi.org/10.1145/3314390
[7]
Taizhou Chen, Tianpei Li, Xingyu Yang, and Kening Zhu. 2023. Efring: Enabling thumb-to-index-finger microgesture interaction through electric field sensing using single smart ring. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 4 (2023), 1–31.
[8]
Wenqiang Chen, Lin Chen, Yandao Huang, Xinyu Zhang, Lu Wang, Rukhsana Ruby, and Kaishun Wu. 2019. Taprint: Secure Text Input for Commodity Smart Wristbands(MobiCom ’19). Article 17, 16 pages.
[9]
Wenqiang Chen, Maoning Guan, Yandao Huang, Lu Wang, Rukhsana Ruby, Wen Hu, and Kaishun Wu. 2018. ViType: A Cost Efficient On-Body Typing System through Vibration. In 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). 1–9. https://doi.org/10.1109/SAHCN.2018.8397098
[10]
Frederick Choi, Sven Mayer, and Chris Harrison. 2021. 3D Hand Pose Estimation on Conventional Capacitive Touchscreens. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction (Toulouse amp; Virtual, France) (MobileHCI ’21). Association for Computing Machinery, New York, NY, USA, Article 3, 13 pages. https://doi.org/10.1145/3447526.3472045
[11]
Gabe Cohn, Daniel Morris, Shwetak N. Patel, and Desney S. Tan. 2011. Your Noise is My Command: Sensing Gestures Using the Body as an Antenna. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 791–800. https://doi.org/10.1145/1978942.1979058
[12]
Christian Corsten, Simon Voelker, Andreas Link, and Jan Borchers. 2018. Use the Force Picker, Luke: Space-Efficient Value Input on Force-Sensitive Mobile Touchscreens(CHI ’18). 1–12.
[13]
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion Models in Vision: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023), 1–20. https://doi.org/10.1109/TPAMI.2023.3261988
[14]
Dian Ding, Lanqing Yang, Yi-Chao Chen, and Guangtao Xue. 2021. VibWriter: Handwriting Recognition System based on Vibration Signal. In 2021 18th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). 1–9. https://doi.org/10.1109/SECON52354.2021.9491615
[15]
Dian Ding, Lanqing Yang, Yi-Chao Chen, and Guangtao Xue. 2022. Handwriting Recognition System Leveraging Vibration Signal on Smartphones. IEEE Transactions on Mobile Computing (2022), 1–1. https://doi.org/10.1109/TMC.2022.3148172
[16]
Dian Ding, Lanqing Yang, Yi-Chao Chen, and Guangtao Xue. 2022. Leakage or Identification: Behavior-Irrelevant User Identification Leveraging Leakage Current on Laptops. PACM IMWUT. 5, 4, Article 152 (dec 2022), 23 pages.
[17]
David Dobbelstein, Christian Winkler, Gabriel Haas, and Enrico Rukzio. 2017. PocketThumb: A Wearable Dual-Sided Touch Interface for Cursor-Based Control of Smart-Eyewear. PACM IMWUT. 1, 2, Article 9 (jun 2017), 17 pages. https://doi.org/10.1145/3090055
[18]
Fengyi Fang, Hongwei Zhang, Lishuang Zhan, Shihui Guo, Minying Zhang, Juncong Lin, Yipeng Qin, and Hongbo Fu. 2023. Handwriting Velcro: Endowing AR Glasses with Personalized and Posture-Adaptive Text Input Using Flexible Touch Sensor. PACM IMWUT. 6, 4, Article 163 (jan 2023), 31 pages.
[19]
Jacqui Fashimpaur, Kenrick Kin, and Matt Longest. 2020. PinchType: Text Entry for Virtual and Augmented Reality Using Comfortable Thumb to Fingertip Pinches. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (, Honolulu, HI, USA, ) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–7.
[20]
[20] TI FDC2214. 2022. https://www.ti.com/product/FDC2214
[21]
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research 17, 1 (2016), 2096–2030.
[22]
Oliver Glauser, Shihao Wu, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine-Hornung. 2019. Interactive Hand Pose Estimation Using a Stretch-Sensing Soft Glove. ACM Trans. Graph. 38, 4, Article 41 (jul 2019), 15 pages. https://doi.org/10.1145/3306346.3322957
[23]
Alix Goguey, Sylvain Malacria, and Carl Gutwin. 2018. Improving Discoverability and Expert Performance in Force-Sensitive Text Selection for Touch Devices with Mode Gauges. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3174051
[24]
[24] Grepow. [n. d.]. https://www.grepow.com/wearables/smart-ring.html
[25]
Zhengxin Guo, Fu Xiao, Biyun Sheng, Huan Fei, and Shui Yu. 2020. WiReader: Adaptive Air Handwriting Recognition Based on Commercial WiFi Signal. IEEE Internet of Things Journal 7, 10 (2020), 10483–10494. https://doi.org/10.1109/JIOT.2020.2997053
[26]
Aakar Gupta, Muhammed Anwar, and Ravin Balakrishnan. 2016. Porous Interfaces for Small Screen Multitasking Using Finger Identification(UIST ’16). New York, NY, USA, 145–156.
[27]
Aakar Gupta and Ravin Balakrishnan. 2016. DualKey: Miniature Screen Text Entry via Finger Identification. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 59–70. https://doi.org/10.1145/2858036.2858052
[28]
Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces(UIST ’11). 627–636.
[29]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
[30]
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015).
[31]
Kaori Ikematsu, Masaaki Fukumoto, and Itiro Siio. 2019. Ohmic-Sticker: Force-to-Motion Type Input Device That Extends Capacitive Touch Surface(UIST ’19). 1021–1030.
[32]
Kaori Ikematsu and Shota Yamanaka. 2020. ScraTouch: Extending Interaction Technique Using Fingernail on Unmodified Capacitive Touch Surfaces. PACM IMWUT. 4, 3, Article 81 (sep 2020), 19 pages.
[33]
Yan Jiang, Xiaoyu Ji, Kai Wang, Chen Yan, Richard Mitev, Ahmad-Reza Sadeghi, and Wenyuan Xu. 2022. WIGHT: Wired Ghost Touch Attack on Capacitive Touchscreens. In 2022 IEEE Symposium on Security and Privacy (SP). 984–1001. https://doi.org/10.1109/SP46214.2022.9833740
[34]
Niels Jonassen. 1998. Human body capacitance: static or dynamic concept?[ESD]. In Electrical Overstress/Electrostatic Discharge Symposium Proceedings. 1998 (Cat. No. 98TH8347). IEEE, 111–117.
[35]
Wolf Kienzle, Eric Whitmire, Chris Rittaler, and Hrvoje Benko. 2021. ElectroRing: Subtle Pinch and Touch Detection with a Ring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 3, 12 pages. https://doi.org/10.1145/3411764.3445094
[36]
Jiwan Kim and Ian Oakley. 2022. SonarID: Using Sonar to Identify Fingers on a Smartwatch(CHI ’22). Article 287, 10 pages.
[37]
Maxime W Lafarge, Josien PW Pluim, Koen AJ Eppenhof, Pim Moeskops, and Mitko Veta. 2017. Domain-adversarial neural networks to address the appearance variability of histopathology images. In Third International Workshop, DLMIA 2017. Springer, 83–91.
[38]
Huy Viet Le, Thomas Kosch, Patrick Bader, Sven Mayer, and Niels Henze. 2018. PalmTouch: Using the Palm as an Additional Input Modality on Commodity Smartphones(CHI ’18). 1–13.
[39]
Huy Viet Le, Sven Mayer, and Niels Henze. 2018. InfiniTouch: Finger-Aware Interaction on Fully Touch Sensitive Smartphones. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 779–792. https://doi.org/10.1145/3242587.3242605
[40]
Huy Viet Le, Sven Mayer, and Niels Henze. 2019. Investigating the Feasibility of Finger Identification on Capacitive Touchscreens Using Deep Learning(IUI ’19). Association for Computing Machinery, New York, NY, USA, 637–649.
[41]
Chang-Ju Lee, Jong Kang Park, Canxing Piao, Han-Eol Seo, Jaehyuk Choi, and Jung-Hoon Chun. 2018. Mutual Capacitive Sensing Touch Screen Controller for Ultrathin Display with Extended Signal Passband Using Negative Capacitance. Sensors 18, 11 (2018). https://doi.org/10.3390/s18113637
[42]
Yijie Li, Yi-Chao Chen, Xiaoyu Ji, Hao Pan, Lanqing Yang, Guangtao Xue, and Jiadi Yu. 2021. Screenid: Enhancing qrcode security by fingerprinting screens. In IEEE INFOCOM 2021-IEEE Conference on Computer Communications. IEEE, 1–10.
[43]
Yijie Li, Juntao Zhou, Dian Ding, Yi-Chao Chen, Lili Qiu, Jiadi Yu, and Guangtao Xue. 2024. MuDiS: An Audio-independent, Wide-angle, and Leak-free Multi-directional Speaker. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 263–278.
[44]
Yihao Liu, Kai Huang, Xingzhe Song, Boyuan Yang, and Wei Gao. 2020. MagHacker: Eavesdropping on Stylus Pen Writing via Magnetic Sensing from Commodity Mobile Devices(MobiSys ’20). 148–160.
[45]
Zongjian Liu, Jieling He, Jianjiang Feng, and Jie Zhou. 2023. PrinType: Text Entry via Fingerprint Recognition. PACM IMWUT. 6, 4, Article 174 (jan 2023), 31 pages.
[46]
Philipp Markert, Daniel V. Bailey, Maximilian Golla, Markus Dürmuth, and Adam J. Aviv. 2020. This PIN Can Be Easily Guessed: Analyzing the Security of Smartphone Unlock PINs. In IEEE Symposium on Security and Privacy (SP). 286–303. https://doi.org/10.1109/SP40000.2020.00100
[47]
Sven Mayer, Huy Viet Le, and Niels Henze. 2017. Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks(ISS ’17). 220–229.
[48]
Akihito Miyamoto, Sungwon Lee, Nawalage Florence Cooray, Sunghoon Lee, Mami Mori, Naoji Matsuhisa, Hanbit Jin, Leona Yoda, Tomoyuki Yokota, Akira Itoh, 2017. Inflammation-free, gas-permeable, lightweight, stretchable on-skin electronics with nanomeshes. Nature nanotechnology 12, 9 (2017), 907–913.
[49]
Hyoungsik Nam, Ki-Hyuk Seol, Junhee Lee, Hyeonseong Cho, and Sang Won Jung. 2021. Review of Capacitive Touchscreen Technologies: Overview, Research Trends, and Machine Learning Approaches. Sensors 21, 14 (2021). https://doi.org/10.3390/s21144776
[50]
Ian Oakley, Carina Lindahl, Khanh Le, DoYoung Lee, and MD. Rasel Islam. 2016. The Flat Finger: Exploring Area Touches on Smartwatches(CHI ’16). New York, NY, USA, 4238–4249.
[51]
Hao Pan, Yi-Chao Chen, Qi Ye, and Guangtao Xue. 2021. MagicInput: Training-Free Multi-Lingual Finger Input System Using Data Augmentation Based on MNISTs(IPSN ’21). New York, NY, USA, 119–131.
[52]
Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2010), 1345–1359.
[53]
Maicon D. Pereira, Germán A. Alvarez-Botero, and Fernando Rangel de Sousa. 2015. Characterization and Modeling of the Capacitive HBC Channel. IEEE Transactions on Instrumentation and Measurement 64, 10 (2015), 2626–2635. https://doi.org/10.1109/TIM.2015.2420391
[54]
Ronald Pethig. 1987. Dielectric properties of body tissues. Clinical Physics and Physiological Measurement 8, 4A (1987), 5.
[55]
Simon Rogers, John Williamson, Craig Stewart, and Roderick Murray-Smith. 2011. AnglePose: Robust, Precise Capacitive Touch Tracking via 3d Orientation Estimation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2575–2584. https://doi.org/10.1145/1978942.1979318
[56]
Juan-Yao Ruan, Paul C.-P. Chao, and Wei-Dar Chen. 2010. A multi-touch interface circuit for a large-sized capacitive touch panel. In SENSORS, 2010 IEEE. 309–314. https://doi.org/10.1109/ICSENS.2010.5689881
[57]
[57] Nordic Semiconductor. [n. d.]. https://www.nordicsemi.com/Products/nRF52832
[58]
[58] System Usability Scale (SUS). [n. d.]. https://www.usability.gov/how-to-andtools/methods/system-usability-scale.html.
[59]
[59] NASA task load Index (NASA-TLX). [n. d.]. https://humansystems.arc.nasa.gov/groups/tlx/downloads/TLXScale.pdf
[60]
Edward Jay Wang, Jake Garrison, Eric Whitmire, Mayank Goel, and Shwetak Patel. 2017. Carpacio: Repurposing Capacitive Sensors to Distinguish Driver and Passenger Touches on In-Vehicle Screens(UIST ’17). 49–55.
[61]
Shaolei Wang, Baoxin Wang, Jiefu Gong, Zhongyuan Wang, Xiao Hu, Xingyi Duan, Zizhuo Shen, Gang Yue, Ruiji Fu, Dayong Wu, Wanxiang Che, Shijin Wang, Guoping Hu, and Ting Liu. 2020. Combining ResNet and Transformer for Chinese Grammatical Error Diagnosis. In Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications. 36–43.
[62]
Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-Mounted Displays. PACM IMWUT. 1, 3, Article 113 (sep 2017), 21 pages.
[63]
Mathias Wilhelm, Daniel Krakowczyk, and Sahin Albayrak. 2020. PeriSense: ring-based multi-finger gesture interaction utilizing capacitive proximity sensing. Sensors 20, 14 (2020), 3990.
[64]
WK Wong, Filbert H Juwono, and Brendan Teng Thiam Khoo. 2021. Multi-features capacitive hand gesture recognition sensor: A machine learning approach. IEEE sensors journal 21, 6 (2021), 8441–8450.
[65]
Kaishun Wu, Qiang Yang, Baojie Yuan, Yongpan Zou, Rukhsana Ruby, and Mo Li. 2021. EchoWrite: An Acoustic-Based Finger Input System Without Training. IEEE Transactions on Mobile Computing 20, 5 (2021), 1789–1803. https://doi.org/10.1109/TMC.2020.2973094
[66]
Robert Xiao, Julia Schwarz, and Chris Harrison. 2015. Estimating 3D Finger Angle on Commodity Touchscreens(ITS ’15). 47–50.
[67]
Zhenyu Yan, Qun Song, Rui Tan, Yang Li, and Adams Wai Kin Kong. 2019. Towards Touch-to-Access Device Authentication Using Induced Body Electric Potentials(MobiCom ’19). Article 23, 16 pages.
[68]
Shanhe Yi, Zhengrui Qin, Ed Novak, Yafeng Yin, and Qun Li. 2016. GlassGesture: Exploring head gesture interface of smart glasses. In IEEE INFOCOM 2016. 1–9. https://doi.org/10.1109/INFOCOM.2016.7524542
[69]
Huanpu Yin, Anfu Zhou, Guangyuan Su, Bo Chen, Liang Liu, and Huadong Ma. 2020. Learning to Recognize Handwriting Input with Acoustic Features. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 64 (jun 2020), 26 pages.
[70]
Yongsang Yoo and Byong-Deok Choi. 2021. Readout Circuits for Capacitive Sensors. Micromachines 12, 8 (2021).
[71]
Yongzhao Zhang, Wei-Hsiang Huang, Chih-Yun Yang, Wen-Ping Wang, Yi-Chao Chen, Chuang-Wen You, Da-Yuan Huang, Guangtao Xue, and Jiadi Yu. 2020. Endophasia: Utilizing Acoustic-Based Imaging for Issuing Contact-Free Silent Speech Commands. PACM IMWUT. 4, 1, Article 37 (mar 2020), 26 pages.
[72]
Hao Zhou, Taiting Lu, Yilin Liu, Shijia Zhang, Runze Liu, and Mahanth Gowda. 2023. One ring to rule them all: An open source smartring platform for finger motion analytics and healthcare applications. In Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation. 27–38.

Index Terms

  1. HandPad: Make Your Hand an On-the-go Writing Pad via Human Capacitance

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    UIST '24: Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
    October 2024
    2334 pages
    ISBN:9798400706288
    DOI:10.1145/3654777
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 October 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Handwriting Input
    2. Human Capacitance
    3. Key Stroke

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    • NSFC

    Conference

    UIST '24

    Acceptance Rates

    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 375
      Total Downloads
    • Downloads (Last 12 months)375
    • Downloads (Last 6 weeks)75
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media