skip to main content
10.1145/3411763.3451689acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
poster
Open access

AI-on-skin: Enabling On-body AI Inference for Wearable Artificial Skin Interfaces

Published: 08 May 2021 Publication History

Abstract

Existing artificial skin interfaces suffer from the lack of on-skin compute that can provide fast neural network inference for time-critical application scenarios. In this paper, we propose AI-on-skin - a wearable artificial skin interface integrated with a neural network hardware accelerator that can be reconfigured across diverse neural network models and applications. AI-on-skin is designed to scale to the entire body, comprising tiny, low-power, accelerators distributed across the body. We built a prototype of AI-on-skin that covers the entire forearm (17 by 10 cm) based on off-the-shelf FPGAs. Our electronic skin based prototype can perform (a) handwriting recognition with 96% accuracy, (b) gesture recognition with 95% accuracy and (c) handwritten word recognition with 93.5% accuracy. AI-On-Skin achieves  20X and 35X speedup over off-body inference via bluetooth and on-body microcontroller based inference approach respectively. To the best of our knowledge, AI-On-Skin is the first ever wearable prototype to demonstrate skin interfaces with on-body neural network inference.

References

[1]
2014. CC2650 sensortag user’s guide - texas instruments wiki. https://processors.wiki.ti.com/index.php/CC2650_SensorTag_User's_Guide
[2]
2015. How Fast is Realtime? Human Perception and Technology. https://www.pubnub.com/blog/how-fast-is-realtime-human-perception-and-technology/
[3]
2016. DFRobot Beetle BLE - The Smallest Board Based on Arduino Uno with Bluetooth 4.0. https://www.dfrobot.com/product-1259.html
[4]
2019. Teensy 4.0 board specifications. https://www.pjrc.com/teensy/datasheets.html
[5]
2020. Shenjing RTL for AI acceleration. https://github.com/Angela-WangBo/Shenjing-RTL
[6]
2020. Skin-like sensors bring a human touch to wearable tech. Retrieved January 16, 2020 from https://tectales.com/wearables-sensors/skin-like-sensors-bring-a-human-touch-to-wearable-tech.html
[7]
2020. Tensorflow 2.1.0 for microcontrollers benchmarks on Teensy 4.0. Retrieved May 30, 2020 from https://www.stupid-projects.com/tensorflow-2-1-0-for-microcontrollers-benchmarks-on-teensy-4-0/
[8]
Gordon Cheng, Emmanuel Dean-Leon, Florian Bergner, Julio Rogelio Guadarrama Olvera, Quentin Leboutet, and Philipp Mittendorfer. 2019. A Comprehensive Realization of Robot Skin: Sensors, Sensing, Control, and Applications. Proc. IEEE PP (08 2019), 1–18. https://doi.org/10.1109/JPROC.2019.2933348
[9]
Xi Duan, Sebastien Taurand, and Manuchehr Soleimani. 2019. Artificial skin through super-sensing method and electrical impedance data from conductive fabric with aid of deep learning. Scientific reports 9, 1 (2019), 1–11.
[10]
David Fergenson, William D. Richard, Tony Peguero, Dfergenson, Richard V, DigitalConfig, William Orosz, Marty, Oem User, Gary Block, and et al.2018. Cmod A7: Breadboardable Artix-7 FPGA Module. https://store.digilentinc.com/cmod-a7-breadboardable-artix-7-fpga-module/
[11]
Eduardo Garcia-Espinosa, Omar Longoria-Gandara, Ioseth Pegueros-Lepe, and Arturo Veloz-Guerrero. 2018. Power consumption analysis of Bluetooth Low Energy commercial products and their implications for IoT aplications. Electronics 7, 12 (2018), 386.
[12]
Hsin-Liu Kao, Miren Bamforth, David Kim, and Chris Schmandt. 2018. Skinmorph: texture-tunable on-skin interface through thin, programmable gel. In Proceedings of the 2018 ACM International Symposium on Wearable Computers. 196–203.
[13]
Hsin-Liu Kao, Christian Holz, Asta Roseway, Andres Calvo, and Chris Schmandt. 2016. DuoSkin: rapidly prototyping on-skin user interfaces using skin-friendly materials. In Proceedings of the 2016 ACM International Symposium on Wearable Computers. 16–23.
[14]
Hsin-Liu Cindy Kao, Abdelkareem Bedri, and Kent Lyons. 2018. SkinWire: Fabricating a Self-Contained On-Skin PCB for the Hand. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1–23.
[15]
Dooyoung Kim, Junghan Kwon, Seunghyun Han, Yong-Lae Park, and Sungho Jo. 2018. Deep full-body motion network for a soft wearable motion sensing suit. IEEE/ASME Transactions on Mechatronics 24, 1 (2018), 56–66.
[16]
Kyun Kyu Kim, InHo Ha, Min Kim, Joonhwa Choi, Phillip Won, Sungho Jo, and Seung Hwan Ko. 2020. A deep-learned skin sensor decoding the epicentral human motions. Nature Communications 11, 1 (2020), 1–8.
[17]
Liangzhen Lai, Naveen Suda, and Vikas Chandra. 2018. CMSIS-NN: Efficient neural network kernels for arm cortex-m cpus. arXiv preprint arXiv:1801.06601(2018).
[18]
Chris Larson, Josef Spjut, Ross Knepper, and Robert Shepherd. 2019. A deformable interface for human touch recognition using stretchable carbon nanotube dielectric elastomer sensors and deep neural networks. Soft robotics 6, 5 (2019), 611–620.
[19]
Wang Wei Lee, Yu Jun Tan, Haicheng Yao, Si Li, Hian Hian See, Matthew Hon, Kian Ann Ng, Betty Xiong, John S Ho, and Benjamin CK Tee. 2019. A neuro-inspired artificial peripheral nervous system for scalable electronic skins. Science Robotics 4, 32 (2019), eaax2198.
[20]
GY Liu, DY Kong, SG Hu, Q Yu, Z Liu, Teng Peng Chen, Y Yin, Sumio Hosaka, and Y Liu. 2018. Smart electronic skin having gesture recognition function by LSTM neural network. Applied Physics Letters 113, 8 (2018), 084102.
[21]
OpenSystems Media. 2020. DSPs, FPGA, and ARM cores. https://www.embedded-computing.com/embedded-computing-design/dsps-fpga-and-arm-cores
[22]
Aditya Shekhar Nittala, Anusha Withana, Narjes Pourjafarian, and Jürgen Steimle. 2018. Multi-touch skin: A thin and flexible multi-touch sensor for on-skin input. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
[23]
Harshal A Sonar, Aaron P Gerratt, Stéphanie P Lacour, and Jamie Paik. 2020. Closed-loop haptic feedback control using a self-sensing soft pneumatic actuator skin. Soft robotics 7, 1 (2020), 22–29.
[24]
Gabor Soter, Andrew Conn, Helmut Hauser, and Jonathan Rossiter. 2018. Bodily aware soft robots: integration of proprioceptive and exteroceptive sensors. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2448–2453.
[25]
Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik. 2019. Learning the signatures of the human grasp using a scalable tactile glove. Nature 569, 7758 (2019), 698–702.
[26]
Tasbolat Taunyazoz, Weicong Sng, Hian Hian See, Brian Lim, Jethro Kuan, Abdul Fatir Ansari, Benjamin Tee, and Harold Soh. 2020. Event-Driven Visual-Tactile Sensing and Learning for Robots. In Proceedings of Robotics: Science and Systems.
[27]
Marc Teyssier, Gilles Bailly, Catherine Pelachaud, Eric Lecolinet, Andrew Conn, and Anne Roudaut. 2019. Skin-On Interfaces: A Bio-Driven Approach for Artificial Skin Design to Cover Interactive Devices. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 307–322. https://doi.org/10.1145/3332165.3347943
[28]
Bo Wang, Jun Zhou, Weng-Fai Wong, and Li-Shiuan Peh. 2020. Shenjing: A low power reconfigurable neuromorphic accelerator with partial-sum and spike networks-on-chip. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 240–245.
[29]
Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. Iskin: flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2991–3000.
[30]
Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. Skinmarks: Enabling interactions on body landmarks using conformal skin electronics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 3095–3105.
[31]
Binbin Ying, Qiyang Wu, Jianyu Li, and Xinyu Liu. 2020. An ambient-stable and stretchable ionic skin with multimodal sensation. Materials Horizons 7, 2 (2020), 477–488.
[32]
Xinge Yu, Zhaoqian Xie, Yang Yu, Jungyup Lee, Abraham Vazquez-Guardado, Haiwen Luan, Jasper Ruban, Xin Ning, Aadeel Akhtar, Dengfeng Li, 2019. Skin-integrated wireless haptic interfaces for virtual and augmented reality. Nature 575, 7783 (2019), 473–479.
[33]
Yujia Zhang and Tiger H Tao. 2020. A Bioinspired Wireless Epidermal Photoreceptor for Artificial Skin Vision. Advanced Functional Materials 30, 22 (2020), 2000381.

Cited By

View all
  • (2024)WalkingWizard—A Truly Wearable EEG Headset for Everyday UseACM Transactions on Computing for Healthcare10.1145/36481065:2(1-38)Online publication date: 22-Apr-2024
  • (2023)Society's Attitudes Towards Human Augmentation and Performance Enhancement Technologies (SHAPE) ScaleProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109157:3(1-23)Online publication date: 27-Sep-2023
  • (2023)AI-On-Skin: Towards Enabling Fast and Scalable On-body AI Inference for Wearable On-Skin InterfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932397:EICS(1-34)Online publication date: 19-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
May 2021
2965 pages
ISBN:9781450380959
DOI:10.1145/3411763
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Artificial Skin interfaces
  2. Artificial intelligence accelerators
  3. Handwritten text recognition
  4. Neural networks
  5. Wearable Computing

Qualifiers

  • Poster
  • Research
  • Refereed limited

Funding Sources

  • Singapore National Research Foundation

Conference

CHI '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)352
  • Downloads (Last 6 weeks)29
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)WalkingWizard—A Truly Wearable EEG Headset for Everyday UseACM Transactions on Computing for Healthcare10.1145/36481065:2(1-38)Online publication date: 22-Apr-2024
  • (2023)Society's Attitudes Towards Human Augmentation and Performance Enhancement Technologies (SHAPE) ScaleProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109157:3(1-23)Online publication date: 27-Sep-2023
  • (2023)AI-On-Skin: Towards Enabling Fast and Scalable On-body AI Inference for Wearable On-Skin InterfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932397:EICS(1-34)Online publication date: 19-Jun-2023
  • (2023)1.7pJ/SOP Neuromorphic Processor with Integrated Partial Sum Routers for In-Network Computing2023 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS46773.2023.10181759(1-5)Online publication date: 21-May-2023
  • (2022)Next Steps in Epidermal Computing: Opportunities and Challenges for Soft On-Skin DevicesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517668(1-22)Online publication date: 29-Apr-2022

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media