skip to main content
research-article

MultiSoft: Soft Sensor Enabling Real-Time Multimodal Sensing with Contact Localization and Deformation Classification

Published: 18 September 2018 Publication History

Abstract

We introduce MultiSoft, a multilayer soft sensor capable of sensing real-time contact localization, classification of deformation types, and estimation of deformation magnitudes. We propose a multimodal sensing pipeline that carries out both inverse problem solving and machine learning tasks. Specifically, we employ an electrical impedance tomography (EIT) for contact localization and a support vector machine (SVM) for classifying deformations and regressing their magnitudes. We propose a deformation-aware system which enables maintaining a persistent single-point contact localization throughout the deformation. By updating a time-varying distribution of conductivity change caused by deformations, a single-point contact localization can be maintained and restored to support interaction using both contact localization and deformations.We devise a multilayer structure to fabricate a highly stretchable and flexible soft sensor with a short sensor settlement after excitations. Through a series of experiments and evaluations, we validate both raw sensor and multimodal sensing performance with the proposed method. We further demonstrate applicability and feasibility of MultiSoft with example applications.

Supplementary Material

yoon (yoon.zip)
Supplemental movie, appendix, image and software files for, MultiSoft: Soft Sensor Enabling Real-Time Multimodal Sensing with Contact Localization and Deformation Classification

References

[1]
Andy Adler and Robert Guardo. 1996. Electrical impedance tomography: regularized imaging and contrast detection. IEEE Transactions on Medical Imaging 15, 2 (Apr 1996), 170--179.
[2]
Hassan Alirezaei, Akihiko Nagakubo, and Yasuo Kuniyoshi. 2007. A highly stretchable tactile distribution sensor for smooth surfaced humanoids. In 2007 7th IEEE-RAS International Conference on Humanoid Robots. 167--173.
[3]
Hassan Alirezaei, Akihiko Nagakubo, and Yasuo Kuniyoshi. 2009. A tactile distribution sensor which enables stable measurement under high and dynamic stretch. In 3D User Interfaces, 2009. 3DUI 2009. IEEE Symposium on. IEEE, 87--93.
[4]
B H Brown and A D Seagar. 1987. The Sheffield data collection system. Clinical Physics and Physiological Measurement 8, 4A (1987), 91. http://stacks.iop.org/0143-0815/8/i=4A/a=012
[5]
Tao Chen, Rui Hao, Huisheng Peng, and Liming Dai. 2015. High-Performance, Stretchable, Wire-Shaped Supercapacitors. Angewandte Chemie International Edition 54, 2 (2015), 618--622.
[6]
Jingyuan Cheng, Mathias Sundholm, Bo Zhou, Marco Hirsch, and Paul Lukowicz. 2016. Smart-surface: Large scale textile pressure sensors arrays for activity recognition. Pervasive and Mobile Computing 30 (2016), 97--112.
[7]
Jingyuan Cheng, Bo Zhou, Kai Kunze, Carl Christian Rheinländer, Sebastian Wille, Norbert Wehn, Jens Weppner, and Paul Lukowicz. 2013. Activity recognition and nutrition monitoring in every day situations with a textile capacitive neckband. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, 155--158.
[8]
Chin-yu Chien, Rong-Hao Liang, Long-Fei Lin, Liwei Chan, and Bing-Yu Chen. 2015. FlexiBend: Enabling Interactivity of Multi-Part, Deformable Fabrications Using Single Shape-Sensing Strip. In Proceedings of the 28th Annual ACM Symposium on User Interface Software 8 Technology (UIST '15). ACM, New York, NY, USA, 659--663.
[9]
Jean-Baptiste Chossat, Hee-Sup Shin, Yong-Lae Park, and Vincent Duchaine. 2015. Soft tactile skin using an embedded ionic liquid and tomographic imaging. Journal of Mechanisms and Robotics 7, 2 (2015), 021008.
[10]
Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: Low-power Gesture Input with Wrist-worn Pressure Sensors. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 161--166.
[11]
Sean Follmer, Daniel Leithinger, Alex Olwal, Nadia Cheng, and Hiroshi Ishii. 2012. Jamming User Interfaces: Programmable Particle Stiffness and Sensing for Malleable and Shape-changing Devices. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 519--528.
[12]
B. Hagström, A. LUND, and E. Nilsson. 2014. Method of producing a piezoelectric and pyroelectric fiber. (Oct. 9 2014). WO Patent App. PCT/EP2014/056,658.
[13]
Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explor. Newsl. 11, 1 (Nov. 2009), 10--18.
[14]
John-Erik Haugen, Oliver Tomic, and Knut Kvaal. 2000. A calibration method for handling the temporal drift of solid state gas-sensors. Analytica Chimica Acta 407, 1 (2000), 23--39.
[15]
PJRC Inc. 2017. Product Detail. (2017). Retrieved November 1, 2017 from http://www.pjrc.com/teensy/index.html.
[16]
Hsin-Liu (Cindy) Kao, Christian Holz, Asta Roseway, Andres Calvo, and Chris Schmandt. 2016. DuoSkin: Rapidly Prototyping On-skin User Interfaces Using Skin-friendly Materials. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (ISWC '16). ACM, New York, NY, USA, 16--23.
[17]
Johan Kildal, Susanna Paasovaara, and Viljakaisa Aaltonen. 2012. Kinetic Device: Designing Interactions with a Deformable Mobile Interface. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 1871--1876.
[18]
A. KÖLLNBERGER and A. Schwinghammer. 2014. Production of thin silicone films. (June 19 2014). WO Patent App. PCT/EP2013/073,914.
[19]
Gierad Laput, Eric Brockmeyer, Scott E. Hudson, and Chris Harrison. 2015. Acoustruments: Passive, Acoustically-Driven, Interactive Controls for Handheld Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2161--2170.
[20]
Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Synthetic Sensors: Towards General-Purpose Sensing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3986--3999.
[21]
Hyosang Lee, Donguk Kwon, Haedo Cho, Inkyu Park, and Jung Kim. 2017. Soft Nanocomposite Based Multi-point, Multi-directional Strain Mapping Sensor Using Anisotropic Electrical Impedance Tomography. Scientific reports 7 (2017).
[22]
Joanne Leong, Patrick Parzer, Florian Perteneder, Teo Babic, Christian Rendl, Anita Vogl, Hubert Egger, Alex Olwal, and Michael Haller. 2016. proCover: Sensory Augmentation of Prosthetic Limbs Using Smart Textile Covers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 335--346.
[23]
F. Lorussi, Enzo Pasquale Scilingo, M. Tesconi, A. Tognetti, and D. De Rossi. 2005. Strain sensing fabric for hand posture and gesture monitoring. IEEE Transactions on Information Technology in Biomedicine 9, 3 (Sept 2005), 372--381.
[24]
Vinh P. Nguyen, Sang Ho Yoon, Ansh Verma, and Karthik Ramani. 2014. BendID: Flexible Interface for Localized Deformation Recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). ACM, New York, NY, USA, 553--557.
[25]
Patrick Parzer, Adwait Sharma, Anita Vogl, Jürgen Steimle, Olwal Alex, and Michael Haller. 2017. SmartSleeve: Real-time Sensing of Surface and Deformation Gestures on Flexible, Interactive Textiles, using a Hybrid Gesture Detection Pipeline. In Proceedings of the 30th Annual ACM Symposium on User Interface Software 8 Technology (UIST '17). ACM, New York, NY, USA, 565--577.
[26]
Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E Robinson. 2016. Project Jacquard: interactive digital textiles at scale. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4216--4227.
[27]
Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 487--494.
[28]
Christian Rendl, David Kim, Sean Fanello, Patrick Parzer, Christoph Rhemann, Jonathan Taylor, Martin Zirkl, Gregor Scheipl, Thomas Rothländer, Michael Haller, and Shahram Izadi. 2014. FlexSense: A Transparent Self-sensing Deformable Surface. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 129--138.
[29]
Christian Rendl, David Kim, Patrick Parzer, Sean Fanello, Martin Zirkl, Gregor Scheipl, Michael Haller, and Shahram Izadi. 2016. FlexCase: Enhancing Mobile Interaction with a Flexible Sensing and Display Cover. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5138--5150.
[30]
Peter Roberts, Dana D. Damian, Wanliang Shan, Tong Lu, and Carmel Majidi. 2013. Soft-matter capacitive sensor for measuring shear and pressure deformation. In 2013 IEEE International Conference on Robotics and Automation. 3529--3534.
[31]
Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 483--492.
[32]
Munehiko Sato, Rohan S. Puri, Alex Olwal, Yosuke Ushigome, Lukas Franciszkiewicz, Deepak Chandra, Ivan Poupyrev, and Ramesh Raskar. 2017. Zensei: Embedded, Multi-electrode Bioimpedance Sensing for Implicit, Ubiquitous User Recognition. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3972--3985.
[33]
Martin Schmitz, Jürgen Steimle, Jochen Huber, Niloofar Dezfuli, and Max Mühlhäuser. 2017. Flexibles: Deformation-Aware 3D-Printed Tangibles for Capacitive Touchscreens. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1001--1014.
[34]
Su Ryon Shin, Raziyeh Farzad, Ali Tamayol, Vijayan Manoharan, Pooria Mostafalu, Yu Shrike Zhang, Mohsen Akbari, Sung Mi Jung, Duckjin Kim, Mattia Comotto, Nasim Annabi, Faten Ebrahim Al-Hazmi, Mehmet R. Dokmeci, and Ali Khademhosseini. 2016. A Bioactive Carbon Nanotube-Based Ink for Printing 2D and 3D Flexible Electronics. Advanced Materials 28, 17 (2016), 3280--3289.
[35]
Paden Shorey and Audrey Girouard. 2017. Bendtroller:: An Exploration of In-Game Action Mappings with a Deformable Game Controller. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1447--1458.
[36]
David Silvera Tawil, David Rye, and Mari Velonaki. 2012. Interpretation of the modality of touch on an artificial arm covered with an EIT-based sensitive skin. The International Journal of Robotics Research 31, 13 (2012), 1627--1641.
[37]
Ronit Slyper, Ivan Poupyrev, and Jessica Hodgins. 2011. Sensing Through Structure: Designing Soft Silicone Sensors. In Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '11). ACM, New York, NY, USA, 213--220.
[38]
Smooth-On. 2017. Ecoflex. (2017). Retrieved November 1, 2017 from https://www.smooth-on.com/products/.
[39]
Yuta Sugiura, Masahiko Inami, and Takeo Igarashi. 2012. A Thin Stretchable Interface for Tangential Force Measurement. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 529--536.
[40]
Mahmoud Tavakoli, Rui Rocha, Luis Osorio, Miguel Almeida, Anibal de Almeida, Vivek Ramachandran, Arya Tabatabai, Tong Lu, and Carmel Majidi. 2017. Carbon doped PDMS: conductance stability over time and implications for additive manufacturing of stretchable electronics. Journal of Micromechanics and Microengineering 27, 3 (2017), 035010. http://stacks.iop.org/0960-1317/27/i=3/a=035010
[41]
Karen Vanderloock, Vero Vanden Abeele, Johan A.K. Suykens, and Luc Geurts. 2013. The Skweezee System: Enabling the Design and the Programming of Squeeze Interactions. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 521--530.
[42]
M Vauhkonen, W R B Lionheart, L M Heikkinen, P J Vauhkonen, and J P Kaipio. 2001. A MATLAB package for the EIDORS project to reconstruct two-dimensional EIT images. Physiological Measurement 22, 1 (2001), 107. http://stacks.iop.org/0967-3334/22/i=1/a=314
[43]
Anita Vogl, Patrick Parzer, Teo Babic, Joanne Leong, Alex Olwal, and Michael Haller. 2017. StretchEBand: Enabling Fabric-based Interactions Through Rapid Fabrication of Textile Stretch Sensors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 2617--2627.
[44]
Wacker. 2017. ELASTOSIL LR 3162 A/B. (2017). Retrieved November 1, 2017 from https://www.wacker.com.
[45]
Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2991--3000.
[46]
Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3095--3105.
[47]
Michael Wessely, Theophanis Tsandilas, and Wendy E. Mackay. 2016. Stretchis: Fabricating Highly Stretchable User Interfaces. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 697--704.
[48]
Sang Ho Yoon, Ke Huo, Vinh P. Nguyen, and Karthik Ramani. 2015. TIMMi: Finger-worn Textile Input Device with Multimodal Sensing in Mobile Interaction. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 269--272.
[49]
Sang Ho Yoon, Ke Huo, Yunbo Zhang, Guiming Chen, Luis Paredes, Subramanian Chidambaram, and Karthik Ramani. 2017. iSoft: A Customizable Soft Sensor with Real-time Continuous Contact and Stretching Sensing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software 8 Technology (UIST '17). ACM, New York, NY, USA, 665--678.
[50]
Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick: Low-Cost Touch Sensing Using Electric Field Tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1--14.
[51]
Yang Zhang, Robert Xiao, and Chris Harrison. 2016. Advancing Hand Gesture Recognition with High Resolution Electrical Impedance Tomography. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 843--850.

Cited By

View all
  • (2024)Rapid Action Evaluation and Optimization Through Mutual Information Using a Multi-Modal Soft Tactile Sensor2024 IEEE 7th International Conference on Soft Robotics (RoboSoft)10.1109/RoboSoft60065.2024.10521945(678-683)Online publication date: 14-Apr-2024
  • (2024)The design and fabrication of a wearable lattice-patterned 3D sensing skinSensors and Actuators A: Physical10.1016/j.sna.2024.115143369(115143)Online publication date: Apr-2024
  • (2023)LattiSense: A 3D-Printable Resistive Deformation Sensor with Lattice StructuresProceedings of the 8th ACM Symposium on Computational Fabrication10.1145/3623263.3623361(1-14)Online publication date: 8-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 2, Issue 3
September 2018
1536 pages
EISSN:2474-9567
DOI:10.1145/3279953
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2018
Accepted: 01 September 2018
Revised: 01 May 2018
Received: 01 February 2018
Published in IMWUT Volume 2, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Soft user interface
  2. input device
  3. multimodal sensing
  4. sensing technique
  5. wearables

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)102
  • Downloads (Last 6 weeks)20
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Rapid Action Evaluation and Optimization Through Mutual Information Using a Multi-Modal Soft Tactile Sensor2024 IEEE 7th International Conference on Soft Robotics (RoboSoft)10.1109/RoboSoft60065.2024.10521945(678-683)Online publication date: 14-Apr-2024
  • (2024)The design and fabrication of a wearable lattice-patterned 3D sensing skinSensors and Actuators A: Physical10.1016/j.sna.2024.115143369(115143)Online publication date: Apr-2024
  • (2023)LattiSense: A 3D-Printable Resistive Deformation Sensor with Lattice StructuresProceedings of the 8th ACM Symposium on Computational Fabrication10.1145/3623263.3623361(1-14)Online publication date: 8-Oct-2023
  • (2023)A Neural Network-based Low-cost Soft Sensor for Touch Recognition and Deformation CaptureProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3595963(889-903)Online publication date: 10-Jul-2023
  • (2023)uKnit: A Position-Aware Reconfigurable Machine-Knitted Wearable for Gestural Interaction and Passive Sensing using Electrical Impedance TomographyProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580692(1-17)Online publication date: 19-Apr-2023
  • (2023)Stretchable e-skin and transformer enable high-resolution morphological reconstruction for soft robotsNature Machine Intelligence10.1038/s42256-023-00622-85:3(261-272)Online publication date: 23-Feb-2023
  • (2023)An EIT-based piezoresistive sensing skin with a lattice structureMaterials & Design10.1016/j.matdes.2023.112227233(112227)Online publication date: Sep-2023
  • (2022)A Bionic Testbed for Cardiac Ablation ToolsInternational Journal of Molecular Sciences10.3390/ijms23221444423:22(14444)Online publication date: 21-Nov-2022
  • (2022)Fabrication of Nāmya: A Bend and Touch-Sensitive Flexible Smartphone-Sized PrototypeHuman Behavior and Emerging Technologies10.1155/2022/12054202022(1-10)Online publication date: 19-Jul-2022
  • (2022)StretchARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35503056:3(1-26)Online publication date: 7-Sep-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media