skip to main content
10.1145/2642918.2647405acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

FlexSense: a transparent self-sensing deformable surface

Published: 05 October 2014 Publication History

Abstract

We present FlexSense, a new thin-film, transparent sensing surface based on printed piezoelectric sensors, which can reconstruct complex deformations without the need for any external sensing, such as cameras. FlexSense provides a fully self-contained setup which improves mobility and is not affected from occlusions. Using only a sparse set of sensors, printed on the periphery of the surface substrate, we devise two new algorithms to fully reconstruct the complex deformations of the sheet, using only these sparse sensor measurements. An evaluation shows that both proposed algorithms are capable of reconstructing complex deformations accurately. We demonstrate how FlexSense can be used for a variety of 2.5D interactions, including as a transparent cover for tablets where bending can be performed alongside touch to enable magic lens style effects, layered input, and mode switching, as well as the ability to use our device as a high degree-of-freedom input controller for gaming and beyond.

Supplementary Material

suppl.mov (uistf3900-file3.mp4)
Supplemental video

References

[1]
Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., and Singh, K. Exploring Interactive Curve and Surface Manipulation Using a Bend and Twist Sensitive Input Strip. In I3D'99, ACM, 1999, 111--118.
[2]
Caglioti, V., Giusti, A., Mureddu, L., and Taddei, P. A Manipulable Vision-Based 3D Input Device for Space Curves. In Articulated Motion and Deformable Objects. Springer, 2008, 309--318.
[3]
Danisch, L. A., Englehart, K., and Trivett, A. Spatially continuous six-degrees-of-freedom position and orientation sensor. In Photonics East, International Society for Optics and Photonics, 1999, 48--56.
[4]
Evgeniou, T., Pontil, M., and Poggio, T. Regularization Networks and Support Vector Machines. In Advances in Computational Mathematics, 2000.
[5]
Follmer, S., Leithinger, D., Olwal, A., Cheng, N., and Ishii, H. Jamming User Interfaces: Programmable Particle Stiffness and Sensing for Malleable and Shape-changing Devices. In UIST'12, ACM, 2012, 519--528.
[6]
Gallant, D. T., Seniuk, A. G., and Vertegaal, R. Towards More Paper-like Input: Flexible Input Devices for Foldable Interaction Styles. In UIST'08, ACM, Oct. 2008, 283.
[7]
Gomes, A., Nesbitt, A., and Vertegaal, R. MorePhone: A Study of Actuated Shape Deformations for Flexible Thin-Film Smartphone Notifications. In CHI'13, ACM, Apr. 2013, 583.
[8]
Herkenrath, G., Karrer, T., and Borchers, J. TWEND: Twisting and Bending as new Interaction Gesture in Mobile Devices. In CHI'08 EA, ACM, Apr. 2008, 3819.
[9]
Holman, D., Vertegaal, R., Altosaar, M., Troje, N., and Johns, D. PaperWindows: Interaction Techniques for Digital Paper. In CHI'05, ACM, 2005, 591--599.
[10]
Kabsch, W. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica (1976).
[11]
Kato, H., and Billinghurst, M. Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. In IWAR'99, IEEE Computer Society, 1999.
[12]
Kato, T., Yamamoto, A., and Higuchi, T. Shape recognition using piezoelectric thin films. In IEEE Industrial Technology, vol. 1, IEEE, 2003, 112--116.
[13]
Khalilbeigi, M., Lissermann, R., Kleine, W., and Steimle, J. Foldme: Interacting with double-sided foldable displays. In TEI'12, ACM, 2012, 33--40.
[14]
Khalilbeigi, M., Lissermann, R., Muhlhauser, M., and Steimle, J. Xpaaand: Interaction Techniques for Rollable Displays. In CHI'11, ACM, 2011, 2729--2732.
[15]
Kildal, J., Paasovaara, S., and Aaltonen, V. Kinetic Device: Designing Interactions with a Deformable Mobile Interface. In CHI EA'12, May 2012.
[16]
Konieczny, J., Shimizu, C., Meyer, G., and Colucci, D. A Handheld Flexible Display System, 2005.
[17]
Lahey, B., Girouard, A., Burleson, W., and Vertegaal, R. PaperPhone: Understanding the Use of Bend Gestures in Mobile Devices with Flexible Electronic Paper Displays. In CHI'11, ACM, May 2011, 1303.
[18]
Leal, A., Bowman, D., Schaefer, L., Quek, F., and Stiles, C. K. 3D Sketching Using Interactive Fabric for Tangible and Bimanual Input. In GI'11, Canadian Human-Computer Communications Society, 2011, 49--56.
[19]
Lee, J. C., Hudson, S. E., and Tse, E. Foldable interactive displays. In UIST'08, ACM, 2008, 287--290.
[20]
Lee, S.-S. et al. How Users Manipulate Deformable Displays as Input Devices. In CHI'10, ACM, Apr. 2010, 1647.
[21]
Lee, S.-S. et al. FlexRemote: Exploring the effectiveness of deformable user interface as an input device for TV. In HCI International 2011--Posters' Extended Abstracts. Springer, 2011, 62--65.
[22]
Punpongsanon, P., Iwai, D., and Sato, K. DeforMe: Projection-based Visualization of Deformable Surfaces Using Invisible Textures. In ETech SA'13, ACM, 2013.
[23]
Rendl, C. et al. PyzoFlex: Printed Piezoelectric Pressure Sensing Foil. In UIST'12, ACM, 2012.
[24]
Rosenberg, I., and Perlin, K. The UnMousePad: An Interpolating Multi-touch Force-sensing Input Pad. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 2009, 65.
[25]
Roudaut, A., Karnik, A., Lochtefeld, M., and Subramanian, S. Morphees: Toward High "Shape Resolution" in Self-Actuated Flexible Mobile Devices. In CHI'13, ACM, Apr. 2013, 593.
[26]
Sato, T., Mamiya, H., Koike, H., and Fukuchi, K. PhotoelasticTouch: Transparent Rubbery Tangible Interface Using an LCD and Photoelasticity. In UIST'09, ACM, 2009, 43--50.
[27]
Schölkopf, B., Herbrich, R., and Smola, A. A Generalized Representer Theorem. In Conference on Computational Learning Theory, 2001.
[28]
Schwesig, C., Poupyrev, I., and Mori, E. Gummi: A Bendable Computer. In CHI'04, ACM, Apr. 2004, 263--270.
[29]
Smith, R. T., Thomas, B. H., and Piekarski, W. Digital Foam Interaction Techniques for 3D Modeling. In VRST'08, ACM, 2008, 61--68.
[30]
Sorkine, O., and Alexa, M. As-rigid-as-possible surface modeling. In SGP'07, 2007.
[31]
Steimle, J., Jordt, A., and Maes, P. Flexpad: Highly Flexible Bending Interactions for Projected Handheld Displays. In CHI'13, ACM, Apr. 2013, 237.
[32]
Tajika, T., Yonezawa, T., and Mitsunaga, N. Intuitive Page-turning Interface of E-books on Flexible E-paper based on User Studies. In MM'08, ACM, Oct. 2008, 793.
[33]
Tarun, A. P. et al. PaperTab: An Electronic Paper Computer with Multiple Large Flexible Electrophoretic Displays. In CHI EA'13, ACM, 2013, 3131--3134.
[34]
Taylor, J. et al. User-specific hand modeling from monocular depth sequences. In CVPR, 2014.
[35]
Tikhonov, A., Leonov, A., and A.G., Y. Nonlinear Ill-Posed Problems. In Kluwer Academic Publishers, 1998.
[36]
Warren, K., Lo, J., Vadgama, V., and Girouard, A. Bending the Rules: Bend Gesture Classification for Flexible Displays. In CHI'13, ACM, Apr. 2013, 607.
[37]
Watanabe, J., Mochizuki, A., and Horry, Y. Bookisheet: Bendable Device for Browsing Content Using the Metaphor of Leafing Through the Pages. In UbiComp'08, ACM, Sept. 2008, 360.
[38]
Wellner, P. Interacting with paper on the DigitalDesk. Communications of the ACM 36, 7 (1993), 87--96.
[39]
Ye, Z., and Khalid, H. Cobra: Flexible Displays for Mobile Gaming Scenarios. In CHI EA'10, ACM, 2010, 4363--4368.
[40]
Zimmerman, T. G., Lanier, J., Blanchard, C., Bryson, S., and Harvill, Y. A Hand Gesture Interface Device. In CHI'87, ACM, 1987, 189--192.
[41]
Zirkl, M. et al. An All-Printed Ferroelectric Active Matrix Sensor Network Based on Only Five Functional Materials Forming a Touchless Control Interface. Advanced Materials, Volume 23, Issue 18 (2011), 2069--2074.

Cited By

View all
  • (2024)Enhancing VR Sketching with a Dynamic Shape DisplayProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687714(1-11)Online publication date: 9-Oct-2024
  • (2024)An Interactive Hybrid Book Integrating Capacitive, Piezoelectric, and Piezoresistive Polymer‐Based TechnologiesAdvanced Engineering Materials10.1002/adem.20230157126:5Online publication date: 5-Feb-2024
  • (2023)Monitoring concept for powder flow monitoring in Laser-Directed Energy Deposition (L-DED) process based on flexible piezoelectric sensorsMaterials Open Research10.12688/materialsopenres.17427.21(1)Online publication date: 14-Jul-2023
  • Show More Cited By

Index Terms

  1. FlexSense: a transparent self-sensing deformable surface

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology
    October 2014
    722 pages
    ISBN:9781450330695
    DOI:10.1145/2642918
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 October 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. flexible transparent sensor
    2. self-contained
    3. shape and deformation reconstruction

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    UIST '14

    Acceptance Rates

    UIST '14 Paper Acceptance Rate 74 of 333 submissions, 22%;
    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)82
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Enhancing VR Sketching with a Dynamic Shape DisplayProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687714(1-11)Online publication date: 9-Oct-2024
    • (2024)An Interactive Hybrid Book Integrating Capacitive, Piezoelectric, and Piezoresistive Polymer‐Based TechnologiesAdvanced Engineering Materials10.1002/adem.20230157126:5Online publication date: 5-Feb-2024
    • (2023)Monitoring concept for powder flow monitoring in Laser-Directed Energy Deposition (L-DED) process based on flexible piezoelectric sensorsMaterials Open Research10.12688/materialsopenres.17427.21(1)Online publication date: 14-Jul-2023
    • (2023)3D Deformation Capture via a Configurable Self-Sensing IMU Sensor NetworkProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808747:1(1-24)Online publication date: 28-Mar-2023
    • (2023)A Neural Network-based Low-cost Soft Sensor for Touch Recognition and Deformation CaptureProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3595963(889-903)Online publication date: 10-Jul-2023
    • (2023)Electrical Impedance Tomographic Shape Sensing for Soft RobotsIEEE Robotics and Automation Letters10.1109/LRA.2023.32403688:3(1555-1562)Online publication date: Mar-2023
    • (2023)Stretchable Shape‐Sensing SheetsAdvanced Intelligent Systems10.1002/aisy.2023003435:12Online publication date: 15-Oct-2023
    • (2022)Assessment of the Measurement Performance of the Multimodal Fibre Optic Shape Sensing Configuration for a Morphing Wing SectionSensors10.3390/s2206221022:6(2210)Online publication date: 12-Mar-2022
    • (2022)Monitoring concept for powder flow monitoring in Laser-Directed Energy Deposition (L-DED) process based on flexible piezoelectric sensorsMaterials Open Research10.12688/materialsopenres.17427.11(1)Online publication date: 4-May-2022
    • (2022)Squeezy-Feely: Investigating Lateral Thumb-Index Pinching as an Input ModalityProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501981(1-15)Online publication date: 29-Apr-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media