Elsevier

Digital Signal Processing

Volume 116, September 2021, 103091
Digital Signal Processing

Sensing beyond itself: Multi-functional use of ubiquitous signals towards wearable applications

https://doi.org/10.1016/j.dsp.2021.103091Get rights and content

Highlights

  • “Cross-sensing”, a new summarized term for sensing technologies.

  • Realizing functions beyond the sensor's original usage in wearables is reviewed.

  • Realizing wearable applications in a non-intrusive way is reviewed.

  • Future trends and technologies for wearable applications are discussed.

Abstract

Wearable technologies provide a non-invasive way to monitor user's activity, identity, and health in real-time, which have attracted tremendous interests from both academia and industry. Due to constraints in form factor and power consumption, the sensing capabilities and functionalities of the wearables are usually limited by the available sensors. In the past decade, researchers have committed to realizing the sensing capability of multiple sensors via the signal from one sensor, which expanded the functionalities and sensing domains of traditional sensors. For the first time, we defined such sensing approach as “cross-sensing” and provided a comprehensive review on the cross-sensing towards wearable applications (i.e., human-machine interface, health services, and security). Specifically, this paper summarized the applied signal processing and machine learning algorithms, and discussed how cross-sensing would affect the development and innovation trends of wearable electronics.

Introduction

Wearable electronics provide a highly flexible real-time sensing platform for user-centered services [1], [2], [3], [4] such as human-machine interface (HMI) [5], [6], [7], [8], [9], health monitoring [10], [11], [12], [13], [14], [15], security credentials [16], [17], [18], [19], and many other Internet of things (IoT) applications [20], [21], [22]. Progress in mechanics, materials, sensing technologies, and data science drives the research thrusts in wearable electronics in academia. The market has also witnessed the enormous commercial value of smart wearable products. According to the up-to-date forecast from Statista, the global market volume of smart wearable products is projected to reach USD $17.85 billion in 2024 [23].

Compactness and functionalities are two critical considerations for the design of wearable devices [24]. A comfortable wearing experience relies mainly on the small size, lightweight, and soft mechanics of the device. However, the demand for the tiny form factor and battery-powered feature poses many challenges. For example, the encapsulated sensors need room to accommodate bulky batteries for adequate power supply; the low-power design for longer operation time bounds the implementation of computing and memory-consuming applications.

To address the above challenges, many researchers attempt to use the signal from one sensor to realize the sensing capability of multiple types of sensors. We define this sensing paradigm as “cross-sensing”, which exploits extra sensing capabilities beyond the sensor's original designed usage and expands the perceptional dimensions of sensors. (i.e., sensing beyond itself) In the past decade, the advancement of many technologies made cross-sensing possible. For instance, microelectromechanical systems (MEMS) technology [25] shrank the physical size of an individual sensor and integrated multiple sensing components into a single device to create more compact hardware; heterogeneous computing [26], [27], [28], [29] improved computing and signal processing efficiency by introducing multiple co-processors to the wearables' system on chip (SoC); compressive machine learning [30], [31], [32], [33] made the deployment of complex machine learning models on the wearables possible.

Our comprehensive survey summarizes the innovation on three categories of wearables' usages frequently appeared within the scope of cross-sensing (Fig. 1). The first category includes entertainment and HMI realized by devices designed for communication, sound recording. In the second category, RF signal, acoustic signal, visual signal, and motion signal are utilized for health and safety applications such as non-intrusive vital sign monitoring, fall detection, sleep monitoring, and mental state detection. The third category is related to security applications and issues, where cross-sensing not only enables continuous and senseless user authentication but also poses the risk of eavesdropping via wearables.

The rest of this article is organized as follows: Section 2 introduces the entertainment and HMI related application. Section 3 reviews the health and safety services conducted by non-biomedical devices. Section 4 surveys the authentication and security applications. Section 5 presents the summarization and our perspectives on the future development of wearables. For better readability, Table 1 summarizes the mentioned abbreviation of signal processing and machine learning algorithms.

Section snippets

Entertainment and HMI

HMI is one of the most important parts of wearables. This section reviews how cross-sensing utilizes signals in wireless communication systems and acoustic systems to provide interfaces for various entertainment applications and facilitate diversified interaction styles.

Health and safety

Wearables' proximity to the human body is their advantage of hosting health and safety services. Conventional biometric monitoring function in wearables is addressed by biometric sensors like electrocardiogram (ECG), electromyography (EMG), and photoplethysmography (PPG). However, the adoption of multiple biometric sensors in wearables aggravated the battery lifetime shortage and raised the cost significantly. Numerous studies have reported health and safety-related services delivered in

Authentication and security

As personal accessories, the wearables are tightly binding with their owner's identity, making them suitable tools for user authentication. Meanwhile, wearables collect the user's private and sensitive information during usage, which also poses many security attack opportunities. This section reviewed the cross-sensing enabled innovative authentication approaches by ubiquitous signals and discussed the possible information leakage from wearables.

Summaries and perspectives

In this paper, an organized overview of wearable applications realized in cross-sensing has been proposed. Moreover, a taxonomy that categorizes the multiple functions of the ubiquitous signals has been included. This section summarized the above innovative works and put forward our perspectives on future works.

As seen in previous sections, a new terminology has been used to refer to the concept of enabling the sensing capability beyond the signal's original usage and the devices' original

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China under Grants 52007019, 61790551, and 61925106, in part by Guangdong Basic and Applied Basic Research Foundation under Grant 2020A1515110887, in part by Tsinghua-Foshan Innovation Special Fund (TFISF) 2020THFS0109, and in part by the grant 2020GQG1004 from the Institute for Guo Qiang, Tsinghua University.

Zihan Wang received the dual BEng. degrees (1st class Hons.) from School of Telecommunications Engineering, Xidian University and Edinburgh Centre for Robotics, Heriot-Watt University, respectively in 2019. He is currently pursuing his MS degree in Data Science and Information Technology at Smart Sensing and Robotics (SSR) group, Tsinghua University. His research interests include self-powered sensors, Internet of Things (IoTs), and robotics. [[email protected]]

References (189)

  • S. Seneviratne et al.

    A survey of wearable devices and challenges

    IEEE Commun. Surv. Tutor.

    (2017)
  • A. Pentland

    Looking at people: sensing for ubiquitous and wearable computing

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2000)
  • F.V. Paulovich et al.

    A future with ubiquitous sensing and intelligent systems

    ACS Sens.

    (2018)
  • W. Dargie

    Principles and Applications of Ubiquitous Sensing

    (2017)
  • A. Dix et al.

    Human-Computer Interaction

    (2003)
  • R. Lietz et al.

    Survey of mood detection through various input modes

  • H. Hong et al.

    Microwave sensing and sleep: noncontact sleep-monitoring technology with microwave biomedical radar

    IEEE Microw. Mag.

    (2019)
  • A. Alharbi et al.

    Design and evaluation of an authentication framework for wearable devices

    IEEE Access

    (2020)
  • J. Blasco et al.

    A survey of wearable biometric recognition systems

    ACM Comput. Surv.

    (2016)
  • A. Raij et al.

    Privacy risks emerging from the adoption of innocuous wearable sensors in the mobile environment

  • Y. Xue

    A review on intelligent wearables: uses and risks

    Hum. Behav. Emerg. Technol.

    (2019)
  • J. Li et al.

    Triboelectric nanogenerators enabled Internet of things: a survey

    Intell. Converg. Netw.

    (2020)
  • S. Hiremath et al.

    Wearable Internet of things: concept, architectural components and promises for person-centered healthcare

  • S. Cirani et al.

    Wearable computing for the Internet of things

    IT Prof.

    (2015)
  • Wearables - Worldwide market...
  • J. Williamson et al.

    Data sensing and analysis: challenges for wearables

  • J.W. Judy

    Microelectromechanical systems (MEMS): fabrication, design and applications

    Smart Mater. Struct.

    (2001)
  • A.A. Khokhar et al.

    Heterogeneous computing: challenges and opportunities

    Computer

    (1993)
  • J. Liu et al.

    Delay-optimal computation task scheduling for mobile-edge computing systems

  • N. Abbas et al.

    Mobile edge computing: a survey

    IEEE Int. Things J.

    (2017)
  • Y. Mao et al.

    A survey on mobile edge computing: the communication perspective

    IEEE Commun. Surv. Tutor.

    (2017)
  • L. Lai et al.

    Enabling Deep Learning at the LoT Edge, IEEE/ACM International Conference on Computer-Aided Design

    (2018)
  • F. Al-Turjman et al.

    Machine learning for wearable IoT-based applications: a survey

  • P. Warden et al.

    Tinyml: Machine Learning with Tensorflow Lite on Arduino and Ultra-Low-Power Microcontrollers

    (2019)
  • J. Lin et al.

    Mcunet: Tiny Deep Learning on Iot Devices, Neural Information Processing Systems Annual Meeting

    (2020)
  • kinect xbox™
  • Intel® RealSense™
  • Leap Motion
  • F. Adib et al.

    3D tracking via body radio reflections

  • M. Zhao et al.

    RF-based 3D skeletons

  • F. Adib et al.

    Multi-person localization via RF body reflections

  • K. Qian et al.

    Widar: decimeter-level passive tracking via velocity monitoring with commodity Wi-Fi

  • K. Qian et al.

    Widar2. 0: passive human tracking with a single Wi-Fi link

  • C. Wu et al.

    MmTrack: passive multi-person localization using commodity millimeter wave radio

    IEEE Int. Conf. Comput. Commun.

    (2020)
  • A. Jovicic et al.

    Visible light communication: opportunities, challenges and the path to market

    IEEE Commun. Mag.

    (2013)
  • S. Arnon

    Visible Light Communication

    (2015)
  • J. Song et al.

    An indoor broadband broadcasting system based on PLC and VLC

    IEEE Trans. Broadcast.

    (2015)
  • T. Li et al.

    Human sensing using visible light communication

  • T. Li et al.

    Practical human sensing in the light

  • M. Ibrahim et al.

    Visible light based activity sensing using ceiling photosensors

  • Cited by (15)

    • SolareSkin: Self-powered Visible Light Sensing Through a Solar Cell E-Skin

      2023, UbiComp/ISWC 2023 Adjunct - Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2023 ACM International Symposium on Wearable Computing
    View all citing articles on Scopus

    Zihan Wang received the dual BEng. degrees (1st class Hons.) from School of Telecommunications Engineering, Xidian University and Edinburgh Centre for Robotics, Heriot-Watt University, respectively in 2019. He is currently pursuing his MS degree in Data Science and Information Technology at Smart Sensing and Robotics (SSR) group, Tsinghua University. His research interests include self-powered sensors, Internet of Things (IoTs), and robotics. [[email protected]]

    Jiarong Li received the BEng degree (Hons.) in electronic science and technology from University of Electronic Science and Technology of China in 2019. He is currently pursuing the MS degree in data science at Tsinghua University since 2019. His research interests include self-powered sensors, robotics, and deep learning. [[email protected]]

    Yuchao Jin received the B.A.Sc degree in Electrical Engineering from University of Toronto in 2020. He is currently pursuing MS degree in Data Science and Information Technology at Tsinghua University. His research interests include self-powered sensors, Internet of Things (IoTs), and robotics. [[email protected]]

    Jiyu Wang received his B.S. and Ph.D. degree in Electrical Engineering from Chongqing University in 2013 and 2019, respectively. He is currently a Post-Doctoral Fellow in Tsinghua-Berkeley Shenzhen Institute (TBSI). He has authored over 20 journal and conference papers and received the National Scholarship and the Excellent Doctoral Dissertation of Chongqing Province in 2018 and 2020. His main research interests include energy harvesting, self-powered electronics, and additive manufacturing. [[email protected]]

    Fang Yang received his B.S.E. and Ph.D. degrees in electronic engineering from Tsinghua University, Beijing, China, in 2005 and 2009, respectively. He is currently working as an associate professor with the Research Institute of Information Technology, Tsinghua University. He has published over 120 peer-reviewed journal and conference papers. He holds over 40 Chinese patents and two PCT patents. His research interests are in the fields of channel coding, channel estimation, interference cancellation, and signal processing techniques for communication systems, especially in power line communication, visible light communication, and digital television terrestrial broadcasting. He received the IEEE Scott Helt Memorial Award (best paper award in IEEE Transactions in Broadcasting) in 2015. He is the Secretary General of Sub-Committee 25 of the China National Information Technology Standardization (SAC/TC28/SC25), and a fellow of IET. [[email protected]]

    Gang Li received the B.S. and Ph.D. degrees in electronic engineering from Tsinghua University, Beijing, China, in 2002 and 2007, respectively. Since July 2007, he has been with the Faculty of Tsinghua University, where he is a Professor with the Department of Electronic Engineering. From 2012 to 2014, he visited The Ohio State University, Columbus, OH, USA, and Syracuse University, Syracuse, NY, USA. He has authored or coauthored more than 160 journal and conference papers. He is the Author of the book, Advanced Sparsity-Driven Models and Methods for Radar Applications. His research interests include radar signal processing, distributed signal processing, sparse signal processing, remote sensing, and information fusion. [[email protected]]

    Xiaoyue Ni is currently an assistant professor in the Department of Mechanical Engineering & Materials Science at Duke University, where she is working on wearable devices for continuous, noninvasive monitoring of human body mechanics and tissue-level diagnosis. She also develops advanced metastructures for active and smart materials. She received her Ph.D. degree in Materials Science from the California Institute of Technology in 2017, where she worked on nanomechanics focusing on resolving fundamental physics of dislocation-mediated plasticity. She received her M.S. degree in Materials Science from Caltech in 2014. She holds a B.S. degree in Physics and Mathematics from Marietta College in 2012. [[email protected]]

    Wenbo Ding received the BS and PhD degrees (Hons.) from Tsinghua University in 2011 and 2016, respectively. He worked as a postdoctoral research fellow at Georgia Tech under the supervision of Prof. Z.L. Wang from 2016 to 2019. He is now a tenure-track assistant professor and PhD supervisor at Tsinghua-Berkeley Shenzhen Institute, Tsinghua Shenzhen International Graduate School, Tsinghua University, where he leads the Smart Sensing and Robotics (SSR) group. His research interests are diverse and interdisciplinary, which include self-powered sensors, energy harvesting, and wearable devices for health and soft robotics with the help of signal processing, machine learning, and mobile computing. He has received many prestigious awards, including the Gold Medal of the 47th International Exhibition of Inventions Geneva and the IEEE Scott Helt Memorial Award. [[email protected]]

    View full text