Sensing beyond itself: Multi-functional use of ubiquitous signals towards wearable applications
Introduction
Wearable electronics provide a highly flexible real-time sensing platform for user-centered services [1], [2], [3], [4] such as human-machine interface (HMI) [5], [6], [7], [8], [9], health monitoring [10], [11], [12], [13], [14], [15], security credentials [16], [17], [18], [19], and many other Internet of things (IoT) applications [20], [21], [22]. Progress in mechanics, materials, sensing technologies, and data science drives the research thrusts in wearable electronics in academia. The market has also witnessed the enormous commercial value of smart wearable products. According to the up-to-date forecast from Statista, the global market volume of smart wearable products is projected to reach USD $17.85 billion in 2024 [23].
Compactness and functionalities are two critical considerations for the design of wearable devices [24]. A comfortable wearing experience relies mainly on the small size, lightweight, and soft mechanics of the device. However, the demand for the tiny form factor and battery-powered feature poses many challenges. For example, the encapsulated sensors need room to accommodate bulky batteries for adequate power supply; the low-power design for longer operation time bounds the implementation of computing and memory-consuming applications.
To address the above challenges, many researchers attempt to use the signal from one sensor to realize the sensing capability of multiple types of sensors. We define this sensing paradigm as “cross-sensing”, which exploits extra sensing capabilities beyond the sensor's original designed usage and expands the perceptional dimensions of sensors. (i.e., sensing beyond itself) In the past decade, the advancement of many technologies made cross-sensing possible. For instance, microelectromechanical systems (MEMS) technology [25] shrank the physical size of an individual sensor and integrated multiple sensing components into a single device to create more compact hardware; heterogeneous computing [26], [27], [28], [29] improved computing and signal processing efficiency by introducing multiple co-processors to the wearables' system on chip (SoC); compressive machine learning [30], [31], [32], [33] made the deployment of complex machine learning models on the wearables possible.
Our comprehensive survey summarizes the innovation on three categories of wearables' usages frequently appeared within the scope of cross-sensing (Fig. 1). The first category includes entertainment and HMI realized by devices designed for communication, sound recording. In the second category, RF signal, acoustic signal, visual signal, and motion signal are utilized for health and safety applications such as non-intrusive vital sign monitoring, fall detection, sleep monitoring, and mental state detection. The third category is related to security applications and issues, where cross-sensing not only enables continuous and senseless user authentication but also poses the risk of eavesdropping via wearables.
The rest of this article is organized as follows: Section 2 introduces the entertainment and HMI related application. Section 3 reviews the health and safety services conducted by non-biomedical devices. Section 4 surveys the authentication and security applications. Section 5 presents the summarization and our perspectives on the future development of wearables. For better readability, Table 1 summarizes the mentioned abbreviation of signal processing and machine learning algorithms.
Section snippets
Entertainment and HMI
HMI is one of the most important parts of wearables. This section reviews how cross-sensing utilizes signals in wireless communication systems and acoustic systems to provide interfaces for various entertainment applications and facilitate diversified interaction styles.
Health and safety
Wearables' proximity to the human body is their advantage of hosting health and safety services. Conventional biometric monitoring function in wearables is addressed by biometric sensors like electrocardiogram (ECG), electromyography (EMG), and photoplethysmography (PPG). However, the adoption of multiple biometric sensors in wearables aggravated the battery lifetime shortage and raised the cost significantly. Numerous studies have reported health and safety-related services delivered in
Authentication and security
As personal accessories, the wearables are tightly binding with their owner's identity, making them suitable tools for user authentication. Meanwhile, wearables collect the user's private and sensitive information during usage, which also poses many security attack opportunities. This section reviewed the cross-sensing enabled innovative authentication approaches by ubiquitous signals and discussed the possible information leakage from wearables.
Summaries and perspectives
In this paper, an organized overview of wearable applications realized in cross-sensing has been proposed. Moreover, a taxonomy that categorizes the multiple functions of the ubiquitous signals has been included. This section summarized the above innovative works and put forward our perspectives on future works.
As seen in previous sections, a new terminology has been used to refer to the concept of enabling the sensing capability beyond the signal's original usage and the devices' original
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work is supported in part by the National Natural Science Foundation of China under Grants 52007019, 61790551, and 61925106, in part by Guangdong Basic and Applied Basic Research Foundation under Grant 2020A1515110887, in part by Tsinghua-Foshan Innovation Special Fund (TFISF) 2020THFS0109, and in part by the grant 2020GQG1004 from the Institute for Guo Qiang, Tsinghua University.
Zihan Wang received the dual BEng. degrees (1st class Hons.) from School of Telecommunications Engineering, Xidian University and Edinburgh Centre for Robotics, Heriot-Watt University, respectively in 2019. He is currently pursuing his MS degree in Data Science and Information Technology at Smart Sensing and Robotics (SSR) group, Tsinghua University. His research interests include self-powered sensors, Internet of Things (IoTs), and robotics. [[email protected]]
References (189)
- et al.
Wearable inertial sensors to measure gait and posture characteristic differences in older adult fallers and non-fallers: a scoping review
Gait Posture
(2020) - et al.
Fall detection and fall risk assessment in older person using wearable sensors: a systematic review
Int. J. Med. Inform.
(2019) - et al.
The use of wearable devices for walking and running gait analysis outside of the lab: a systematic review
Gait Posture
(2018) - et al.
Increasing trend of wearables and multimodal interface for human activity monitoring: a review
Biosens. Bioelectron.
(2017) - et al.
Biosignal monitoring using wearables: observations and opportunities
Biomed. Signal Process. Control
(2017) - et al.
Stretchable respiration sensors: advanced designs and multifunctional platforms for wearable physiological monitoring
Biosens. Bioelectron.
(2020) - et al.
Application of data fusion techniques and technologies for wearable health monitoring
Med. Eng. Phys.
(2017) - et al.
Stress detection in daily life scenarios using smart phones and wearable sensors: a survey
J. Biomed. Inform.
(2019) - et al.
FingerSound: recognizing unistroke thumb gestures using a ring
Proc. ACM Interact., Mob., Wearable Ubiquitous Technol.
(2017) - et al.
Wavelet variance maximization: a contactless respiration rate estimation method based on remote photoplethysmography
Biomed. Signal Process. Control
(2021)
A survey of wearable devices and challenges
IEEE Commun. Surv. Tutor.
Looking at people: sensing for ubiquitous and wearable computing
IEEE Trans. Pattern Anal. Mach. Intell.
A future with ubiquitous sensing and intelligent systems
ACS Sens.
Principles and Applications of Ubiquitous Sensing
Human-Computer Interaction
Survey of mood detection through various input modes
Microwave sensing and sleep: noncontact sleep-monitoring technology with microwave biomedical radar
IEEE Microw. Mag.
Design and evaluation of an authentication framework for wearable devices
IEEE Access
A survey of wearable biometric recognition systems
ACM Comput. Surv.
Privacy risks emerging from the adoption of innocuous wearable sensors in the mobile environment
A review on intelligent wearables: uses and risks
Hum. Behav. Emerg. Technol.
Triboelectric nanogenerators enabled Internet of things: a survey
Intell. Converg. Netw.
Wearable Internet of things: concept, architectural components and promises for person-centered healthcare
Wearable computing for the Internet of things
IT Prof.
Data sensing and analysis: challenges for wearables
Microelectromechanical systems (MEMS): fabrication, design and applications
Smart Mater. Struct.
Heterogeneous computing: challenges and opportunities
Computer
Delay-optimal computation task scheduling for mobile-edge computing systems
Mobile edge computing: a survey
IEEE Int. Things J.
A survey on mobile edge computing: the communication perspective
IEEE Commun. Surv. Tutor.
Enabling Deep Learning at the LoT Edge, IEEE/ACM International Conference on Computer-Aided Design
Machine learning for wearable IoT-based applications: a survey
Tinyml: Machine Learning with Tensorflow Lite on Arduino and Ultra-Low-Power Microcontrollers
Mcunet: Tiny Deep Learning on Iot Devices, Neural Information Processing Systems Annual Meeting
3D tracking via body radio reflections
RF-based 3D skeletons
Multi-person localization via RF body reflections
Widar: decimeter-level passive tracking via velocity monitoring with commodity Wi-Fi
Widar2. 0: passive human tracking with a single Wi-Fi link
MmTrack: passive multi-person localization using commodity millimeter wave radio
IEEE Int. Conf. Comput. Commun.
Visible light communication: opportunities, challenges and the path to market
IEEE Commun. Mag.
Visible Light Communication
An indoor broadband broadcasting system based on PLC and VLC
IEEE Trans. Broadcast.
Human sensing using visible light communication
Practical human sensing in the light
Visible light based activity sensing using ceiling photosensors
Cited by (15)
Integrated sensing, lighting and communication based on visible light communication: A review
2024, Digital Signal Processing: A Review JournalFlexible Sensing Antenna for Overbending Detection in Intelligent Packaging Applications
2023, IEEE Sensors LettersSolareSkin: Self-powered Visible Light Sensing Through a Solar Cell E-Skin
2023, UbiComp/ISWC 2023 Adjunct - Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2023 ACM International Symposium on Wearable ComputingRecent Advances in MXene-Based Fibers, Yarns, and Fabrics for Wearable Energy Storage Devices Applications
2023, ACS Applied Electronic Materials
Zihan Wang received the dual BEng. degrees (1st class Hons.) from School of Telecommunications Engineering, Xidian University and Edinburgh Centre for Robotics, Heriot-Watt University, respectively in 2019. He is currently pursuing his MS degree in Data Science and Information Technology at Smart Sensing and Robotics (SSR) group, Tsinghua University. His research interests include self-powered sensors, Internet of Things (IoTs), and robotics. [[email protected]]
Jiarong Li received the BEng degree (Hons.) in electronic science and technology from University of Electronic Science and Technology of China in 2019. He is currently pursuing the MS degree in data science at Tsinghua University since 2019. His research interests include self-powered sensors, robotics, and deep learning. [[email protected]]
Yuchao Jin received the B.A.Sc degree in Electrical Engineering from University of Toronto in 2020. He is currently pursuing MS degree in Data Science and Information Technology at Tsinghua University. His research interests include self-powered sensors, Internet of Things (IoTs), and robotics. [[email protected]]
Jiyu Wang received his B.S. and Ph.D. degree in Electrical Engineering from Chongqing University in 2013 and 2019, respectively. He is currently a Post-Doctoral Fellow in Tsinghua-Berkeley Shenzhen Institute (TBSI). He has authored over 20 journal and conference papers and received the National Scholarship and the Excellent Doctoral Dissertation of Chongqing Province in 2018 and 2020. His main research interests include energy harvesting, self-powered electronics, and additive manufacturing. [[email protected]]
Fang Yang received his B.S.E. and Ph.D. degrees in electronic engineering from Tsinghua University, Beijing, China, in 2005 and 2009, respectively. He is currently working as an associate professor with the Research Institute of Information Technology, Tsinghua University. He has published over 120 peer-reviewed journal and conference papers. He holds over 40 Chinese patents and two PCT patents. His research interests are in the fields of channel coding, channel estimation, interference cancellation, and signal processing techniques for communication systems, especially in power line communication, visible light communication, and digital television terrestrial broadcasting. He received the IEEE Scott Helt Memorial Award (best paper award in IEEE Transactions in Broadcasting) in 2015. He is the Secretary General of Sub-Committee 25 of the China National Information Technology Standardization (SAC/TC28/SC25), and a fellow of IET. [[email protected]]
Gang Li received the B.S. and Ph.D. degrees in electronic engineering from Tsinghua University, Beijing, China, in 2002 and 2007, respectively. Since July 2007, he has been with the Faculty of Tsinghua University, where he is a Professor with the Department of Electronic Engineering. From 2012 to 2014, he visited The Ohio State University, Columbus, OH, USA, and Syracuse University, Syracuse, NY, USA. He has authored or coauthored more than 160 journal and conference papers. He is the Author of the book, Advanced Sparsity-Driven Models and Methods for Radar Applications. His research interests include radar signal processing, distributed signal processing, sparse signal processing, remote sensing, and information fusion. [[email protected]]
Xiaoyue Ni is currently an assistant professor in the Department of Mechanical Engineering & Materials Science at Duke University, where she is working on wearable devices for continuous, noninvasive monitoring of human body mechanics and tissue-level diagnosis. She also develops advanced metastructures for active and smart materials. She received her Ph.D. degree in Materials Science from the California Institute of Technology in 2017, where she worked on nanomechanics focusing on resolving fundamental physics of dislocation-mediated plasticity. She received her M.S. degree in Materials Science from Caltech in 2014. She holds a B.S. degree in Physics and Mathematics from Marietta College in 2012. [[email protected]]
Wenbo Ding received the BS and PhD degrees (Hons.) from Tsinghua University in 2011 and 2016, respectively. He worked as a postdoctoral research fellow at Georgia Tech under the supervision of Prof. Z.L. Wang from 2016 to 2019. He is now a tenure-track assistant professor and PhD supervisor at Tsinghua-Berkeley Shenzhen Institute, Tsinghua Shenzhen International Graduate School, Tsinghua University, where he leads the Smart Sensing and Robotics (SSR) group. His research interests are diverse and interdisciplinary, which include self-powered sensors, energy harvesting, and wearable devices for health and soft robotics with the help of signal processing, machine learning, and mobile computing. He has received many prestigious awards, including the Gold Medal of the 47th International Exhibition of Inventions Geneva and the IEEE Scott Helt Memorial Award. [[email protected]]