Skip to main content

Advertisement

SpringerLink
Log in

A fine-grained gesture tracking system based on millimeter-wave

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

As an essential part of modern human-computer interaction, gesture recognition is widely used in industry, society, medical care and entertainment. Existing gesture recognition solutions either rely on computer vision, which suffers from the light condition, or use inertial sensors, which are limited by the battery life. In this paper, we propose a millimeter-wave based solution to recognize the human gesture for natural interaction, which can efficiently solve the above limitations. We leverage a frequency-modulated continuous wave radar to implement a cm-level fine-grained gesture tracking system. In order to obtain the target position, we build a theoretical model between signals and position by applying a multi-dimensional fast Fourier transform to the captured data. To handle the interference of the multipath effect, we propose a distance-based voting method to delete unreasonable points and optimize the trajectory. Furthermore, we design an optimal estimation algorithm based on measurements with different precision to obtain more acceptable estimation. Finally, in view of the sparse points, we propose an interpolation method of fitting traces in different dimensions to augment the point cloud. We have implemented a prototype system, and the experimental results show that our system can accurately track the gestures without knowing the gesture in advance. The average trajectory position error is 0.94 cm, and the average normalized location error is 0.027 for gestures with the size of about 20 cm \({\times }\)40 cm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  • Abdelnasser, H., Youssef, M., Harras, K.A.: Wigest: A ubiquitous wifi-based gesture recognition system. In: 2015 IEEE Conference on Computer Communications (INFOCOM), pp. 1472–1480 (2015). IEEE

  • Bu, Y., Xie, L., Yin, Y., Wang, C., Ning, J., Cao, J., Lu, S.: Handwriting-assistant: Reconstructing continuous strokes with millimeter-level accuracy via attachable inertial sensors. In: Proceedings of the ACM International on Ubiquitous Computing (UbiComp), pp. 1–25 (2021)

  • Foehr, J., Germelmann, C.C.: Alexa, can i trust you? Exploring consumer paths to trust in smart voice-interaction technologies. J. Assoc. Consumer Res. 5(2), 181–205 (2020)

    Article  Google Scholar 

  • Gao, R., Zhang, M., Zhang, J., Li, Y., Yi, E., Wu, D., Wang, L., Zhang, D.: Towards position-independent sensing for gesture recognition with wi-fi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5(2), 1–28 (2021)

    Article  Google Scholar 

  • Jiang, C., Guo, J., He, Y., Jin, M., Li, S., Liu, Y.: mmvib: micrometer-level vibration measurement with mmwave radar. In: Proceedings of the ACM International Conference on Mobile Computing and Networking (MobiCom), pp. 1–13 (2020)

  • Kowalski, J., Jaskulska, A., Skorupska, K., Abramczuk, K., Biele, C., Kopeć, W., Marasek, K.: Older adults and voice interaction: A pilot study with google home. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI EA), pp. 1–6 (2019)

  • Lien, J., Gillian, N., Karagozler, M.E., Amihood, P., Schwesig, C., Olson, E., Raja, H., Poupyrev, I.: Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Trans. Graph. (TOG) 35(4), 1–19 (2016)

    Article  Google Scholar 

  • Liu, H., Zhou, A., Dong, Z., Sun, Y., Zhang, J., Liu, L., Ma, H., Liu, J., Yang, N.: M-gesture: Person-independent real-time in-air gesture recognition using commodity millimeter wave radar. IEEE Internet Things J. 9(5), 3397–3415 (2021)

    Article  Google Scholar 

  • Liu, G., Gu, Y., Yin, Y., Yu, C., Wang, Y., Mi, H., Shi, Y.: Keep the phone in your pocket: Enabling smartphone operation with an imu ring for visually impaired people. In: Proceedings of the ACM International on Ubiquitous Computing (UbiComp), pp. 1–23 (2020)

  • Lv, Z., Feng, S., Feng, L., Li, H.: Extending touch-less interaction on vision based wearable device. In: Proceedings of the IEEE International Symposium Virtual Reality (VR), pp. 231–232 (2015)

  • Melgarejo, P., Zhang, X., Ramanathan, P., Chu, D.: Leveraging directional antenna capabilities for fine-grained gesture recognition. In: Proceedings of the ACM International on Ubiquitous Computing (UbiComp), pp. 541–551 (2014)

  • Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: A neural network based on spd manifold learning for skeleton-based hand gesture recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12036–12045 (2019)

  • Owda, A.Y., Salmon, N., Casson, A.J., Owda, M.: The reflectance of human skin in the millimeter-wave band. Sensors 20(5), 1480 (2020)

    Article  Google Scholar 

  • Pu, Q., Gupta, S., Gollakota, S., Patel, S.: Whole-home gesture recognition using wireless signals. In: Proceedings of the ACM International Conference on Mobile Computing and Networking (MobiCom), pp. 27–38 (2013)

  • Rao, S., Ahmad, A., Roh, J.C., Bharadwaj, S.: 77ghz single chip radar sensor enables automotive body and chassis applications. Tex, Instrum (2017)

    Google Scholar 

  • Shi, Y., Zhang, H., Zhao, K., Cao, J., Sun, M., Nanayakkara, S.: Ready, steady, touch! sensing physical contact with a finger-mounted imu. In: Proceedings of the ACM International on Ubiquitous Computing (UbiComp), pp. 1–25 (2020)

  • Sun, K., Yu, C., Shi, W., Liu, L., Shi, Y.: Lip-interact: Improving mobile device interaction with silent speech commands. In: Proceedings of ACM Symposium on User Interface Software and Technology (UIST), pp. 581–593 (2018)

  • Wei, T., Zhang, X.: mtrack: High-precision passive tracking using millimeter wave radios. In: Proceedings of the ACM International Conference on Mobile Computing and Networking (MobiCom), pp. 117–129 (2015)

  • Weng, Y., Yu, C., Shi, Y., Zhao, Y., Yan, Y., Shi, Y.: Facesight: Enabling hand-to-face gesture interaction on ar glasses with a downward-facing camera vision. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (2021)

  • Yu, Y., Wang, D., Zhao, R., Zhang, Q.: Rfid based real-time recognition of ongoing gesture with adversarial learning. In: Proceedings of the 17th Conference on Embedded Networked Sensor Systems, pp. 298–310 (2019)

  • Yu, C., Wei, X., Vachher, S., Qin, Y., Liang, C., Weng, Y., Gu, Y., Shi, Y.: Handsee: Enabling full hand interaction on smartphone with front camera-based stereo vision. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), pp. 1–13 (2019)

  • Zhang, Q., Wang, D., Zhao, R., Yu, Y.: Soundlip: Enabling word and sentence-level lip interaction for smart devices. In: Proceedings of the ACM International on Ubiquitous Computing (UbiComp), pp. 1–128 (2021)

  • Zou, Y., Xiao, J., Han, J., Wu, K., Li, Y., Ni, L.M.: Grfid: a device-free rfid-based gesture recognition system. IEEE Trans. Mob. Comput. 16(2), 381–393 (2016)

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported in part by National Natural Science Foundation of China under Grant Nos. 61902175, 61872174, 61832008; Jiangsu Natural Science Foundation under Grant No. BK20190293; the Key K & D Program of Jiangsu Province under Grant BE2020001-3; the Fundamental Research Funds for the Central Universities No. 2022300296 (0202/14380096). This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. Chuyu Wang is the corresponding author.

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Yiwen Feng, Chuyu Wang, Lei Xie & Sanglu Lu

Authors
  1. Yiwen Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chuyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sanglu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chuyu Wang.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, Y., Wang, C., Xie, L. et al. A fine-grained gesture tracking system based on millimeter-wave. CCF Trans. Pervasive Comp. Interact. 4, 357–369 (2022). https://doi.org/10.1007/s42486-022-00119-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-022-00119-0

Keywords

Search

Navigation