Skip to main content
SpringerLink
Log in

Gesture recognition with RFID: an experimental study

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

Wireless signal-based gesture recognition attracts more and more attention in many fields due to its convenience and non-invasiveness without privacy issues. However, existing wireless gesture recognition approaches suffer a lot from the surrounding environment and relative deployment positions of wireless devices, which limits their application and generalization. As one of the most popular wireless sensing techniques, radio frequency identification (RFID) has been widely exploited for gesture recognition in recent years. In this paper, we propose a single gesture and sequential gestures recognition system ExpGRF with an experimental study via commercial RFID devices. ExpGRF leverages the elaborated signal processing pipeline and adversarial learning to extract gesture-discriminative, user-independent, environment-independent and deployment-independent features from RFID signals. Extensive experiments show that ExpGRF obtains strong effectiveness and robustness to user, environment and deployment diversity. Its accuracy achieves 97.2% for gesture sequence recognition in the non-light-of-sight (NLOS) scenario with new users and new deployments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Ali, K., Liu, A.X., Wang, W., Shahzad, M.: Recognizing keystrokes using wifi devices. IEEE J. Sel. Areas Commun. 35(5), 1175–1190 (2017)

    Article  Google Scholar 

  • Chang, L., Xiong, J., Wang, J., Chen, X., Wang, Y., Tang, Z., Fang, D.: Rf-copybook: a millimeter level calligraphy copybook based on commodity RFID. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(4), 1–19 (2018)

    Article  Google Scholar 

  • Chen, L., Xiong, J., Chen, X., Lee, S.I., Zhang, D., Yan, T., Fang, D.: Lungtrack: towards contactless and zero dead-zone respiration monitoring with commodity RFIDs. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3(3), 1–22 (2019)

    Google Scholar 

  • Cui, R., Liu, H., Zhang, C.: Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 1610–1618 (2017)

  • Dian, C., Wang, D., Zhang, Q., Zhao, R., Yu, Y.: Towards domain-independent complex and fine-grained gesture recognition with RFID. Proc. ACM Human-Comput. Interact. 4(ISS), 1–22 (2020)

    Article  Google Scholar 

  • Ding, H., Shangguan, L., Yang, Z., Han, J., Zhou, Z., Yang, P., Xi, W., Zhao, J.: Femo: a platform for free-weight exercise monitoring with RFIDs. In: Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, ACM, pp. 141–154 (2015)

  • Fan, X., Gong, W., Liu, J.: Tagfree activity identification with RFIDs. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2(1), 7 (2018)

    Article  Google Scholar 

  • Fang, B., Co, J., Zhang, M.: Deepasl: enabling ubiquitous and non-intrusive word and sentence-level sign language translation. In: Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems, ACM, pp. 5 (2017)

  • Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 2030–2096 (2016)

    MathSciNet  MATH  Google Scholar 

  • Global, E.: EPC radio-frequency identity protocols class-1 generation-2 uhf RFID protocol for communications at 860–960 MHz. Version 1, 23 (2008)

    Google Scholar 

  • Gong, J., Yang, X.D., Irani, P.: Wristwhirl: one-handed continuous smartwatch input using wrist gestures. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ACM, pp. 861–872 (2016)

  • Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning, pp. 369–376 (2006)

  • Gummeson, J., Mccann, J., Yang, C.J., Ranasinghe, D., Hudson, S., Sample, A.: RFID light bulb: enabling ubiquitous deployment of interactive RFID systems. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(2), 12:1-12:16 (2017)

    Article  Google Scholar 

  • He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)

  • Hoai, M., De la Torre, F.: Max-margin early event detectors. Int. J. Comput. Vis. 107(2), 191–202 (2014)

    Article  MathSciNet  Google Scholar 

  • Huang, A., Wang, D., Zhao, R., Zhang, Q.: Au-id: automatic user identification and authentication through the motions captured from sequential human activities using RFID. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3(2), 1–26 (2019)

    Article  Google Scholar 

  • Impinj (2021) Reader modes (rf modes) made easy. https://support.impinj.com/hc/en-us/articles/360000046899-Reader-Modes-Made-Easy

  • Itoh, K.: Analysis of the phase unwrapping algorithm. Appl. Opt. 21(14), 2470–2470 (1982)

    Article  Google Scholar 

  • Jiang, W., Miao, C., Ma, F., Yao, S., Wang, Y., Yuan, Y., Xue, H., Song, C., Ma, X., Koutsonikolas, D., et al.: Towards environment independent device free human activity recognition. In: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pp. 289–304 (2018)

  • Jiang, W., Li, S., Zhao, Y., Tu, H., Liu, C.: Fine-grained hand gesture recognition based on active acoustic signal for VR systems. CCF Trans. Pervasive Comput. Interact. 2(4), 329–339 (2020)

    Article  Google Scholar 

  • Li, X., Chang, L., Song, F., Wang, J., Chen, X., Tang, Z., Wang, Z.: Crossgr: accurate and low-cost cross-target gesture recognition using Wi-Fi. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5(1), 1–23 (2021)

    Article  Google Scholar 

  • Lien, J., Gillian, N., Karagozler, M.E., Amihood, P., Schwesig, C., Olson, E., Raja, H., Poupyrev, I.: Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Trans. Graph. (TOG) 35(4), 142 (2016)

    Article  Google Scholar 

  • Ling, K., Dai, H., Liu, Y., Liu, AX.: Ultragesture: fine-grained gesture sensing and recognition. In: 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), IEEE, pp. 1–9 (2018)

  • Liu, T., Yang, L., Li, X., Huang, H., Liu, Y.: Tagbooth: deep shopping data acquisition powered by RFID tags. In: 2015 IEEE Conference on Computer Communications (INFO- COM), pp. 1670–1678 (2015)

  • Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:150804025 (2015)

  • Lvd, Maaten, Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)

    MATH  Google Scholar 

  • Ma, Y., Zhou, G., Wang, S., Zhao, H., Jung, W.: Signfi: sign language recognition using Wi-Fi. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2(1), 23 (2018)

    Article  Google Scholar 

  • Meng, Z., Li, J., Chen, Z., Zhao, Y., Mazalov, V., Gong, Y., Juang, B.H.: Speaker-invariant training via adversarial learning. In: 2018 IEEE International Conference on Acoustics, pp. 5969–5973. Speech and Signal Processing (ICASSP), IEEE (2018)

  • Parate, A., Chiu, M.C., Chadowitz, C., Ganesan, D., Kalogerakis, E.: Risq: recognizing smoking gestures with inertial sensors on a wristband. In: Proceedings of the 12th annual international conference on Mobile systems, applications, and services, ACM, pp. 149–161 (2014)

  • Ramos-Garcia, R.I., Muth, E.R., Gowdy, J.N., Hoover, A.W.: Improving the recognition of eating gestures using intergesture sequential dependencies. IEEE J. Biomed. Health Inform. 19(3), 825–831 (2015)

    Article  Google Scholar 

  • Santurkar, S., Tsipras, D., Ilyas, A., Madry, A.: How does batch normalization help optimization? In: Advances in Neural Information Processing Systems, pp. 2483–2493 (2018)

  • Shangguan, L., Zhou, Z., Jamieson, K.: Enabling gesture-based interactions with objects. In: Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, ACM, pp. 239–251 (2017)

  • Shangguan, L., Zhou, Z., Zheng, X., Yang, L., Liu, Y., Han, J.: Shopminer: mining customer shopping behavior in physical clothing stores with cots RFID devices. In: Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, ACM, pp. 113–125 (2015)

  • Tan, S., Zhang, L., Wang, Z., Yang, J.: Multitrack: multi-user tracking and activity recognition using commodity Wi-Fi. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2019)

  • Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. Comput. Vis. Pattern Recogn. 1, 4 (2017)

    Google Scholar 

  • Wang, W., Liu, A.X., Sun, K.: Device-free gesture tracking using acoustic signals. In: Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking, ACM, pp. 82–94 (2016)

  • Wang, Y., Zheng, Y.: Modeling RFID signal reflection for contact-free activity recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2(4), 1–22 (2018)

    Google Scholar 

  • Wang, J., Vasisht, D., Katabi, D.: RF-idraw: virtual touch screen in the air using RF signals. ACM SIGCOMM Comput. Commun. Rev. ACM 44, 235–246 (2014)

    Article  Google Scholar 

  • Xie, L., Wang, C., Liu, A.X., Sun, J., Lu, S.: Multi-touch in the air: concurrent micromovement recognition using RF signals. IEEE/ACM Trans. Networking 26(1), 231–244 (2017)

    Article  Google Scholar 

  • Xu, Q., Liu, X., Luo, J., Tang, Z.: Emotion monitoring with RFID: an experimental study. CCF Trans. Pervasive Comput. Interact. 2(4), 299–313 (2020)

    Article  Google Scholar 

  • Yang, L., Chen, Y., Li, X.Y., Xiao, C., Li, M., Liu, Y.: Tagoram: real-time tracking of mobile RFID tags to high precision using cots devices. In: Proceedings of the 20th annual international conference on Mobile computing and networking, ACM, pp. 237–248 (2014)

  • Yang, X., Chen, X., Cao, X., Wei, S., Zhang, X.: Chinese sign language recognition based on an optimized tree-structure framework. IEEE J. Biomed. Health Inform. 21(4), 994–1004 (2017)

    Article  Google Scholar 

  • Yu, Y., Wang, D., Zhao, R., Zhang, Q.: RFID based real-time recognition of ongoing gesture with adversarial learning. In: Proceedings of the 17th Conference on Embedded Networked Sensor Systems, pp. 298–310 (2019)

  • Zhang, Q., Wang, D., Zhao, R., Yu, Y.: Myosign: enabling end-to-end sign language recognition with wearables. In: Proceedings of the 24th International Conference on Intelligent User Interfaces, ACM, pp. 650–660 (2019a)

  • Zhang, S., Yang, C., Kui, X., Wang, J., Liu, X., Guo, S.: Reactor: real-time and accurate contactless gesture recognition with RFID. In: 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), IEEE, pp. 1–9 (2019b)

  • Zhao, M., Yue, S., Katabi, D., Jaakkola, T.S., Bianchi, M.T.: Learning sleep stages from radio signals: a conditional adversarial architecture. In: International Conference on Machine Learning, pp. 4100–4109 (2017)

  • Zou, Y., Xiao, J., Han, J., Wu, K., Li, Y., Ni, L.M.: Grfid: a device-free RFID-based gesture recognition system. IEEE Trans. Mob. Comput. 16(2), 381–393 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Software, Shanghai Jiao Tong University, Shanghai, China

    Run Zhao, Qian Zhang, Dian Cao, Zhiyao Sheng & Dong Wang

Authors
  1. Run Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dian Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiyao Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dong Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, R., Zhang, Q., Cao, D. et al. Gesture recognition with RFID: an experimental study. CCF Trans. Pervasive Comp. Interact. 3, 397–412 (2021). https://doi.org/10.1007/s42486-021-00079-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-021-00079-x

Keywords

Search

Navigation