Skip to main content
SpringerLink
Log in

An End-to-End Object Detector with Spatiotemporal Context Learning for Machine-Assisted Rehabilitation

  • Conference paper
  • First Online:
Intelligent Robotics and Applications (ICIRA 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13455))

Included in the following conference series:

  • 2463 Accesses

Abstract

Recently, object detection technologies applied in rehabilitation systems are mainly based on the ready-made technology of CNNs. This paper proposes an DETR-based detector which is an end-to-end object detector with spatiotemporal context learning for machine-assisted rehabilitation. To improve the performance of small object detection, first, the multi-level features of the RepVGG are fused with the SE attention mechanism to build a SEFP-RepVGG. To make the encoder-decoder structure more suitable, next, the value of the encoder is generated by using feature maps with more detailed information than key/query. To reduce computation, Patch Merging is finally imported to modify the feature map scale of the input encoder. The proposed detector has higher real-time performance than DETR and obtains the competitive detection accuracy on the ImageNet VID benchmark. Some typical samples from the NTU RGB-D 60 dataset are selected to build a new limb-detection dataset for further evaluation. The results show the effectiveness of the proposed detector in the rehabilitation scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kadu, A., Singh, M.: Comparative analysis of e-health care telemedicine system based on internet of medical things and artificial intelligence. In: 2nd International Conference on Smart Electronics and Communication (ICOSEC), pp. 1768–1775 (2021). https://doi.org/10.1109/ICOSEC51865.2021.9591941

  2. Debnath, B., O’Brien, M., Yamaguchi, M., Behera, A.: A review of computer vision-based approaches for physical rehabilitation and assessment. Multimedia Syst. 28(1), 209–239 (2021). https://doi.org/10.1007/s00530-021-00815-4

    Article  Google Scholar 

  3. Chae, S.H., Kim, Y., Lee, K.S., Park, H.S.: Development and clinical evaluation of a web-based upper limb home rehabilitation system using a smartwatch and machine learning model for chronic stroke survivors: prospective comparative study. JMIR Mhealth Uhealth 8(7), e17216 (2020). https://doi.org/10.2196/17216

    Article  Google Scholar 

  4. Chiang, A.T., Chen, Q., Wang, Y., Fu, M.R.: Kinect-based in-home exercise system for lymphatic health and lymphedema intervention. IEEE J. Transl. Eng. Health Med. 6, 1–13 (2018). https://doi.org/10.1109/JTEHM.2018.2859992

    Article  Google Scholar 

  5. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017). https://doi.org/10.1109/TPAMI.2016.2577031

    Article  Google Scholar 

  6. Redmon, J.,, Farhadi, A.: YOLOv3: an incremental improvement. eprint arXiv:1804.02767 (2018). https://doi.org/10.48550/arXiv.1804.02767

  7. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  8. Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: RepVGG: making VGG-style ConvNets great again. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13728–13737. IEEE, New York (2021). https://doi.org/10.1109/CVPR46437.2021.01352

  9. Zhang, X., Gao, Y., Ye, F., Liu, Q., Zhang, K.: An approach to improve SSD through skip connection of multiscale feature maps. Comput. Intell. Neurosci. 2020, 13 (2020). https://doi.org/10.1155/2020/2936920

    Article  Google Scholar 

  10. Yao, Z., Ai, J., Li, B., Zhang, C.: Efficient DETR: improving end-to-end object detector with dense prior. eprint arXiv: 2104.01318 (2021). https://doi.org/10.48550/arxiv.2104.01318

  11. Li, S., et al.: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. eprint arXiv: 1907.00235 (2019). https://doi.org/10.48550/arxiv.1907.00235

  12. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992–10002. IEEE, New York (2021). https://doi.org/10.48550/arxiv.2103.14030

  13. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  14. Shahroudy, A., Liu, J., Ng, T.-T., Wang, G.: NTU RGB+D: a large scale dataset for 3D human activity analysis. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1010–1019. IEEE, New York (2016). https://doi.org/10.1109/CVPR.2016.115

  15. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: deformable transformers for end-to-end object detection. eprint arXiv: 2010.04159 (2020). https://doi.org/10.48550/arxiv.2010.04159

  16. Meng, D., et al.: Conditional DETR for fast training convergence. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3631–3640. IEEE, New York (2021). https://doi.org/10.1109/ICCV48922.2021.00363

  17. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995. IEEE, New York (2017). https://doi.org/10.1109/CVPR.2017.634

  18. Howard, A., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. eprint arXiv: 1704.04861 (2017). https://doi.org/10.48550/arxiv.1704.04861

  19. Child, R., Gray, S., Radford, A., Sutskever, I: Generating long sequences with sparse transformers. eprint arXiv: 1904.10509 (2019). https://doi.org/10.48550/arxiv.1904.10509

  20. Zhou, N.: Research on video object detection based on temporal characteristics. J. China Acad. Electron. Inf. 16(02), 157–164 (2021)

    Google Scholar 

  21. Chai, Y.: Patchwork: a patch-wise attention network for efficient object detection and segmentation in video streams. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3414–3423. IEEE, New York (2019). https://doi.org/10.1109/ICCV.2019.00351

  22. Tan, M., Pang, R., Le, Q.V.: EfficientDet: scalable and efficient object detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10778–10787. IEEE, New York (2019). https://doi.org/10.1109/CVPR42600.2020.0107

  23. Chen, X., Yu, J., Wu, Z.: Temporally identity-aware SSD with attentional LSTM. IEEE Trans. Cybern. 50(6), 2674–2686 (2020). https://doi.org/10.1109/TCYB.2019.2894261

    Article  Google Scholar 

  24. Kang, K., et al.: Object detection in videos with tubelet proposal networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 889–897. IEEE, New York (2017). https://doi.org/10.1109/CVPR.2017.101

Download references

Acknowledgments

The authors would like to acknowledge the support from the AiBle project co-financed by the European Regional Development Fund, National Key R&D Program of China (Grant No. 2018YFB1304600), CAS Interdisciplinary Innovation Team (Grant No. JCTD-2018-11), Liaoning Province Higher Education Innovative Talents Program Support Project (Grant No. LR2019058), and National Natural Science Foundation of China (grant No. 52075530, 51575412, and 62006204). LiaoNing Province Joint Open Fund for Key Scientific and Technological Innovation Bases (Grant No. 2021-KF-12-05).

Author information

Authors and Affiliations

  1. School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, 110159, Liaoning, China

    Xuna Wang, Hongwei Gao & Tianyu Ma

  2. Department of Biomedical Engineering, Zhejiang University, Hangzhou, 310027, China

    Jiahui Yu

Authors
  1. Xuna Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongwei Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tianyu Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiahui Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongwei Gao .

Editor information

Editors and Affiliations

  1. Harbin Institute of Technology, Shenzhen, China

    Honghai Liu

  2. Huazhong University of Science and Technology, Wuhan, China

    Zhouping Yin

  3. Shenyang Institute of Automation, Shenyang, Liaoning, China

    Lianqing Liu

  4. Harbin Institute of Technology, Harbin, China

    Li Jiang

  5. Shanghai Jiao Tong University, Shanghai, China

    Guoying Gu

  6. Shenzhen Institute of Advanced Technology, Shenzhen, China

    Xinyu Wu

  7. Harbin Institute of Technology, Shenzhen, China

    Weihong Ren

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, X., Gao, H., Ma, T., Yu, J. (2022). An End-to-End Object Detector with Spatiotemporal Context Learning for Machine-Assisted Rehabilitation. In: Liu, H., et al. Intelligent Robotics and Applications. ICIRA 2022. Lecture Notes in Computer Science(), vol 13455. Springer, Cham. https://doi.org/10.1007/978-3-031-13844-7_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-13844-7_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-13843-0

  • Online ISBN: 978-3-031-13844-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation