skip to main content
research-article

SwingNet: Ubiquitous Fine-Grained Swing Tracking Framework via Stochastic Neural Architecture Search and Adversarial Learning

Published: 14 September 2021 Publication History

Abstract

Sports analytics in the wild (i.e., ubiquitously) is a thriving industry. Swing tracking is a key feature in sports analytics. Therefore, a centimeter-level tracking resolution solution is required. Recent research has explored deep neural networks for sensor fusion to produce consistent swing-tracking performance. This is achieved by combining the advantages of two sensor modalities (IMUs and depth sensors) for golf swing tracking. Here, the IMUs are not affected by occlusion and can support high sampling rates. Meanwhile, depth sensors produce significantly more accurate motion measurements than those produced by IMUs. Nevertheless, this method can be further improved in terms of accuracy and lacking information for different domains (e.g., subjects, sports, and devices). Unfortunately, designing a deep neural network with good performance is time consuming and labor intensive, which is challenging when a network model is deployed to be used in new settings. To this end, we propose a network based on Neural Architecture Search (NAS), called SwingNet, which is a regression-based automatic generated deep neural network via stochastic neural network search. The proposed network aims to learn the swing tracking feature for better prediction automatically. Furthermore, SwingNet features a domain discriminator by using unsupervised learning and adversarial learning to ensure that it can be adaptive to unobserved domains. We implemented SwingNet prototypes with a smart wristband (IMU) and smartphone (depth sensor), which are ubiquitously available. They enable accurate sports analytics (e.g., coaching, tracking, analysis and assessment) in the wild.
Our comprehensive experiment shows that SwingNet achieves less than 10 cm errors of swing tracking with a subject-independent model covering multiple sports (e.g., golf and tennis) and depth sensor hardware, which outperforms state-of-the-art approaches.

References

[1]
Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In 11th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 14). 317--329.
[2]
Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. 2018. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5167--5176.
[3]
Nguyen Khoa Nam Cao, Young Soo Suh, and Quoc Khanh Dang. 2013. 3D dynamics analysis of a golf full swing by fusing inertial sensor and vision data. In 2013 13th International Conference on Control, Automation and Systems (ICCAS 2013). IEEE, 1300--1303.
[4]
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. arXiv preprint arXiv:1812.08008 (2018).
[5]
Changhao Chen, Xiaoxuan Lu, Andrew Markham, and Niki Trigoni. 2018. IONet: Learning to cure the curse of drift in inertial odometry. In Thirty-Second AAAI Conference on Artificial Intelligence.
[6]
Sungkuk Chun, Donghoon Kang, Hyeong-Rae Choi, Anjin Park, Ki-Kwang Lee, and Jinwook Kim. 2014. A sensor-aided self coaching model for uncocking improvement in golf swing. Multimedia Tools and Applications 72, 1 (2014), 253--279.
[7]
Jessica Maria Echterhoff, Juan Haladjian, and Bernd Brügge. 2018. Gait and jump classification in modern equestrian sports. In Proceedings of the 2018 ACM International Symposium on Wearable Computers. 88--91.
[8]
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural Architecture Search: A Survey. Journal of Machine Learning Research 20, 55 (2019), 1--21. http://jmlr.org/papers/v20/18-598.html
[9]
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. 2019. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 3224--3234.
[10]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (NIPS'14). MIT Press, Cambridge, MA, USA, 2672--2680. http://dl.acm.org/citation.cfm?id=2969033.2969125
[11]
Mahanth Gowda, Ashutosh Dhekne, Sheng Shen, Romit Roy Choudhury, Lei Yang, Suresh Golwalkar, and Alexander Essanian. 2017. Bringing IoT to sports analytics. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ( {NSDI} 17). 499--513.
[12]
Aric Hagberg, Dan Schult, Pieter Swart, D Conway, L Séguin-Charbonneau, C Ellison, B Edwards, and J Torrents. 2013. Networkx. High productivity software for complex networks. Webová strá nka https://networkx.lanl.gov/wiki (2013).
[13]
Munawar Hayat, Mohammed Bennamoun, and Amar A El-Sallam. 2016. An RGB-D based image set classification for robust face recognition from Kinect data. Neurocomputing 171 (2016), 889--900.
[14]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[15]
Fang Hu, Peng He, Songlin Xu, Yin Li, and Cheng Zhang. 2020. FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 71 (June 2020), 24 pages. https://doi.org/10.1145/3397306
[16]
Forrest Iandola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. 2014. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv:1404.1869 (2014).
[17]
Jeya Vikranth Jeyakumar, Liangzhen Lai, Naveen Suda, and Mani Srivastava. 2019. SenseHAR: a robust virtual activity sensor for smartphones and wearables. In Proceedings of the 17th Conference on Embedded Networked Sensor Systems. ACM, 15--28.
[18]
Hong Jia, Yuezhong Wu, Jun Liu, Lina Yao, and Wen Hu. 2019. Mobile Golf Swing Tracking Using Deep Learning with Data Fusion: Poster Abstract. In Proceedings of the 17th Conference on Embedded Networked Sensor Systems (SenSys '19). Association for Computing Machinery, New York, NY, USA, 422--423. https://doi.org/10.1145/3356250.3361968
[19]
Yubing Jian, Mohit Agarwal, Yuchen Liu, Douglas M Blough, and Raghupathy Sivakumar. 2019. Poster: Hawkeye-Predictive Positioning of a Ceiling-Mounted Mobile AP in mmWave WLANs for Maximizing Line-of-sight. In The 25th Annual International Conference on Mobile Computing and Networking. 1--3.
[20]
Wenjun Jiang, Chenglin Miao, Fenglong Ma, Shuochao Yao, Yaqing Wang, Ye Yuan, Hongfei Xue, Chen Song, Xin Ma, Dimitrios Koutsonikolas, Wenyao Xu, and Lu Su. 2018. Towards Environment Independent Device Free Human Activity Recognition. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking (MobiCom '18). Association for Computing Machinery, New York, NY, USA, 289--304. https://doi.org/10.1145/3241539.3241548
[21]
Rushil Khurana, Karan Ahuja, Zac Yu, Jennifer Mankoff, Chris Harrison, and Mayank Goel. 2018. GymCam: Detecting, Recognizing and Tracking Simultaneous Exercises in Unconstrained Scenes. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 185 (Dec. 2018), 17 pages. https://doi.org/10.1145/3287063
[22]
Ho-Han Kim and Sung-Young Kim. 2019. Implementation of Golf Swing Analysis System Based on Swing Trajectories Analysis. JOURNAL OF ADVANCED INFORMATION TECHNOLOGY AND CONVERGENCE 9, 2 (2019), 65--74.
[23]
Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. Spotfi: Decimeter level localization using wifi. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 269--282.
[24]
Kensaku Kumada, Yutaka Usui, and Katsuya Kondo. 2013. Golf swing tracking and evaluation using Kinect sensor and particle filter. In 2013 International Symposium on Intelligent Signal Processing and Communication Systems. IEEE, 698--703.
[25]
Hyeokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D. Abowd, Nicholas D. Lane, and Thomas Plötz. 2020. IMUTube: Automatic Extraction of Virtual on-Body Accelerometry from Video for Human Activity Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 87 (Sept. 2020), 29 pages. https://doi.org/10.1145/3411841
[26]
Mingyang Li, Hongsheng Yu, Xing Zheng, and Anastasios I Mourikis. 2014. High-fidelity sensor modeling and self-calibration in vision-aided inertial navigation. In 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 409--416.
[27]
Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, and Song Han. 2020. Mcunet: Tiny deep learning on iot devices. arXiv preprint arXiv:2007.10319 (2020).
[28]
Xinye Lin, Yixin Chen, Xiao-Wen Chang, Xue Liu, and Xiaodong Wang. 2018. SHOW: Smart Handwriting on Watches. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 151 (Jan. 2018), 23 pages. https://doi.org/10.1145/3161412
[29]
David B Lindell, Matthew O'Toole, and Gordon Wetzstein. 2018. Single-photon 3D imaging with deep sensor fusion. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--12.
[30]
Yilin Liu, Fengyang Jiang, and Mahanth Gowda. 2020. Finger Gesture Tracking for Interactive Applications: A Pilot Study with Sign Languages. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 112 (Sept. 2020), 21 pages. https://doi.org/10.1145/3414117
[31]
Slobodan Milanko and Shubham Jain. 2020. LiftRight: Quantifying strength training performance using a wearable sensor. Smart Health (2020), 100115.
[32]
Cao Nguyen Khoa Nam, Hee Jun Kang, and Young Soo Suh. 2014. Golf swing motion tracking using inertial sensors and a stereo camera. IEEE Transactions on Instrumentation and measurement 63, 4 (2014), 943--952.
[33]
Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016. Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision. Springer, 483--499.
[34]
Soonchan Park, Ju Yong Chang, Hyuk Jeong, Jae-Ho Lee, and Ji-Young Park. 2017. Accurate and efficient 3d human pose estimation algorithm using single depth images for pose analysis in golf. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 49--57.
[35]
Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. 2017. Coarse-to-fine volumetric prediction for single-image 3D human pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7025--7034.
[36]
Natural Point. 2009. Inc.: Optitrack-optical motion tracking solutions.
[37]
E Real, A Aggarwal, Y Huang, and QV Le. [n.d.]. Regularized evolution for image classifier architecture search. arXiv 2018. arXiv preprint arXiv:1802.01548 ([n. d.]).
[38]
Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 429--444.
[39]
Pete Shinners et al. 2011. PyGame-Python game development. Retrieved from 785 (2011).
[40]
Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning. PMLR, 6105--6114.
[41]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
[42]
Chuyu Wang, Jian Liu, Yingying Chen, Lei Xie, Hong Bo Liu, and Sanclu Lu. 2018. RF-Kinect: A Wearable RFID-Based Approach Towards 3D Body Movement Tracking. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1, Article 41 (March 2018), 28 pages. https://doi.org/10.1145/3191773
[43]
Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: virtual touch screen in the air using RF signals. ACM SIGCOMM Computer Communication Review 44, 4 (2014), 235--246.
[44]
Jiann-Jyh Wang, Pei-Feng Yang, Wei-Hua Ho, and Tzyy-Yuang Shiang. 2015. Determine an effective golf swing by swing speed and impact precision tests. Journal of Sport and Health Science 4, 3 (2015), 244--249.
[45]
Lihong Wang and Kuiliang Xia. 2019. Data Fusion Algorithms for Wireless Sensor Networks Based on Deep Learning Model. In Proceedings of the 3rd International Conference on High Performance Compilation, Computing and Communications (HP3C '19). ACM, New York, NY, USA, 155--158. https://doi.org/10.1145/3318265.3318297
[46]
Teng Wei and Xinyu Zhang. 2015. mtrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 117--129.
[47]
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. 2019. Exploring randomly wired neural networks for image recognition. arXiv preprint arXiv:1904.01569 (2019).
[48]
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. 2018. SNAS: stochastic neural architecture search. arXiv preprint arXiv:1812.09926 (2018).
[49]
Lei Yang, Yekui Chen, Xiang-Yang Li, Chaowei Xiao, Mo Li, and Yunhao Liu. 2014. Tagoram: Real-time tracking of mobile RFID tags to high precision using COTS devices. In Proceedings of the 20th annual international conference on Mobile computing and networking. 237--248.
[50]
Xuehan Ye, Shuo Huang, Yongcai Wang, Wenping Chen, and Deying Li. 2019. Unsupervised Localization by Learning Transition Model. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 65 (June 2019), 23 pages. https://doi.org/10.1145/3328936
[51]
Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. 15--29.
[52]
Cheng Zhang, Fan Yang, Gang Li, Qiang Zhai, Yi Jiang, and Dong Xuan. 2018. Mv-sports: A motion and vision sensor integration-based sports analysis system. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1070--1078.
[53]
H. Zhao, S. Wang, G. Zhou, and W. Jung. 2019. TennisEye: Tennis Ball Speed Estimation using a Racket-mounted Motion Sensor. In 2019 18th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). 241--252. https://doi.org/10.1145/3302506.3310404
[54]
Han Zhou, Yi Gao, Xinyi Song, Wenxin Liu, and Wei Dong. 2019. LimbMotion: Decimeter-Level Limb Tracking for Wearable-Based Human-Computer Interaction. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 4, Article 161 (Dec. 2019), 24 pages. https://doi.org/10.1145/3369836
[55]
Xiaowei Zhou, Menglong Zhu, Georgios Pavlakos, Spyridon Leonardos, Konstantinos G Derpanis, and Kostas Daniilidis. 2019. Monocap: Monocular human motion capture using a cnn coupled with a geometric prior. IEEE transactions on pattern analysis and machine intelligence 41, 4 (2019), 901--914.

Cited By

View all
  • (2024)Silent Impact: Tracking Tennis Shots from the Passive ArmProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676403(1-15)Online publication date: 13-Oct-2024
  • (2024)Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314157:4(1-25)Online publication date: 12-Jan-2024
  • (2024)Enhancing accuracy and convenience of golf swing tracking with a wrist-worn single inertial sensorScientific Reports10.1038/s41598-024-59949-w14:1Online publication date: 22-Apr-2024
  • Show More Cited By

Index Terms

  1. SwingNet: Ubiquitous Fine-Grained Swing Tracking Framework via Stochastic Neural Architecture Search and Adversarial Learning

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 5, Issue 3
    Sept 2021
    1443 pages
    EISSN:2474-9567
    DOI:10.1145/3486621
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 14 September 2021
    Published in IMWUT Volume 5, Issue 3

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Adversarial Learning
    2. Neural Architecture Search
    3. Sport Analytics
    4. Swing Tracking

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)81
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Silent Impact: Tracking Tennis Shots from the Passive ArmProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676403(1-15)Online publication date: 13-Oct-2024
    • (2024)Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314157:4(1-25)Online publication date: 12-Jan-2024
    • (2024)Enhancing accuracy and convenience of golf swing tracking with a wrist-worn single inertial sensorScientific Reports10.1038/s41598-024-59949-w14:1Online publication date: 22-Apr-2024
    • (2023)Ubiquitous, Secure, and Efficient Mobile Sensing SystemsProceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services10.1145/3581791.3597511(629-630)Online publication date: 18-Jun-2023

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media