skip to main content
10.1145/3586183.3606796acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

AR-Enhanced Workouts: Exploring Visual Cues for At-Home Workout Videos in AR Environment

Published: 29 October 2023 Publication History

Abstract

In recent years, with growing health consciousness, at-home workout has become increasingly popular for its convenience and safety. Most people choose to follow video guidance during exercising. However, our preliminary study revealed that fitness-minded people face challenges when watching exercise videos on handheld devices or fixed monitors, such as limited movement comprehension due to static camera angles and insufficient feedback. To address these issues, we reviewed popular workout videos, identified user requirements, and came up with an augmented reality (AR) solution. Following a user-centered iterative design process, we proposed a design space of AR visual cues for workouts and implemented an AR-based application. Specifically, we captured users’ exercise performance with pose-tracking technology and provided feedback via AR visual cues. Two user experiments showed that incorporating AR visual cues could improve movement comprehension and enable users to adjust their movements based on real-time feedback. Finally, we presented several suggestions to inspire future design and apply AR visual cues to sports training.

Supplemental Material

ZIP File
Supplemental File
ZIP File
Appendix and Video Demo

References

[1]
2023. 8Fit. Retrieved January 22, 2023 from https://8fit.com/
[2]
2023. Covid-19 has forced us to exercise indoors. But home fitness has been shaping our lives for decades.Retrieved January 22, 2023 from https://www.bbc.com/worklife/article/20200504-covid-19-update-quarantine-home-workouts-during-coronavirus
[3]
2023. Forma Gym. Retrieved January 22, 2023 from https://formagym.com/isotonic-or-isometric-exercise-which-will-help-you-most/
[4]
2023. Keep. Retrieved January 22, 2023 from https://www.gotokeep.com/
[5]
2023. Kinovea. Retrieved January 22, 2023 from https://www.kinovea.org/
[6]
2023. MediaPipe. Retrieved January 22, 2023 from https://google.github.io/mediapipe/
[7]
2023. Mixamo. Retrieved January 22, 2023 from https://www.mixamo.com/
[8]
2023. MotionPro. Retrieved January 22, 2023 from https://www.motionprosoftware.com/index.htm
[9]
2023. MRTK. Retrieved January 22, 2023 from https://github.com/microsoft/MixedRealityToolkit-Unity
[10]
2023. MVN Link. Retrieved January 22, 2023 from https://www.movella.com/products/motion-capture/xsens-mvn-link
[11]
2023. OnForm. Retrieved January 22, 2023 from https://www.getonform.com/sports
[12]
2023. OpenPose. Retrieved January 22, 2023 from https://github.com/CMU-Perceptual-Computing-Lab/openpose
[13]
2023. The pandemic’s home-workout revolution may be here to stay.Retrieved January 22, 2023 from https://www.washingtonpost.com/road-to-recovery/2021/01/07/home-fitness-boom/
[14]
2023. Youtube. Retrieved January 22, 2023 from https://www.youtube.com/
[15]
Sarah Fdili Alaoui, Frederic Bevilacqua, and Christian Jacquemin. 2015. Interactive visuals as metaphors for dance movement qualities. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 3 (2015), 1–24.
[16]
Hashem A Almusawi, Christopher M Durugbo, and Afaf M Bugawa. 2021. Innovation in physical education: Teachers’ perspectives on readiness for wearable technology integration. Computers & Education 167 (2021), 104185.
[17]
Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: enhancing movement training with an augmented reality mirror. In Proceedings of the ACM symposium on User Interface Software and Technology. 311–320.
[18]
Alberto Cannavò, Filippo Gabriele Pratticò, Giuseppe Ministeri, and Fabrizio Lamberti. 2018. A movement analysis system based on immersive virtual reality and wearable technology for sport training. In Proceedings of the International Conference on Virtual Reality. 26–31.
[19]
Jacky CP Chan, Howard Leung, Jeff KT Tang, and Taku Komura. 2010. A virtual reality dance training system using motion capture technology. IEEE Transactions on Learning Technologies 4, 2 (2010), 187–195.
[20]
Hua-Tsung Chen, Yu-Zhen He, and Chun-Chieh Hsu. 2018. Computer-assisted yoga training system. Multimedia Tools and Applications 77 (2018), 23969–23991.
[21]
Philo Tan Chua, Rebecca Crivella, Bo Daly, Ning Hu, Russ Schaaf, David Ventura, Todd Camill, Jessica Hodgins, and Randy Pausch. 2003. Training for physical tasks in virtual environments: Tai Chi. In Proceedings of the International Conference on Virtual Reality (VR). 87–94.
[22]
Christopher Clarke, Doga Cavdir, Patrick Chiu, Laurent Denoue, and Don Kimber. 2020. Reactive video: adaptive video playback based on user motion for supporting physical activity. In Proceedings of the ACM Symposium on User Interface Software and Technology. 196–208.
[23]
Caleb Conner and Gene Michael Poor. 2016. Correcting exercise form using body tracking. In Proceedings of the SIGCHI Conference Extended Abstracts on Human Factors in Computing Systems. 3028–3034.
[24]
Dazhen Deng, Weiwei Cui, Xiyu Meng, Mengye Xu, Yu Liao, Haidong Zhang, and Yingcai Wu. 2022. Revisiting the Design Patterns of Composite Visualizations. IEEE Transactions on Visualization and Computer Graphics (2022), 1–16. https://doi.org/10.1109/TVCG.2022.3213565
[25]
Dazhen Deng, Aoyu Wu, Haotian Li, Ji Lan, Yong Wang, Huamin Qu, and Yingcai Wu. 2022. KB4VA: A Knowledge Base of Visualization Designs for Visual Analytics. arxiv:2211.02567 [cs.HC]
[26]
Dazhen Deng, Jiang Wu, Jiachen Wang, Yihong Wu, Xiao Xie, Zheng Zhou, Hui Zhang, Xiaolong (Luke) Zhang, and Yingcai Wu. 2021. EventAnchor: Reducing Human Interactions in Event Annotation of Racket Sports Videos. In Proceedings of CHI Conference on Human Factors in Computing Systems. 1–13.
[27]
Bhat Dittakavi, Divyagna Bavikadi, Sai Vikas Desai, Soumi Chakraborty, Nishant Reddy, Vineeth N Balasubramanian, Bharathi Callepalli, and Ayon Sharma. 2022. Pose Tutor: An explainable system for pose correction in the wild. In Proceedings of the CVF Conference on Computer Vision and Pattern Recognition. 3540–3549.
[28]
Mihai Fieraru, Mihai Zanfir, Silviu Cristian Pirlea, Vlad Olaru, and Cristian Sminchisescu. 2021. AIFit: Automatic 3d human-interpretable feedback models for fitness training. In Proceedings of the CVF Conference on Computer Vision and Pattern Recognition. 9919–9928.
[29]
David Hamilton, Jim McKechnie, Edward Edgerton, and Claire Wilson. 2021. Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design. Journal of Computers in Education 8, 1 (2021), 1–32.
[30]
Felix Hülsmann, Jan Philip Göpfert, Barbara Hammer, Stefan Kopp, and Mario Botsch. 2018. Classification of motor errors to provide real-time feedback for sports coaching in virtual reality—A case study in squats and Tai Chi pushes. Computers & Graphics 76 (2018), 47–59.
[31]
Eakta Jain, Yaser Sheikh, Moshe Mahler, and Jessica Hodgins. 2010. Augmenting Hand Animation with Three-dimensional Secondary Motion. In Symposium on Computer Animation. 93–102.
[32]
Raine Kajastila, Leo Holsti, and Perttu Hämäläinen. 2016. The augmented climbing wall: High-exertion proximity interaction on a wall-sized interactive surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 758–769.
[33]
Taihei Kojima, Atsushi Hiyama, Takahiro Miura, and Michitaka Hirose. 2014. Training archived physical skill through immersive virtual environment. In Human Interface and the Management of Information.51–58.
[34]
Mohammad Amin Kuhail, Areej ElSayary, Shahbano Farooq, and Ahlam Alghamdi. 2022. Exploring Immersive Learning Experiences: A Survey. In Informatics, Vol. 9. 75.
[35]
Matthew Kyan, Guoyu Sun, Haiyan Li, Ling Zhong, Paisarn Muneesawang, Nan Dong, Bruce Elder, and Ling Guan. 2015. An approach to ballet dance training through ms kinect and visualization in a cave virtual reality environment. ACM Transactions on Intelligent Systems and Technology (TIST) 6, 2 (2015), 1–37.
[36]
Qing Lei, Ji-Xiang Du, Hong-Bo Zhang, Shuang Ye, and Duan-Sheng Chen. 2019. A survey of vision-based human action evaluation methods. Sensors 19, 19 (2019), 4129.
[37]
Tica Lin, Rishi Singh, Yalong Yang, Carolina Nobre, Johanna Beyer, Maurice A Smith, and Hanspeter Pfister. 2021. Towards an understanding of situated ar visualization for basketball free-throw training. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–13.
[38]
Tica Lin, Yalong Yang, Johanna Beyer, and Hanspeter Pfister. 2020. SportsXR–Immersive Analytics in Sports. arXiv preprint arXiv:2004.08010 (2020).
[39]
Jingyuan Liu, Nazmus Saquib, Zhutian Chen, Rubaiat Habib Kazi, Li-Yi Wei, Hongbo Fu, and Chiew-Lan Tai. 2022. PoseCoach: A Customizable Analysis and Visualization System for Video-based Running Coaching. IEEE Transactions on Visualization and Computer Graphics (2022), To appear.
[40]
Laura Marchal-Crespo, Mark van Raai, Georg Rauter, Peter Wolf, and Robert Riener. 2013. The effect of haptic guidance and visual feedback on learning a complex tennis task. Experimental Brain Research 231 (2013), 277–291.
[41]
Takashi Matsumoto, Erwin Wu, and Hideki Koike. 2022. Skiing, Fast and Slow: Evaluation of Time Distortion for VR Ski Training. In Augmented Humans. 142–151.
[42]
Stefan C Michalski, Ancret Szpak, and Tobias Loetscher. 2019. Using virtual environments to improve real-world motor skills in sports: a systematic review. Frontiers in psychology 10 (2019), 2159.
[43]
Moritz Mödinger, Alexander Woll, and Ingo Wagner. 2022. Video-based visual feedback to enhance motor learning in physical education—a systematic review. German Journal of Exercise and Sport Research 52, 3 (2022), 447–460.
[44]
Tamara Munzner. 2014. Visualization analysis and design.
[45]
Heather L O’Brien and Elaine G Toms. 2010. The development and evaluation of a survey to measure user engagement. Journal of the American Society for Information Science and Technology 61, 1 (2010), 50–69.
[46]
Katerina El Raheb, Marina Stergiou, Akrivi Katifori, and Yannis Ioannidis. 2019. Dance interactive learning systems: A study on interaction workflow and teaching approaches. ACM Computing Surveys (CSUR) 52, 3 (2019), 1–37.
[47]
Katerina El Raheb, George Tsampounaris, Akrivi Katifori, and Yannis Ioannidis. 2018. Choreomorphy: A whole-body interaction experience for dance improvisation and visual experimentation. In Proceedings of the International Conference on Advanced Visual Interfaces. 1–9.
[48]
Alessandra Semeraro and Laia Turmo Vidal. 2022. Visualizing instructions for physical training: Exploring visual cues to support movement learning from instructional videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–16.
[49]
Cezary Sieluzycki, Patryk Kaczmarczyk, Janusz Sobecki, Kazimierz Witkowski, Jarosław Maśliński, and Wojciech Cieśliński. 2016. Microsoft Kinect as a tool to support training in professional sports: augmented reality application to Tachi-Waza techniques in judo. In Third European Network Intelligence Conference (ENIC). 153–158.
[50]
Pooya Soltani and Antoine HP Morice. 2020. Augmented reality tools for sports education and training. Computers & Education 155 (2020), 103923.
[51]
Robert Spence. 2001. Information visualization. Vol. 1.
[52]
Marina Stergiou, Katerina El Raheb, and Yannis Ioannidis. 2019. Imagery and metaphors: From movement practices to digital and immersive environments. In Proceedings of the International Conference on Movement and Computing. 1–8.
[53]
Richard Tang, Xing-Dong Yang, Scott Bateman, Joaquim Jorge, and Anthony Tang. 2015. Physio@Home: Exploring visual guidance and feedback techniques for physiotherapy exercises. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 4123–4132.
[54]
Wan-Lun Tsai, Li-wen Su, Tsai-Yen Ko, Cheng-Ta Yang, and Min-Chun Hu. 2019. Improve the decision-making skill of basketball players by an action-aware VR training system. In Proceedings of the International Conference on Virtual Reality and 3D User Interfaces (VR). 1193–1194.
[55]
Georgios Tsampounaris, Katerina El Raheb, Vivi Katifori, and Yannis Ioannidis. 2016. Exploring visualizations in real-time motion capture for dance education. In Proceedings of the Pan-Hellenic Conference on Informatics. 1–6.
[56]
Laia Turmo Vidal, Elena Márquez Segura, Christopher Boyer, and Annika Waern. 2019. Enlightened yoga: Designing an augmented class with wearable lights to support instruction. In Proceedings of the Designing Interactive Systems Conference. 1017–1031.
[57]
Jarke J Van Wijk. 2005. The value of visualization. In Proceedings of the IEEE International Conference on Visualization. 79–86.
[58]
Eduardo Velloso, Andreas Bulling, and Hans Gellersen. 2013. MotionMA: motion modelling and analysis by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1309–1318.
[59]
Jianbo Wang, Kai Qiu, Houwen Peng, Jianlong Fu, and Jianke Zhu. 2019. AI coach: Deep human pose estimation and analysis for personalized athletic training assistance. In Proceedings of the ACM International Conference on Multimedia. 374–382.
[60]
Paweł W Woźniak, Monika Zbytniewska, Francisco Kiss, and Jasmin Niess. 2021. Making Sense of Complex Running Metrics Using a Modified Running Shoe. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–11.
[61]
Erwin Wu, Takayuki Nozawa, Florian Perteneder, and Hideki Koike. 2020. VR alpine ski training augmentation using visual cues of leading skier. In Proceedings of the CVF Conference on Computer Vision and Pattern Recognition Workshops. 878–879.
[62]
Erwin Wu, Mitski Piekenbrock, Takuto Nakumura, and Hideki Koike. 2021. SPinPong-virtual reality table tennis skill acquisition using visual, haptic and temporal cues. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2566–2576.
[63]
Shuainan Ye, Zhutian Chen, Xiangtong Chu, Yifan Wang, Siwei Fu, Lejun Shen, Kun Zhou, and Yingcai Wu. 2020. ShuttleSpace: Exploring and analyzing movement trajectory in immersive visualization. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 860–869.
[64]
Ziyi Zhao, Sena Kiciroglu, Hugues Vinzant, Yuan Cheng, Isinsu Katircioglu, Mathieu Salzmann, and Pascal Fua. 2022. 3D Pose Based Feedback For Physical Exercises. In Proceedings of the Asian Conference on Computer Vision. 1316–1332.
[65]
Qiushi Zhou, Andrew Irlitti, Difeng Yu, Jorge Goncalves, and Eduardo Velloso. 2022. Movement guidance using a mixed reality mirror. In Proceedings of the Designing Interactive Systems Conference. 821–834.
[66]
Liyuan Zou, Takatoshi Higuchi, Haruo Noma, Lopez-Gulliver Roberto, and Tadao Isaka. 2019. Evaluation of a virtual reality-based baseball batting training system using instantaneous bat swing information. In Proceedings of the International Conference on Virtual Reality and 3D User Interfaces (VR). 1289–1290.

Cited By

View all
  • (2025)ChartGPT: Leveraging LLMs to Generate Charts From Abstract Natural LanguageIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.336862131:3(1731-1745)Online publication date: Mar-2025
  • (2024)Low Vision Boxing: Participatory Design of Adaptive Kickboxing Experiences with Low Vision PersonProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675619(1-18)Online publication date: 27-Oct-2024
  • (2024)MR MANE: MR Microsurgical Suturing Skill Acquisition for Novice Using Imitation of ExampleProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656660(1-5)Online publication date: 3-Jun-2024
  • Show More Cited By

Index Terms

  1. AR-Enhanced Workouts: Exploring Visual Cues for At-Home Workout Videos in AR Environment

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
    October 2023
    1825 pages
    ISBN:9798400701320
    DOI:10.1145/3586183
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 29 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. SportsXR
    2. argumented reality
    3. movement learning

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    • National Key R&D Program of China
    • NSFC

    Conference

    UIST '23

    Acceptance Rates

    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)613
    • Downloads (Last 6 weeks)35
    Reflects downloads up to 10 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)ChartGPT: Leveraging LLMs to Generate Charts From Abstract Natural LanguageIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.336862131:3(1731-1745)Online publication date: Mar-2025
    • (2024)Low Vision Boxing: Participatory Design of Adaptive Kickboxing Experiences with Low Vision PersonProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675619(1-18)Online publication date: 27-Oct-2024
    • (2024)MR MANE: MR Microsurgical Suturing Skill Acquisition for Novice Using Imitation of ExampleProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656660(1-5)Online publication date: 3-Jun-2024
    • (2024)VisCourt: In-Situ Guidance for Interactive Tactic Training in Mixed RealityProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676466(1-14)Online publication date: 13-Oct-2024
    • (2024)FitSight: Tracking and Feedback Engine for Personalized Fitness TrainingProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659547(223-231)Online publication date: 22-Jun-2024
    • (2024)Development of Intelligent Workout Environment for VR Devices2024 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON60312.2024.10578849(1-5)Online publication date: 8-May-2024
    • (2024)Adaptive 360° video timeline exploration in VR environmentComputers & Graphics10.1016/j.cag.2024.104108125(104108)Online publication date: Dec-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media