Skip to main content
SpringerLink
Log in

Sample-Based Human Movement Detection for Interactive Videos Applied to Performing Arts

  • Conference paper
  • First Online:
Human-Computer Interaction – INTERACT 2023 (INTERACT 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14144))

Included in the following conference series:

  • 675 Accesses

Abstract

In the performing arts context, video analysis plays a significant role in analyzing the artists’ movement in educational and professional artistic environments. However, current research lacks user-centered systems for artists to directly engage with movement detection in their respective practice recordings. Our framework presents an annotation tool capable of automatically analyzing human poses of individual examples in real-time and presenting motion data in the form of simple reports.

At the core of this system, augmenting multimedia content by manually adding different types of annotations to the intended video segment is possible. Therefore, we developed novel features that combine automatic intelligent movement analysis with manual annotation capabilities for a more rewarding and efficient user experience. We demonstrate how pose estimation techniques can be employed to set up customized sets of human poses that can be automatically identified in videos. In addition, this paper discusses the results obtained from two system evaluation phases, where we report our findings based on expert feedback.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A version of the annotation tool is available at https://motion-notes.di.fct.unl.pt/.

References

  1. Torre, I., Galluccio, I., Coccoli, M.: Video augmentation to support video-based learning. In: ACM International Conference Proceeding Series (2022)

    Google Scholar 

  2. Dias Pereira Dos, A., Loke, L., Martinez-Maldonado, R.: Exploring video annotation as a tool to support dance teaching. In: ACM International Conference Proceeding Series, pp. 448–452 (2018)

    Google Scholar 

  3. Paul, S., Saha, R., Padhi, S., Majumdar, S., Das, P.P., Rao, S.: NrityaManch: an annotation and retrieval system for bharatanatyam dance ACM reference format. Proceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation, p. 22 (2022). https://doi.org/10.1145/3574318

  4. Singh, V., Latulipe, C., Carroll, E., Lottridge, D.: The choreographer’s notebook-a video annotation system for dancers and choreographers. C and C 2011 - Proceedings of the 8th ACM Conference on Creativity and Cognition, pp. 197–206 (2011). https://doi.org/10.1145/2069618.2069653

  5. Lagrue, S., et al.: An ontology web application-based annotation tool for intangible culture heritage dance videos. SUMAC 2019 - Proceedings of the 1st Workshop on Structuring and Understanding of Multimedia heritAge Contents, co-located with MM 2019, pp. 75–81 (2019). https://doi.org/10.1145/3347317.3357245

  6. Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., Liu, Y.: Deep learning for sensor-based human activity recognition. ACM Computing Surveys (CSUR), p. 54 (2021). https://doi.org/10.1145/3447744

  7. Kuffner dos Anjos, R., Ribeiro, C., Fernandes, C., Sartor, J.B.: Three-Dimensional Visualization of Movement Qualities in Contemporary Dance (2018). https://doi.org/10.1145/3212721.3212812

  8. Ho, C., Tsai, W.T., Lin, K.S., Chen, H.H.: Extraction and alignment evaluation of motion beats for street dance. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing – Proceedings, pp. 2429–2433 (2013). https://doi.org/10.1109/ICASSP.2013.6638091

  9. Ashwin, T.S., Prakash, V., Rajendran, R.: A systematic review of intelligent tutoring sys-tems based on Gross body movement detected using computer vision. Computers and Education: Artificial Intell. 4, 100125 (2023). https://doi.org/10.1016/J.CAEAI.2023.100125

    Article  Google Scholar 

  10. Kong, Y., Fu, Y.: Human action recognition and prediction: a survey. Int J Comput Vis. 130, 1366–1401 (2022). https://doi.org/10.1007/S11263-022-01594-9

    Article  Google Scholar 

  11. Munea, T.L., Jembre, Y.Z., Weldegebriel, H.T., Chen, L., Huang, C., Yang, C.: The progress of human pose estimation: a survey and taxonomy of models applied in 2D human pose estimation. IEEE Access. 8, 133330–133348 (2020). https://doi.org/10.1109/ACCESS.2020.3010248

    Article  Google Scholar 

  12. Difini, G.M., Martins, M.G., Barbosa, J.L.V.: Human pose estimation for training assistance: a systematic literature review. ACM International Conference Proceeding Series, pp.189–196 (2021). https://doi.org/10.1145/3470482.3479633

  13. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (2017)

    Google Scholar 

  14. Li, Y.C., Chang, C.T., Cheng, C.C., Huang, Y.L.: Baseball swing pose estimation using openpose. 2021 IEEE International Conference on Robotics, Automation and Artificial Intelligence, RAAI 2021, pp. 6–9 (2021). https://doi.org/10.1109/RAAI52226.2021.9507807

  15. Potempski, F., Sabo, A., Patterson, K.K.: Technical note: quantifying music-dance synchrony during salsa dancing with a deep learning-based 2D pose estimator. J Biomech 141, 111178 (2022). https://doi.org/10.1016/J.JBIOMECH.2022.111178

  16. Muhammad, K., et al.: Human action recognition using attention based LSTM network with dilated CNN features. Future Generation Computer Syst. 125, 820–830 (2021). https://doi.org/10.1016/J.FUTURE.2021.06.045

  17. Yen, C.C., Pin, T., Xu, H.: Bilateral pose transformer for human pose estimation. ACM International Conference Proceeding Series. Par F180473, pp. 21–29 (2022). https://doi.org/10.1145/3532342.3532346

  18. Xia, H., Zhang, Q.: VitPose: multi-view 3D human pose estimation with vision transformer. 2022 IEEE 8th International Conference on Computer and Communications, ICCC 2022, pp. 1922–1927 (2022). https://doi.org/10.1109/ICCC56324.2022.10065997

  19. Janardhanan, J., Umamaheswari, S.: A comprehensive study on human pose estimation. 8th International Conference on Advanced Computing and Communication Systems, ICACCS 2022, pp. 535–541 (2022). https://doi.org/10.1109/ICACCS54159.2022.9784965

  20. Raheb, K.E., Stergiou, M., Katifori, A., Ioannidis, Y.: Dance interactive learning systems. ACM Comput Surv. 52, 1–37 (2020). https://doi.org/10.1145/3323335

    Article  Google Scholar 

  21. Towey, D., et al.: Students as partners in a multi-media note-taking app development: best practices. In: Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering Companion, ICSE-C, pp. 334–335. Institute of Electrical and Electronics Engineers Inc. (2017)

    Google Scholar 

  22. Rodrigues, R., Neves Madeira, R.: Studying natural user interfaces for smart video annotation towards ubiquitous environments. In: 20th International Conference on Mobile and Ubiquitous Multimedia (MUM 2021), pp. 1–18. ACM, New York, NY, USA, Leuven, Belgium (2021)

    Google Scholar 

  23. de Sousa, L., Richter, B., Nel, C.: The effect of multimedia use on the teaching and learning of Social Sciences at tertiary level: a case study. Yesterday and Today, pp. 1–22 (2017). https://doi.org/10.17159/2223-0386/2017/n17a1

  24. Sidi, Y., Blau, I., Shamir-Inbal, T.: Mapping active and collaborative learning in higher education through annotations in hyper-video by learning analytics. J. Comput. Assist Learn. 38, 1752–1764 (2022). https://doi.org/10.1111/JCAL.12714

    Article  Google Scholar 

  25. Tseng, S.S.: The influence of teacher annotations on student learning engagement and video watching behaviors. Int. J. Educ. Technol. High. Educ. 18, 1–17 (2021). https://doi.org/10.1186/S41239-021-00242-5/FIGURES/5

    Article  Google Scholar 

  26. Mirriahi, N., Jovanović, J., Lim, L.A., Lodge, J.M.: Two sides of the same coin: video an-notations and in-video questions for active learning. Education Tech. Research Dev. 69, 2571–2588 (2021). https://doi.org/10.1007/S11423-021-10041-4/TABLES/3

    Article  Google Scholar 

  27. Rijmer, S.: Negotiating deliberate choice-making: Insights from an interdisciplinary and mul-timodal encounter during the making of a new contemporary dance. In: Dance Data, Cognition, and Multimodal Communication, pp. 15–37. Routledge, London (2022)

    Google Scholar 

  28. Jurgens, S.: Three methods of designing a workflow with multimodal video annotation in interdisciplinary choreographic processes. Multimodality and Performance, Fernandes, C. (Ed.). Cambridge Scholars Publishing, Newcastle upon Tyne, pp. 159–178 (2016)

    Google Scholar 

  29. Butterworth, J., Wildschut, L.: Contemporary Choreography: A Critical Reader. Routledge (2009)

    Google Scholar 

  30. Harvie, J., Lavender, A.: Making Contemporary Theatre: International Rehearsal Processes, p. 252 (2010)

    Google Scholar 

  31. Jürgens, S., Fernandes, C.: Choreographic practice-as-research. In: Performance as Research, pp. 249–274. Routledge, First edition. Routledge, New York (2017)

    Google Scholar 

  32. Bargeron, D., Gupta, A., Grudin, J., Sanocki, E.: Annotations for streaming video on the web. Conference on Human Factors in Computing Systems – Proceedings, pp. 278–279 (1999). https://doi.org/10.1145/632716.632887

  33. Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., Sloetjes, H.: ELAN: a professional framework for multimodality research. In: Proceedings of the 5th International Conference on Language Resources and Evaluation, LREC 2006, pp. 1556–1559 (2006)

    Google Scholar 

  34. Buono, P., Desolda, G., Lanzilotti, R.: Scenes extraction from videos of telementored surgeries. Proceedings: DMS 2013 - 19th International Conference on Distributed Multimedia Systems, pp. 106–111 (2013)

    Google Scholar 

  35. Pless, A., Hari, R., Brem, B., Woermamm, U., Schnabel, K.P.: Using self and peer video annotations of simulated patient encounters in communication training to facilitate the reflection of communication skills: an implementation study. GMS J Med Educ. 38 (2021). https://doi.org/10.3205/ZMA001451

  36. Deng, D., et al.: EventAnchor: reducing Human Interactions in Event Annotation of Racket Sports Videos KEYWORDS. https://doi.org/10.1145/3411764.3445431

  37. Risko, E.F., Foulsham, T., Dawson, S., Kingstone, A.: The collaborative lecture annotation system (CLAS): a new TOOL for distributed learning. IEEE Trans. Learn. Technol. 6, 4–13 (2013). https://doi.org/10.1109/TLT.2012.15

    Article  Google Scholar 

  38. Cabral, D., Valente, J., Silva, J., Aragão, U., Fernandes, C., Correia, N.: A creation-tool for contemporary dance using multimodal video annotation. MM’11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops, pp. 905–908 (2011). https://doi.org/10.1145/2072298.2071899

  39. Ribeiro, C., Kuffner, R., Fernandes, C., Pereira, J.: 3D annotation in contemporary dance. In: Proceedings of the 3rd International Symposium on Movement and Computing, pp. 1–4. ACM, New York, NY, USA (2016)

    Google Scholar 

  40. de Lahunta, S., Jenett, F.: Making digital choreographic objects interrelate. In: Performing the Digital, pp. 63–80. transcript Verlag (2016)

    Google Scholar 

  41. Rizzo, A., et al.: WhoLoDancE: whole-body interaction learning for dance education. In: CEUR Workshop Proceedings, pp. 41–50 (2018)

    Google Scholar 

  42. Lagrue, S., et al.: An Ontology Web Application-based Annotation Tool for Intangible Culture Heritage Dance Videos (2019). https://doi.org/10.1145/3347317.3357245

  43. Rivière, J.P., Alaoui, S.F., Caramiaux, B., Mackay, W.E.: Capturing movement decomposition to support learning and teaching in contemporary dance. Proc ACM Hum Comput Interact. 3 (2019). https://doi.org/10.1145/3359188

  44. Rodrigues, R., Madeira, R.N., Correia, N.: Exploring the user interaction with a multimodal web-based video annotator. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST. 429 LNICST, pp. 13–22 (2022). https://doi.org/10.1007/978-3-030-99188-3_2/COVER

  45. Rodrigues, R., Madeira, R.N., Correia, N., Fernandes, C., Ribeiro, S.: Multimodal web based video annotator with real-time human pose estimation. Lecture Notes in Comput-er Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 11872 LNCS, pp. 23–30 (2019). https://doi.org/10.1007/978-3-030-33617-2_3/COVER

  46. Diogo, J., Rodrigues, R., Madeira, R., Correia, N.: Video annotation tool using human pose estimation for sports training. ACM International Conference Proceeding Series, pp. 262–265 (2022). https://doi.org/10.1145/3568444.3570592

Download references

Acknowledgments

This work is funded by Fundação para a Ciência e Tecnologia through Studentship grants: 2020.09417.BD (Ph.D.), UIDB/04516/2020/BIM/11 (MSc). It was supported by the project WEAVE, Grant Agreement Number: INEA/CEF/ICT/A2020/ 2288018. It is also supported by NOVA LINCS (UIDB/04516/2020) with the financial support of FCT.IP. Lastly, we would like to thank Virgínia Gonçalves, Ana Carvalho, Alina Teles, Bárbara Salvador, Berta Berton, Carolina Ramalho, Inês Costa and Mariana Colaço from Centro Cultural e Recreativo do Alto do Moinho for their feedback.

Author information

Authors and Affiliations

  1. NOVA LINCS, NOVA School of Science and Technology, NOVA University Lisbon, Lisbon, Portugal

    Rui Rodrigues, João Diogo & Nuno Correia

  2. SUSTAIN.RD, Escola Superior de Tecnologia de Setúbal, Instituto Politécnico de Setúbal, Setúbal, Portugal

    Rui Rodrigues

  3. ITI/LARSYS, Funchal, Portugal

    Stephan Jurgens

  4. ICNOVA, FCSH, NOVA University of Lisboa, Lisbon, Portugal

    Carla Fernandes

  5. CITUR, Instituto Politécnico de Leiria, Leiria, Portugal

    Carla Fernandes

Authors
  1. Rui Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. João Diogo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephan Jurgens
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carla Fernandes
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nuno Correia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nuno Correia .

Editor information

Editors and Affiliations

  1. University of West London, London, UK

    José Abdelnour Nocera

  2. Reykjavik University, Reykjavik, Iceland

    Marta Kristín Lárusdóttir

  3. University of York, York, UK

    Helen Petrie

  4. University of Bari Aldo Moro, Bari, Italy

    Antonio Piccinno

  5. Université Côte d’Azur, Sophia Antipolis Cedex, France

    Marco Winckler

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rodrigues, R., Diogo, J., Jurgens, S., Fernandes, C., Correia, N. (2023). Sample-Based Human Movement Detection for Interactive Videos Applied to Performing Arts. In: Abdelnour Nocera, J., Kristín Lárusdóttir, M., Petrie, H., Piccinno, A., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2023. INTERACT 2023. Lecture Notes in Computer Science, vol 14144. Springer, Cham. https://doi.org/10.1007/978-3-031-42286-7_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42286-7_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42285-0

  • Online ISBN: 978-3-031-42286-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation