Skip to main content
SpringerLink
Log in

Generation of Action Recognition Training Data Through Rotoscoping and Augmentation of Synthetic Animations

  • Conference paper
  • First Online:
Augmented Reality, Virtual Reality, and Computer Graphics (AVR 2019)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11614))

  • 2071 Accesses

Abstract

In this paper, we present a method to synthetically generate the training material needed by machine learning algorithms to perform human action recognition from 2D videos. As a baseline pipeline, we consider a 2D video stream passing through a skeleton extractor (OpenPose), whose 2D joint coordinates are analyzed by a random forest. Such a pipeline is trained and tested using real live videos. As an alternative approach, we propose to train the random forest using automatically generated 3D synthetic videos. For each action, given a single reference live video, we edit a 3D animation (in Blender) using the rotoscoping technique. This prior animation is then used to produce a full training set of synthetic videos via perturbation of the original animation curves. Our tests, performed on live videos, show that our alternative pipeline leads to comparable accuracy, with the advantage of drastically reducing both the human effort and the computing power needed to produce the live training material.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.microsoft.com/en-us/hololens – Feb 8th, 2019.

  2. 2.

    https://www.metavision.com/ – Feb 8th, 2019.

  3. 3.

    https://www.ted.com/talks/meron_gribetz_a_glimpse_of_the_future_through_an_augmented_reality_headset?language=it.

  4. 4.

    https://en.wikipedia.org/wiki/Rotoscoping.

  5. 5.

    https://www.blender.org/ – Apr 30, 2019.

  6. 6.

    https://github.com/animate1978/MB-Lab – Feb 9th, 2019.

  7. 7.

    https://www.mathworks.com/help/stats/select-data-and-validation-for-classification-problem.html – Apr 30, 2019.

References

  1. Nilsson, M.: Action and intention recognition in human interaction with autonomous vehicles (2015)

    Google Scholar 

  2. D’Agostini, J., et al.: An augmented reality virtual assistant to help mild cognitive impaired users in cooking a system able to recognize the user status and personalize the support. In: 2018 Workshop on Metrology for Industry 4.0 and IoT, pp. 12–17. IEEE (2018)

    Google Scholar 

  3. Dariush, B., Fujimura, K., Sakagami, Y.: Vision based human activity recognition and monitoring system for guided virtual rehabilitation. US Patent App. 12/873,498, 3 March 2011

    Google Scholar 

  4. Gorecky, D., Schmitt, M., Loskyll, M., Zühlke, D.: Human-machine-interaction in the industry 4.0 era. In: 2014 12th IEEE International Conference on Industrial Informatics (INDIN), pp. 289–294. IEEE (2014)

    Google Scholar 

  5. Mizumoto, T., Fornaser, A., Suwa, H., Yasumoto, K., De Cecco, M.: Kinect-based micro-behavior sensing system for learning the smart assistance with human subjects inside their homes. In: 2018 Workshop on Metrology for Industry 4.0 and IoT, pp. 1–6. IEEE (2018)

    Google Scholar 

  6. Bloom, V., Makris, D., Argyriou, V.: G3D: a gaming action dataset and real time action recognition evaluation framework. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 7–12. IEEE (2012)

    Google Scholar 

  7. Papadopoulos, G.T., Axenopoulos, A., Daras, P.: Real-time skeleton-tracking-based human action recognition using kinect data. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds.) MMM 2014. LNCS, vol. 8325, pp. 473–483. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-04114-8_40

    Chapter  Google Scholar 

  8. Wang, C., Wang, Y., Yuille, A.L.: An approach to pose-based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 915–922 (2013)

    Google Scholar 

  9. Li, W., Zhang, Z., Liu, Z.: Action recognition based on a bag of 3D points. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 9–14. IEEE (2010)

    Google Scholar 

  10. Blank, M., Gorelick, L., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes, pp. 1395–1402. IEEE (2005)

    Google Scholar 

  11. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local SVM approach. In: Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, vol. 3, pp. 32–36. IEEE (2004)

    Google Scholar 

  12. Weinland, D., Ronfard, R., Boyer, E.: Free viewpoint action recognition using motion history volumes. Comput. Vis. Image Underst. 104(2–3), 249–257 (2006)

    Article  Google Scholar 

  13. Soomro, K., Zamir, A.R., Shah, M.: Ucf101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)

  14. Poppe, R.: A survey on vision-based human action recognition. Image Vis. Comput. 28(6), 976–990 (2010)

    Article  Google Scholar 

  15. Marin, J., Vázquez, D., Gerónimo, D., López, A.M.: Learning appearance in virtual scenarios for pedestrian detection. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 137–144. IEEE (2010)

    Google Scholar 

  16. Abbondanza, P., Giancola, S., Sala, R., Tarabini, M.: Accuracy of the microsoft kinect system in the identification of the body posture. In: Perego, P., Andreoni, G., Rizzo, G. (eds.) MobiHealth 2016. LNICST, vol. 192, pp. 289–296. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58877-3_37

    Chapter  Google Scholar 

  17. Lie, W.-N., Le, A.T., Lin, G.-H.: Human fall-down event detection based on 2D skeletons and deep learning approach. In: 2018 International Workshop on Advanced Image Technology (IWAIT), pp. 1–4. IEEE (2018)

    Google Scholar 

  18. Simon, T., Wei, S.-E., Joo, H., Sheikh, Y., Hidalgo, G., Cao, Z.: OpenPose (2018)

    Google Scholar 

  19. Fornaser, A., De Cecco, M., Bosetti, P., Mizumoto, T., Yasumoto, K.: Sigma-z random forest, classification and confidence. Meas. Sci. Technol. 30(2), 025002 (2018)

    Article  Google Scholar 

  20. Fornaser, A., Mizumoto, T., Suwa, H., Yasumoto, K., De Cecco, M.: The influence of measurements and feature types in automatic micro-behavior recognition in meal preparation. IEEE Instrum. Meas. Mag. 21(6), 10–14 (2018)

    Article  Google Scholar 

  21. Yoo, J.-H., Hwang, D., Nixon, M.S.: Gender classification in human gait using support vector machine. In: Blanc-Talon, J., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2005. LNCS, vol. 3708, pp. 138–145. Springer, Heidelberg (2005). https://doi.org/10.1007/11558484_18

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, University of Trento, Trento, Italy

    Nicola Covre, Alberto Fornaser & Mariolino De Cecco

  2. German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany

    Fabrizio Nunnari

Authors
  1. Nicola Covre
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fabrizio Nunnari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alberto Fornaser
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mariolino De Cecco
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicola Covre .

Editor information

Editors and Affiliations

  1. University of Salento, Lecce, Italy

    Lucio Tommaso De Paolis

  2. University of Paris-Sud, Orsay, France

    Patrick Bourdot

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Covre, N., Nunnari, F., Fornaser, A., De Cecco, M. (2019). Generation of Action Recognition Training Data Through Rotoscoping and Augmentation of Synthetic Animations. In: De Paolis, L., Bourdot, P. (eds) Augmented Reality, Virtual Reality, and Computer Graphics. AVR 2019. Lecture Notes in Computer Science(), vol 11614. Springer, Cham. https://doi.org/10.1007/978-3-030-25999-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-25999-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-25998-3

  • Online ISBN: 978-3-030-25999-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation