Skip to main content
SpringerLink
Log in

Automatic Indexing of Virtual Camera Features from Japanese Anime

  • Conference paper
  • First Online:
Image Analysis and Processing. ICIAP 2022 Workshops (ICIAP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13373))

Included in the following conference series:

  • 907 Accesses

Abstract

The position, orientation, and the distance of the camera in relation to the subject(s) in a movie scene, namely camera level, camera angle, and shot scale, are essential features in the film-making process due to their influence on the viewer’s perception of the scene. Since animation techniques exploit drawings or computer graphics objects for making films instead of camera shooting, the automatic understanding of such “virtual camera” features appears harder if compared to live-action movies. Exploiting a dataset of animated movies from popular directors such as Hayao Miyazaki, Hideaki Anno and Mamoru Oshii, we finetune pre-trained convolutional neural networks and use One Cycle Learning Rate to reach convergence in a limited number of epochs. While some difficulties are revealed and discussed in classifying complex features, like camera angle and camera level, showing F1 scores of respectively 61% and 68%, the classification of the shot scale reaches a score of about 80%, which is comparable with state-of-the-art methods applied on live-action movies. The developed models will be useful in conducting automated movie annotation for a wide range of applications, such as in stylistic analysis, video recommendation, and studies in media psychology. The database and a demo of the classifiers are available at https://cinescale.github.io/anime

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/

  2. Arijon, D.: Grammar of the Film Language. Silman-James Press, Beverly Hills, CA (1976)

    Google Scholar 

  3. Bak, H.Y., Park, S.B.: Comparative study of movie shot classification based on semantic segmentation. Appl. Sci. 10, 3390 (2020)

    Article  Google Scholar 

  4. Benini, S., Savardi, M., Bálint, K., Kovács, A.B., Signoroni, A.: On the influence of shot scale on film mood and narrative engagement in film viewers. IEEE Trans. Affect. Comput. 1 (2019). https://doi.org/10.1109/TAFFC.2019.2939251

  5. Benini, S., Svanera, M., Adami, N., Leonardi, R., Kovács, A.B.: Shot scale distribution in art films. Multimed. Tools Appl. 75(23), 16499–16527 (2016). https://doi.org/10.1007/s11042-016-3339-9

    Article  Google Scholar 

  6. Bhattacharya, S., Mehran, R., Sukthankar, R., Shah, M.: Classification of cinematographic shots using lie algebra and its application to complex event recognition. IEEE Trans. Multimed. 16(3), 686–696 (2014). https://doi.org/10.1109/TMM.2014.2300833

    Article  Google Scholar 

  7. Cherif, I., Solachidis, V., Pitas, I.: Shot type identification of movie content. In: 2007 9th International Symposium on Signal Processing and Its Applications, pp. 1–4 (2007)

    Google Scholar 

  8. Chollet, F., et al.: Keras (2015). https://keras.io

  9. Cores Sarría, L.: The influence of camera angle in film narratives. Ph.D. thesis (2015). https://doi.org/10.13140/RG.2.2.23950.69444

  10. Cutting, J.E., Armstrong, K.L.: Facial expression, size, and clutter: inferences from movie structure to emotion judgments and back. Atten. Percept. Psychophys. 78(3), 891–901 (2015). https://doi.org/10.3758/s13414-015-1003-5

    Article  Google Scholar 

  11. Devries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. CoRR abs/1708.04552 (2017). http://arxiv.org/abs/1708.04552

  12. Duan, L.Y., Xu, M., Tian, Q., Xu, C.S., Jin, J.: A unified framework for semantic shot classification in sports video. IEEE Trans. Multimedia 7, 1066–1083 (2006). https://doi.org/10.1109/TMM.2005.858395

    Article  Google Scholar 

  13. Ekin, A., Tekalp, A.M.: Robust dominant color region detection with applications to sports video analysis (2003)

    Google Scholar 

  14. Gugger, S.: The 1cycle policy. https://sgugger.github.io/the-1cycle-policy.html

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  16. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks (2018)

    Google Scholar 

  17. Huang, W., Olson, J.S., Olson, G.M.: Camera angle affects dominance in video-mediated communication. In: CHI 2002 Extended Abstracts on Human Factors in Computing Systems, pp. 716–717. Association for Computing Machinery, NY (2002). https://doi.org/10.1145/506443.506562

  18. Jain, N., Gupta, V., Shubham, S., Madan, A., Chaudhary, A., Santosh, K.C.: Understanding cartoon emotion using integrated deep neural network on large dataset. Neural Comput. Appl. 1–21 (2021). https://doi.org/10.1007/s00521-021-06003-9

  19. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980

  20. Lin, J.C., et al.: Coherent deep-net fusion to classify shots in concert videos. IEEE Trans. Multimed. 20(11), 3123–3136 (2018). https://doi.org/10.1109/TMM.2018.2820904

    Article  Google Scholar 

  21. Marín-Reyes, P.A., Lorenzo-Navarro, J., Santana, M.C., Sánchez-Nielsen, E.: Shot classification and keyframe detection for vision based speakers diarization in parliamentary debates. In: CAEPIA (2016)

    Google Scholar 

  22. Minhas, R.A., Javed, A., Irtaza, A., Mahmood, M.T., Joo, Y.B.: Shot classification of field sports videos using AlexNet convolutional neural network. Appl. Sci. 9(3), 483 (2019). https://doi.org/10.3390/app9030483

    Article  Google Scholar 

  23. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  24. Sätteli, H.P.: The effect of different vertical camera-angles on faceperception (2010)

    Google Scholar 

  25. Savardi, M., Signoroni, A., Migliorati, P., Benini, S.: Shot scale analysis in movies by convolutional neural networks. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 2620–2624 (2018). https://doi.org/10.1109/ICIP.2018.8451474

  26. Selvaraju, R.R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., Batra, D.: Grad-CAM: why did you say that? Visual explanations from deep networks via gradient-based localization. CoRR abs/1610.02391 (2016). http://arxiv.org/abs/1610.02391

  27. Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017, Santa Rosa, CA, 24–31 March 2017, pp. 464–472. IEEE Computer Society (2017). https://doi.org/10.1109/WACV.2017.58

  28. Smith, L.N.: A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. CoRR abs/1803.09820 (2018). http://arxiv.org/abs/1803.09820

  29. Svanera, M., Savardi, M., Signoroni, A., Kovács, A.B., Benini, S.: Who is the film’s director? Authorship recognition based on shot features. IEEE Multimed. 26(4), 43–54 (2019). https://doi.org/10.1109/MMUL.2019.2940004

    Article  Google Scholar 

  30. Tarvainen, J., Laaksonen, J., Takala, T.: Film mood and its quantitative determinants in different types of scenes. IEEE Trans. Affect. Comput. 11(2), 313–326 (2020). https://doi.org/10.1109/TAFFC.2018.2791529

    Article  Google Scholar 

  31. Tsingalis, I., Vretos, N., Nikolaidis, N., Pitas, I.: SVM-based shot type classification of movie content. In: Proceedings of 9th Mediterranean Electro Technical Conference. Istanbul, pp. 104–107 (2012)

    Google Scholar 

  32. Wang, H.L., Cheong, L.F.: Taxonomy of directing semantics for film shot classification. IEEE Trans. Circuits Syst. Video Technol. 19(10), 1529–1542 (2009). https://doi.org/10.1109/TCSVT.2009.2022705

    Article  Google Scholar 

  33. Xie, L., Chang, F., Divakaran, A., Sun, H.: Structure analysis of soccer video with hidden Markov models. In: 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (2001). https://doi.org/10.1109/ICASSP.2002.5745558

  34. Zheng, Y., et al.: Cartoon face recognition: a benchmark dataset, pp. 2264–2272. Association for Computing Machinery, NY (2020). https://doi.org/10.1145/3394171.3413726

  35. Zhou, Y.H., Cao, Y.D., Zhang, L.F., Zhang, H.X.: An SVM-based soccer video shot classification. In: 2005 International Conference on Machine Learning and Cybernetics, vol. 9, pp. 5398–5403 (2005). https://doi.org/10.1109/ICMLC.2005.1527898

Download references

Author information

Authors and Affiliations

  1. Department of Information Engineering, University of Brescia, Brescia, Italy

    Gianluca Gualandris, Mattia Savardi & Sergio Benini

  2. Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Brescia, Italy

    Alberto Signoroni

Authors
  1. Gianluca Gualandris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mattia Savardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alberto Signoroni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sergio Benini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sergio Benini .

Editor information

Editors and Affiliations

  1. National Research Council, Lecce, Italy

    Pier Luigi Mazzeo

  2. Università Politecnica delle Marche, Ancona, Italy

    Emanuele Frontoni

  3. Boston University, Boston, MA, USA

    Stan Sclaroff

  4. National Research Council, Lecce, Italy

    Cosimo Distante

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gualandris, G., Savardi, M., Signoroni, A., Benini, S. (2022). Automatic Indexing of Virtual Camera Features from Japanese Anime. In: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (eds) Image Analysis and Processing. ICIAP 2022 Workshops. ICIAP 2022. Lecture Notes in Computer Science, vol 13373. Springer, Cham. https://doi.org/10.1007/978-3-031-13321-3_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-13321-3_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-13320-6

  • Online ISBN: 978-3-031-13321-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation