Skip to main content
SpringerLink
Log in

Automated Image and Video Quality Assessment for Computational Video Editing

  • Conference paper
  • First Online:
Analysis of Images, Social Networks and Texts (AIST 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12602))

Abstract

We study non-reference image and video quality assessment methods, which are of great importance for computational video editing. The object of our work is image quality assessment (IQA) applicable for fast and robust frame-by-frame multipurpose video quality assessment (VQA) for short videos.

We present a complex framework for assessing the quality of images and videos. The scoring process consists of several parallel steps of metric collection with final score aggregation step. Most of the individual scoring models are based on deep convolutional neural networks (CNN). The framework can be flexibly extended or reduced by adding or removing these steps. Using Deep CNN-Based Blind Image Quality Predictor (DIQA) as a baseline for IQA, we proposed improvements based on two patching strategies, such as uniform patching and object-based patching, and add intelligent pre-training step with distortion classification.

We evaluated our model on three IQA benchmark image datasets (LIVE, TID2008, and TID2013) and manually collected short YouTube videos. We also consider interesting for automated video editing metrics used for video scoring based on the scale of a scene, face presence in frame and compliance of the shot transitions with the shooting rules. The results of this work are applicable to the development of intelligent video and image processing systems.

Ilya Makarov—This research is partially based on the work supported by Samsung Research, Samsung Electronics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amel, A.M., Abdessalem, B.A., Abdellatif, M.: Video shot boundary detection using motion activity descriptor. arXiv preprint arXiv:1004.4605 (April 2010). http://arxiv.org/abs/1004.4605

  2. Bosse, S., Maniry, D., Wiegand, T., Samek, W.: A deep neural network for image quality assessment. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 3773–3777. IEEE (September 2016). https://doi.org/10.1109/ICIP.2016.7533065, http://ieeexplore.ieee.org/document/7533065/

  3. Brown, B.: Cinematography: Theory and Practice: Image Making for Cinematographers and Directors. Taylor & Francis (2016). https://books.google.ru/books?id=GiQlDwAAQBAJ

  4. Cherif, I., Solachidis, V., Pitas, I.: Shot type identification of movie content. In: 2007 9th International Symposium on Signal Processing and Its Applications, pp. 1–4. IEEE (February 2007). https://doi.org/10.1109/ISSPA.2007.4555491, http://ieeexplore.ieee.org/document/4555491/

  5. Ekin, A., Tekalp, A.: Robust Dominant Color Region Detection with Applications to Sports Video Analysis. In: Proceedings 2003 International Conference on Image Processing (Cat. No. 03CH37429) (2003)

    Google Scholar 

  6. Ferman, A., Tekalp, A.: Two-stage hierarchical video summary extraction to match low-level user browsing preferences. IEEE Trans. Multimedia 5(2), 244–256 (2003). https://doi.org/10.1109/TMM.2003.811617, http://ieeexplore.ieee.org/document/1208494/

  7. Hassanien, A., Elgharib, M., Selim, A., Bae, S.H., Hefeeda, M., Matusik, W.: Large-scale, Fast and Accurate Shot Boundary Detection through Spatio-temporal Convolutional Neural Networks. arXiv preprint arXiv:1705.03281 (May 2017). http://arxiv.org/abs/1705.03281

  8. Kang, L., Ye, P., Li, Y., Doermann, D.: Convolutional Neural Networks for No-Reference Image Quality Assessment. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

    Google Scholar 

  9. Kim, J., Nguyen, A., Lee, S.: Deep CNN-based blind image quality predictor. IEEE Trans. Neural Netw. Learn. Syst. 30(1), 11–24 (2019). https://doi.org/10.1109/TNNLS.2018.2829819

    Article  Google Scholar 

  10. Li, Y., et al.: No-reference image quality assessment with shearlet transform and deep neural networks. Neurocomputing 154, 94–109 (2015). https://doi.org/10.1016/j.neucom.2014.12.015, https://linkinghub.elsevier.com/retrieve/pii/S0925231214016798

  11. Liu, L., et al.: Deep learning for generic object detection: a survey. Int. J. Comput. Vis. 128(2), 261–318 (2020). https://doi.org/10.1007/s11263-019-01247-4, http://link.springer.com/10.1007/s11263-019-01247-4

  12. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695–4708 (2012). https://doi.org/10.1109/TIP.2012.2214050

    Article  MathSciNet  MATH  Google Scholar 

  13. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “Completely Blind” image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2013). https://doi.org/10.1109/LSP.2012.2227726

    Article  Google Scholar 

  14. Moorthy, A.K., Bovik, A.C.: Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Trans. Image Process. 20(12), 3350–3364 (2011)

    Article  MathSciNet  Google Scholar 

  15. Pass, G., Zabih, R., Miller, J.: Comparing images using color coherence vectors. In: Proceedings of the fourth ACM international conference on Multimedia - MULTIMEDIA 1996. pp. 65–73. ACM Press, New York, USA (1996). https://doi.org/10.1145/244130.244148, http://portal.acm.org/citation.cfm?doid=244130.244148

  16. Ponomarenko, N., et al.: Image database TID2013: Peculiarities, results and perspectives. Signal Process. Image Commun. 30, 57–77 (2015). https://doi.org/10.1016/j.image.2014.10.009, https://linkinghub.elsevier.com/retrieve/pii/S0923596514001490

  17. Ponomarenko, N., Lukin, V., Zelensky, A., Egiazarian, K., Carli, M., Battisti, F.: TID2008-a database for evaluation of full-reference visual quality assessment metrics. Adv. Mod. Radioelectronics 10(4), 30–45 (2009)

    Google Scholar 

  18. Somani, R.: AI For Filmmaking: Recognising Shot Types with ResNets (2019). https://rsomani95.github.io/ai-film-1.html

  19. Saad, M.A., Bovik, A.C., Charrier, C.: Blind image quality assessment: a natural scene statistics approach in the DCT domain. IEEE Trans. Image Process. 21(8), 3339–3352 (2012). https://doi.org/10.1109/TIP.2012.2191563

    Article  MathSciNet  MATH  Google Scholar 

  20. Sheikh, H.R., Wang, Z., Cormack, L., Bovik, A.C.: LIVE image quality assessment database release 2 (2005). http://live.ece.texas.edu/research/quality (2005)

  21. Souček, T., Moravec, J., Lokoč, J.: TransNet: a deep network for fast detection of common shot transitions. arXiv preprint arXiv:1906.03363 (June 2019). http://arxiv.org/abs/1906.03363

  22. Tabii, Y., Djibril, M.O., Hadi, Y., Thami, R.O.H.: A new method for video soccer shot classification. In: VISAPP (1), pp. 221–224 (2007)

    Google Scholar 

  23. Zhang, H., Kankanhalli, A., Smoliar, S.W.: Automatic partitioning of full-motion video. Multimedia Syst. 1(1), 10–28 (1993). https://doi.org/10.1007/BF01210504, http://link.springer.com/10.1007/BF01210504

  24. Zhang, P., Zhou, W., Wu, L., Li, H.: SOM: semantic obviousness metric for image quality assessment. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. HSE University, Moscow, Russia

    Konstantin Lomotin & Ilya Makarov

  2. Samsung-PDMI Joint AI Center, St. Petersburg Department of Steklov Institute of Mathematics, St. Petersburg, Russia

    Ilya Makarov

Authors
  1. Konstantin Lomotin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ilya Makarov
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. RWTH Aachen University, Aachen, Germany

    Wil M. P. van der Aalst

  2. University of Ljubljana, Ljubljana, Slovenia

    Vladimir Batagelj

  3. National Research University Higher School of Economics, Moscow, Russia

    Dmitry I. Ignatov

  4. Krasovskii Institute of Mathematics and Mechanics, Yekaterinburg, Russia

    Michael Khachay

  5. National Research University Higher School of Economics, St. Petersburg, Russia

    Olessia Koltsova

  6. University of Oslo, Oslo, Norway

    Andrey Kutuzov

  7. National Research University Higher School of Economics, Moscow, Russia

    Sergei O. Kuznetsov

  8. National Research University Higher School of Economics, Moscow, Russia

    Irina A. Lomazova

  9. Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  10. LORIA, Vandœuvre lès Nancy, France

    Amedeo Napoli

  11. Skolkovo Institute of Science and Technology, Moscow, Russia

    Alexander Panchenko

  12. University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

  13. Università Ca' Foscari Venezia, Venice, Italy

    Marcello Pelillo

  14. National Research University Higher School of Economics, Nizhny Novgorod, Russia

    Andrey V. Savchenko

  15. Kazan Federal University, Kazan, Russia

    Elena Tutubalina

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lomotin, K., Makarov, I. (2021). Automated Image and Video Quality Assessment for Computational Video Editing. In: van der Aalst, W.M.P., et al. Analysis of Images, Social Networks and Texts. AIST 2020. Lecture Notes in Computer Science(), vol 12602. Springer, Cham. https://doi.org/10.1007/978-3-030-72610-2_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72610-2_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72609-6

  • Online ISBN: 978-3-030-72610-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation