Skip to main content
SpringerLink
Log in

Deep learning for content-based video retrieval in film and television production

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

While digitization has changed the workflow of professional media production, the content-based labeling of image sequences and video footage, necessary for all subsequent stages of film and television production, archival or marketing is typically still performed manually and thus quite time-consuming. In this paper, we present deep learning approaches to support professional media production. In particular, novel algorithms for visual concept detection, similarity search, face detection, face recognition and face clustering are combined in a multimedia tool for effective video inspection and retrieval. The analysis algorithms for concept detection and similarity search are combined in a multi-task learning approach to share network weights, saving almost half of the computation time. Furthermore, a new visual concept lexicon tailored to fast video retrieval for media production and novel visualization components are introduced. Experimental results show the quality of the proposed approaches. For example, concept detection achieves a mean average precision of approximately 90% on the top-100 video shots, and face recognition clearly outperforms the baseline on the public Movie Trailers Face Dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://taglichtmedia.de

  2. http://trecvid.nist.gov

References

  1. Blanco G, Bedo MVN, Cazzolato MT, Santos LFD, Jorge AES, Traina C, Azevedo-Marques PM, Traina AJM (2016) A label-scaled similarity measure for content-based image retrieval. In: 2016 IEEE international symposium on multimedia (ISM), pp 20–25

  2. Breuel TM, Ul-Hasan A, Al-Azawi MA, Shafait F (2013) High-performance OCR for printed english and fraktur using LSTM networks. In: proceedings of international conference on document analysis and recognition, pp 683–687

  3. Chatfield K, Simonyan K, Vedaldi A, Zisserman A (2014) Return of the devil in the details: delving deep into convolutional nets. In: Proceedings of the British machine vision conference, pp 1–11

  4. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) ImageNet: A large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition (CVPR ’09), pp 2–9

  5. Ding C, Tao D (2016) Trunk-branch ensemble convolutional neural networks for video-based face recognition. arXiv:1607.05427

  6. Ewerth R, Freisleben B (2004) Video cut detection without thresholds. In: Proceedings of the 11th international workshop on signals, systems and image processing (IWSSIP ’04). Poznan, Poland, pp 227–230

  7. Ewerth R, Freisleben B (2009) Unsupervised detection of gradual video shot changes with motion-based false alarm removal. In: International conference on advanced concepts for intelligent vision systems, pp 253–264

  8. Ewerth R, Mühling M, Freisleben B (2007) Self-supervised learning of face appearances in tv casts and movies. Int J Semant Comput 1(2):185–204

    Article  Google Scholar 

  9. Farfade SS, Saberian MJ, Li LJ (2015) Multi-view face detection using deep convolutional neural networks. In: Proceedings of the 5th ACM on international conference on multimedia retrieval. ACM, pp 643–650

  10. Gong Y, Jia Y, Leung T, Toshev A, Ioffe S (2013) Deep convolutional ranking for multilabel image annotation. arXiv:1312.4894

  11. Graves A, Mohamed A, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: IEEE international conference on acoustics, speech and signal processing (ICASSP 2013), pp 6645–6649

  12. Guo Y, Zhang L, Hu Y, He X, Gao J (2016) Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In: European conference on computer vision. Springer, pp 87–102

  13. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. arXiv:1512.03385

  14. Hudelist MA, Cobârzan C, Beecks C, Van de Werken R, Kletz S, Hürst W, Schoeffmann K (2016) Collaborative video search combining video retrieval with human-based visual inspection. In: 22nd international conference on multimedia modelling. Springer International Publishing, FL, USA, pp 400–405

  15. Jain V, Learned-Miller E (2010) Fddb: A benchmark for face detection in unconstrained settings. UMass Amherst Technical Report

  16. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of ACM international conference on multimedia, pp 675–678

  17. Jiang H, Learned-Miller E (2016) Face detection with the faster r-cnn. arXiv:1606.03473

  18. Klare BF, Klein B, Taborsky E, Blanton A, Cheney J, Allen K, Grother P, Mah A, Burge M, Jain AK (2015) Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1931–1939

  19. Krizhevsky A, Hinton G (2011) Using very deep autoencoders for content-based image retrieval. In: Proceedings of European symposium on artificial neural networks, pp 1–7

  20. Krizhevsky A, Sutskever I, Hinton G (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1–9

  21. Kumar N, Berg AC, Belhumeur PN, Nayar SK (2009) Attribute and simile classifiers for face verification. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 365–372

  22. Kumar V, Namboodiri AM, Jawahar C (2014) Face recognition in videos by label propagation. In: 22nd international conference on pattern recognition (ICPR). IEEE, pp 303–308

  23. Learned-Miller GBHE (2014) Labeled faces in the wild: updates and new reporting procedures. Technical Report UM-CS-2014-003. University of Massachusetts, Amherst

    Google Scholar 

  24. Lin K, Yang HF, Hsiao JH, Chen CS (2015) Deep learning of binary hash codes for fast image retrieval Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 27–35

    Google Scholar 

  25. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: Single shot multibox detector. In: European conference on computer vision. Springer, pp 21–37

  26. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the 7th IEEE international conference on computer vision, vol 2, pp 1150–1157

  27. Masi I, Tran AT, Leksut JT, Hassner T, Medioni G (2016) Do we really need to collect millions of faces for effective face recognition? arXiv:1603.07057

  28. Meddeb M, Karray H, Alimi AM (2016) Content-based arabic speech similarity search and emotion detection. In: Hassanien AE, Shaalan K, Gaber T, Azar AT, Tolba MF (eds) Proceedings of the international conference on advanced intelligent systems and informatics. Springer International Publishing, pp 530–539

  29. Nesterov Y (1983) A method for unconstrained convex minimization problem with the rate of convergence o (1/k2). In: Doklady an SSSR, vol 269, pp 543–547

  30. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  31. Ortiz EG, Wright A, Shah M (2013) Face recognition in movie trailers via mean sequence sparse representation-based classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3531–3538

  32. Otto C, Wang D, Jain AK (2016) Clustering millions of faces by identity. arXiv:1604.00989

  33. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: British machine vision conference, pp 1–6

  34. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  35. Salakhutdinov R, Hinton G (2009) Semantic hashing. Int J Approx Reason 50(7):969–978

    Article  Google Scholar 

  36. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

  37. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  38. Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  39. Song M (2008) Handbook of research on text and web mining technologies. IGI Global

  40. Sun Y, Liang D, Wang X, Tang X (2015) Deepid3: Face recognition with very deep neural networks. arXiv:1502.00873

  41. Sutskever I, Martens J, Dahl GE, Hinton G (2013) On the importance of initialization and momentum in deep learning 30th international conference on machine learning, vol 28, pp 1139–1147

  42. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9

  43. Taigman Y, Yang M, Ranzato M, Wolf L (2014) DeepFace: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1–8

  44. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, vol 1, pp 511–518

  45. Wan J, Wang D, Hoi SCH, Wu P (2014) Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of the ACM international conference on multimedia (MM), pp 157–166

  46. Wang J, Shi L, Wang H, Meng J, Wang JJ, Sun Q, Gu Y (2016) Optimizing top precision performance measure of content-based image retrieval by learning similarity function. In: 23nd international conference on pattern recognition (ICPR)

  47. Wei Y, Xia W, Huang J, Ni B, Dong J, Zhao Y, Yan S (2014) CNN: Single-label to Multi-label. pp 1–14

  48. Yang S, Luo P, Loy CC, Tang X (2016) Wider face: A face detection benchmark. In: IEEE conference on computer vision and pattern recognition (CVPR)

  49. Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv:1411.7923

  50. Yianilos PN (1993) Data structures and algorithms for nearest neighbor search in general metric spaces. In: Annual ACM-SIAM symposium on discrete algorithms, pp 311–321

  51. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, pp 818–833

  52. Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene recognition using places database. Adv Neural Inf Process Syst 27:487–495

    Google Scholar 

Download references

Acknowledgements

This work is financially supported by the German Federal Ministry for Economic Affairs and Energy (BMWi) in the ZIM Programme.

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, University of Marburg, Hans-Meerwein-Str. 6, D-35032, Marburg, Germany

    Markus Mühling, Nikolaus Korfhage & Bernd Freisleben

  2. taglicht media Film- & Fernsehproduktion GmbH, Cäsarstraße 58, 50968, Köln, Germany

    Thomas Langelage & Uli Veith

  3. German National Library of Science and Technology (TIB), Welfengarten 1B, D-30167, Hannover, Germany

    Eric Müller, Christian Otto, Matthias Springstein & Ralph Ewerth

  4. L3S Research Center, Leibniz Universität Hannover, Appelstr. 4, D-30167, Hannover, Germany

    Eric Müller, Christian Otto & Ralph Ewerth

Authors
  1. Markus Mühling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikolaus Korfhage
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christian Otto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Matthias Springstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Thomas Langelage
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Uli Veith
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ralph Ewerth
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Bernd Freisleben
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Markus Mühling.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mühling, M., Korfhage, N., Müller, E. et al. Deep learning for content-based video retrieval in film and television production. Multimed Tools Appl 76, 22169–22194 (2017). https://doi.org/10.1007/s11042-017-4962-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4962-9

Keywords

Search

Navigation