Skip to main content
SpringerLink
Log in

OPFSumm: on the video summarization using Optimum-Path Forest

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Video summarization attempts at encoding a given video into a compact representation for a better storage and retrieval purposes. This work copes with the problem of static video summarization using the unsupervised Optimum-Path Forest (OPF). We sampled the encoded video sequence into frames and extracted features based on color information or spectral properties. After that, meaningless frames are removed, and OPF models the problem of video summarization as a clustering process. Possible redundant keyframes are filtered, and at last the video summary is created based on non-redundant keyframes. We presented a more in-depth study that also considers temporal information to obtain better video representations. The experiments over three public datasets were analyzed through F-measure evaluation metric and showed the robustness of OPF for automatic video summarization: 0.19 for SumMe dataset, 0.728 concerning Open Video dataset, and 0.451 regarding YouTube dataset..

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. https://github.com/jppbsi/OPFSumm/wiki

  2. http://www.ffmpeg.org/ (As of February 2017)

  3. We applied this procedure on each subset.

  4. The threshold was set up empirically.

  5. http://www.open-video.org/ (As of February 2017)

  6. http://www.youtube.com/ (As of February 2017)

  7. https://people.ee.ethz.ch/gyglim/vsum/ (As of February 2017)

  8. We considered two color descriptors only since [16] observed that GCH and CCV achieved better performances compared to the five other descriptors.

References

  1. Almeida J, Leite NJ, Torres RS (2012) VISON: VIdeo Summarization for ONline applications. Pattern Recogn Lett 33(4):397–409

    Article  Google Scholar 

  2. Almeida J, Leite NJ, Torres RS (2013) Online video summarization on compressed domain. J Vis Commun Image Represent 24(6):729–738

    Article  Google Scholar 

  3. Amorim WP, Falcão AX, Papa JP, Carvalho MH (2016) Improving semi-supervised learning through optimum connectivity. Pattern Recogn 60:72–85

    Article  Google Scholar 

  4. Avila SEF, Lopes APB, Luz A Jr, Araújo AA (2011) VSUMM: a mechanism designed to produce static video summaries and a novel evaluation method. Pattern Recogn Lett 32(1):56–68

    Article  Google Scholar 

  5. Castelo-Fernández C, Calderón-Ruiz G (2015) Progress in pattern recognition, image analysis, computer vision, and applications. In: Pardo A, Kittler J (eds) 20th Iberoamerican congress on pattern recognition, lecture notes in computer science, vol 9423, chap. Automatic video summarization using the optimum-path forest unsupervised classifier. Springer International Publishing, New York, pp 760–767

  6. Choi YS, Kim KJ (2004) In: Laganá A, Gavrilova ML, Kumar V, Mun YS, Tan CJK, Gervasi O (eds) International conference on computational science and its applications. Lecture notes in computer science, vol 3043, chap. Video summarization using fuzzy one-class support vector machine. Springer, Berlin, pp 49–56

  7. DeMenthon D, Kobla V, Doermann DS (1998) Video summarization by curve simplification. In: ACM international conference multimedia (MM’08), pp 211–218

  8. Furini M, Geraci F, Montangero M, Pellegrini M (2010) STIMO: STIll and MOving video storyboard for the web scenario. Multimed Tools Appl 46(1):47–69

    Article  Google Scholar 

  9. Gong B, Chao WL, Grauman K, Sha F (2014) Diverse sequential subset selection for supervised video summarization. In: Proceedings of the 27th international conference on neural information processing systems, NIPS’14. MIT Press, Cambridge, pp 2069–2077. http://dl.acm.org/citation.cfm?id=2969033.2969058

  10. Gygli M, Grabner H, Riemenschneider H, Van Gool L (2014) Creating summaries from user videos. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) 13th European conference on computer vision. Springer International Publishing, pp 505–520

  11. Gygli M, Grabner H, Van Gool L (2015) Video summarization by learning submodular mixtures of objectives. In: CVPR

  12. Huang J, Kumar R, Mitra M, Zhu WJ, Zabih R (1997) Image indexing using color correlograms. In: IEEE international conference computer vision and pattern recognition (CVPR’97), pp 762–768

  13. Jacob H, Pádua FLC, Lacerda A, Pereira ACM (2017) A video summarization approach based on the emulation of bottom-up mechanisms of visual attention. J Intell Inf Syst 1–19. https://doi.org/10.1007/s10844-016-0441-4

  14. Jacobs CE, Finkelstein A, Salesin D (1995) Fast multiresolution image querying. In: International conference on computer graphics and interactive techniques (SIGGRAPH’95), pp 277–286

  15. Mahasseni B, Lam M, Todorovic S (2017) Unsupervised video summarization with adversarial lstm networks. In: The IEEE conference on computer vision and pattern recognition (CVPR). IEEE

  16. Martins GB, Afonso LCS, Osaku D, Almeida JG, Papa JP (2014) Progress in pattern recognition, image analysis, computer vision, and applications. In: Bayro-Corrochano E, Hancock E (eds) Lecture Notes in Computer Science, vol. 8827, chap. Static Video Summarization through Optimum-Path Forest Clustering, pp 893–900. Springer International Publishing. 19th Iberoamerican congress on pattern recognition

  17. Martins GB, Papa JP, Almeida J (2016) Temporal-and spatial-driven video summarization using optimum-path forest. In: 29th SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), pp 335–339

  18. Minetto R, Spina TV, Falcão AX, Leite NJ, Papa JP, Stolfi J (2012) IFTrace: video segmentation of deformable objects using the image foresting transform. Comput Vis Image Underst 116(2):274–291

    Article  Google Scholar 

  19. Money AG, Agius HW (2008) Video summarization: a conceptual framework and survey of the state of the art. J Vis Commun Image Represent 19(2):121–143

    Article  Google Scholar 

  20. Mundur P, Rao Y, Yesha Y (2006) Keyframe-based video summarization using Delaunay clustering. Int J Digit Libr 6(2):219–232

    Article  Google Scholar 

  21. Panda R, Kuanar SK, Chowdhury AS (2014) Scalable video summarization using skeleton graph and random walk. In: Proceedings of the 2014 22nd international conference on pattern recognition, ICPR ’14. IEEE Computer Society, Washington, DC, pp 3481–3486

  22. Papa JP, Falcão AX (2008) A new variant of the optimum-path forest classifier. In: Bebis G, Boyle R, Parvin B, Koracin D, Remagnino P, Porikli F, Peters J, Klosowski J, Arns L, Chun Y, Rhyne TM, Monroe L (eds) Advances in visual computing, lecture notes in computer science, vol 5358. Springer, Berlin, pp 935–944

  23. Papa JP, Falcão AX (2009) A learning algorithm for the optimum-path forest classifier. In: Torsello A, Escolano F, Brun L (eds) Graph-based representations in pattern recognition, lecture notes in computer science, vol 5534. Springer, Berlin, pp 195–204

  24. Papa JP, Falcão AX, Suzuki CTN (2009) Supervised pattern classification based on optimum-path forest. Int J Imaging Syst Technol 19(2):120–131

    Article  Google Scholar 

  25. Papa JP, Falcão AX, Albuquerque VHC, Tavares JMRS (2012) Efficient supervised optimum-path forest classification for large datasets. Pattern Recogn 45 (1):512–520

    Article  Google Scholar 

  26. Papa JP, Scheirer W, Cox DD (2016) Fine-tuning deep belief networks using harmony search. Appl Soft Comput 46:875–885

    Article  Google Scholar 

  27. Papa JP, Fernandes SEN, Falcão AXs (2017) Optimum-path forest based on k-connectivity: theory and applications. Pattern Recogn Lett 87:117–126

    Article  Google Scholar 

  28. Papadopoulos DP, Chatzichristofis AA, Papamarkos N (2011) 5th international conference on computer vision/computer graphics collaboration techniques. chap. Video summarization using a self-growing and self-organized neural gas network. Springer, Berlin, pp 216–226

  29. Pass G, Zabih R, Miller J (1996) Comparing images using color coherence vectors. In: ACM international conference on multimedia, pp 65–73

  30. Pisani RJ, Nakamura RYM, Riedel PS, Zimback CRL, Falcão AX, Papa JP (2014) Toward satellite-based land cover classification through optimum-path forest. IEEE Trans Geosci Remote Sens 52(10):6075–6085

    Article  Google Scholar 

  31. Rocha LM, Cappabianco FAM, Falcão AX (2009) Data clustering as an optimum-path forest problem with applications in image analysis. Int J Imaging Syst Technol 19(2):50–68

    Article  Google Scholar 

  32. Rosa GH, Costa KAP, Passos LA Jr, Papa JP, Falcão AX, Tavares JMRS (2014) On the training of artificial neural networks with radial basis function using optimum-path forest clustering. In: 22nd international conference on pattern recognition, pp 1472–1477

  33. Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888–905

    Article  Google Scholar 

  34. Souza GB, Santos DFS, Pires RG, Marana AN, Papa JP (2017) Deep boltzmann machines for robust fingerprint spoofing attack detection. In: Proceedings of the international joint conference on neural networks, pp 1863–1870

  35. Stehling RO, Nascimento MA, Falcão AX (2002) A compact and efficient image retrieval approach based on border/interior pixel classification. In: ACM international conference on information and knowledge management (CIKM’02), pp 102–109

  36. Sun K, Zhu J, Lei Z, Hou X, Zhang Q, Duan J, Qiu G (2017) Learning deep semantic attributes for user video summarization. In: The IEEE international conference on multimedia and expo (ICME). IEEE

  37. Swain MJ, Ballard BH (1991) Color indexing. Int J Comput Vis 7(1):11–32

    Article  Google Scholar 

  38. Zhang D, Lu G (2002) Shape-based image retrieval using generic fourier descriptor. Signal Process Image Commun 17(10):825–848

    Article  Google Scholar 

  39. Zhang L, Xia Y, Mao K, Ma H, Shan Z (2015) An effective video summarization framework toward handheld devices. IEEE Trans Ind Electron 62 (2):1309–1316

    Article  Google Scholar 

  40. Zhang K, Chao WL, Sha F, Grauman K (2016) Summary transfer: exemplar-based subset selection for video summarization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1059–1067

  41. Zhang K, Chao WL, Sha F, Grauman K (2016) Video summarization with long short-term memory. Springer International Publishing, Berlin, pp 766–782

    Google Scholar 

Download references

Acknowledgments

The authors acknowledge “Coordination for the Improvement of Higher Education Personnel”, “São Paulo Research Foundation” grants 2013/07375-0, 2014/16250-9, 2014/12236-1, 2016/06441-7, and 2016/19403-6, and “National Council for Scientific and Technological Development” grants 306166/2014-3, 423228/2016-1, 304315/2017-6, and 307066/2017-7).

Author information

Authors and Affiliations

  1. Department of Computing, São Paulo State University, Bauru, Brazil

    Guilherme B. Martins, Danillo R. Pereira & João Paulo Papa

  2. Institute of Science and Technology, Federal University of São Paulo, São Paulo, Brazil

    Jurandy G. Almeida

  3. Graduate Program in Applied Informatics, University of Fortaleza, Fortaleza, CE, Brazil

    Victor Hugo C. de Albuquerque

Authors
  1. Guilherme B. Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danillo R. Pereira
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jurandy G. Almeida
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Victor Hugo C. de Albuquerque
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. João Paulo Papa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Victor Hugo C. de Albuquerque.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Martins, G.B., Pereira, D.R., Almeida, J.G. et al. OPFSumm: on the video summarization using Optimum-Path Forest. Multimed Tools Appl 79, 11195–11211 (2020). https://doi.org/10.1007/s11042-018-5874-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5874-z

Keywords

Search

Navigation