Skip to main content
SpringerLink
Log in

Scalable storyboards in handheld devices: applications and evaluation metrics

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Summaries are an essential component of video retrieval and browsing systems. Most research in video summarization has focused on content analysis to obtain compact yet comprehensive representations of video items. However, important aspects such as how they can be effectively integrated in mobile interfaces and how to predict the quality and usability of the summaries have not been investigated. Conventional summaries are limited to a single instance with certain length (i.e. a single scale). In contrast, scalable summaries target representations with multiple scales, that is, a set of summaries with increasing length in which longer summaries include more information about the video. Thus, scalability provides high flexibility that can be exploited in devices such as smartphones or tablets to provide versions of the summary adapted to the limited visualization area. In this paper, we explore the application of scalable storyboards to summary adaptation and zoomable video navigation in handheld devices. By introducing a new adaptation dimension related with the summarization scale, we can formulate navigation and adaptation in a two-dimensional adaptation space, where different navigation actions modify the trajectory in that space. We also describe the challenges to evaluate scalable summaries and some usability issues that arise from having multiple scales, proposing some objective metrics that can provide useful insight about their potential quality and usability without requiring very costly user studies. Experimental results show a reasonable agreement with the trends shown in subjective evaluations. Experiments also show that content-based scalable storyboards are less redundant and useful than the content-blind baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Adami N, Signoroni A, Leonardi R (2007) State-of-the-art and trends in scalable video compression with wavelet-based approaches. IEEE Trans Circ Systx Video Technol 17(9):1238–1255

    Article  Google Scholar 

  2. Ahmad I, Wei X, Sun Y, Zhang YQ (2005) Video transcoding: an overview of various techniques and research issues. IEEE Trans Multimed 7(5):793–804

    Article  Google Scholar 

  3. Albanese M, Fayzullin M, Picariello A, Subrahmanian V (2006) The priority curve algorithm for video summarization. Inf Syst 31(7):679–695

    Article  Google Scholar 

  4. Benini S, Bianchetti A, Leonardi R, Migliorati P (2006) Extraction of significant video summaries by dendrogram analysis. In: Proceedings IEEE International Conference on Image Processing, pp 133–136

  5. Benini S, Migliorati P, Leonardi R (2010) Statistical skimming of feature films. Int J Digit Multimed Broadcast 2010:11

    Google Scholar 

  6. Bescos J, Martinez JM, Herranz L, Tiburzi F (2007) Content-driven adaptation of on-line video. Signal Process: Image Commun 22:651–668

    Google Scholar 

  7. Chang SF, Vetro A (2005) Video adaptation: concepts, technologies, and open issues. IEEE Proc 93(1):148–158

    Article  Google Scholar 

  8. Cong Y., Yuan J., Luo J. (2012) Towards scalable summarization of consumer videos via sparse dictionary selection. IEEE Trans Multimed 14(1):66–75

    Article  Google Scholar 

  9. Dong P, Xia Y, Wang S, Zhuo L, Feng D (2014) An iteratively reweighting algorithm for dynamic video summarization Multimedia Tools and Applications, pp 1–25. doi:10.1007/s11042-014-2126-8

  10. Dumont E, Merialdo B (2010) Rushes video summarization and evaluation. Multimed Tools Appl 48:51–68

    Article  Google Scholar 

  11. Friedland G, Gottlieb L, Janin A (2013) Narrative theme navigation for sitcoms supported by fan-generated scripts. Multimed Tools Appl 63(2):387–406. doi:10.1007/s11042-011-0877-z

    Article  Google Scholar 

  12. Gong Y, Liu X (2000) Video summarization using singular value decomposition. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition, vol 2, pp 174–180

  13. Haesen M, Meskens J, Luyten K, Coninx K, Becker J, Tuytelaars T, Poulisse GJ, Pham P, Moens MF (2013) Finding a needle in a haystack: an interactive video archive explorer for professional video searchers. Multimed Tools Appl 63(2):331–356

    Article  Google Scholar 

  14. Hanjalic A, Zhang H (1999) An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis. IEEE Trans Circ Syst Video Technol 9(8):1280–1289

    Article  Google Scholar 

  15. Herranz L, Calic J, Martínez JM, Mrak M (2012) Scalable comic-like video summaries and layout disturbance. IEEE Trans Multimed 14(4):1290–1297

    Article  Google Scholar 

  16. Herranz L, Martinez JM (2008) Generation of scalable summaries based on iterative GOP ranking. In: Proceedings IEEE International Conference on Image Processing, pp 2544–2547

  17. Herranz L, Martínez JM (2009) An integrated approach to summarization and adaptation using H.264/MPEG-4 SVC. Signal Process: Image Commun 24(6):499–509

    Google Scholar 

  18. Herranz L, Martínez JM (2010) A framework for scalable summarization of video. IEEE Trans Circ Syst Video Technol 20(9):1265–1270

    Article  Google Scholar 

  19. Hürst W, Darzentas D (2012) History: a hierarchical storyboard interface design for video browsing on mobile devices. In: Proceedings International Conference on Mobile and Ubiquitous Multimedia, pp 17:1–17:4

  20. Irie G, Satou T, Kojima A, Yamasaki T, Aizawa K (2010) Automatic trailer generation. In: Proceedings of the International Conference on Multimedia, MM ’10. ACM, NY, USA, pp 839–842

    Book  Google Scholar 

  21. Li Y, Merialdo B (2010) Vert: automatic evaluation of video summaries. In: Proceedings of the international conference on Multimedia, pp 851–854

  22. Likert R (1932) A technique for the measurement of attitudes. Arch Psychol 22 (140):1–55

    Google Scholar 

  23. Liu H, Liu Y, Sun F (2014) Video key-frame extraction for smart phones. Multimed Tools Appl:1–19. doi:10.1007/s11042-014-2390-7

  24. Marchionini G, Wildemuth BM, Geisler G (2006) The Open Video digital library: A M?bius strip of research and practice. J Am Soc Inf Sci Technol 57(12):1629–1643

    Article  Google Scholar 

  25. Mohanta PP, Saha SK, Chanda B (2009) Generation of size constrained video storyboard using spanning tree. In: Proceedings of the First International Conference on Internet Multimedia Computing and Service, pp 179–182

  26. Money AG, Agius H (2008) Video summarisation: A conceptual framework and survey of the state of the art. J Vis Commun Image Represent 19(2):121–143

    Article  Google Scholar 

  27. Mukherjee D, Said A, Liu S (2005) A framework for fully format-independent adaptation of scalable bit streams. IEEE Trans Circ Syst Video Technol 15(10):1280–1290

    Article  Google Scholar 

  28. Mundur P, Rao Y, Yesha Y (2006) Keyframe-based video summarization using delaunay clustering. Int J Digit Libr 6(2):219–232

    Article  Google Scholar 

  29. Ohm JR (2005) Advances in scalable video coding. IEEE Proc 93(1):42–56

    Article  Google Scholar 

  30. Over P, Smeaton AF, Awad G (2008) The TRECVid 2008 BBC rushes summarization evaluation. In: Proceedings 2nd ACM TRECVid Video Summarization Workshop. ACM, pp 1–20

  31. Over P, Smeaton AF, Kelly P The TRECVid 2007 bbc rushes summarization evaluation pilot. In: TRECVid

  32. Santini S (2007) Who needs video summarization anyway? In: Proceedings International Conference Semantic Computing ICSC 2007, pp 177–184

  33. Scherp A, Mezaris V (2014) Survey on modeling and indexing events in multimedia. Multimed Tools Appl 70(1):7–23. doi:10.1007/s11042-013-1427-7

    Article  Google Scholar 

  34. Schoeffmann K (2014) A user-centric media retrieval competition: The video browser showdown 2012-2014. IEEE MultiMedia 21(4):8–13. doi:10.1109/MMUL.2014.56

    Article  Google Scholar 

  35. Schoeffmann K, Ahlstrom D, Hudelist M (2014) 3-d interfaces to improve the performance of visual known-item search. Multimed, IEEE Trans 16(7):1942–1951. doi:10.1109/TMM.2014.2333666

    Article  Google Scholar 

  36. Schwarz H, Marpe D, Wiegand T (2007) Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans Circ Syst Video Technol 17(9):1103–1120

    Article  Google Scholar 

  37. Sibson R (1973) Slink: an optimally efficient algorithm for the single-link cluster method. Comput J 16(1):30–34

    Article  MathSciNet  Google Scholar 

  38. Smeaton AF, Over P, Kraaij W (2006) Evaluation campaigns and TRECVid. In: Proceedings of the ACM international workshop on Multimedia information retrieval. ACM, pp 321–330

  39. Smeulders A, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  40. Truong BT, Venkatesh S (2007) Video abstraction: A systematic review and classification. ACM Trans Multimed Comput Commun Appl 3(1):3

    Article  Google Scholar 

  41. Valdes V, Martinez JM (2012) Automatic evaluation of video summaries. ACM Trans Multimed Comput Commun Appl 8(3):25:1–25:21

    Article  Google Scholar 

  42. Vetro A (2004) MPEG-21 digital item adaptation: Enabling universal multimedia access. IEEE Multimed 11(1):84–87

    Article  Google Scholar 

  43. Xin J, Lin CW, Sun MT (2005) Digital video transcoding. IEEE Proc 93(1):84–97

    Article  Google Scholar 

  44. Yang Y, Ma Z, Xu Z, Yan S, Hauptmann AG (2013) How related exemplars help complex event detection in web videos? In: IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1–8, pp 2104–2111

  45. Yuan Z, Lu T, Wu D, Huang Y, Yu H (2011) Video summarization with semantic concept preservation. In: Proceedings International Conference on Mobile and Ubiquitous Multimedia. ACM, pp 109–112

  46. Zhang L, Gao Y, Hong C, Feng Y, Zhu J, Cai D (2014) Feature correlation hypergraph: Exploiting high-order potentials for multimodal recognition. Cybern, IEEE Trans 44(8):1408–1419. doi:10.1109/TCYB.2013.2285219

    Article  Google Scholar 

  47. Zhao S, Yao H, Sun X (2013) Video classification and recommendation based on affective analysis of viewers. Neurocomputing 119:101–110

    Article  Google Scholar 

  48. Zhao S, Yao H, Sun X, Jiang X, Xu P (2013) Flexible presentation of videos based on affective content analysis. In: Li S, El Saddik A, Wang M, Mei T, Sebe N, Yan S, Hong R, Gurrin C (eds) Proceedings International Conference on Multimedia Modeling, Lecture Notes in Computer Science, vol 7732, pp 368–379

  49. Zhu X, Fan J, Elmagarmid AK, Wu X (2003) Hierarchical video content description and summarization using unified semantic and visual similarity. Multimed Syst 9(1):31–53

    Article  Google Scholar 

  50. Zhu XQ, Wu XD, Fan JP, Elmagarmid AK, Aref WF (2004) Exploring video content structure for hierarchical summarization. Multimed Syst 10(2):98–115

    Article  Google Scholar 

  51. Zhuang Y, Rui Y, Huang T, Mehrotra S (1998) Adaptive key frame extraction using unsupervised clustering. In: Proceedings of International Conference on Image Processing, pp 866–870

Download references

Acknowledgements

This work was supported in part by the National Basic Research Program of China (973 Program): 2012CB316400, in part by the National Natural Science Foundation of China: 61322212 and 61350110237, in part by the National Hi-Tech Development Program (863 Program) of China: 2014AA015202, and in part by the Chinese Academy of Sciences Fellowships for Young International Scientists: 2011Y1GB05. This work was also funded by Lenovo Outstanding Young Scientists Program (LOYS).

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China

    Luis Herranz & Shuqiang Jiang

Authors
  1. Luis Herranz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuqiang Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luis Herranz.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Herranz, L., Jiang, S. Scalable storyboards in handheld devices: applications and evaluation metrics. Multimed Tools Appl 75, 12597–12625 (2016). https://doi.org/10.1007/s11042-014-2421-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2421-4

Keywords

Search

Navigation