research-article

Automatic evaluation of video summaries

Authors:

Jose M. MartinezAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 8, Issue 3

Article No.: 25, Pages 1 - 21

https://doi.org/10.1145/2240136.2240138

Published: 06 August 2012 Publication History

Abstract

This article describes a method for the automatic evaluation of video summaries based on the training of individual predictors for different quality measures from the TRECVid 2008 BBC Rushes Summarization Task. The obtained results demonstrate that, with a large set of evaluation data, it is possible to train fully automatic evaluation systems based on visual features automatically extracted from the summaries. The proposed approach will enable faster and easier estimation of the results of newly developed abstraction algorithms and the study of which summary characteristics influence their perceived quality.

References

[1]

Bai, L., Lao, S., Smeaton, A. F., and O'Connor, N. E. 2008. Automatic summarization of rushes video using bipartite graphs. In Proceedings of the 3rd International Conference on Semantic and Digital Media Technologies (SAMT '08). Springer-Verlag, 3--14.

Digital Library

[2]

Bailer, W. and Thallinger, G. 2008. Comparison of content selection methods for skimming rushes video. In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (TVS '08). ACM, 85--89.

Digital Library

[3]

Bredin, H., Byrne, D., Lee, H., O'Connor, N. E., and Jones, G. J. 2008. Dublin city university at the trecvid 2008 BBC rushes summarisation task. In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (TVS '08). ACM, 45--49.

Digital Library

[4]

Breiman, L., Friedman, J., Stone, C. J., and Olshen, R. A. 1984. Classification and Regression Trees. Chapman and Hall.

[5]

Christel, M. G., Hauptmann, A. G., Lin, W.-H., Chen, M.-Y., Yang, J., Maher, B., and Baron, R. V. 2008. Exploring the utility of fast-forward surrogates for BBC rushes. In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (TVS '08). ACM, 35--39.

Digital Library

[6]

Christel, M. G., Smith, M. A., Taylor, C. R., and Winkler, D. B. 1998. Evolving video skims into useful multimedia abstractions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98). ACM, 171--178.

Digital Library

[7]

Ding, W., Marchionini, G., and Tse, T. 1997. Previewing video data: Browsing key frames at high rates using a video slide show. In Proceedings of the International Symposium on Research, Development and Practice in Digital Libraries (ISDL '97). 151--158.

[8]

Dumont, E. and Merialdo, B. 2007. Split-screen dynamically accelerated video summaries. In Proceedings of the International Workshop on TRECVID Video Summarization (TVS '07). ACM, 55--59.

Digital Library

[9]

Dumont, E. and Merialdo, B. 2008a. Automatic evaluation method for rushes summarization: experimentation and analysis. In Proceedings of the 6th International Workshop on Content-Based Multimedia Indexing (CBMI '08). 518--525.

[10]

Dumont, E. and Merialdo, B. 2008b. Sequence alignment for redundancy removal in video rushes summarization. In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (TVS '08). ACM, 55--59.

Digital Library

[11]

Dumont, E. and Merialdo, B. 2010. Rushes video summarization and evaluation. Multimed. Tools Appl. 48, 1, 51--68.

Digital Library

[12]

Ekin, A., Tekalp, A., and Mehrotra, R. 2003. Automatic soccer video analysis and summarization. IEEE Trans. Image Process. 12, 7, 796--807.

Digital Library

[13]

Ferman, A. and Tekalp, A. 2003. Two-stage hierarchical video summary extraction to match low-level user browsing preferences. IEEE Trans. Multimed. 5, 2, 244--256.

Digital Library

[14]

Hauptmann, A. G., Christel, M. G., Lin, W.-H., Maher, B., Yang, J., Baron, R. V., and Xiang, G. 2007. Clever clustering vs. simple speed-up for summarizing rushes. In Proceedings of the International Workshop on TRECVID Video Summarization (TVS '07). ACM, 20--24.

Digital Library

[15]

Kasutani, E. and Yamada, A. 2001. The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval. In Proceedings of the International Conference on Image Processing (ICIP '01). 674--677.

[16]

Over, P., Smeaton, A. F., and Awad, G. 2008. The trecvid 2008 BBC rushes summarization evaluation. In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (TVS '08). ACM, 1--20.

Digital Library

[17]

Over, P., Smeaton, A. F., and Kelly, P. 2007. The trecvid 2007 bbc rushes summarization evaluation pilot. In Proceedings of the International Workshop on TRECVID Video Summarization (TVS '07). ACM, 1--15.

Digital Library

[18]

Ren, T., Liu, Y., and Wu, G. 2008. Full-reference quality assessment for video summary. In Proceedings of the IEEE International Conference on Data Mining Workshops. IEEE Computer Society, Los Alamitos, CA, 874--883.

Digital Library

[19]

Taskiran, C., Pizlo, Z., Amir, A., Ponceleon, D., and Delp, E. 2006. Automated video program summarization using speech transcripts. IEEE Trans. Multimed. 8, 4, 775--791.

Digital Library

[20]

Valdes, V. and Martinez, J. M. 2008. Binary tree based on-line video summarization. In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (TVS '08). ACM, 134--138.

Digital Library

[21]

Yamasaki, K., Shinoda, K., and Furui, S. 2008. Automatically estimating number of scenes for rushes summarization. In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (TVS '08). ACM, 129--133.

Digital Library

Cited By

Sen DVivekraj V(2024)Multi-Reference Evaluation of Dynamic Video Summaries Using Granule-Aware F-MeasureIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33698558:4(3040-3054)Online publication date: Aug-2024
https://doi.org/10.1109/TETCI.2024.3369855
Liu JYu DTang Z(2021)Video summary generation by visual shielding compressed sensing coding and double-layer affinity propagationJournal of Visual Communication and Image Representation10.1016/j.jvcir.2021.103321(103321)Online publication date: Sep-2021
https://doi.org/10.1016/j.jvcir.2021.103321
Herranz LJiang S(2016)Scalable storyboards in handheld devicesMultimedia Tools and Applications10.1007/s11042-014-2421-475:20(12597-12625)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1007/s11042-014-2421-4
Show More Cited By

Index Terms

Automatic evaluation of video summaries
1. Information systems
  1. Information retrieval
    1. Document representation
2. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Abstraction

Recommendations

Discrepancy between automatic and manual evaluation of summaries
NAACL HLT '12: Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization

Today, automatic evaluation metrics such as ROUGE have become the de-facto mode of evaluating an automatic summarization system. However, based on the DUC and the TAC evaluation results, (Conroy and Schlesinger, 2008; Dang and Owczarzak, 2008) showed ...
Automatic Summarization Evaluation: Methods and Practices
Testing Software and Systems
Abstract
Automatic summarization systems extract relevant information from documents to provide concise summaries that maintain the same information. Traditionally a human task, automatic summarization has been a challenge since the 1950s and continues to ...
Supervised automatic evaluation for summarization with voted regression model

The high quality evaluation of generated summaries is needed if we are to improve automatic summarization systems. Although human evaluation provides better results than automatic evaluation methods, its cost is huge and it is difficult to reproduce the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 8, Issue 3

July 2012

143 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/2240136

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 August 2012

Accepted: 01 April 2011

Revised: 01 August 2010

Received: 01 April 2010

Published in TOMM Volume 8, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Spanish Government

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
386
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sen DVivekraj V(2024)Multi-Reference Evaluation of Dynamic Video Summaries Using Granule-Aware F-MeasureIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33698558:4(3040-3054)Online publication date: Aug-2024
https://doi.org/10.1109/TETCI.2024.3369855
Liu JYu DTang Z(2021)Video summary generation by visual shielding compressed sensing coding and double-layer affinity propagationJournal of Visual Communication and Image Representation10.1016/j.jvcir.2021.103321(103321)Online publication date: Sep-2021
https://doi.org/10.1016/j.jvcir.2021.103321
Herranz LJiang S(2016)Scalable storyboards in handheld devicesMultimedia Tools and Applications10.1007/s11042-014-2421-475:20(12597-12625)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1007/s11042-014-2421-4
Shun-Hsing Ou Chia-Han Lee Somayazulu VYen-Kuang Chen Shao-Yi Chien (2015)On-Line Multi-View Video Summarization for Wireless Video Sensor NetworkIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2014.23319169:1(165-179)Online publication date: Feb-2015
https://doi.org/10.1109/JSTSP.2014.2331916
Gong BChao WGrauman KSha F(2014)Diverse sequential subset selection for supervised video summarizationProceedings of the 28th International Conference on Neural Information Processing Systems - Volume 210.5555/2969033.2969058(2069-2077)Online publication date: 8-Dec-2014
https://dl.acm.org/doi/10.5555/2969033.2969058
Khosla AHamid RLin CSundaresan N(2013)Large-Scale Video Summarization Using Web-Image PriorsProceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition10.1109/CVPR.2013.348(2698-2705)Online publication date: 23-Jun-2013
https://dl.acm.org/doi/10.1109/CVPR.2013.348

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents