Abstract
Recent years have witnessed the explosive growth of video and clips with the popularization of mobile devices that are deeply shaping people’s life. Video summarization is an important approach to highlight and label the videos. Since the requirement of video summarization in practice is diverse due to the consumer majority and videos themselves, it is difficult for a neural network that extracts specific features to be adapted according to the changing requirements. In this paper, we propose a novel method of video summarization that employs the characteristics of the bullet-screen comments as a reference and train a novel neural network to generate a short clip as the video summary. First, we propose a method to select candidate clips for video summarization based on the distribution of the bullet-screen comments and these candidates are annotated manually to construct our labelled dataset from the Bilibili video library. Second, we propose and train a novel network combined with the pointer network and the General Adversarial Networks, named by diffPointer-GAN, to directly identify the start and end of the summary clip, other than the previous process composed of scoring, segmentation and selecting. Finally, we demonstrate that the proposed method has a definite priority to two baseline methods that are implemented on our dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aradhye, H., Toderici, G., Yagnik, J.: Video2text: learning to annotate video content. In: 2009 IEEE International Conference on Data Mining Workshops, pp. 144–151. IEEE (2009)
Calic, J., Thomas, B.: Spatial analysis in key-frame extraction using video segmentation. In: Workshop on Image Analysis for Multimedia Interactive Services (2004)
De Avila, S.E.F., Lopes, A.P.B., da Luz, A., Jr., de Albuquerque Araújo, A.: Vsumm a mechanism designed to produce static video summaries and a novel evaluation method. Pattern Recogn. Lett. 32(1), 56–68 (2011)
Gygli, M., Grabner, H., Riemenschneider, H., Van Gool, L.: Creating summaries from user videos. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 505–520. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_33
Gygli, M., Grabner, H., Van Gool, L.: Video summarization by learning submodular mixtures of objectives. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3090–3098 (2015)
Kawashima, T., Tateyama, K., Iijima, T., Aoki, Y.: Indexing of baseball telecast for content-based video retrieval. In: Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No. 98CB36269), vol. 1, pp. 871–874. IEEE (1998)
Kuramoto, M., Masaki, T., Kitamura, Y., Kishino, F.: Video database retrieval based on gestures and its application. IEEE Trans. Multimedia 4(4), 500–508 (2002)
Li, G., Ma, S., Han, Y.: Summarization-based video caption via deep neural networks. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 1191–1194 (2015)
Mahasseni, B., Lam, M., Todorovic, S.: Unsupervised video summarization with adversarial LSTM networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 202–211 (2017)
Mei, S., Guan, G., Wang, Z., Wan, S., He, M., Feng, D.D.: Video summarization via minimum sparse reconstruction. Pattern Recogn. 48(2), 522–533 (2015)
See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368 (2017)
Song, W., Yongguo, H., Yadong, W., Zhang, S.: Video key frame extraction method based on image dominant color (in Chinese). Comput. Appl. 33(09), 2631–2635 (2013)
Song, Y., Vallmitjana, J., Stent, A., Jaimes, A.: Tvsum: summarizing web videos using titles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5179–5187 (2015)
Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Yu, L., Zhang, W., Wang, J., SeqGAN, Y.Y.: Sequence generative adversarial nets with policy gradient. arxiv e-prints, arXiv preprint arXiv:1609.05473 (2016)
Zhang, K., Chao, W.-L., Sha, F., Grauman, K.: Video summarization with long short-term memory. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 766–782. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_47
Zhuang, Y., Rui, Y., Huang, T.S., Mehrotra, S.: Adaptive key frame extraction using unsupervised clustering. In: Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No. 98cb36269), vol. 1, pp. 866–870. IEEE (1998)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ke, F., Li, P., Lu, W. (2021). Video Summarization by DiffPointer-GAN. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Communications in Computer and Information Science, vol 1517. Springer, Cham. https://doi.org/10.1007/978-3-030-92310-5_70
Download citation
DOI: https://doi.org/10.1007/978-3-030-92310-5_70
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92309-9
Online ISBN: 978-3-030-92310-5
eBook Packages: Computer ScienceComputer Science (R0)