Abstract
Extracting sports highlights and summarizing the most exciting moments of the match is an important task for the broadcast media. However, it requires intensive video editing. We propose a Multimodal Analysis Approach (MAA) for auto-curating sports highlights, and use it to create a real-world system for editing aids of soccer highlight reels. MAA fuses information from the players actions (action recognition), the landmarks (image classification), the scores on the scoreboard (OCR) and the commentators tone of the voice (audio detection) to determine the most exciting moments of a match. In addition, we use the face recognition technology to make star highlights. The shot-boundary detection method is developed to accurately identify the start and end frames of highlights for content summaries. The proposed system has performed real-time highlights extraction from the video stream of the FIFA world cup 2018. Moreover, MAA produces highlights with better quality by comprehensive user studies from multiple participants subjects.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)
Chen, S.C., Shyu, M.L., Chen, M., Zhang, C.: A decision tree-based multimodal data mining framework for soccer goal detection. In: 2004 IEEE International Conference on Multimedia and Expo, ICME 2004, Vol. 1, pp. 265–268. IEEE, June 2004
Xie, Z., Shyu, M.L., Chen, S.C.: Video event detection with combined distance-based and rule-based data mining techniques. In: Multimedia and Expo, pp. 2026–2029. IEEE, July 2007
Leonardi, R., Migliorati, P., Prandini, M.: Semantic indexing of soccer audio-visual sequences: a multimodal approach based on controlled Markov chains. IEEE Trans. Circuits Syst. Video Technol. 14(5), 634–643 (2004)
Huang, C.L., Shih, H.C., Chao, C.Y.: Semantic analysis of soccer video using dynamic Bayesian network. IEEE Trans. Multimed. 8(4), 749–760 (2006)
Yang, Y., Lin, S., Zhang, Y., Tang, S.: Highlights extraction in soccer videos based on goal-mouth detection. In: 9th International Symposium on Signal Processing and Its Applications, ISSPA 2007, pp. 1–4. IEEE, February 2007
Kang, Y.L., Lim, J.H., Kankanhalli, M.S., Xu, C.S., Tian, Q.:. Goal detection in soccer video using audio/visual keywords. In: 2004 International Conference on Image Processing, ICIP 2004, vol. 3, pp. 1629–1632. IEEE, October 2004
Zawbaa, H.M., El-Bendary, N., Hassanien, A.E., Kim, T.H.: Event detection based approach for soccer video summarization using machine learning. Int. J. Multimed. Ubiquitous Eng. 7(2), 63–80 (2012)
Li, B., Pan, H., Sezan, I.: A general framework for sports video summarization with its application to soccer. In: Proceedings 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, (ICASSP 2003), vol. 3, pp. III–169. IEEE, April 2003
Tjondronegoro, D.W., Chen, Y.P.P.: Knowledge-discounted event detection in sports video. IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 40(5), 1009–1024 (2010)
Liu, H.: Highlight extraction in soccer videos by using multimodal analysis. In: 2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 2169–2173. IEEE, July 2017
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, pp. 568–576 (2014)
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.:Temporal segment networks: towards good practices for deep action recognition. In: European Conference on Computer Vision, pp. 20–36. Springer, Cham, October 2016
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4489–4497 (2015)
Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3D residual networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5534–5542. IEEE, October 2017
Carreira, J., Zisserman, A.: Quo Vadis, action recognition? A new model and the kinetics dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4733. IEEE, July 2017
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., Suleyman, M.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. arXiv preprint arXiv:1801.04381 (2018)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Smith, R.: An overview of the Tesseract OCR engine. In: Ninth International Conference on Document Analysis and Recognition, ICDAR 2007, vol. 2, pp. 629–633. IEEE, September 2007
Son, W., Cho, H.T., Yoon, K., Lee, S.P.: Sub-fingerprint masking for a robust audio fingerprinting system in a real-noise environment for portable consumer devices. IEEE Trans. Consum. Electron. 56(1), 156–160 (2010)
Malekesmaeili, M., Ward, R.K.: A local fingerprinting approach for audio copy detection. Sig. Process. 98, 308–321 (2014)
Smith, J.O.: Spectral audio signal processing, vol. 1334027739. W3K (2011)
Deng, J., Guo, J., Xue, N., Zafeiriou, S.: ArcFace: additive angular margin loss for deep face recognition. arXiv preprint arXiv:1801.07698 (2018)
Boreczky, J.S., Rowe, L.A.: Comparison of video shot boundary detection techniques. J. Electron. Imaging 5(2), 122–129 (1996)
Merler, M., Joshi, D., Nguyen, Q.B., Hammer, S., Kent, J., Smith, J.R., Feris, R.S.: Automatic curation of golf highlights using multimodal excitement features. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 57–65. IEEE, July 2017
Acknowledgment
This work was supported by the National Natural Science Foundation of China (No. 61871046).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yuan, W., Zhang, Y., Hu, X., Song, M. (2020). Automatic Curation System Using Multimodal Analysis Approach (MAA). In: Bi, Y., Bhatia, R., Kapoor, S. (eds) Intelligent Systems and Applications. IntelliSys 2019. Advances in Intelligent Systems and Computing, vol 1038. Springer, Cham. https://doi.org/10.1007/978-3-030-29513-4_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-29513-4_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29512-7
Online ISBN: 978-3-030-29513-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)