Abstract
Automated analysis of sports video summarization is challenging due to variations in cameras, replay speed, illumination conditions, editing effects, game structure, genre, etc. To address these challenges, we propose an effective video summarization framework based on shot classification and replay detection for field sports videos. Accurate shot classification is mandatory to better structure the input video for further processing, i.e., key events or replay detection. Therefore, we present a lightweight convolutional neural network based method for shot classification. Then we analyze each shot for replay detection and specifically detect the successive batch of logo transition frames that identify the replay segments from the sports videos. For this purpose, we propose local octa-pattern features to represent video frames and train the extreme learning machine for classification as replay or non-replay frames. The proposed framework is robust to variations in cameras, replay speed, shot speed, illumination conditions, game structure, sports genre, broadcasters, logo designs and placement, frame transitions, and editing effects. The performance of our framework is evaluated on a dataset containing diverse YouTube sports videos of soccer, baseball, and cricket. Experimental results demonstrate that the proposed framework can reliably be used for shot classification and replay detection to summarize field sports videos.
摘要
由于摄像机、回放速度、光照条件、剪辑效果、比赛结构和类型等方面的差异, 体育视频摘要的自动分析具有挑战性。为了解决这些问题, 针对户外运动视频, 本文提出一种基于镜头分类和回放检测的有效视频摘要框架。准确的镜头分类对于更好地安排输入视频从而进行进一步处理 (关键事件或回放检测) 是必要的。因此, 提出一种基于轻量级卷积神经网络的镜头分类方法。该方法对每一个镜头进行回放检测; 特别地, 检测出从体育视频中识别出回放片段的连续标识转换帧。为此, 提出局部八元模式特征来表示视频帧, 并训练极限学习机分为回放或非回放两类。所提框架对于摄像机、回放速度、镜头速度、光照条件、比赛结构、运动类型、广播公司、标识设计和位置、帧转换和剪辑效果具有鲁棒性。基于YouTube体育视频集中的足球、棒球和板球运动对所提框架进行性能评估。实验结果证明所提框架能够可靠地用于户外运动视频摘要的镜头分类和回放检测。
Similar content being viewed by others
References
Bagheri-Khaligh A, Raziperchikolaei R, Moghaddam ME, 2012. A new method for shot classification in soccer sports video based on SVM classifier. Proc IEEE Southwest Symp on Image Analysis and Interpretation, p.109–112. https://doi.org/10.1109/SSIAI.2012.6202465
Chen CM, Chen LH, 2014. Novel framework for sports video analysis: a basketball case study. Proc Int Conf on Image Processing, p.961–965. https://doi.org/10.1109/ICIP.2014.7025193
Chen CM, Chen LH, 2015. A novel method for slow motion replay detection in broadcast basketball video. Multimed Tools Appl, 74(21):9573–9593. https://doi.org/10.1007/s11042-014-2137-5
Choros K, Gogol A, 2016. Improved method of detecting replay logo in sports videos based on contrast feature and histogram difference. Proc 8th Int Conf on Computational Collective Intelligence, p.542–552. https://doi.org/10.1007/978-3-319-45243-2_50
Dang ZH, Du J, Huang QM, et al., 2007. Replay detection based on semi-automatic logo template sequence extraction in sports video. Proc 4th Int Conf on Image and Graphics, p.839–844. https://doi.org/10.1109/ICIG.2007.73
Duan LY, Xu M, Tian Q, et al., 2004. Mean shift-based video segment representation and applications to replay detection. Proc 29th IEEE Int Conf on Acoustics, Speech, and Signal Processing, p.709–712. https://doi.org/10.1109/ICASSP.2004.1327209
Ekin A, Tekalp AM, Mehrotra R, 2003. Automatic soccer video analysis and summarization. IEEE Trans Image Process, 12(7):796–807. https://doi.org/10.1109/TIP.2003.812758
Eldib MY, Zaid BSA, Zawbaa HM, et al., 2009. Soccer video summarization using enhanced logo detection. Proc 16th IEEE Int Conf on Image Processing, p.4345–4348. https://doi.org/10.1109/ICIP.2009.5413649
Fani M, Yazdi M, Clausi DA, et al., 2017. Soccer video structure analysis by parallel feature fusion network and hidden-to-observable transferring Markov model. IEEE Access, 5:27322–27336. https://doi.org/10.1109/ACCESS.2017.2769140
Javed A, Bajwa KB, Malik H, et al., 2016. An efficient framework for automatic highlights generation from sports videos. IEEE Signal Process Lett, 23(7):954–958. https://doi.org/10.1109/LSP.2016.2573042
Javed A, Irtaza A, Khaliq Y, et al., 2019. Replay and keyevents detection for sports video summarization using confined elliptical local ternary patterns and extreme learning machine. Appl Intell, 49(8):2899–2917. https://doi.org/10.1007/s10489-019-01410-x
Javed A, Malik KM, Irtaza A, et al., 2020. A decision tree framework for shot classification of field sports videos. J Supercomput, 76(9):7242–7267. https://doi.org/10.1007/s11227-020-03155-8
Jiang H, Zhang M, 2011. Tennis video shot classification based on support vector machine. Proc IEEE Int Conf on Computer Science and Automation Engineering, p.757–761. https://doi.org/10.1109/CSAE.2011.5952612
Kapela R, McGuinness K, O’Connor NE, 2017. Real-time field sports scene classification using colour and frequency space decompositions. J Real-Time Image Process, 13(4):725–737. https://doi.org/10.1007/s11554-014-0437-7
Li W, Chen SJ, Wang HB, 2009. A rule-based sports video event detection method. Proc Int Conf on Computational Intelligence and Software Engineering, p.1–4. https://doi.org/10.1109/CISE.2009.5366226
Minhas RA, Javed A, Irtaza A, et al., 2019. Shot classification of field sports videos using AlexNet convolutional neural network. Appl Sci, 9(3):483. https://doi.org/10.3390/app9030483
Murala S, Maheshwari RP, Balasubramanian R, 2012. Local tetra patterns: a new feature descriptor for content-based image retrieval. IEEE Trans Image Process, 21(5):2874–2886. https://doi.org/10.1109/TIP.2012.2188809
Nguyen N, Yoshitaka A, 2012. Shot type and replay detection for soccer video parsing. Proc IEEE Int Symp on Multimedia, p.344–347. https://doi.org/10.1109/ISM.2012.69
Pan H, Van Beek P, Sezan MI, 2001. Detection of slow-motion replay segments in sports video for highlights generation. Proc IEEE Int Conf on Acoustics, Speech, and Signal Processing, p.1649–1652. https://doi.org/10.1109/ICASSP.2001.941253
Pan H, Li BX, Sezan MI, 2002. Automatic detection of replay segments in broadcast sports programs by detection of logos in scene transitions. Proc IEEE Int Conf on Acoustics, Speech, and Signal Processing, p.IV-3385–IV-3388. https://doi.org/10.1109/ICASSP.2002.5745380
Raventós A, Quijada R, Torres L, et al., 2015. Automatic summarization of soccer highlights using audio-visual descriptors. SpringerPlus, 4(1):301. https://doi.org/10.1186/s40064-015-1065-9
Su PC, Lan CH, Wu CS, et al., 2013. Transition effect detection for extracting highlights in baseball videos. EURASIP J Image Video Process, 2013(1):27. https://doi.org/10.1186/1687-5281-2013-27
Tavassolipour M, Karimian M, Kasaei S, 2014. Event detection and summarization in soccer videos using Bayesian network and copula. IEEE Trans Circ Syst Video Technol, 24(2):291–304. https://doi.org/10.1109/TCSVT.2013.2243640
Tien MC, Chen HT, Chen YW, et al., 2007. Shot classification of basketball videos and its application in shooting position extraction. Proc IEEE Int Conf on Acoustics, Speech and Signal Processing, p.1085–1088. https://doi.org/10.1109/ICASSP.2007.366100
Wang DH, Tian Q, Gao S, et al., 2004. News sports video shot classification with sports play field and motion features. Proc IEEE Conf on Image Processing, p.2247–2250. https://doi.org/10.1109/ICIP.2004.1421545
Wang JJ, Chang E, Xu CS, 2005. Soccer replay detection using scene transition structure analysis. Proc IEEE Int Conf on Acoustics, Speech, and Signal Processing, p.433–436. https://doi.org/10.1109/ICASSP.2005.1415434
Wang L, Liu X, Lin S, et al., 2004. Generic slow-motion replay detection in sports video. Proc Int Conf on Image Processing, p.1585–1588. https://doi.org/10.1109/ICIP.2004.1421370
Wu X, He R, Sun ZN, et al., 2018. A light CNN for deep face representation with noisy labels. IEEE Trans Inform Forens Secur, 13(11):2884–2896. https://doi.org/10.1109/TIFS.2018.2833032
Xu W, Yi Y, 2011. A robust replay detection algorithm for soccer video. IEEE Signal Process Lett, 18(9):509–512. https://doi.org/10.1109/LSP.2011.2161287
Zhao F, Dong Y, Wei Z, et al., 2012. Matching logos for slow motion replay detection in broadcast sports video. Proc IEEE Int Conf on Acoustics, Speech and Signal Processing, p.1409–1412. https://doi.org/10.1109/ICASSP.2012.6288154
Zhao Z, Jiang SQ, Huang QM, et al., 2006. Highlight summarization in sports video based on replay detection. Proc Int Conf on Multimedia and Expo, p.1613–1616. https://doi.org/10.1109/ICME.2006.262855
Author information
Authors and Affiliations
Contributions
Ali JAVED developed the proposed method and designed the research. Amen ALI KHAN collected and processed the dataset. Ali JAVED and Amen ALI KHAN wrote the code, performed the experimentation, and drafted the paper. Ali JAVED revised and finalized the paper.
Corresponding author
Additional information
Compliance with ethics guidelines
Ali JAVED and Amen ALI KHAN declare that they have no conflict of interest.
Project supported by the Directorate of Advanced Studies, Research & Technological Development, University of Engineering and Technology Taxila (No. UET/ASRTD/RG-1002-3)
Rights and permissions
About this article
Cite this article
Javed, A., Ali Khan, A. Shot classification and replay detection for sports video summarization. Front Inform Technol Electron Eng 23, 790–800 (2022). https://doi.org/10.1631/FITEE.2000414
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2000414
Key words
- Extreme learning machine
- Lightweight convolutional neural network
- Local octa-patterns
- Shot classification
- Replay detection
- Video summarization