Abstract
With the development of mobile Internet, a large amount of short video data is generated on the Internet. The urgent problem of short video classification is how to better fuse the information of different multimodal information. This paper proposes a short video multimodal fusion (SV-MF) scheme based on deep learning combined with pre-trained models to complete the classification task of short video. The main innovations of the SV-MF scheme are as follows: (1) We find that text modalities contain higher-order information and tend to perform better than audio and visual modalities, and with the use of pre-trained language models, text modalities have been further improved in multimodal video classification. (2) Due to the strong semantic representation ability of text. The SVMF scheme proposes a local fusion method based on Transformer for low-order visual and audio modal information to alleviate the information deviation caused by multi-mode fusion. (3) The SV-MF scheme proposes a post processing strategy based on keywords to further improve the classification accuracy of the model. Experimental results based on a multimodal short video classification dataset derived from social networks show that the performance of the SV-MF scheme is better than the previous video fusion scheme.
Yan, H. and Cao, X. contributed equally to this paper and should be considered as co-first authors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Shutsko, A.: User-generated short video content in social media. a case study of TikTok. International Conference on Human-Computer Interaction. Springer, pp. 108–125 (2020)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25 (2012)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems, 1 (2014)
Soomro K, Zamir A R, Shah M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild[J]. Computer Science, 2012
Kuehne, H., Jhuang, H., Stiefelhagen, R., et al.: HMDB: A Large Video Database for Human Motion Recognition. Springer, Berlin Heidelberg (2013)
Long, X., Gan, C., De Melo, G., et al.: Attention clusters: Purely attention based local feature integration for video classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7834–7843 (2018)
Li, L.H., Yatskar, M., Yin, D., et al.: Visualbert: A Simple and Performant Baseline for Vision and Language. arXiv preprint arXiv:1908.03557 (2019)
Devlin, J., Chang, M.W., Lee, K., et al.: Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018)
Xie, S., Girshick, R., Dollár, P., et al.: Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)
Hershey, S., Chaudhuri, S., Ellis, D.P.W., et al.: CNN architectures for large-scale audio classification. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 131–135 (2017)
Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. Advances in Neural Information Processing Systems 30 (2017)
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. JMLR Workshop and Conference Proceedings, pp. 249–256 (2010)
He, K., Zhang, X., Ren, S., et al.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Cao, X. et al. (2024). Design of a Multimodal Short Video Classification Model. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1963. Springer, Singapore. https://doi.org/10.1007/978-981-99-8138-0_7
Download citation
DOI: https://doi.org/10.1007/978-981-99-8138-0_7
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8137-3
Online ISBN: 978-981-99-8138-0
eBook Packages: Computer ScienceComputer Science (R0)