Abstract
Video classification is a complex task that involves analyzing audio and video signals using deep neural models. To reliably classify these signals, researchers have developed multimodal fusion techniques that combine audio and video data into compact, quickly processed representations. However, previous approaches to multimodal data fusion have relied heavily on manually designed attention mechanisms. To address these limitations, we propose the Multimodal Auto Attention Fusion (MAF) model, which uses Neural Architecture Search (NAS) to automatically identify effective attentional representations for a wide range of tasks. Our approach includes a custom-designed search space that allows for the automatic generation of attention representations. Using automated Key, Query, and Value representation design, the MAF model enhances its self-attentiveness, allowing for the creation of highly effective attention representation designs. Compared to other multimodal fusion methods, our approach exhibits competitive performance in detecting modality interactions. We conducted experiments on three large datasets (UCF101, ActivityNet, and YouTube-8M), which confirmed the effectiveness of our approach and demonstrated its superior performance compared to other popular models. Furthermore, our approach exhibits robust generalizability across diverse datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jenni, S., Jin, H.: Time-equivariant contrastive video representation learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9970–9980 (2021)
Astrid, M., Zaheer, M.Z., Lee, S.I.: Synthetic temporal anomaly guided end-to-end video anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 207–214 (2021)
Sankarapandian, S., et al.: A pathology deep learning system capable of triage of melanoma specimens utilizing dermatopathologist consensus as ground truth. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 629–638 (2021)
Rao, A., Park, J., Woo, S., Lee, J.Y., Aalami, O.: Studying the effects of self-attention for medical image analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3416–3425 (2021)
Zhang, K., Peng, J., Fu, J., Liu, D.: Exploiting optical flow guidance for transformer-based video inpainting. arXiv preprint arXiv:2301.10048 (2023)
Abu-El-Haija, S., et al.: Youtube-8m: a large-scale video classification benchmark. arXiv preprint arXiv:1609.08675 (2016)
Ghanem, B., et al.: The activitynet large-scale activity recognition challenge 2018 summary. arXiv preprint arXiv:1808.03766 (2018)
Soomro, K., Zamir, A.R., Shah, M.: UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
Saxena, N., Wu, R., Jain, R.: Towards one shot search space poisoning in neural architecture search. arXiv preprint arXiv:2111.07138 (2021)
Shen, Y., et al.: ProxyBO: accelerating neural architecture search via Bayesian optimization with zero-cost proxies. arXiv preprint arXiv:2110.10423 (2021)
Yin, Y., Huang, S., Zhang, X.: BM-NAS: bilevel multimodal neural architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, pp. 8901–8909 (2022)
Xu, Z., So, D.R., Dai, A.M.: MUFASA: multimodal fusion architecture search for electronic health records. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, pp. 10532–10540 (2021)
Sato, R., Sakuma, J., Akimoto, Y.: AdvantageNAS: efficient neural architecture search with credit assignment. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, pp. 9489–9496 (2021)
White, C., Neiswanger, W., Savani, Y.: BANANAS: Bayesian optimization with neural architectures for neural architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, pp. 10293–10301 (2021)
Zhang, W., et al.: Transformer-based multimodal information fusion for facial expression analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2428–2437 (2022)
Kong, C., Zheng, K., Liu, Y., Wang, S., Rocha, A., Li, H.: M3FAS: an accurate and robust multimodal mobile face anti-spoofing system. arXiv preprint arXiv:2301.12831 (2023)
Huang, R., et al.: Make-an-audio: text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661 (2023)
Xu, M., Yuan, X., Miret, S., Tang, J.: ProtST: multi-modality learning of protein sequences and biomedical texts. arXiv preprint arXiv:2301.12040 (2023)
Wang, L., Koniusz, P., Huynh, D.Q.: Hallucinating IDT descriptors and I3D optical flow features for action recognition with CNNs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8698–8708 (2019)
Toering, M., Gatopoulos, I., Stol, M., Hu, V.T.: Self-supervised video representation learning with cross-stream prototypical contrasting. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 108–118 (2022)
Lin, Y., Guo, X., Lu, Y.: Self-supervised video representation learning with meta-contrastive network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8239–8249 (2021)
Tong, Z., Song, Y., Wang, J., Wang, L.: VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602 (2022)
Gowda, S.N., Rohrbach, M., Sevilla-Lara, L.: SMART frame selection for action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 2, pp. 1451–1459 (2021)
Li, H., Wu, Z., Shrivastava, A., Davis, L.S.: 2D or not 2D? Adaptive 3D convolution selection for efficient video recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6155–6164 (2021)
Wu, W., et al.: DSANet: dynamic segment aggregation network for video-level representation learning. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 1903–1911 (2021)
Zhu, H., Wang, Z., Shi, Y., Hua, Y., Xu, G., Deng, L.: Multimodal fusion method based on self-attention mechanism. Wirel. Commun. Mob. Comput. 2020, 1–8 (2020)
Long, X., et al.: Multimodal keyless attention fusion for video classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018)
Guan, C., Wang, X., Zhu, W.: AutoAttend: automated attention representation search. In: International Conference on Machine Learning, pp. 3864–3874. PMLR (2021)
Acknowledgement
This material is based upon work partially supported by the National Science Foundation under NSF grants IIS 1914489 and IIS 2008202. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, C., Ding, W., Shen, S., Chen, P. (2023). MAF: Multimodal Auto Attention Fusion for Video Classification. In: Fujita, H., Wang, Y., Xiao, Y., Moonis, A. (eds) Advances and Trends in Artificial Intelligence. Theory and Applications. IEA/AIE 2023. Lecture Notes in Computer Science(), vol 13925. Springer, Cham. https://doi.org/10.1007/978-3-031-36819-6_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-36819-6_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36818-9
Online ISBN: 978-3-031-36819-6
eBook Packages: Computer ScienceComputer Science (R0)