Abstract
Present advancements in digital content have resulted in an enhanced interest in video understanding. The Temporal Answer Grounding in Video Corpus (TAGVC) aims to pinpoint the visual response within an extensive array of untrimmed instructional videos using language-based questions. This research explores TAGVC, a notably complex task involving an intricate combination of skills including video retrieval and comprehension, visual answer localization, and collaboration between vision and language, posing challenges greater than the initial Temporal Answer Grounding in a Single Video (TAGSV). This paper outlines a novel approach to tackling such challenges, proposing a Fine-grained Modality Alignment and Local-Global Optimization Framework(FMALG) for TAGVC. By combining the strengths of visual and textual predictions, this system offers a resilient solution. The fine-grained modality alignment is used to understand each video segment’s context succinctly. In addition, the local-global optimization technique is implemented to learn the global retrieval capabilities and visualize answer localization. The subtitle quality is also improved using OpenAI’s ChatGPT. The efficacy of the proposed methods is evidenced through extensive experiments, where we achieved first place on track 3 and second place on track 2.
S. Cheng, Z. Zhou and J. Liu—Equal contribution
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Li, B., Weng, Y., Sun, B., Li, S.: Learning to locate visual answer in video corpus using question. In: ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5 (2023)
Weng, Y., Li, B.: Visual answer localization with cross-modal mutual knowledge transfer. In: ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5 (2023)
Tapaswi, M., et al.: Understanding stories in movies through question-answering. In: CVPR pp. 4631–4640 (2016)
Na, S., Lee, S., Kim, J., Kim, G.: A read-write memory network for movie story understanding. In: ICCV, pp. 677–685. IEEE Computer Society (2017)
Kim, J., Ma, M., Kim, K., Kim, S., Yoo, C.D.: Progressive attention memory network for movie story question answering. In: CVPR, pp. 8337–8346 (2019)
Jang, Y., Song, Y., Yu, Y., Kim, Y., Kim, G.: Tgif-qa: toward spatio-temporal reasoning in visual question answering. In: CVPR, pp. 2758–2766 (2017)
Lei, J., Licheng, Y., Bansal, M., Berg, T.L.: Localized, compositional video question answering. In: EMNLP, Tvqa (2018)
Lei, J., Yu, L., Berg, T.L., Bansal, M.: Tvqa+: spatio-temporal grounding for video question answering. In: ACL (2020)
Kim, H., Tang, Z., Bansal, M.: Dense-caption matching and frame-selection gating for temporal localization in videoqa. In: ACL (2020)
Kim, J., Ma, M., Pham, T.X., Kim, K., Yoo, C.D.: Modality shifting attention network for multi-modal video question answering. In CVPR. IEEE (2020)
Gao, J., Sun, C., Yang, Z., Nevatia, R.: Tall: temporal activity localization via language query. In: ICCV, pp. 5267–5275 (2017)
Hendricks, L.A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., Russell, B.C.: Localizing moments in video with natural language. In: ICCV, pp. 5804–5813. IEEE Computer Society (2017)
Yuan, Y., Mei, T., Zhu, W.: To find where you talk: temporal sentence localization in video with attention based location regression. In: AAAI, pp. 9159–9166. AAAI Press (2019)
Ma, Z., Ye, J., Yang, X., Liu, J.: Hcld: a hierarchical framework for zero-shot cross-lingual dialogue system. In: Proceedings of the 29th International Conference on Computational Linguistics, pp. 4492–4498 (2022)
Bojanowski, P., et al.: Weakly-supervised alignment of video with text. In: ICCV, pp. 4462–4470. IEEE Computer Society (2015)
Mithun, N.C., Paul, S., Roy-Chowdhury, A.K.: Weakly supervised video moment retrieval from text queries. In: CVPR, pp. 11592–11601. Computer Vision Foundation/IEEE (2019)
Liu, J., Cheng, S., Zhou, Z., Gu, Y., Ye, J., Luo, H.: Enhancing multilingual document-grounded dialogue using cascaded prompt-based post-training models. In: Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, Toronto, Canada, pp. 44–51. Association for Computational Linguistics (July 2023)
Ma, Z., Liu, Z., Ye, J.:. Sldt: sequential latent document transformer for multilingual document-based dialogue. In: Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pp. 57–67 (2023)
Lin, Z., Zhao, Z., Zhang, Z., Wang, Q., Liu, H.: Weakly-supervised video moment retrieval via semantic completion network. In: AAAI, pp. 11539–11546. AAAI Press (2020)
Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 305–321 (2018)
Zhang, J., et al.: Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR, abs/ arXiv: 2209.02970 (2022)
Cui, Y., et al.: Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101 (2019)
Zhao, Z., Guo, L., He, X., Shao, S., Yuan, Z., Liu, J.: Mamo: masked multimodal modeling for fine-grained vision-language representation learning. arXiv preprint arXiv:2210.04183 (2022)
Acknowledgment
The research work is supported by National Key R &D Program of China (No.202 2YFB3904700), Key Research and Development Program of in Shandong Province (2019JZZY020102), Key Research and Development Program of Jiangsu Province (No.BE2018084), Industrial Internet Innovation and Development Project in 2021 (TC210A02M, TC210804D), Opening Project of Beijing Key Laboratory of Mobile Computing and Pervasive Device.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cheng, S., Zhou, Z., Liu, J., Ye, J., Luo, H., Gu, Y. (2023). A Unified Framework for Optimizing Video Corpus Retrieval and Temporal Answer Grounding: Fine-Grained Modality Alignment and Local-Global Optimization. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14304. Springer, Cham. https://doi.org/10.1007/978-3-031-44699-3_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-44699-3_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44698-6
Online ISBN: 978-3-031-44699-3
eBook Packages: Computer ScienceComputer Science (R0)
