Abstract
In this paper, we present an interactive video retrieval system named VideoCLIP 2.0 developed for the Video Browser Showdown in 2024. Building upon the foundation of the previous year’s system, VideoCLIP, this upgraded version incorporates several enhancements to support novice users in solving retrieval tasks quickly and effectively. Firstly, the revised system enables search using a variety of modalities, such as rich text, dominant colour, OCR, query-by-image, and now relevance feedback. Additionally a new keyframe selection technique and a new embedding model to replace the existing CLIP model have been employed. This new model aims to obtain richer visual representations in order to improve search performance in the live interactive challenge. Lastly, the user interface has been refined to enable quicker inspection and user-friendly navigation, particularly beneficial for novice users. In this paper we describe the updates to VideoCLIP.
T.-N. Nguyen and L.M. Quang—Contributed equally to this research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amato, G., et al.: VISIONE at video browser showdown 2023. In: Dang-Nguyen, D.T., et al. (eds.) MultiMedia Modeling, MMM 2023. LNCS, vol. 13833, pp. 615–621. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-27077-2_48
Berns, F., Rossetto, L., Schoeffmann, K., Beecks, C., Awad, G.: V3C1 dataset: an evaluation of content characteristics. In: Proceedings of the 2019 on International Conference on Multimedia Retrieval, ICMR 2019, pp. 334–338, New York, NY, USA. Association for Computing Machinery (2019)
Hezel, N., Schall, K., Jung, K., Barthel, K.U.: Efficient search and browsing of large-scale video collections with vibro. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13142, pp. 487–492. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_43
Lokoč, J., et al.: Interactive video retrieval in the age of effective joint embedding deep models: lessons from the 11th VBS. Multimedia Syst. 29(6), 3481–3504 (2023)
Lokoč, J., et al.: Is the reign of interactive search eternal? Findings from the video browser showdown 2020. ACM Trans. Multimedia Comput. Commun. Appl. 17(3), 1–26 (2021)
Nguyen, T.N., et al.: VideoCLIP: an interactive CLIP-based video retrieval system at VBS2023. In: Dang-Nguyen, D.T., et al. (eds.) MultiMedia Modeling, MMM 2023. LNCS, vol. 13833, pp. 671–677. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-27077-2_57
Nguyen, T.-N., Puangthamawathanakun, B., Healy, G., Nguyen, B.T., Gurrin, C., Caputo, A.: Videofall - a hierarchical search engine for VBS2022. In: Þór Jónsson, B., et al. (eds.) MMM 2022, Part II. LNCS, vol. 13142, pp. 518–523. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_48
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, 18–24 July 2021, vol. 139 of Proceedings of Machine Learning Research, pp. 8748–8763. PMLR (2021)
Rossetto, L., Schoeffmann, K., Bernstein, A.: Insights on the V3C2 Dataset. CoRR, abs/2105.01475 (2021)
Sauter, L., et al.: Exploring effective interactive text-based video search in vitrivr. In: Dang-Nguyen, D.T., et al. (eds.) Proceedings of the 29th International Conference on MultiMedia Modeling, MMM 2023, Part I, Bergen, Norway, 9–12 January 2023, vol. 13833, pp. 646–651. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-27077-2_53
Schall, K., Hezel, N., Jung, K., Barthel, K.U.: Vibro: video browsing with semantic and visual image embeddings. In: Dang-Nguyen, D.T., et al. (eds.) Proceedings of the 29th International Conference on MultiMedia Modeling, MMM 2023, Bergen, Norway, 9–12 January 2023, Part I, vol. 13833, pp. 665–670. Springer Heidelberg (2023). https://doi.org/10.1007/978-3-031-27077-2_56
Schoeffmann, K., Stefanics, D., Leibetseder, A.: divexplore at the video browser showdown 2023. In: Dang-Nguyen, D.T., et al. (eds.) Proceedings of the 29th International Conference on MultiMedia Modeling, MMM 2023, Part I, Bergen, Norway, 9–12 January 2023, vol. 13833, pp. 684–689. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-27077-2_59
Truong, Q.-T., et al.: Marine video kit: a new marine video dataset for content-based analysis and retrieval. In: Dang-Nguyen, D.T., et al. (eds.) Proceedings of the 29th International Conference on MultiMedia Modeling, MMM 2023, Bergen, Norway, 9–12 January 2023, vol. 13833, pp. 539–550. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-27077-2_42
Weng, Z., Yang, X., Li, A., Wu, Z., Jiang, Y.-G.: Open-VCLIP: transforming clip to an open-vocabulary video model via interpolated weight optimization. In: Proceedings of the 40th International Conference on Machine Learning, ICML 2023. JMLR.org (2023)
Acknowledgments
This research was conducted with the financial support of Science Foundation Ireland under Grant Agreement No. 18/CRT/6223, and 13/RC/2106_P2 at the ADAPT SFI Research Centre at DCU. ADAPT, the SFI Research Centre for AI-Driven Digital Content Technology is funded by Science Foundation Ireland through the SFI Research Centres Programme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, TN., Quang, L.M., Healy, G., Nguyen, B.T., Gurrin, C. (2024). VideoCLIP 2.0: An Interactive CLIP-Based Video Retrieval System for Novice Users at VBS2024. In: Rudinac, S., et al. MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14557. Springer, Cham. https://doi.org/10.1007/978-3-031-53302-0_37
Download citation
DOI: https://doi.org/10.1007/978-3-031-53302-0_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53301-3
Online ISBN: 978-3-031-53302-0
eBook Packages: Computer ScienceComputer Science (R0)