Abstract
In this paper, we present an interactive video retrieval system named VideoCLIP developed for the Video Browser Showdown 2023. To support users in solving retrieval tasks, the system enables search using a variety of modalities, such as rich text, dominant colour, OCR, and query-by-image. Moreover, a new search modality has been added to empower our core engine, which is inherited from the Contrastive Language-Image Pre-training (CLIP) model. Finally, the user interface is enhanced to display results in groups in order to reduce the effort for a user when locating potentially relevant targets.
T.-N. Nguyen and B. Puangthamawathanakun—Contributed equally to this research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amato, G., et al.: VISIONE at video browser showdown 2022. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13142, pp. 543–548. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_52
Berns, F., Rossetto, L., Schoeffmann, K., Beecks, C., Awad, G.: V3C1 Dataset: an evaluation of content characteristics. In: Proceedings of the 2019 on International Conference on Multimedia Retrieval, ICMR ’19 pp. 334–338, New York, NY, USA, 2019. Association for Computing Machinery
Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for Image Recognition at Scale (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)
Hezel, N., Schall, K., Jung, K., Barthel, K.U.: Efficient search and browsing of large-scale video collections with vibro. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13142, pp. 487–492. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_43
Jocher, G., et al.: Ultralytics/YOLOv5: v6.2 - YOLOv5 Classification Models, Apple M1, Reproducibility, ClearML and Deci.ai integrations, Aug. (2022)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Lokoč, J., et al.: Is the reign of interactive search eternal? Findings from the video browser showdown 2020. ACM Trans. Multimedia Comput. Commun. Appl., 17(3), Jul (2021)
Lokoč, J., Mejzlík, F., Souček, T., Dokoupil, P., Peška, L.: Video search with context-aware ranker and relevance feedback. In: Þór Jónsson, B., et al. (eds.) MultiMedia Modeling. Lecture Notes in Computer Science, pp. 505–510. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_46
Nguyen, T.-N., Puangthamawathanakun, B., Healy, G., Nguyen, B.T., Gurrin, C., Caputo, A.: Videofall - a hierarchical search engine for VBS2022. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13142, pp. 518–523. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_48
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T., eds, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748–8763. PMLR, 18–24 Jul 2021
Rossetto, L., Schoeffmann, K., Bernstein, A.: Insights on the V3C2 Dataset. CoRR, abs/2105.01475 (2021)
Truong, Q.-T., et al.: Marine video kit: A new marine video dataset for content-based analysis and retrieval. In: Marine video kit: a new marine video dataset for content-based analysis and retrieval MMM 2023, Bergen, Norway, January 9–12, 2023. Springer (2023)
Acknowledgments
This research was conducted with the financial support of Science Foundation Ireland under Grant Agreement No. 18/CRT/6223, and 13/RC/2106_P2 at the ADAPT SFI Research Centre at DCU. ADAPT, the SFI Research Centre for AI-Driven Digital Content Technology, is funded by Science Foundation Ireland through the SFI Research Centres Programme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, TN. et al. (2023). VideoCLIP: An Interactive CLIP-based Video Retrieval System at VBS2023. In: Dang-Nguyen, DT., et al. MultiMedia Modeling. MMM 2023. Lecture Notes in Computer Science, vol 13833. Springer, Cham. https://doi.org/10.1007/978-3-031-27077-2_57
Download citation
DOI: https://doi.org/10.1007/978-3-031-27077-2_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27076-5
Online ISBN: 978-3-031-27077-2
eBook Packages: Computer ScienceComputer Science (R0)