Ralph Gasser
ABSTRACT
The steady growth of multimedia collections - both in terms of size and heterogeneity - necessitates systems that are able to conjointly deal with several types of media as well as large volumes of data. This is especially true when it comes to satisfying a particular information need, i.e., retrieving a particular object of interest from a large collection. Nevertheless, existing multimedia management and retrieval systems are mostly organized in silos and treat different media types separately. Hence, they are limited when it comes to crossing these silos for accessing objects. In this paper, we present vitrivr, a general-purpose content-based multimedia retrieval stack. In addition to the keyword search provided by most media management systems, vitrivr also exploits the object's content in order to facilitate different types of similarity search. This can be done within and, most importantly, across different media types giving rise to new, interesting use cases. To the best of our knowledge, the full vitrivr stack is unique in that it seamlessly integrates support for four different types of media, namely images, audio, videos, and 3D models.
- George Awad, Asad Butt, Keith Curtis, Yooyoung Lee, Jonathan Fiscus, Afzal Godil, David Joy, Andrew Delgado, Alan F. Smeaton, Yvette Graham, Wessel Kraaij, Georges Quénot, Joao Magalhaes, David Semedo, and Saverio Blasi. 2018. TRECVID 2018: Benchmarking Video Activity Detection, Video Captioning and Matching, Video Storytelling Linking and Video Search. In Proceedings of TRECVID 2018 . NIST, USA.Google Scholar
- Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. 2003. On Visual Similarity based 3D Model Retrieval. In Computer Graphics Forum, Vol. 22. Wiley Online Library, 223--232.Google Scholar
- Claudiu Cobârzan, Klaus Schoeffmann, Werner Bailer, Wolfgang Hürst, Adam Blavz ek, Jakub Lokovc, Stefanos Vrochidis, Kai Uwe Barthel, and Luca Rossetto. 2017. Interactive video Search Tools: a Detailed Analysis of the Video Browser Showdown 2015. Multimedia Tools and Applications, Vol. 76, 4 (2017), 5539--5571. Google ScholarDigital Library
- Myron Flickner, Harpreet Sawhney, Wayne Niblack, Jonathan Ashley, Qian Huang, Byron Dom, Monika Gorkani, Jim Hafner, Denis Lee, Dragutin Petkovic, et almbox. 1995. Query by Image and Video Content: The QBIC System. Computer, Vol. 28 (1995), 23--32. Google ScholarDigital Library
- Jonathan T Foote. 1997. Content-based Retrieval of Music and Audio. In Proc. SPIE 3229, Multimedia Storage and Archiving Systems II. 138--147.Google Scholar
- Ralph Gasser, Luca Rossetto, and Heiko Schuldt. 2019. Towards an All-Purpose Content-Based Multimedia Information Retrieval System. arXiv preprint arXiv:1902.03878 (2019).Google Scholar
- Ivan Giangreco and Heiko Schuldt. 2016. ADAM$_pro$: Database Support for Big Multimedia Retrieval . Datenbank-Spektrum, Vol. 16, 1 (2016), 17--26.Google ScholarCross Ref
- Emilia Gó mez. 2006. Tonal Description of Music Audio Signals. Doctoral Dissertation. Universitat Pompeu Fabra, Barcelona.Google Scholar
- Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz. 2003. Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors. In Eurographics Symposium on Geometry Processing, Vol. 43. 156--164.Google Scholar
- Patrick M. Kelly, Michael Cannon, and Donald R. Hush. 1995. Query by image example: the CANDID approach.Google Scholar
- Serkan Kiranyaz, Kerem Caglar, Esin Guldogan, Olcay Guldogan, and Moncef Gabbouj. 2003. MUVIS: a Content-based Multimedia Indexing and Retrieval Framework. In Proceedings of the Seventh International Symposium on Signal Processing and Its Applications (ISSPA), Vol. 1. 1--8.Google ScholarCross Ref
- Goujun Lu. 2001. Indexing and retrieval of Audio: A Survey. Multimedia Tools and Applications, Vol. 15, 3 (2001), 269--290. Google ScholarDigital Library
- Meinard Muller, Frank Kurth, and Michael Clausen. 2005. Chroma-based Statistical Audio Features for Audio Matching. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2005. IEEE, New Paltz, NY, USA, 275--278.Google Scholar
- Luca Rossetto, Ivan Giangreco, and Heiko Schuldt. 2014. Cineast: a Multi-feature Sketch-based Video Retrieval Engine. In 2014 IEEE International Symposium on Multimedia. IEEE, Taichung, Taiwan, 18--23.Google ScholarDigital Library
- Luca Rossetto, Ivan Giangreco, and Heiko Schuldt. 2015a. OSVC-Open Short Video Collection 1.0. Technical Report CS-2015-002 (2015).Google Scholar
- Luca Rossetto, Ivan Giangreco, Heiko Schuldt, Stéphane Dupont, Omar Seddati, Metin Sezgin, and Yusuf Sahillioug lu. 2015b. IMOTION -- a Content-based Video Retrieval Engine. In International Conference on Multimedia Modeling. Springer, 255--260.Google ScholarCross Ref
- Luca Rossetto, Ivan Giangreco, Claudiu Tua nase, and Heiko Schuldt. 2016. vitrivr: A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, Amsterdam, the Netherlands, 1183--1186.Google ScholarDigital Library
- Luca Rossetto, Ivan Giangreco, Claudiu Tua nase, Heiko Schuldt, Stéphane Dupont, and Omar Seddati. 2017. Enhanced Retrieval and Browsing in the IMOTION System. In International Conference on Multimedia Modeling. Springer, 469--474.Google Scholar
- Luca Rossetto, Mahnaz Amiri Parian, Ralph Gasser, Ivan Giangreco, Silvan Heller, and Heiko Schuldt. 2019 a. Deep Learning-Based Concept Detection in vitrivr. In International Conference on Multimedia Modeling. Springer, 616--621.Google ScholarCross Ref
- Luca Rossetto, Heiko Schuldt, George Awad, and Asad A Butt. 2019 b. V3C--A Research Video Collection. In International Conference on Multimedia Modeling . Springer, 349--360.Google ScholarCross Ref
- Dietmar Saupe and Dejan V. Vranić. 2001. 3D Model Retrieval with Spherical Harmonics and Moments. In Proceedings of the 23rd DAGM-Symposium on Pattern Recognition, Vol. 2191. Springer, 392--397. Google ScholarDigital Library
- Klaus Schoeffmann, David Ahlström, Werner Bailer, Claudiu Cobârzan, Frank Hopfgartner, Kevin McGuinness, Cathal Gurrin, Christian Frisson, Duy-Dinh Le, Manfred Del Fabro, Hongliang Bai, and Wolfgang Weiss. 2014. The Video Browser Showdown: a Live Evaluation of Interactive Video Search Tools. International Journal of Multimedia Information Retrieval, Vol. 3, 2 (2014), 113--127.Google Scholar
- Johan WH Tangelder and Remco C Veltkamp. 2004. A survey of Content Based 3D Shape Retrieval Methods. In Shape Modeling Applications, 2004. Proceedings. IEEE, 145--156. Google ScholarDigital Library
- Rainer Typke, Frans Wiering, and Remco C Veltkamp. 2005. A Survey of Music Information Retrieval Systems. In Proceedings of the 6th International Conference on Music Information Retrieval. Queen Mary, University of London, 153--160.Google Scholar
- Avery Wang. 2006. The Shazam Music Recognition Service. Commun. ACM, Vol. 49, 8 (2006), 44--48. Google ScholarDigital Library
Index Terms
- Multimodal Multimedia Retrieval with vitrivr
Recommendations
Retrieval of Structured and Unstructured Data with vitrivr
LSC '19: Proceedings of the ACM Workshop on Lifelog Search ChallengeWith the increase in sensory capability of mobile devices, the data that can be generated and used in a lifelogging context gets increasingly diverse. Such data is special in the context of multimedia, not only because of its close personal relationship ...
Interactive Lifelog Retrieval with vitrivr
LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search ChallengeThe variety and amount of data being collected in our everyday life poses unique challenges for multimedia retrieval. In the Lifelog Search Challenge (LSC), multimedia retrieval systems compete in finding events based on descriptions containing hints ...
A Survey on Content-Based Retrieval for Multimedia Databases
Conventional database systems are designed for managing textual and numerical data, and retrieving such data is often based on simple comparisons of text/numerical values. However, this simple method of retrieval is no longer adequate for the multimedia ...
Comments