Skip to main content
Springer Nature Link
Log in

A Task Category Space for User-Centric Comparative Multimedia Search Evaluations

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13141))

Included in the following conference series:

  • 2394 Accesses

Abstract

In the last decade, user-centric video search competitions have facilitated the evolution of interactive video search systems. So far, these competitions focused on a small number of search task categories, with few attempts to change task category configurations. Based on our extensive experience with interactive video search contests, we have analyzed the spectrum of possible task categories and propose a list of individual axes that define a large space of possible task categories. Using this concept of category space, new user-centric video search competitions can be designed to benchmark video search systems from different perspectives. We further analyse the three task categories considered so far at the Video Browser Showdown and discuss possible (but sometimes challenging) shifts within the task category space.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.robots.ox.ac.uk/~vgg/challenges/video-pentathlon/challenge.html.

References

  1. Awad, G., et al.: TRECVID 2020: a comprehensive campaign for evaluating video retrieval tasks across multiple application domains. In: Proceedings of the TRECVID, Gaithersburg, MD, USA (2020)

    Google Scholar 

  2. Awad, G., et al.: TRECVID 2019: an evaluation campaign to benchmark video activity detection, video captioning and matching, and video search & retrieval. In: Proceedings of the TRECVID, Gaithersburg, MD, USA (2019)

    Google Scholar 

  3. Awad, G., et al.: TRECVID 2016: evaluating video search, video event detection, localization, and hyperlinking. In: Proceedings of the TREC Video Retrieval Evaluation (TRECVID), Gaithersburg, MD, USA (2016)

    Google Scholar 

  4. Bates, M.J.: The design of browsing and berrypicking techniques for the online search interface. Online Rev. 13(5), 407–424 (1989)

    Article  Google Scholar 

  5. Belkin, N.J., Marchetti, P.G., Cool, C.: BRAQUE: design of an interface to support user interaction in information retrieval. Inf. Process. Manage. 29(3), 325–344 (1993)

    Article  Google Scholar 

  6. Buckland, M.K.: On types of search and the allocation of library resources. J. Am. Soc. Inf. Sci. 30(3), 143–147 (1979)

    Article  Google Scholar 

  7. Christel, M.G.: Carnegie Mellon University traditional informedia digital video retrieval system. In: Proceedings of the ACM Conference on Image and Video Retrieval (CIVR), Amsterdam, The Netherlands, p. 647 (2007)

    Google Scholar 

  8. Constantin, M.G., Hicks, S., Larson, M., Nguyen, N.T.: MediaEval multimedia evaluation benchmark: tenth anniversary and counting. ACM SIGMM Rec. 12(2) (2020)

    Google Scholar 

  9. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), 5:1–5:60 (2008)

    Google Scholar 

  10. Gurrin, C., et al.: Introduction to the third annual lifelog search challenge (LSC’20). In: Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR), pp. 584–585 (2020)

    Google Scholar 

  11. Gurrin, C., et al.: Comparing approaches to interactive lifelog search at the Lifelog Search Challenge (LSC2018). ITE Trans. Media Technol. Appl. 7(2), 46–59 (2019)

    Google Scholar 

  12. Heller, S., Sauter, L., Schuldt, H., Rossetto, L.: Multi-stage queries and temporal scoring in vitrivr. In: Proceedings of the IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1–5 (2020)

    Google Scholar 

  13. Hu, W., Xie, N., Li, L., Zeng, X., Maybank, S.: A survey on visual content-based video indexing and retrieval. IEEE Trans. Syst. Man Cybern. Part C 41(6), 797–819 (2011)

    Article  Google Scholar 

  14. Kraaij, W., Smeaton, A.F., Over, P., Arlandis, J.: TRECVID 2004 - an overview. In: Proceedings of the TRECVID, Gaithersburg, MD, USA (2004)

    Google Scholar 

  15. Larson, M.A., Soleymani, M., Gravier, G., Ionescu, B., Jones, G.J.F.: The benchmarking initiative for multimedia evaluation: MediaEval 2016. IEEE Multimedia 24(1), 93–96 (2017)

    Article  Google Scholar 

  16. Lee, J.H., Renear, A., Smith, L.C.: Known-item search: variations on a concept. In: Proceedings of the ASIS&T Annual Meeting, Austin, TX, USA (2006)

    Google Scholar 

  17. Li, X., Xu, C., Yang, G., Chen, Z., Dong, J.: W2VV++: fully deep learning for ad-hoc video search. In: Proceedings of the ACM Multimedia, Nice, France (2019)

    Google Scholar 

  18. Li, Y., Belkin, N.J.: A faceted approach to conceptualizing tasks in information seeking. Inf. Process. Manage. 44(6), 1822–1837 (2008)

    Article  Google Scholar 

  19. Lokoč, J., Bailer, W., Schoeffmann, K., Münzer, B., Awad, G.: On influential trends in interactive video retrieval: video browser showdown 2015–2017. IEEE Trans. Multimedia 20(12), 3361–3376 (2018)

    Article  Google Scholar 

  20. Lokoč, J., et al.: A W2VV++ case study with automated and interactive text-to-video retrieval. In: Proceedings of the ACM Multimedia (Virtual Event), Seattle, WA, USA, pp. 2553–2561 (2020)

    Google Scholar 

  21. Over, P., et al.: TRECVID 2010 - an overview of the goals, tasks, data, evaluation mechanisms, and metrics. In: Proceedings of the TRECVID, Gaithersburg, MD, USA (2010)

    Google Scholar 

  22. Over, P., et al.: TRECVID 2013 - an overview of the goals, tasks, data, evaluation mechanisms and metrics. In: Proceedings of the TRECVID, Gaithersburg, MD, USA (2013)

    Google Scholar 

  23. Rossetto, L., Bailer, W., Bernstein, A.: Considering human perception and memory in interactive multimedia retrieval evaluations. In: Proceedings of the International Conference on Multimedia Modeling (MMM), Prague, Czech Republic, pp. 605–616 (2021)

    Google Scholar 

  24. Rossetto, L., et al.: On the user-centric comparative remote evaluation of interactive video search systems. IEEE Multimedia 28, 18–28 (2021)

    Article  Google Scholar 

  25. Rossetto, L., Gasser, R., Sauter, L., Bernstein, A., Schuldt, H.: A system for interactive multimedia retrieval evaluations. In: Proceedings of the International Conference on MultiMedia Modeling (MMM), Prague, Czech Republic (2021)

    Google Scholar 

  26. Salton, G.: Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley (1989)

    Google Scholar 

  27. Schoeffmann, K.: A user-centric media retrieval competition: the video browser showdown 2012–2014. IEEE Multimedia 21(4), 8–13 (2014)

    Article  Google Scholar 

  28. Schoeffmann, K.: Video browser showdown 2012–2019: a review. In: Proceedings of the International Conference on Content-Based Multimedia Indexing (CBMI), Dublin, Ireland, pp. 1–4 (2019)

    Google Scholar 

  29. Smeaton, A.F., Kraaij, W., Over, P.: TRECVID 2003 - an overview. In: Proceedings of the TRECVID, Gaithersburg, MD, USA (2003)

    Google Scholar 

  30. Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and TRECVID. In: Proceedings of the ACM International Workshop on Multimedia Information Retrieval (MIR), Santa Barbara, California, USA, pp. 321–330 (2006)

    Google Scholar 

  31. Snoek, C.G., Worring, M., de Rooij, O., van de Sande, K.E., Yan, R., Hauptmann, A.G.: VideOlympics: real-time evaluation of multimedia retrieval systems. IEEE Multimedia 15(1), 86–91 (2008)

    Article  Google Scholar 

  32. Walker, G., Janes, J.: Online retrieval: a dialogue of theory and practice. Libraries Unlimited (1999)

    Google Scholar 

  33. Wildemuth, B.M., O’Neill, A.L.: The “known’’ in known-item searches: empirical support for user-centered design. Coll. Res. Lib. 56(3), 265–281 (1995)

    Article  Google Scholar 

  34. Worring, M., Snoek, C., de Rooij, O., Nguyen, G., van Balen, R., Koelma, D.: Mediamill: advanced browsing in news video archives. In: Proceedings of the ACM Conference on Image and Video Retrieval (CIVR), Tempe, AZ, USA, pp. 533–536 (2006)

    Google Scholar 

  35. Wu, J., Ngo, C.W.: Interpretable embedding for ad-hoc video search. In: Proceedings of the ACM Multimedia (Virtual Event), Seattle, WA, USA, pp. 3357–3366 (2020)

    Google Scholar 

  36. Wu, J., Nguyen, P.A., Ma, Z., Ngo, C.W.: SQL-like interpretable interactive video search. In: Proceedings of the International Conference on MultiMedia Modeling (MMM), Prague, Czech Republic, pp. 391–397 (2021)

    Google Scholar 

Download references

Acknowledgments

This work has been supported by Czech Science Foundation (GAČR) project 19-22071Y, by Science Foundation Ireland under grant number SFI/12/RC/2289_2, and by EU’s Horizon 2020 research and innovation programs, under grant agreements n\(^\circ \) 951911 AI4Media (https://ai4media.eu) and n\(^\circ \) 825079 MindSpaces (https://mindspaces.eu/).

Author information

Authors and Affiliations

  1. Charles University, Prague, Czech Republic

    Jakub Lokoč & Ladislav Peška

  2. JOANNEUM RESEARCH, Graz, Austria

    Werner Bailer

  3. HTW Berlin, Berlin, Germany

    Kai Uwe Barthel

  4. Dublin City University, Dublin, Ireland

    Cathal Gurrin

  5. University of Basel, Basel, Switzerland

    Silvan Heller

  6. IT University of Copenhagen, Copenhagen, Denmark

    Björn þór Jónsson

  7. University of Zurich, Zurich, Switzerland

    Luca Rossetto

  8. Klagenfurt University, Klagenfurt, Austria

    Klaus Schoeffmann

  9. ISTI CNR, Pisa, Italy

    Lucia Vadicamo

  10. Centre for Research and Technology Hellas, Thessaloniki, Greece

    Stefanos Vrochidis

  11. City University of Hong Kong, Hong Kong, China

    Jiaxin Wu

Authors
  1. Jakub Lokoč
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Werner Bailer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kai Uwe Barthel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cathal Gurrin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Silvan Heller
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Björn þór Jónsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ladislav Peška
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Luca Rossetto
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Klaus Schoeffmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Lucia Vadicamo
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Stefanos Vrochidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Jiaxin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jakub Lokoč .

Editor information

Editors and Affiliations

  1. IT University of Copenhagen, Copenhagen, Denmark

    Björn Þór Jónsson

  2. Dublin City University, Dublin, Ireland

    Cathal Gurrin

  3. University of Science, VNU-HCM, Ho Chi Minh City, Vietnam

    Minh-Triet Tran

  4. University of Bergen, Bergen, Norway

    Duc-Tien Dang-Nguyen

  5. National Tsing Hua University, Hsinchu, Taiwan

    Anita Min-Chun Hu

  6. Hanoi University of Science and Technology, Hanoi, Vietnam

    Binh Huynh Thi Thanh

  7. Median Technologies, Valbonne, France

    Benoit Huet

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lokoč, J. et al. (2022). A Task Category Space for User-Centric Comparative Multimedia Search Evaluations. In: Þór Jónsson, B., et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13141. Springer, Cham. https://doi.org/10.1007/978-3-030-98358-1_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-98358-1_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-98357-4

  • Online ISBN: 978-3-030-98358-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation