Skip to main content
SpringerLink
Log in

A Conversational Search Framework for Multimedia Archives

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14612))

Included in the following conference series:

  • 274 Accesses

Abstract

Conversational search system seek to support users in their search activities to improve the effectiveness and efficiency of search while reducing their cognitive load. The challenges of multimedia search mean that search supports provided by conversational search have the potential to improve the user search experience. For example, by assisting users in constructing better queries and making more informed decisions in relevance feedback stages whilst searching. However, previous research on conversational search has been focused almost exclusively on text archives. This demonstration illustrates the potential for the application of conversational methods in multimedia search. We describe a framework to enable multimodal conversational search for use with multimedia archives. Our current prototype demonstrates the use of an conversational AI assistant during the multimedia information retrieval process for both image and video collections.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://rasa.com/.

  2. 2.

    https://whoosh.readthedocs.io/.

  3. 3.

    https://github.com/apotyagalova/conversational-search-agent.

References

  1. Hanjalic, A., Lienhart, R., Ma, W.Y., Smith, J.R.: The holy grail of multimedia information retrieval: so close or yet so far away? Proc. IEEE 96(4), 541–547 (2008). https://doi.org/10.1109/JPROC.2008.916338

    Article  Google Scholar 

  2. Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: data, models and evaluation metrics. J. Artif. Intell. Res. 47, 853–899 (2013)

    Article  MathSciNet  Google Scholar 

  3. Kaushik, A., Jacob, B., Velavan, P.: An exploratory study on a reinforcement learning prototype for multimodal image retrieval using a conversational search interface. Knowledge 2(1), 116–138 (2022)

    Article  Google Scholar 

  4. Kim, H., Kim, D., Yoon, S., Dernoncourt, F., Bui, T., Bansal, M.: Caise: conversational agent for image search and editing. In: Proceedings of the The Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-2022) (2022)

    Google Scholar 

  5. Nie, L., Jiao, F., Wang, W., Wang, Y., Tian, Q.: Conversational image search. IEEE Trans. Image Process. 30, 7732–7743 (2021)

    Article  Google Scholar 

  6. Radlinski, F., Craswell, N.: A theoretical framework for conversational search. In: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pp. 117–126 (2017)

    Google Scholar 

  7. Xu, J., Mei, T., Yao, T., Rui, Y.: MSR-VTT: a large video description dataset for bridging video and language. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5288–5296 (2016)

    Google Scholar 

  8. Zamani, H., Trippas, J.R., Dalton, J., Radlinski, F.: Conversational information seeking. arXiv preprint arXiv:2201.08808 (2022)

Download references

Acknowledgments

This work was conducted with the financial support of the Science Foundation Ireland Centre for Research Training in Digitally-Enhanced Reality (d-real) under Grant No. 18/CRT/6224, and partially as part of the ADAPT Centre at DCU (Grant No. 13/RC/2106_P2) (www.adaptcentre.ie). For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.

Author information

Authors and Affiliations

  1. ADAPT Centre, School of Computing, Dublin City University, Dublin 9, Ireland

    Anastasia Potyagalova & Gareth J. F. Jones

Authors
  1. Anastasia Potyagalova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gareth J. F. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anastasia Potyagalova .

Editor information

Editors and Affiliations

  1. Georgetown University, Washington, WA, USA

    Nazli Goharian

  2. University of Pisa, PISA, Pisa, Italy

    Nicola Tonellotto

  3. King's College London, London, UK

    Yulan He

  4. University College London, London, UK

    Aldo Lipani

  5. University of Glasgow, Glasgow, UK

    Graham McDonald

  6. University of Glasgow, Glasgow, UK

    Craig Macdonald

  7. University of Glasgow, Glasgow, UK

    Iadh Ounis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Potyagalova, A., Jones, G.J.F. (2024). A Conversational Search Framework for Multimedia Archives. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14612. Springer, Cham. https://doi.org/10.1007/978-3-031-56069-9_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56069-9_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56068-2

  • Online ISBN: 978-3-031-56069-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation