Skip to main content
SpringerLink
Log in

MIRRE approach: nonlinear and multimodal exploration of MIR aggregated search results

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Nowadays, web users frequently explore multimedia contents to satisfy their information needs. The exploration approaches usually provide linear interaction mechanisms and do not exploit the multiple information modalities associated with results. They cannot treat multimedia documents as aggregated entities. The aggregation of results in multimedia documents and nonlinear navigation in them is usually not possible. The exploration of multimedia content segregated in multiple verticals is tedious. In this research, we propose an approach to address the core issues in multimedia contents exploration. We provide a nonlinear and multimodal exploration of multimedia document results. We generate result spaces by exploiting multimodal similarity and semantic relationships in results and enable their nonlinear and multimodal exploration via a search user interface (SUI) design. The result space connects retrieved multimedia documents and their aggregated media objects via multimodal similarity and semantic relationships, respectively. The SUI provides lookup, preview/view/access, browsing, and visualization of multimedia documents by introducing various types of interface components. The approach has been instantiated over a real dataset of aggregated multimedia documents and evaluated via usability tests. Our investigation reveals that 93.33% users had completed exploration tasks within time limits. The overall usability scores assessed via SUS and PUEU instruments are 72.1% and 61.6%, respectively. The user satisfaction tested via QUIS-instrument is 70.3%. The usage of elementary SUI components providing the nonlinear exploration is high. The ploy-representation of results improves the overall information gain. The evaluation reveals that approach is usable, gives a satisfactory exploration mechanism, and SUI components collectively provide exploration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. https://www.google.com/

  2. https://www.bing.com/

  3. https://yandex.com/

  4. http://vcl.iti.gr/multimodal-search-and-retrieval/

  5. http://iti.gr/iti/projects/I-SEARCH.html

  6. https://lucenenet.apache.org/

  7. http://chatzichristofis.info

  8. http://sixrevisions.com/javascript/

  9. https://www.youtube.com/watch?v=Ues5PaeBKXA

  10. http://farm6.staticflickr.com/5068/5674693075_3bb4039812_b.jpg

  11. https://www.youtube.com/watch?v=Ues5PaeBKXA

  12. accessed from http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

  13. accessed from https://garyperlman.com/quest/quest.cgi?form=PUEU

  14. accessed from https://garyperlman.com/quest/quest.cgi?form=QUIS

References

  1. Achsas S et al (2018) Improving relational aggregated search from big data sources using stacked autoencoders. Cogn Syst Res 51:61–71

    Article  Google Scholar 

  2. Arampatzis A, Zagoris K, Chatzichristofis SA (2012) A study of awareness in multimedia search. Inf Process Manag 48(1):32–46

    Article  Google Scholar 

  3. Arampatzis A, Zagoris K, Chatzichristofis SA (2013) Dynamic two-stage image retrieval from large multimedia databases. Inf Process Manag 49(1):274–285

    Article  Google Scholar 

  4. Arguello J (2017) Aggregated search. Found Trends Inf Ret 10 (5):365–502

    Article  Google Scholar 

  5. Axenopoulos A, Daras P, Malassiotis S, Croce V, Lazzaro M, Etzold J, Grimm P, Massari A, Camurri A, Steiner T et al (2012) I-search: a unified framework for multimodal search and retrieval. In: The future internet. Springer, pp. 130–141

  6. Axenopoulos A, Manolopoulou S, Daras P (2012) Optimizing multimedia retrieval using multimodal fusion and relevance feedback techniques. Adv Multimed Model 7131:716–727

    Article  Google Scholar 

  7. Azad HK, Deepak A (2019) Query expansion techniques for information retrieval: a survey. Inf Process Manag 56(5):1698–1735

    Article  Google Scholar 

  8. Azzopardi L, Thomas P, Craswell N (2018) Measuring the utility of search engine result pages: an information foraging based measure. In: The 41st International ACM SIGIR conference on research & development in information retrieval, pp 605–614

  9. Bangor A, Kortum P, Miller J (2009) Determining what individual sus scores mean: Adding an adjective rating scale. J Usability Stud 4(3):114–123

    Google Scholar 

  10. Bogdanov D, Wack N, Gómez E, Gulati S, Herrera P, Mayor O, Roma G, Salamon J, Zapata J, Serra X (2013) Essentia: an open-source library for sound and music analysis. In: Proceedings of the 21st ACM international conference on Multimedia. ACM, pp 855–858

  11. Bracamonte T, Bustos B, Poblete B, Schreck T (2018) Extracting semantic knowledge from web context for multimedia ir: a taxonomy, survey and challenges. Multimed Tools Appl 1–37

  12. Brooke J (2013) Sus: a retrospective. J Usability Stud 8(2):29–40

    Google Scholar 

  13. Brooke J et al (1996) Sus-a quick and dirty usability scale. Usability Evaluation in Industry 189(194):4–7

    Google Scholar 

  14. Carrera CC, Perez JLS, Cantero JdlT (2018) Teaching with ar as a tool for relief visualization: Usability and motivation study. Int Res Geogr Environ Educ 27(1):69–84

    Article  Google Scholar 

  15. Carrion B, Onorati T, Díaz P, Triga V (2019) A taxonomy generation tool for semantic visual analysis of large corpus of documents. Multimed Tools Appl 78(23):32919–32937

    Article  Google Scholar 

  16. Chatzichristofis SA, Zagoris K, Boutalis YS, Papamarkos N (2010) Accurate image retrieval based on compact composite descriptors and relevance feedback information. Int J Pattern Recognit Artif Intell 24(02):207–244

    Article  Google Scholar 

  17. Chin JP, Diehl VA, Norman KL (1988) Development of an instrument measuring user satisfaction of the human-computer interface. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 213–218

  18. Daras P, Axenopoulos A, Darlagiannis V, Tzovaras D, Bourdon XL, Joyeux L, Verroust-Blondet A, Croce V, Steiner T, Massari A et al (2011) Introducing a unified framework for content object description. Int J Multimed Intell Secur 2(3):351–375

    Google Scholar 

  19. Daras P, Manolopoulou S, Axenopoulos A (2012) Search and retrieval of rich media objects supporting multiple multimodal queries. IEEE Trans Multimed 14(3):734–746

    Article  Google Scholar 

  20. Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 319–340

  21. Dourado IC, Pedronette DCG, da Silva Torres R (2019) Unsupervised graph-based rank aggregation for improved retrieval. Inf Process Manag 56(4):1260–1279

    Article  Google Scholar 

  22. Gasser R, Rossetto L, Schuldt H (2019) Multimodal multimedia retrieval with vitrivr. In: Proceedings of the 2019 on international conference on multimedia retrieval. pp 391–394

  23. Gialampoukidis I, Moumtzidou A, Liparas D, Tsikrika T, Vrochidis S, Kompatsiaris I (2017) Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression. Multimed Tools Appl 76(21):22383–22403

    Article  Google Scholar 

  24. Halvey M, Vallet D, Hannah D, Jose JM (2014) Supporting exploratory video retrieval tasks with grouping and recommendation. Inf Process Manag 50 (6):876–898

    Article  Google Scholar 

  25. Hart SG (2006) Nasa-task load index (nasa-tlx); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting, vol 50. Sage Publications, Los Angeles, pp 904–908

  26. Hearst M (2009) Search user interfaces. Cambridge University Press, Cambridge

    Book  Google Scholar 

  27. Heu JU, Qasim I, Lee DH (2015) Fodosu: multi-document summarization exploiting semantic analysis based on social folksonomy. Inf Process Manag 51(1):212–225

    Article  Google Scholar 

  28. Huang X, Qi J, Sun Y, Zhang R, Zheng HT (2019) Carl: Aggregated search with context-aware module embedding learning. In: 2019 International joint conference on neural networks (IJCNN). IEEE, pp 1–8

  29. Kannan R, Ghinea G, Swaminathan S (2015) What do you wish to see? a summarization system for movies based on user preferences. Inf Process Manag 51(3):286–305

    Article  Google Scholar 

  30. Kofler C, Larson M, Hanjalic A (2016) User intent in multimedia search: a survey of the state of the art and future challenges. ACM Comput Surv (CSUR) 49(2):36

    Article  Google Scholar 

  31. Kopliku A, Pinel-Sauvagnat K, Boughanem M (2014) Aggregated search: a new information retrieval paradigm. ACM Comput Surv 46(3):1–31

    Article  Google Scholar 

  32. Lee OJ, Jung JJ (2019) Integrating character networks for extracting narratives from multimodal data. Inf Process Manag 56(5):1894–1923

    Article  Google Scholar 

  33. Lefortier D, Serdyukov P, Romanenko F, de Rijke M (2014) Blending vertical and web results. In: Advances in information retrieval. Springer, pp 184–196

  34. Lewis JR (1995) Ibm computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human-Comput Int 7(1):57–78

    Article  Google Scholar 

  35. Lund AM (2001) Measuring usability with the use questionnaire12. Usability Interface 8(2):3–6

    Google Scholar 

  36. Lux M, Chatzichristofis SA (2008) Lire: lucene image retrieval: an extensible java cbir library. In: Proceedings of the 16th ACM international conference on multimedia. ACM, pp 1085–1088

  37. Marchionini G (2006) Exploratory search: from finding to understanding. Commun ACM 49(4):41–46

    Article  Google Scholar 

  38. Missaoui S, Kassem F, Viviani M, Agostini A, Faiz R, Pasi G (2019) Looker: a mobile, personalized recommender system in the tourism domain based on social media user-generated content. Pers Ubiquit Comput 1–17

  39. Petkos G, Schinas M, Papadopoulos S, Kompatsiaris Y (2017) Graph-based multimodal clustering for social multimedia. Multimed Tools Appl 76 (6):7897–7919

    Article  Google Scholar 

  40. Pinto-Cáceres SM, Almeida J, Baranauskas MCC, Torres RdS (2015) Fisir: A flexible framework for interactive search in image retrieval systems. In: Lecture notes in computer science, vol 8935. Springer, pp 335–347

  41. Pouyanfar S, Yang Y, Chen SC, Shyu ML, Iyengar S (2018) Multimedia big data analytics: A survey. ACM Comput Surv (CSUR) 51(1):10

    Article  Google Scholar 

  42. Rashid U, Bhatti MA (2015) Towards a conceptual framework to implement multiple media information search system. In: Digital information management (ICDIM), 2015 Tenth international conference on. IEEE, pp 154–159

  43. Rashid U, Bhatti MA (2017) A framework to explore results in multiple media information aggregated search. Multimed Tools Appl 76(24):25787–25826

    Article  Google Scholar 

  44. Rashid U, Viviani M, Pasi G (2016) A graph-based approach for visualizing and exploring a multimedia search result space. Inform Sci 370:303–322

    Article  Google Scholar 

  45. Rashid U, Viviani M, Pasi G, Bhatti MA (2016) The browsing issue in multimodal information retrieval: a navigation tool over a multiple media search result space. In: Flexible query answering systems 2015. Springer, pp 271–282

  46. Ren H, Renoust B, Melanċon G, Viaud ML, Satoh S (2018) Exploring temporal communities in mass media archives. In: Proceedings of the 26th ACM international conference on multimedia. pp 1247–1249

  47. Rinaldi AM, Russo C (2018) User-centered information retrieval using semantic multimedia big data. In: 2018 IEEE International conference on big data (Big Data). IEEE, pp 2304–2313

  48. Rinaldi AM, Russo C (2020) Using a multimedia semantic graph for web document visualization and summarization. Multimed Tools Appl 1–41

  49. Saddal M, Rashid U, Khattak AS (2019) A browsing approach to explore web image search results. In: 22nd International multitopic conference (INMIC). IEEE, pp 1–6

  50. Sánchez-Nielsen E, Chávez-Gutiérrez F, Lorenzo-Navarro J (2019) A semantic parliamentary multimedia approach for retrieval of video clips with content understanding. Multimed Syst 25(4):337–354

    Article  Google Scholar 

  51. Seifert C, Jurgovsky J, Granitzer M (2014) Facetscape: A visualization for exploring the search space. In: Proceedings of the 18th international conference on information visualization. IEEE, pp 94–101

  52. Tian F, Liu X, Liu Z, Sun N, Wang M, Wang H, Zhang F (2019) Multimedia integrated annotation based on common space learning. Multimed Tools Appl 78(1):437–456

    Article  Google Scholar 

  53. Tullis TS, Stetson JN (2004) A comparison of questionnaires for assessing website usability. In: Proceedings of the usability professional’s association conference. pp 1–12. Usability Professional Association

  54. Vaizman Y, McFee B, Lanckriet G (2014) Codebook-based audio feature representation for music information retrieval. IEEE/ACM Trans Audio Speech Lang Process 22(10):1483–1493

    Article  Google Scholar 

  55. Vila-Suero D, Rico M, Botezan I, Gómez-Pérez A (2019) Evaluating the impact of semantic technologies on bibliographic systems: A user-centered and comparative approach. J Web Semantics

  56. Wang J, Antonenko P, Celepkolu M, Jimenez Y, Fieldman E, Fieldman A (2019) Exploring relationships between eye tracking and traditional usability testing data. Int J Human Comput Int 35(6):483–494

    Article  Google Scholar 

  57. Wang X, Liu X, Peng SJ, Zhong B, Chen Y, Du JX (2020) Semi-supervised discrete hashing for efficient cross-modal retrieval. Multimed Tools Appl 79(35):25335–25356

    Article  Google Scholar 

  58. Yang Y, Wu F, Xu D, Zhuang Y, Chia LT (2010) Cross-media retrieval using query dependent search methods. Pattern Recogn Lett 43(8):2927–2936

    Article  Google Scholar 

  59. Yazici A, Koyuncu M, Yilmaz T, Sattari S, Sert M, Gulen E (2018) An intelligent multimedia information system for multimodal content extraction and querying. Multimed Tools Appl 77(2):2225–2260

    Article  Google Scholar 

  60. Zhuang YT, Yang Y, Wu F (2008) Mining semantic correlation of heterogeneous multimedia data for cross-media retrieval. IEEE Trans Multimed 10 (2):221–229

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the provision of the research facilities provided by the Department of Computer Sciences, Quaid-i-Azam University, Islamabad, Pakistan, to conduct this research study.

Author information

Authors and Affiliations

  1. Department of Computer Sciences, Quaid-i-Azam University, Islamabad, Pakistan

    Umer Rashid, Khalid Saleem & Adeel Ahmed

Authors
  1. Umer Rashid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khalid Saleem
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adeel Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Umer Rashid.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rashid, U., Saleem, K. & Ahmed, A. MIRRE approach: nonlinear and multimodal exploration of MIR aggregated search results. Multimed Tools Appl 80, 20217–20253 (2021). https://doi.org/10.1007/s11042-021-10603-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10603-x

Keywords

Search

Navigation