Skip to main content
Springer Nature Link
Log in

A multimedia document browser based on multilayer networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Querying and retrieving relevant information still remains a difficult task, one with a relatively high cognitive cost for users, who usually focus only on the first few pages of results. This issue drives effort to support the exploration of search results through clustering and visualization. This paper contributes to this challenge by providing a visual analytics system that is designed to support search tasks in multimedia document archives. The system provides complex querying, semantic overviews of time, and visual, and textual concepts combined with analysis. All search tasks are supported with linked-highlighting and leapfrog interactions. This is made possible all in a single data structure thanks to multilayer network modelling.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

Notes

  1. see: https://www.imdb.com/

  2. Available from Kaggle at kaggle.com/carolzhangdc/imdb-5000-movie-dataset

  3. See microsoft.com/translator

  4. A demonstration video from our original paper [62] is available at https://youtu.be/VfGwa6T94t8.

  5. Available at https://github.com/renoust/visualclouds

  6. See https://semantic-ui.com/ and https://d3js.org/

  7. See www.jasondavies.com/wordcloud/about/

References

  1. Aigner W, Miksch S, Müller W, Schumann H, Tominski C (2007) Visualizing time-oriented data—a systematic view. Comput Graph 31 (3):401–409

    Article  Google Scholar 

  2. Amar R, Stasko J (2004) A knowledge task-based framework for design and evaluation of information visualizations. In: 2004. INFOVIS 2004. IEEE symposium on Information visualization. IEEE, pp 143–150

  3. Amos B, Ludwiczuk B, Satyanarayanan M (2016) Openface: A general-purpose face recognition library with mobile applications. Tech. rep., Technical report, CMU-CS-16-118 CMU

  4. Archambault D, Munzner T, Auber D (2008) Grouseflocks: Steerable exploration of graph hierarchy space. IEEE Trans Vis Comput Graph 14(4):900–913

    Article  Google Scholar 

  5. Azzopardi L, Kelly D, Brennan K (2013) How query cost affects search behavior. In: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. ACM, pp 23–32

  6. Baeza-Yates R, Davis E (2004) Web page ranking using link attributes. In: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters. ACM, pp 328–329

  7. Barry AM (1997) Visual intelligence: perception, image, and manipulation in visual communication. SUNY Press

  8. Barth L, Kobourov SG, Pupyrev S (2013) An experimental study of algorithms for semantics-preserving word cloud layout. University of Arizona Report

  9. Becker RA, Cleveland WS (1987) Brushing scatterplots. Technometrics 29(2):127–142

    Article  MathSciNet  Google Scholar 

  10. Blondel VD, Guillaume J.L, Lambiotte R, Lefebvre E (2008) Fast unfolding of communities in large networks. J Stat Mech Theory Exper 10:8. https://doi.org/10.1088/1742-5468/2008/10/P10008

    MATH  Google Scholar 

  11. Brehmer M, Munzner T (2013) A multi-level typology of abstract visualization tasks. IEEE TVCG 19(12):2376–2385

    Google Scholar 

  12. Brin S, Page L (2012) Reprint of: The anatomy of a large-scale hypertextual web search engine. Comput Netw 56(18):3825–3833

    Article  Google Scholar 

  13. Burt R, Scott T (1985) Relation content in multiple networks. Soc Sci Res 14:287–308

    Article  Google Scholar 

  14. Carpineto C, Osiński S, Romano G, Weiss D (2009) A survey of web clustering engines. ACM Comput Surv (CSUR) 41(3):17

    Article  Google Scholar 

  15. Chinchor NA, Thomas JJ, Wong PC, Christel MG, Ribarsky W (2010) Multimedia analysis+ visual analytics= multimedia analytics. IEEE Comput Graph Appl 30(5):52–60

    Article  Google Scholar 

  16. Clarkson E, Desai K, Foley J (2009) Resultmaps: Visualization for search interfaces. IEEE TVCG 15(6)

  17. De Domenico M, Porter MA, Arenas A (2015) Muxviz: a tool for multilayer analysis and visualization of networks. J Compl Netw 3(2):159–176

    Article  Google Scholar 

  18. Demmel J, Dumitriu I, Holtz O (2007) Fast linear algebra is stable. Numer Math 108(1):59–91

    Article  MathSciNet  Google Scholar 

  19. Di Marco A, Navigli R (2013) Clustering and diversifying web search results with graph-based word sense induction. Comput Linguist 39(3):709–754

    Article  Google Scholar 

  20. Ferragina P, Gullì A (2004) Knowledge Discovery in Databases: PKDD 2004, chap. The Anatomy of SnakeT: A Hierarchical Clustering Engine for Web-Page Snippets. Springer, Berlin, pp 506–508. https://doi.org/10.1007/978-3-540-30116-5_48

  21. Fujimura K, Toda H, Inoue T, Hiroshima N, Kataoka R, Sugizaki M (2006) Blogranger—a multi-faceted blog search engine. In: Proceedings of WWW, Weblogging Ecosystem

  22. Gao J, Buldyrev SV, Havlin S, Stanley HE (2011) Robustness of a network of networks. Phys Rev Lett 107(19):195,701

    Article  Google Scholar 

  23. Ghoniem M, Fekete JD, Castagliola P (2004) A comparison of the readability of graphs using node-link and matrix-based representations. In: IEEE Symposium on information visualization. IEEE, pp 17–24

  24. Ghoniem M, Mcgee F, Melanċon G, Otjacques B, Pinaud B (2019) The state of the art in multilayer network visualization. Computer Graphics Forum. https://doi.org/10.1111/cgf.13610

  25. Gomez-Nieto E, San Roman F, et al. (2014) Similarity preserving snippet-based visualization of web search results. IEEE TVCG 20(3):457–470

    Google Scholar 

  26. Hascoët M, Dragicevic P (2012) Interactive graph matching and visual comparison of graphs and clustered graphs. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, pp 522–529

  27. Havre S, Hetzler B, Nowell L (2000) Themeriver: Visualizing theme changes over time. In: IEEE Symposium on information visualization 2000. INFOVIS 2000. Proceedings. IEEE, pp 115–123

  28. Hervé N, Viaud ML, Thièvre J, Saulnier A, Champ J, Letessier P, Buisson O, Joly A (2013) Otmedia: the french transmedia news observatory. In: Proceedings of ACM multimedia. ACM, pp 441–442

  29. Ide I, Nack F (2013) Explain this to me!. ITE Trans Media Technol Appl 1(2):101–117

    Article  Google Scholar 

  30. Ide I et al (2004) Topic threading for structuring a large-scale news video archive. Image Video Retr 1(1):123–131

    Article  Google Scholar 

  31. Itoh M, Toyoda M, Zhu CZ, Satoh S, Kitsuregawa M (2014) Image flows visualization for inter-media comparison. In: IEEE Pacific visualization symposium. IEEE, pp 129–136

  32. Johnson J, Douze M, Jégou H (2019) Billion-scale similarity search with gpus. IEEE Transactions on Big Data

  33. Käki M (2005) Findex: Search result categories help users when document ranking fails. In: SIGCHI Conference on human factors in computing systems, pp 131–140

  34. Kivelä M, Arenas A, Barthelemy M, Gleeson JP, Moreno Y, Porter MA (2014) Multilayer networks. J Compl Netw 2(3):203–271

    Article  Google Scholar 

  35. Kochtchi A, Landesberger TV, Biemann C (2014) Networks of names: Visual exploration and semi-automatic tagging of social networks from newspaper articles. In: Computer graphics forum, vol 33-3. Wiley online library, pp 211–220

  36. Koshman S (2006) Visualization-based information retrieval on the web. Library Inf Sci Res 28(2): 192–207

    Article  Google Scholar 

  37. Koshman S, Spink A, Jansen BJ (2006) Web searching on the vivisimo search engine. J Amer Soc Inf Sci Technol 57(14):1875–1887

    Article  Google Scholar 

  38. Krovetz R, Croft B (1992) Lexical ambiguity and information retrieval. ACM Trans Inf Syst (TOIS) 10(2):115–141

    Article  Google Scholar 

  39. Krstajic M, Najm-Araghi M, Mansmann F, Keim DA (2012) Incremental visual text analytics of news story development. In: Visualization and data analysis, pp 829407

  40. Le DD, Satoh S (2011) Indexing faces in broadcast news video archives. In: 2011 IEEE 11Th ICDM workshops, pp 519–526

  41. Le TN, Luqman MM, Burie JC, Ogier JM (2015) A comic retrieval system based on multilayer graph representation and graph mining. In: International workshop on graph-based representations in pattern recognition. Springer, pp 355–364

  42. Le DD, Phan S, Nguyen VT, Renoust B, Nguyen TA, Hoang VN, Ngo TD, Tran MT, Watanabe Y, Klinkigt M et al (2016) Nii-hitachi-uit at trecvid 2016. In: TRECVID Workshop. NIST, Gaithersburg

  43. Li H, Jou B, Ellis JG, Morozoff D, Chang SF (2013) News Rover: Exploring topical structures and serendipity in heterogeneous multimedia news. In: Proceedings of multimedia. ACM, pp 449–450

  44. Luo H, Fan J, Yang J, Ribarsky W, Satoh S (2006) Exploring large-scale video news via interactive visualization. In: IEEE VAST. IEEE, pp 75–82

  45. Maaten L.v.d., Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9:2579–2605

    MATH  Google Scholar 

  46. Marchionini G (2006) Exploratory search: from finding to understanding. Commun ACM 49(4):41–46

    Article  Google Scholar 

  47. Matsui Y, Ogaki K, Yamasaki T, Aizawa K (2017) Pqk-means: Billion-scale clustering for product-quantized codes. In: Proceedings of the 25th ACM international conference on Multimedia, pp 1725–1733

  48. Matsui Y, Yamasaki T, Aizawa K (2018) Pqtable: Nonexhaustive fast search for product-quantized codes using hash tables. IEEE Trans Multimed 20 (7):1809–1822

    Article  Google Scholar 

  49. Matsuo Y, Sakaki T, Uchiyama K, Ishizuka M (2006) Graph-based word clustering using a web search engine. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp 542–550

  50. McCormack G (2008) Japan and north korea: The long and twisted path toward normalcy. Technical report, Working Paper. US-Korea Inst. at SAIS

  51. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119

  52. Misue K, Eades P, Lai W, Sugiyama K (1995) Layout adjustment and the mental map. J Vis Lang Comput 6(2):183–210

    Article  Google Scholar 

  53. Moltmann F (2017) Natural language ontology Oxford Research Encyclopedia of Linguistics. https://doi.org/10.1093/acrefore/9780199384655.013.330

  54. Munzner T (2009) A nested model for visualization design and validation. IEEE Trans Vis Comput Graph 15(6):921–928

    Article  Google Scholar 

  55. Navigli R (2009) Word sense disambiguation: a survey. ACM Comput Surv 41(2):10

    Article  Google Scholar 

  56. Ngo Td et al (2013) Face retrieval in large-scale news video datasets. IEICE Trans Inf Syst 96(8): 1811–1825

    Article  Google Scholar 

  57. Noack A (2009) Modularity clustering is force-directed layout. Phys Rev E 79(2):026,102

    Article  Google Scholar 

  58. Nocaj A, Brandes U (2012) Organizing search results with a reference map. IEEE TVCG 18(12): 2546–2555

    Google Scholar 

  59. Osiński S, Weiss D (2005) Carrot2: Design of a flexible and efficient web information retrieval framework. In: International atlantic web intelligence conference. Springer, pp 439–444

  60. Page L, Brin S, Motwani R, Winograd T (1999) The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab

  61. Peat HJ, Willett P (1991) The limitations of term co-occurrence data for query expansion in document retrieval systems. J Amer Soc Inf Sci 42(5):378–383

    Article  Google Scholar 

  62. Ren H, Renoust B, Viaud ML, Melanċon G, Satoh S (2018) Generating “visual clouds” from multiplex networks for tv news archive query visualization. In: 2018 International conference on content-based multimedia indexing (CBMI). IEEE, pp 1–6

  63. Renoust B, Melanċon G, Viaud ML (2014) Entanglement in multiplex networks: understanding group cohesion in homophily networks. In: Social network analysis. Springer, pp 89–117

  64. Renoust B, Melancon G, Munzner T (2015) Detangler: Visual analytics for multiplex networks. In: Computer graphics forum, vol 34-3. Wiley online library, pp 321–330

  65. Renoust B, Kobayashi T, Ngo TD, Le DD, Satoh S (2016) When face-tracking meets social networks: a story of politics in news videos. Appl Netw Sci 1(1):4

    Article  Google Scholar 

  66. Scaiella U, Ferragina P, Marino A, Ciaramita M (2012) Topical clustering of search results. In: Proceedings of WSDM. ACM, pp 223–232

  67. Selberg E, Etzioni O (1995) Multi-service search and comparison using the metacrawler. In: Inproceedings of the 4th International World Wide Web Conference, pp 195–208

  68. Shi J, Tomasi C (1994) Good features to track. In: Proceedings CVPR’94. IEEE pp 593–600

  69. Shneiderman B (1996) The eyes have it: a task by data type taxonomy for information visualizations. In: Visual languages. IEEE, pp 336–343

  70. Sinclair J, Cardew-Hall M (2008) The folksonomy tag cloud: when is it useful? J Inf Sci 34(1): 15–29

    Article  Google Scholar 

  71. Singer JB (2008) Five ws and an h: Digital challenges in newspaper newsrooms and boardrooms. Int J Media Manag 10(3):122–129

    Article  Google Scholar 

  72. Spink A, Jansen BJ, Wolfram D, Saracevic T (2002) From e-sex to e-commerce: Web search changes. Computer 35(3):107–109

    Article  Google Scholar 

  73. Spink A, Wolfram D, Jansen MB, Saracevic T (2001) Searching the web: The public and their queries. J Amer Soc Inf Sci Technol 52(3):226–234

    Article  Google Scholar 

  74. Teevan J, Alvarado C, Ackerman MS, Karger DR (2004) The perfect search engine is not enough: a study of orienteering behavior in directed search. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, pp 415–422

  75. Viegas FB, Wattenberg M, Feinberg J (2009) Participatory visualization with wordle. IEEE TVCG 15(6):1137–1144

    Google Scholar 

  76. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  77. Wang W, Wang H, Dai G, Wang H (2006) Visualization of large hierarchical data by circle packing. In: Proceedings of SIGCHI, CHI ’06. ACM, New York, pp 517–520. https://doi.org/10.1145/1124772.1124851

  78. Wang M, Li H, Tao D, Lu K, Wu X (2012) Multimodal graph-based reranking for web image search. IEEE Trans Image Process 21(11):4649–4661

    Article  MathSciNet  Google Scholar 

  79. Wasserman S, Faust K (1994) Social network analysis, methods and applications. Structural analysis in the social sciences. Cambridge University Press. https://doi.org/10.1017/CBO9780511815478

  80. Wilson ML, Kules B, Shneiderman B, et al. (2010) From keyword search to exploration: Designing future search interfaces for the web. Found Trends®; Web Sci 2(1):1–97

    Article  Google Scholar 

  81. Wu Y, Provan T, Wei F, Liu S, Ma KL (2011) Semantic-preserving word clouds by seam carving. Comput Graph Forum 30(3):741–750

    Article  Google Scholar 

  82. Xu J, Tao Y, Lin H (2016) Semantic word cloud generation based on word embeddings. In: Pacifvis, pp 239–243

  83. Zhang D, Dong Y (2004) Advanced Web Technologies and Applications: 6th Asia-Pacific Web Conference, APWeb 2004, chap. Semantic, Hierarchical, Online Clustering of Web Search Results. Springer, Berlin, pp 69–78. https://doi.org/10.1007/978-3-540-24655-8_8

  84. Zhang B, Li H, Liu Y, Ji L, Xi W, Fan W, Chen Z, Ma WY (2005) Improving web search results using affinity graph. In: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, pp 504–511

  85. Zhao Y, Karypis G (2002) Evaluation of hierarchical clustering algorithms for document datasets. In: Proc. CIKM, CIKM ’02. ACM, New York, pp 515–524. https://doi.org/10.1145/584792.584877

Download references

Acknowledgments

We would like to dedicate this work in memory of our co-author Marie-Luce Viaud, who left us too soon during the process of completing this work.

Author information

Authors and Affiliations

  1. Institute for Datability Science (IDS), Osaka University, Osaka, Japan

    Benjamin Renoust

  2. National Institute of Informatics (NII), Tokyo, Japan

    Benjamin Renoust & Shin’ichi Satoh

  3. French National Audiovisual Institute (INA), Paris, France

    Haolin Ren & Marie-Luce Viaud

  4. LaBRI CNRS UMR 5800, University of Bordeaux, Bordeaux, France

    Haolin Ren & Guy Melançon

Authors
  1. Benjamin Renoust
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Haolin Ren
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Guy Melançon
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Marie-Luce Viaud
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Shin’ichi Satoh
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Benjamin Renoust or Haolin Ren.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Renoust, B., Ren, H., Melançon, G. et al. A multimedia document browser based on multilayer networks. Multimed Tools Appl 80, 22551–22588 (2021). https://doi.org/10.1007/s11042-020-09872-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09872-9

Keywords