Skip to main content
SpringerLink
Log in

Graph-based multimodal clustering for social multimedia

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Real world datasets often consist of data expressed through multiple modalities. Clustering such datasets is in most cases a challenging task as the involved modalities are often heterogeneous. In this paper we propose a graph-based multimodal clustering approach. The proposed approach utilizes an example relevant clustering in order to learn a model of the “same cluster” relationship between a pair of items. This model is subsequently used in order to organize the items of the collection to be clustered in a graph, where the nodes represent the items and a link between a pair of nodes exists if the model predicted that the corresponding pair of items belong to the same cluster. Eventually, a graph clustering algorithm is applied on the graph in order to produce the final clustering. The proposed approach is applied on two problems that are typically treated using clustering techniques; in particular, it is applied on the problem of detecting social events and to the problem of discovering different landmark views in collections of social multimedia.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Aggarwal C, Subbian K (2014) Evolutionary network analysis: a survey. ACM Comput Surv 47(1):10:1–10:36. doi:10.1145/2601412

    Article  MATH  Google Scholar 

  2. Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-Up Robust features (SURF). Comp Vis Image Underst 110(3):346–359. doi:10.1016/j.cviu.2007.09.014

    Article  Google Scholar 

  3. Becker H, Naaman M, Gravano L (2010) Learning similarity metrics for event identification in social media. In: Proceedings of the third ACM international conference on Web search and data mining, WSDM ’10. ACM, New York, pp 291–300. doi:10.1145/1718487.1718524

    Chapter  Google Scholar 

  4. Bekkerman R, Jeon J (2007) Multi-modal clustering for multimedia collections. In: CVPR

  5. Blei D , Ng A, Jordan M (2003) Latent dirichlet allocation. J Mach Learn Res 3(1532–4435):993–1022. http://dl.acm.org/citation.cfm?id=944919.944937

    MATH  Google Scholar 

  6. Boteanu B, Mironica I, Ionescu B (2015) Hierarchical clustering pseudo-relevance feedback for social image search result diversification. In: 2015 13th international workshop on Content-based multimedia indexing (CBMI), pp 1–6

  7. Brenner M, Izquierdo E (2011) Mediaeval benchmark: social event detection in collaborative photo collections. In: Mediaeval, CEUR workshop proceedings

  8. Cai X, Nie F, Huang H, Kamangar F (2011) Heterogeneous image feature integration via multi-modal spectral clustering. In: 2011 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1977–1984. doi:10.1109/CVPR.2011.5995740

  9. Clauset A, Newman MEJ, Moore C (2004) Finding community structure in very large networks. Phys Rev E:1–6. doi:10.1103/PhysRevE.70.066111

  10. Dang-Nguyen D, Piras L, Giacinto G, Boato G, Natale FGBD (2014) Retrieval of diverse images by pre-filtering and hierarchical clustering. In: Working notes proceedings of the mediaeval 2014 workshop, Barcelona

  11. Fortunato S (2010) Community detection in graphs. Phys Rep 486(3-5):75–174

    Article  MathSciNet  Google Scholar 

  12. Gînsca A, Popescu A, Rekabsaz N (2014) CEA List’s participation at the mediaeval 2014 retrieving diverse social images task. In: Working notes proceedings of the mediaeval 2014 workshop, Barcelona

  13. Goder A, Filkov V (2008) Consensus clustering algorithms: Comparison and refinement. In: Munro JI, Wagner D (eds) Proceedings of the workshop on algorithm engineering and experiments, ALENEX 2008. SIAM, San Francisco, pp 109–117

  14. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explorations Newsletter 11(1):10–18. doi:10.1145/1656274.1656278

    Article  Google Scholar 

  15. Ionescu B, Popescu A, Lupu M, Gînsca A, Müller H (2014) Retrieving diverse social images at mediaeval 2014: challenge, dataset and evaluation. In: Working notes proceedings of the mediaeval 2014 workshop, Barcelona

  16. Jégou H, Douze M, Schmid C, Pérez P (2010) Aggregating local descriptors into a compact image representation. In: 23rd IEEE conference on computer vision & pattern recognition (CVPR ’10). doi:10.1109/CVPR.2010.5540039. IEEE Computer Society, San Francisco, pp 3304–3311

  17. Jegou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33(1):117–128

    Article  Google Scholar 

  18. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. arXiv:1408.5093

  19. Jian M, Dong J, Ma J (2011) Image retrieval using wavelet-based salient regions. Imaging Sci J 59(4):219–231. doi:10.1179/136821910X12867873897355

    Article  Google Scholar 

  20. Khalidov V, Forbes F, Horaud RP (2011) Conjugate mixture models for clustering multimodal data. Neural Comput 23(2):517–557. http://perception.inrialpes.fr/Publications/2011/KFH11

    Article  MathSciNet  MATH  Google Scholar 

  21. Li Y, Crandall DJ, Huttenlocher DP (2009) Landmark classification in large-scale image collections. In: IEEE 12th international conference on computer vision, ICCV 2009. IEEE, Kyoto, pp 1957–1964. doi:10.1109/ICCV.2009.5459432

    Google Scholar 

  22. Liu X, Troncy R, Huet B (2011) Using social media to identify events. In: WSM’11, ACM Multimedia 3rd workshop on social media, november 18-december 1st, 2011, Scottsdale

  23. Manning CD, Raghavan P, Schütze H (2008) Introduction to information retrieval. Cambridge University Press, New York

    Book  MATH  Google Scholar 

  24. Nguyen NP, Dinh TN, Xuan Y, Thai MT (2011) Adaptive algorithms for detecting community structure in dynamic social networks. In: INFOCOM 2011. 30th IEEE International Conference on Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, 10-15 April 2011. IEEE, Shanghai, pp 2282–2290. doi:10.1109/INFCOM.2011.5935045

    Google Scholar 

  25. Papadopoulos S, Kompatsiaris Y, Vakali A, Spyridonos P (2012) Community detection in social media. Data Min Knowl Disc 24(3):515–554

    Article  Google Scholar 

  26. Papadopoulos S, Schinas E, Mezaris V, Troncy R, Kompatsiaris Y (2012) Social Event Detection at MediaEval 2012: challenges, dataset and evaluation. In: Mediaeval. Workshop, Pisa, p 2012

  27. Papadopoulos S, Troncy R, Mezaris V, Huet B, Kompatsiaris I (2011) Social event detection at mediaeval 2011: challenges, dataset and evaluation Mediaeval, CEUR workshop proceedings

  28. Papadopoulos S, Zigkolis C, Kompatsiaris Y, Vakali A (2011) CERTH@Mediaeval 2011 social event detection task. In: Mediaeval, CEUR workshop proceedings

  29. Papadopoulos S, Zigkolis C, Kompatsiaris Y, Vakali A (2011) Cluster-based landmark and event detection for tagged photo collections. Multimedia, IEEE 18 (1):52–63. doi:10.1109/MMUL.2010.68

    Article  Google Scholar 

  30. Petkos G, Papadopoulos S, Kompatsiaris Y (2012) Social event detection using multimodal clustering and integrating supervisory signals. In: Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, ICMR ’12. ACM, New York, pp 23:1–23:8. doi:10.1145/2324796.2324825

  31. Petkos G, Papadopoulos S, Mezaris V, Kompatsiaris Y (2014) Social event detection at mediaeval 2014: challenges, datasets, and evaluation. In: Working notes proceedings of the mediaeval 2014 workshop, Barcelona

  32. Petkos G, Papadopoulos S, Schinas E, Kompatsiaris Y (2014) Graph-based multimodal clustering for social event detection in large collections of images Multimedia modeling - 20th anniversary international conference, MMM 2014, Dublin, Ireland, January 6–10, 2014, Proceedings, Part I, pp 146–158

  33. Phuvipadawat S, Murata T (2010) Breaking news detection and tracking in twitter. IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology 3:120–123. http://doi.ieeecomputersociety.org/10.1109/WI-IAT.2010.205

    Article  Google Scholar 

  34. Rendle S, Schmidt-Thieme L (2008) Scaling record linkage to non-uniform distributed class sizes. In: Proceedings of the 12th Pacific-Asia conference on Advances in knowledge discovery and data mining, PAKDD’08. Springer, Berlin, pp 308–319. http://dl.acm.org/citation.cfm?id=1786574.1786605

    Chapter  Google Scholar 

  35. Reuter T, Cimiano P (2012) Event-based classification of social media streams. In: Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, ICMR ’12. ACM, New York, pp 22:1–22:8. doi:10.1145/2324796.2324824

    Google Scholar 

  36. Reuter T, Papadopoulos S, Petkos G, Mezaris V, Kompatsiaris Y, Cimiano P, Vries CMD, Geva S (2013) Social event detection at mediaeval 2013: challenges, datasets, and evaluation. In: Proceedings of the mediaeval 2013 multimedia benchmark workshop, Barcelona

  37. Samangooei S, Hare JS, Dupplaw D, Niranjan M, Gibbins N, Lewis PH, Davies J, Jain N, Preston J (2013) Social event detection via sparse multi-modal feature selection and incremental density based clustering. In: Proceedings of the mediaeval 2013 multimedia benchmark workshop, Barcelona

  38. Snoek CGM, Worring M, Smeulders AWM (2005) Early versus late fusion in semantic video analysis. In: Proceedings of the 13th annual ACM international conference on Multimedia, MULTIMEDIA ’05. ACM, New York, pp 399–402. doi:10.1145/1101149.1101236

    Chapter  Google Scholar 

  39. Spiliopoulou M., Aggarwal CC (2011) Evolution in social networks: a survey. In: Social network data analytics, Springer, pp 149–175

  40. Xu X, Yuruk N, Feng Z, Schweiger TAJ (2007) Scan: a structural clustering algorithm for networks. In: Proceedings of the 13th ACM SIGKDD, KDD ’07. ACM, NY, pp 824–833. doi:10.1145/1281192.1281280

  41. Ye Z, Hu S, Yu J (2008) Adaptive clustering algorithm for community detection in complex networks. Phys Rev E 78(4)

  42. Zaharieva M, Schopfhauser D, del Fabro M, Zeppelzauer M (2014) Clustering and retrieval of social events in flickr. In: Working notes proceedings of the mediaeval 2014 workshop, Barcelona

  43. Zaharieva M, Schwab P (2014) A unified framework for retrieving diverse social images. In: Working notes proceedings of the mediaeval 2014 workshop, Barcelona

  44. Zhang T, Ramakrishnan R, Livny M (1996) Birch: An efficient data clustering method for very large databases. SIGMOD Rec 25(2):103–114. doi:10.1145/235968.233324

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CERTH - ITI, Thessaloniki, Greece

    Georgios Petkos, Manos Schinas, Symeon Papadopoulos & Yiannis Kompatsiaris

Authors
  1. Georgios Petkos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manos Schinas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Symeon Papadopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yiannis Kompatsiaris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Georgios Petkos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Petkos, G., Schinas, M., Papadopoulos, S. et al. Graph-based multimodal clustering for social multimedia. Multimed Tools Appl 76, 7897–7919 (2017). https://doi.org/10.1007/s11042-016-3378-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3378-2

Keywords

Search

Navigation