Stevan Rudinac
ABSTRACT
In this paper we investigate whether and how the human choice of images for summarizing a visual collection is influenced by the semantic concepts depicted in them. More specifically, by analysing a large collection of human-created visual summaries obtained through crowdsourcing, we aim at automatically identifying the objects, settings, actions and events that make an image a good candidate for inclusion in a visual summary. Informed by the outcomes of this analysis, we show that the distribution of semantic concepts can be successfully utilized for learning to rank the images based on their likelihood of inclusion in the summary by a human, and that it can be easily combined with other features related to image content, context, aesthetic appeal and sentiment. Our experiments demonstrate the promise of using semantic concept detectors for automatically analysing crowdsourced user preferences at a large scale.
- D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In ACM MM '13, pages 223--232, 2013. Google Scholar
Digital Library
- Y.-L. Boureau, J. Ponce, and Y. LeCun. A theoretical analysis of feature pooling in visual recognition. In ICML '10, pages 111--118, 2010.Google Scholar
- L. Cao, J. Luo, A. Gallagher, X. Jin, J. Han, and T. Huang. A worldwide tourism recommendation system based on geotagged web photos. IEEE ICASSP'10, pages 2274--2277, 2010.Google Scholar
- J. Carbonell and J. Goldstein. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In ACM SIGIR '98, pages 335--336, 1998. Google ScholarDigital Library
- O. Chapelle and S. Keerthi. Effcient algorithms for ranking with svms. Inform. Retrieval, 13(3):201--215, 2010. Google ScholarDigital Library
- B. J. Frey and D. Dueck. Clustering by passing messages between data points. Science, 315:972--976, 2007.Google ScholarCross Ref
- B. Ionescu, A.-L. Radu, M. Mendendez, H. Muller, A. Popescu, and B. Loni. Div400: A social image retrieval result diversification dataset. In ACM MMSys '14, pages 29--34, 2014. Google ScholarDigital Library
- L. S. Kennedy and M. Naaman. Generating diverse and representative image search results for landmarks. In WWW '08, pages 297--306, 2008. Google ScholarDigital Library
- A. Khosla, A. Das Sarma, and R. Hamid. What makes an image popular? In WWW '14, pages 867--876, 2014. Google ScholarDigital Library
- M. Lestari Paramita, M. Sanderson, and P. Clough. Diversity in photo retrieval: Overview of the imageclefphoto task 2009. In LNCS, volume 6242, pages 45--59. 2010. Google ScholarDigital Library
- A. Nenkova, R. Passonneau, and K. McKeown. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Trans. Speech Lang. Process., 4(2), 2007. Google ScholarDigital Library
- K. Owczarzak and H. T. Dang. Overview of the TAC 2011 summarization track: Guided task and AESOP task. In TAC '11, 2011.Google Scholar
- Y. Pang, Q. Hao, Y. Yuan, T. Hu, R. Cai, and L. Zhang. Summarizing tourist destinations by mining user-generated travelogues and photos. Comput. Vis. Image Und., 115:352--363, 2011. Google ScholarDigital Library
- S. Rudinac, M. Larson, and A. Hanjalic. Learning crowdsourced user preferences for visual summarization of image collections. IEEE Trans. Multimedia, 15(6):1231--1243, 2013. Google ScholarDigital Library
- C. G. M. Snoek, K. E. A. van de Sande, A. Habibian, S. Kordumova, Z. Li, M. Mazloom, S. L. Pintea, R. Tao, D. C. Koelma, and A. W. M. Smeulders. The mediamill trecvid 2012 semantic video search engine. In TRECVID Workshop, 2012.Google Scholar
Index Terms
- Making use of Semantic Concept Detection for Modelling Human Preferences in Visual Summarization
Recommendations
Transductive learning over automatically detected themes for multi-document summarization
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information RetrievalWe propose a new method for query-biased multi-document summarization, based on sentence extraction. The summary of multiple documents is created in two steps. Sentences are first clustered; where each cluster corresponds to one of the main themes ...
Video summarization with semantic concept preservation
MUM '11: Proceedings of the 10th International Conference on Mobile and Ubiquitous MultimediaA compelling video summarization should allow viewers to understand the summary content and recover the original plot correctly. To this end, we materialize the abstract elements that are cognitively informative for viewers as concepts. They implicitly ...
Combining concept hierarchies and statistical topic models
CIKM '08: Proceedings of the 17th ACM conference on Information and knowledge managementStatistical topic models provide a general data-driven framework for automated discovery of high-level knowledge from large collections of text documents. While topic models can potentially discover a broad range of themes in a data set, the ...
Comments