skip to main content
10.1145/1873951.1873956acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Leveraging loosely-tagged images and inter-object correlations for tag recommendation

Published: 25 October 2010 Publication History

Abstract

Large-scale loosely-tagged images (i.e., multiple object tags are given loosely at the image level) are available on Internet, and it is very attractive to leverage such loosely-tagged images for automatic image annotation applications. In this paper, a multi-task structured SVM algorithm is developed to leverage both the inter-object correlations and the loosely-tagged images for achieving more effective training of a large number of inter-related object classifiers. To leverage the loosely-tagged images for object classifier training, each loosely-tagged image is partitioned into a set of image instances (image regions) and a multiple instance learning algorithm is developed for instance label identification by automatically identifying the correspondences between multiple tags (given at the image level) and the image instances. An object correlation network is constructed for characterizing the inter-object correlations explicitly and identifying the inter-related learning tasks automatically. To enhance the discrimination power of a large number of inter-related object classifiers, a multi-task structured SVM algorithm is developed to model the inter-task relatedness more precisely and leverage the inter-object correlations for classifier training. Our experiments on a large number of inter-related object classes have provided very positive results.

References

[1]
Flickr, http://www.flickr.com.
[2]
J. Fan, Y. Shen, N. Zhou, Y. Gao, “Harvesting large-scale weakly-tagged image databases from the web", IEEE CVPR, 2010.
[3]
MSRC, http://research.microsoft.com/.
[4]
Y. Deng, B.S. Manjunath, "Color image segmentation", IEEE CVPR, 1999.
[5]
B. Russell, A. Efros, J. Sivic, W. Freeman, A. Zisserman, "Using multiple segmentations to discover objects and their extent in image collections", IEEE CVPR, 2006.
[6]
B.J. Frey, D. Dueck, "Clustering by passing messages between data points", Science, vol.315, 2007.
[7]
S. Vijayanarasimhan, K. Grauman, "Keywords to visual categories: Multiple-instance learning for weakly supervised object categorization", IEEE CVPR 2008.
[8]
Q. Zhang, W. Yu, S. A. Goldman, J. E. Fritts, "Content-based image retrieval using multiple-instance learning", ICML, 2002.
[9]
O. Maron, A. L. Ratan, "Multiple-instance learning for natural scene classification", ICML, 1998.
[10]
Y. Chen, J. Bi, J. Z. Wang, "MILES: multiple instance learning via embedded instance selection", IEEE Trans. PAMI, vol.28, no.12, pp.1931--1947, 2006.
[11]
M. R. Boutell, J. Luo, X. Shen, C.M. Brown,"Learning multi-label scene classification", Pattern Recognition, vol. 37, no.9, pp. 1757--1771, 2004.
[12]
Z.-H. Zhu, M.-L. Zhang, "Multi-instance multi-label learning with application to scene classification", NIPS, 2006.
[13]
J. Fan, Y. Gao, H. Luo, "Integrating concept ontology and multi-task learning to achieve more effective classifier training for multi-level image annotation", IEEE Trans. on Image Processing, vol. 17, no.3, pp.407--426, 2008.
[14]
T. Evgeniou, C.A. Micchelli, M. Pontil, "Learning multiple tasks with kernel methods", Journal of Machine Learning Research, vol.6, pp.615--637, 2005.
[15]
S. Kumar, M. Hebert, "Discriminative random fields", Intl. Journal of Computer Vision, vol.68, no.2, pp.179--201, 2006.
[16]
J. Yang, Y. Liu, E. X. Ping, A.G. Hauptmann, "Harmonium models for semantic video representation and classification", SIAM Conf. on Data Mining, 2007.
[17]
A. Torralba, K. P. Murphy, W. T. Freeman, "Sharing features: efficient boosting procedures for multiclass object detection", IEEE CVPR, 2004.
[18]
W. Jiang, S.-F. Chang, A. Loui, "Context-based concept fusion with boosted conditional random fields", IEEE ICASSP, 2007.
[19]
J. Tang, X. Hua, M. Wang, Z. Gu, G. Qi, X. Wu, "Correlative linear neighborhood propagation for video annotation", IEEE Trans. on SMC, vol. 39, no.2, pp.409--416, 2009.
[20]
J. Liu, M. Li, W.-Y. Ma, Q. Liu, H. Lu, "An adaptive graph model for automatic image annotation", ACM Multimedia Workshop on MIR, 2006.
[21]
G.-J. Qi, X.-S. Hua, Y. Rui, J. Tang, T. Mei, H.-J. Zhang, "Correlative multi-label video annotation", ACM Multimedia, pp.17--26, 2007.
[22]
Z. Zha, X.-S. Hua, T. Mei, J. Wang, G.-J. Qi, Z. Wang, "Joint multi-label multi-instance learning for image classification", IEEE CVPR, 2008.
[23]
I. Tsochantaridis, T. Joachims, T. Hofmann,Y. Altun, "Large margin methods for structured and interdependent output variables", Journal of Machine Learning Research, vol.6, pp.1453--1484, 2005.
[24]
T. Joachims, T. Finley, C. Yu, "Cutting-plane training of structural SVMs", Machine Learning, vol. 77, no.1, pp.27--59, 2009.
[25]
J.A. Hanley, B.J. Mcneil, "The meaning and use of the area under a receiver operating characteristic (roc) curve", Radiology, vol.143, no.1, pp.29--36, 1982.
[26]
J. Fan, Y. Gao, H. Luo, "Multi-level annotation of natural scenes using dominant image components and semantic image concepts", ACM Multimedia, 2004.
[27]
H.P. Graf, E. Cosatto, L. Bottou, I. Durdanovic, V. Vapnik, "Parallel support vector machines: The cascade SVM", NIPS 2004.
[28]
R. Fan, P. Chen, C.-J. Lin, "Working set selection using the second order information for training SVM", Journal of Machine Learning Research, vol. 6, pp.1889--1918, 2005.

Cited By

View all
  • (2017)iPrivacy: Image Privacy Protection by Identifying Sensitive Objects via Deep Multi-Task LearningIEEE Transactions on Information Forensics and Security10.1109/TIFS.2016.263609012:5(1005-1016)Online publication date: May-2017
  • (2017)Social tag relevance learning via ranking-oriented neighbor votingMultimedia Tools and Applications10.1007/s11042-016-3512-176:6(8831-8857)Online publication date: 1-Mar-2017
  • (2017)Personalized Tag RecommendationUnderstanding-Oriented Multimedia Content Analysis10.1007/978-981-10-3689-7_4(75-99)Online publication date: 27-May-2017
  • Show More Cited By

Index Terms

  1. Leveraging loosely-tagged images and inter-object correlations for tag recommendation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '10: Proceedings of the 18th ACM international conference on Multimedia
    October 2010
    1836 pages
    ISBN:9781605589336
    DOI:10.1145/1873951
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 25 October 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. loosely-tagged images
    2. multi-task structured svm
    3. multiple instance learning
    4. object correlation network

    Qualifiers

    • Research-article

    Conference

    MM '10
    Sponsor:
    MM '10: ACM Multimedia Conference
    October 25 - 29, 2010
    Firenze, Italy

    Acceptance Rates

    Overall Acceptance Rate 554 of 2,551 submissions, 22%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)2
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 24 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2017)iPrivacy: Image Privacy Protection by Identifying Sensitive Objects via Deep Multi-Task LearningIEEE Transactions on Information Forensics and Security10.1109/TIFS.2016.263609012:5(1005-1016)Online publication date: May-2017
    • (2017)Social tag relevance learning via ranking-oriented neighbor votingMultimedia Tools and Applications10.1007/s11042-016-3512-176:6(8831-8857)Online publication date: 1-Mar-2017
    • (2017)Personalized Tag RecommendationUnderstanding-Oriented Multimedia Content Analysis10.1007/978-981-10-3689-7_4(75-99)Online publication date: 27-May-2017
    • (2017)Collaborative Filtering Fusing Label Features Based on SDAEAdvances in Data Mining. Applications and Theoretical Aspects10.1007/978-3-319-62701-4_17(223-236)Online publication date: 1-Jul-2017
    • (2016)A survey of tag-based information retrievalInternational Journal of Multimedia Information Retrieval10.1007/s13735-016-0115-66:2(99-113)Online publication date: 9-Dec-2016
    • (2016)Integrating multiple types of features for event identification in social imagesMultimedia Tools and Applications10.1007/s11042-014-2436-x75:6(3301-3322)Online publication date: 1-Mar-2016
    • (2016)Modelling multilevel data in multimediaMultimedia Tools and Applications10.1007/s11042-014-2394-375:9(4933-4955)Online publication date: 1-May-2016
    • (2015)Parallel AP Clustering and Re-ranking for Automatic Image-Text Alignment and Large-Scale Web Image SearchProceedings of the 5th ACM on International Conference on Multimedia Retrieval10.1145/2671188.2749294(451-454)Online publication date: 22-Jun-2015
    • (2015)Content-Irrelevant Tag Cleansing via Bi-Layer Clustering and Peer CooperationJournal of Signal Processing Systems10.1007/s11265-014-0895-y81:1(29-44)Online publication date: 1-Oct-2015
    • (2015)An efficient framework of Bregman divergence optimization for co-ranking images and tags in a heterogeneous networkMultimedia Tools and Applications10.1007/s11042-014-1873-x74:15(5635-5660)Online publication date: 1-Jul-2015
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media