Skip to main content
SpringerLink
Log in

User-adaptive image retrieval via fusing pointwise and pairwise labels

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

In many Web image retrieval applications, adapting the retrieval results according to some model of the user is a desired feature as the returned images can be made specifically relevant to a user’s needs. Making retrieval user-adaptive faces several practical challenges, including the ambiguity of user query, the lack of user-adaptive training data, and lack of proper mechanisms for supporting adaptive learning. To address some of these challenges, we propose a hybrid learning strategy that fuses knowledge from both pointwise and pairwise training data into one framework for attribute-based, user-adaptive image retrieval. An online learning algorithm is developed for updating the ranking performance based on user feedback. The framework is also derived into a kernel form allowing easy application of kernel techniques. We use both synthetic and real-world datasets to evaluate the performance of the proposed algorithm. Comparison with other state-of-the-art approaches suggests that our method achieves obvious performance gains over ranking and zero-shot learning. Further, our online learning algorithm was found to be able to deliver much better performance than batch learning, given the same elapsed running time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Berg TL, Berg AC, Shih J (2010) Automatic attribute discovery and characterization from noisy web data. In: Proc. of ECCV’10, Springer-Verlag, Berlin, pp 663–676. http://dl.acm.org/citation.cfm?id=1886063.1886114

  2. Burges C, Shaked T, Renshaw E, Lazier A, Deeds M, Hamilton N, Hullender G (2005) Learning to rank using gradient descent. In: Proc. of ICML ’05, pp 89–96. ACM. doi:10.1145/1102351.1102363

  3. Cao Y, Xu J, Liu TY, Li H, Huang Y, Hon HW (2006) Adapting ranking svm to document retrieval. In: Proc. of SIGIR ’06, ACM, pp 186–193. doi:10.1145/1148170.1148205

  4. Chen L, Zhang P, Li B (2014) Instructive video retrieval based on hybrid ranking and attribute learning: a case study on surgical skill training. In: Proceedings of the ACM International Conference on Multimedia, MM ’14, ACM, New York, pp 1045–1048. doi:10.1145/2647868.2655050

  5. Chen L, Zhang P, Li B (2015) Fusing pointwise and pairwise labels for supporting user-adaptive image retrieval. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, ICMR ’15, ACM, New York, pp 67–74. doi:10.1145/2671188.2749358

  6. Chen L, Zhang Q, Li B (2014) Predicting multiple attributes via relative multi-task learning. In: Proc. of CVPR ’14, pp 1027–1034

  7. Elahi N, Karlsen R, Holsbø EJ (2013) Personalized photo recommendation by leveraging user modeling on social network. In: Proc. of IIWAS ’13, ACM, pp 68:68–68:71. doi:10.1145/2539150.2539232

  8. Farhadi A, Endres I, Hoiem D, Forsyth D (2009) Describing objects by their attributes. In: Proc. of CVPR ’09, pp 1778–1785. doi:10.1109/CVPR.2009.5206772

  9. Freund Y, Iyer R, Schapire RE, Singer Y (2003) An efficient boosting algorithm for combining preferences. J Mach Learn Res 4:933–969. http://dl.acm.org/citation.cfm?id=945365.964285

  10. Joachims T (2002) Optimizing search engines using clickthrough data. In: Proc. of KDD ’02, pp 133–142. ACM. doi:10.1145/775047.775067

  11. Kimeldorf G, Wahba G (1971) Some results on tchebycheffian spline functions. J Math Anal Appl 33(1):82–95. doi:10.1016/0022-247X(71)90184-3. http://www.sciencedirect.com/science/article/pii/0022247X71901843

  12. Kovashka A, Grauman K (2013) Attribute adaptation for personalized image search. In: Proc. of ICCV ’13, pp 3432–3439. doi:10.1109/ICCV.2013.426

  13. Kovashka A, Parikh D, Grauman K (2012) Whittlesearch: Image search with relative attribute feedback. In: Proc. of CVPR’12, pp 2973–2980. doi:10.1109/CVPR.2012.6248026

  14. Kumar N, Belhumeur P, Nayar S (2008) Facetracer: A search engine for large collections of images with faces. In: Proc. of ECCV ’08, Springer-Verlag, Berlin, pp 340–353

  15. Kumar N, Berg A, Belhumeur P, Nayar S (2009) Attribute and simile classifiers for face verification. In: Proc. of CVPR ’09, pp 365–372. doi:10.1109/ICCV.2009.5459250

  16. Lampert C, Nickisch H, Harmeling S (2009) Learning to detect unseen object classes by between-class attribute transfer. In: Proc. of CVPR ’09, pp 951–958. doi:10.1109/CVPR.2009.5206594

  17. Li P, Wu Q, Burges CJ (2008) Mcrank: Learning to rank using multiple classification and gradient boosting. In: Proc. of NIPS ’08, Curran Associates, Inc., pp 897–904. http://papers.nips.cc/paper/3270-mcrank-learning-to-rank-using-multiple-classification-and-gradient-boosting.pdf

  18. Moon T, Smola A, Chang Y, Zheng Z (2010) Intervalrank: Isotonic regression with listwise and pairwise constraints. In: WSDM. doi:10.1145/1718487.1718507

  19. Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vision 42(3):145–175. doi:10.1023/A:1011139631724

    Article  MATH  Google Scholar 

  20. Parikh D, Grauman K (2011) Relative attributes. In: Proc. of ICCV ’11, pp 503–510. doi:10.1109/ICCV.2011.6126281

  21. Renjifo C, Carmen C (2012) The discounted cumulative margin penalty: rank-learning with a list-wise loss and pair-wise margins. In: MLSP. doi:10.1109/MLSP.2012.6349807

  22. Sang J, Xu C, Lu D (2012) Learn to personalized image search from the photo sharing websites. Multim IEEE Trans 14(4):963–974. doi:10.1109/TMM.2011.2181344

    Article  Google Scholar 

  23. Sculley D (2010) Combined regression and ranking. In: Proc. of SIGKDD’10, ACM, New York, pp 979–988. doi:10.1145/1835804.1835928

  24. Shalev-Shwartz S, Singer Y, Srebro N (2007) Pegasos: Primal estimated sub-gradient solver for svm. In: Proc. of ICML ’07, ACM, pp 807–814. doi:10.1145/1273496.1273598

  25. Shashua A, Levin A (2003) Ranking with large margin principle: two approaches. In: Proc. of NIPS ’03, pp 961–968. MIT Press. http://papers.nips.cc/paper/2269-ranking-with-large-margin-principle-two-approaches.pdf

  26. Shen X, Tan B, Zhai C (2005) Context-sensitive information retrieval using implicit feedback. In: Proc. of SIGIR ’05, ACM, pp 43–50. doi:10.1145/1076034.1076045

  27. Song Y, Wang H, He X (2014) Adapting deep ranknet for personalized search. In: Proc. of WSDM ’14, ACM, pp 83–92.doi:10.1145/2556195.2556234

  28. Sontag D, Collins-Thompson K, Bennett PN, White RW, Dumais S, Billerbeck B (2012) Probabilistic models for personalizing web search. In: Proc. of WSDM ’12, ACM, pp 433–442. doi:10.1145/2124295.2124348

  29. Tsai MF, Liu TY, Qin T, Chen HH, Ma WY (2007) Frank: A ranking method with fidelity loss. In: Proc. of SIGIR ’07, ACM, pp 383–390. doi:10.1145/1277741.1277808

  30. Vaquero DA, Feris RS, Tran D, Brown L, Hampapur A, Turk M (2009) Attribute-based people search in surveillance environments. In: Applications of computer vision (WACV), Workshop on Dec 2009, pp 1–8. doi:10.1109/WACV.2009.5403131

  31. Wang H, He X, Chang MW, Song Y, White RW, Chu W (2013) Personalized ranking model adaptation for web search. In: Proc. of SIGIR ’13, ACM, pp 323–332. doi:10.1145/2484028.2484068

  32. Wang Y, Mori G (2010) A discriminative latent model of object classes and attributes. In: Proc. of ECCV’10, Springer-Verlag, pp 155–168. http://dl.acm.org/citation.cfm?id=1888150.1888163

  33. White RW, Bennett PN, Dumais ST (2010) Predicting short-term interests using activity-based search context. In: Proc. of CIKM ’10, ACM, pp 1009–1018. doi:10.1145/1871437.1871565

  34. Zhang H, Zha ZJ, Yan S, Bian J, Chua TS (2012) Attribute feedback. In: Proceedings of the 20th ACM international conference on multimedia, MM ’12, ACM, New York, pp. 79–88. doi:10.1145/2393347.2393365

  35. Zhang H, Zha ZJ, Yang Y, Yan S, Gao Y, Chua TS (2013) Attribute-augmented semantic hierarchy: towards bridging semantic gap and intention gap in image retrieval. In: Proceedings of the 21st ACM international conference on multimedia, MM ’13, ACM, New York, pp 33–42. doi:10.1145/2502081.2502093

Download references

Acknowledgments

The work was supported in part by a grant from the Army Research Office (ARO). Any opinions expressed in this material are those of the authors and do not necessarily reflect the views of the ARO.

Author information

Authors and Affiliations

  1. Computer Science, Arizona State University, Tempe, AZ, USA

    Lin Chen & Baoxin Li

  2. Alibaba Group, Hangzhou, Zhejiang, People’s Republic of China

    Peng Zhang

Authors
  1. Lin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Baoxin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lin Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, L., Zhang, P. & Li, B. User-adaptive image retrieval via fusing pointwise and pairwise labels. Int J Multimed Info Retr 5, 19–33 (2016). https://doi.org/10.1007/s13735-015-0092-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13735-015-0092-1

Keywords

Search

Navigation