Skip to main content
SpringerLink
Log in

Contextual modeling on auxiliary points for robust image reranking

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Image reranking is an effective post-processing step to adjust the similarity order in image retrieval. As key components of initialized ranking lists, top-ranked neighborhoods of a given query usually play important roles in constructing dissimilarity measure. However, the number of pertinent candidates varies with respect to different queries. Thus the images with short lists of ground truth suffer from insufficient contextual information. It consequently introduces noises when using k-nearest neighbor rule to define the context. In order to alleviate this problem, this paper proposes auxiliary points which are added as assistant neighbors in an unsupervised manner. These extra points act on revealing implicit similarity in the metric space and clustering matched image pairs. By isometrically embedding each constructed metric space into the Euclidean space, the image relationships on underlying topological manifolds are locally represented by distance descriptions. Furthermore, by combining Jaccard index with our auxiliary points, we present a contextual modeling on auxiliary points (CMAP) method for image reranking.With richer contextual activations, the Jaccard similarity coefficient defined by local distribution achieves more reliable outputs as well as more stable parameters. Extensive experiments demonstrate the robustness and effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Jégou H, Douze M, Schmid C. Improving bag-of-features for large scale image search. International Journal of Computer Vision, 2010, 87(3): 316–336

    Article  Google Scholar 

  2. Song G, Tan X. Hierarchical deep hashing for image retrieval. Frontiers of Computer Science, 2017, 11(2): 253–265

    Article  Google Scholar 

  3. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110

    Article  Google Scholar 

  4. Li Y, Kong X, Zheng L, Tian Q. Exploiting hierarchical activations of neural network for image retrieval. In: Proceedings of the 24nd ACM International Conference on Multimedia. 2016, 132–136

    Chapter  Google Scholar 

  5. Jégou H, Perronnin F, Douze M, Sanchez J, Perez P, Schmid C. Aggregating local image descriptors into compact codes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(9): 1704–1716

    Article  Google Scholar 

  6. Liu Z, Wang S, Tian Q. Fine-residual VLAD for image retrieval. Neurocomputing, 2016, 173: 1183–1191

    Article  Google Scholar 

  7. Zheng L, Wang S, Liu Z, Tian Q. Packing and padding: coupled multiindex for accurate image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1939–1946

    Google Scholar 

  8. Chum O, Mikulik A, Perdoch M, Matas J. Total recall II: query expansion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 889–896

    Google Scholar 

  9. Bai S, Bai X, Tian Q, Latecki L J. Regularized diffusion process for visual retrieval. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2017, 3967–3973

    Google Scholar 

  10. Bai S, Zhou Z, Wang J, Bai X, Latecki L J, Tian Q. Ensemble diffusion for retrieval. In: Proceedings of the IEEE Conference on Computer Vision. 2017, 774–783

    Google Scholar 

  11. Köknar-Tezel S, Latecki L J. Improving svm classification on imbalanced time series data sets with ghost points. Knowledge and Information Systems, 2011, 28(1): 1–23

    Article  Google Scholar 

  12. Jégou H, DouzeM, Schmid C. On the burstiness of visual elements. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1169–1176

    Google Scholar 

  13. Zhu Y, Jiang J, Han W, Ding Y, Tian Q. Interpretation of users’ feedback via swarmed particles for content-based image retrieval. Information Sciences, 2017, 375: 246–257

    Article  Google Scholar 

  14. Zheng L, Yang Y, Tian Q. Sift meets CNN: a decade survey of instance retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(5): 1224–1244

    Article  Google Scholar 

  15. Chen K, Ding G, Han J. Attribute-based supervised deep learning model for action recognition. Frontiers of Computer Science, 2017, 11(2): 219–229

    Article  Google Scholar 

  16. Gong Y, Wang L, Guo R, Lazebnik S. Multi-scale orderless pooling of deep convolutional activation features. In: Proceedings of the European Conference on Computer Vision. 2014, 392–407

    Google Scholar 

  17. Babenko A, Slesarev A, Chigorin A, Lempitsky V. Neural codes for image retrieval. In: Proceedings of the European Conference on Computer Vision. 2014, 584–599

    Google Scholar 

  18. Kalantidis Y, Mellina C, Osindero S. Cross-dimensional weighting for aggregated deep convolutional features. In: Proceedings of the European Conference on Computer Vision. 2016, 685–701

    Google Scholar 

  19. Ng J Y, Yang F, Davis L S. Exploiting local features from deep networks for image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2015, 53–61

    Google Scholar 

  20. Babenko A, Lempitsky V. Aggregating local deep features for image retrieval. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 1269–1277

    Google Scholar 

  21. Chum O, Philbin J, Sivic J, Isard M, Zisserman A. Total recall: automatic query expansion with a generative feature model for object retrieval. In: Proceedings of the IEEE International Conference on Computer Vision. 2007, 1–8

    Google Scholar 

  22. Qin D, Gammeter S, Bossard L, Quack T, Gool L V. Hello neighbor: accurate object retrieval with k-reciprocal nearest neighbors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 777–784

    Google Scholar 

  23. Jégou H, Schmid C, Harzallah H, Verbeek J. Accurate image search using the contextual dissimilarity measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(1): 2–11

    Article  Google Scholar 

  24. Sun S, Li Y, Zhou W, Tian Q, Li H. Local residual similarity for image re-ranking. Information Sciences, 2017, 417: 143–153

    Article  Google Scholar 

  25. Arandjelovi´c R, Zisserman A. Three things everyone should know to improve object retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 2911–2918

    Google Scholar 

  26. Yang X, Prasad L, Latecki L J. Affinity learning with diffusion on tensor product graph. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 28–38

    Article  Google Scholar 

  27. Donoser M, Bischof H. Diffusion processes for retrieval revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1320–1327

    Google Scholar 

  28. Bai S, Bai X, Tian Q. Scalable person re-identification on supervised smoothed manifold. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 3356–3365

    Google Scholar 

  29. Bai S, Bai X, Tian Q, Latecki L J. Regularized diffusion process on bidirectional context for object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 99: 1

    Google Scholar 

  30. Bai S, Bai X. Sparse contextual activation for efficient visual reranking. IEEE Transactions on Image Processing, 2016, 25(3): 1056–1069

    Article  MathSciNet  MATH  Google Scholar 

  31. Nister D, Stewenius H. Scalable recognition with a vocabulary tree. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2006, 2161–2168

    Google Scholar 

  32. Jégou H, Douze M, Schmid C. Hamming embedding and weak geometric consistency for large scale image search. In: Proceedings of the European Conference on Computer Vision. 2008, 304–317

    Google Scholar 

  33. Balntas V, Lenc K, Vedaldi A, Mikolajczyk K. Hpatches: a benchmark and evaluation of handcrafted and learned local descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 3852–3861

    Google Scholar 

  34. Philbin J, Chum O, Isard M, Sivic J, Zisserman A. Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8

    Google Scholar 

  35. Huiskes MJ, Lew MS. The mir flickr retrieval evaluation. In: Proceedings of the ACM International Conference on Multimedia Information Retrieval. 2008, 39–43

    Google Scholar 

  36. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T. Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia. 2014, 675–678

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (NSFC) (Grant No. 71421001), in part by the National Natural Science Foundation of China (NSFC) (Grant Nos. 61502073, 61772111 and 61429201), in part by the Fundamental Research Funds for the Central Universities (DUT18JC02), and in part to Dr. Qi Tian by ARO (W911NF-15- 1-0290) and Faculty Research Gift Awards by NEC Laboratories of America and Blippar. This work was supported in part by the China Scholarship Council.

Author information

Authors and Affiliations

  1. School of Information and Communication Engineering, Dalian University of Technology, Dalian, 116024, China

    Ying Li, Xiangwei Kong & Haiyan Fu

  2. Department of Computer Science, University of Texas at San Antonio, San Antonio, Texas, 78249, USA

    Qi Tian

Authors
  1. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiangwei Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haiyan Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qi Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiangwei Kong.

Additional information

Ying Li received her BE degree in electronics and information engineering and the MS degree in signal and information processing from Dalian University of Technology, in 2012 and 2015, respectively, and she is pursuing the PhD degree from the School of Information and Communication Engineering, Dalian University of Technology, China. She is currently a Visiting Graduate Student with the Department of Computer Science, the University of Texas at San Antonio (UTSA), Texas, USA, funded by the China Scholarship Council (CSC). Her research interests include multimedia retrieval, computer vision, and image forensics.

Xiangwei Kong received the PhD degree in management science and engineering from Dalian University of Technology, China in 2003. From 2006 to 2007, she was a Visiting Scholar with the Department of Computer Science, Purdue University, USA. From 2014 to 2015, she was a senior research scientist with the Department of Computer Science, New York University, USA. She is currently a Professor with the School of Information and Communication Engineering, and the Director of Research Center of Multimedia Information Processing and Security, Dalian University of Technology, China. She has published four edited books and more than 185 research papers in refereed international journals and conferences in the areas of cross-modal retrieval, multimedia information security, knowledge mining, and business intelligence.

Haiyan Fu received her PhD degree from Dalian University of Technology, China in 2014. She is currently an associate professor in the School of Information and Communication Engineering, Dalian University of Technology. Her research interests are in the areas of image retrieval, image hashing, and computer vision.

Qi Tian received the BE degree in electronic engineering from Tsinghua University, China in 1992, and the MS degree in ECE from Drexel University, USA in 1996, and the PhD degree in ECE from University of Illinois at Urbana-Champaign, USA in 2002. He is currently a full professor with the Department of Computer Science, the University of Texas at San Antonio (UTSA), USA. During 2008 and 2009, he took one-year Faculty Leave at Microsoft Research Asia, Beijing, China, as a Lead Researcher in the Media Computing Group. He has authored or coauthored more than 340 refereed journal and conference papers. He is a fellow of IEEE. His research interests include multimedia information retrieval, computer vision, and pattern recognition.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Kong, X., Fu, H. et al. Contextual modeling on auxiliary points for robust image reranking. Front. Comput. Sci. 13, 1010–1022 (2019). https://doi.org/10.1007/s11704-018-7403-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-018-7403-7

Keywords

Search

Navigation