Skip to main content
SpringerLink
Log in

An efficient indexing technique for billion-scale nearest neighbor search

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Approximate nearest neighbor search is an indispensable component in many computer vision applications. To index more data, such as images, on one commercial server, Douze et al. introduced L&C that works on operating points considering 64–128 bytes per vector. While the idea is inspiring, we observe that L&C still suffers the accuracy saturation problem, which it is aimed to solve. To this end, we propose a simple yet effective two-layer graph index structure, together with dual residual encoding, to attain higher accuracy. Particularly, we partition vectors into multiple clusters and build the top-layer graph using the corresponding centroids. For each cluster, a subgraph is created with compact codes of the first-level vector residuals. Such an index structure provides better graph search precision as well as saves quite a few bytes for compression. We employ the second-level residual quantization to re-rank the candidates obtained through graph traversal, which is more efficient than regression-from-neighbors adopted by L&C. Comprehensive experiments show that our proposal obtains over 10% and 30% higher recall@1 than the state-of-the-arts, and achieves up to 7.7x and 6.1x speedup over L&C on Deep1B and Sift1B, respectively. Our proposal also attains 90%+ recall@10 and recall@100 on two billion-sized datasets at the cost of 10ms per query.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

All data generated or analysed during this study are included in this published article (and its supplementary information files)

Materials Availability

SIFT1B:http://corpus-texmex.irisa.fr/

DEEP1B:http://sites.skoltech.ru/compvision/noimi/

Notes

  1. Both vectors and the search graph are considered.

  2. The first 100M vectors in Deep1B.

  3. https://github.com/facebookresearch/faiss

References

  1. André F, Kermarrec A, Scouarnec N.L (2018) Quicker ADC unlocking the hidden potential of product quantization with SIMD. CoRR arXiv:1812.09162

  2. Aoyama K, Saito K, Sawada H, Ueda N (2011) Fast approximate similarity search based on degree-reduced neighborhood graphs. In: SIGKDD, pp 1055–1063

  3. Arya S, Mount DM (1993) Approximate nearest neighbor queries in fixed dimensions. In: SODA, vol 93, pp 271–280

  4. Babenko A, Lempitsky V (2014) Additive quantization for extreme vector compression. In: CVPR, pp 931–938

  5. Babenko A, Lempitsky V (2014) The inverted multi-index. IEEE Trans Pattern Anal Mach Intell 37(6):1247–1260

    Article  Google Scholar 

  6. Babenko A, Lempitsky V (2014) Improving bilayer product quantization for billion-scale approximate nearest neighbors in high dimensions. arXiv:1404.1831

  7. Babenko A, Lempitsky V (2015) Tree quantization for large-scale similarity search and classification. In: CVPR, pp 4240–4248

  8. Babenko A, Lempitsky V (2016) Efficient indexing of billion-scale datasets of deep descriptors. In: CVPR, pp 2055–2063

  9. Banerjee P, Bhunia AK, Bhattacharyya A, Roy PP, Murala S (2018) Local neighborhood intensity pattern-a new texture feature descriptor for image retrieval. Expert Syst Appl 113:100–115. https://doi.org/10.1016/j.eswa.2018.06.044

    Article  Google Scholar 

  10. Baranchuk D, Persiyanov D, Sinitsin A, Babenko A (2019) Learning to route in similarity graphs. In: ICML, vol 97, pp 475–484

  11. Beis JS, Lowe DG (1997) Shape indexing using approximate nearest-neighbour search in high-dimensional spaces. In: CVPR, pp 1000–1006. IEEE

  12. Beyer K, Goldstein J, Ramakrishnan R, Shaft U (1999) When is nearest neighbor meaningful?. In: ICDT, pp 217–235. Springer

  13. Bhunia AK, Bhattacharyya A, Banerjee P, Roy PP, Murala S (2020) A novel feature descriptor for image retrieval by combining modified color histogram and diagonally symmetric co-occurrence texture pattern. Pattern Anal Appl 23(2):703–723. https://doi.org/10.1007/s10044-019-00827-x

    Article  Google Scholar 

  14. Böhm C, Berchtold S, Keim DA (2001) Searching in high-dimensional spaces: index structures for improving the performance of multimedia databases. ACM Computing Surveys (CSUR) 33(3):322–373

    Article  Google Scholar 

  15. Datar M, Immorlica N, Indyk P, Mirrokni VS (2004) Locality-sensitive hashing scheme based on p-stable distributions. In: SoCG, pp 253–262

  16. Douze M, Jégou H, Perronnin F (2016) Polysemous codes. In: ECCV, pp 785–801. Springer

  17. Douze M, Sablayrolles A, Jégou H (2018) Link and code: fast indexing with graphs and compact regression codes. In: CVPR, pp 3646–3654

  18. Douze M, Szlam A, Hariharan B, Jégou H (2018) Low-shot learning with large-scale diffusion. In: CVPR, pp 3349–3358

  19. Fu C, Xiang C, Wang C, Cai D (2019) Fast approximate nearest neighbor search with the navigating spreading-out graph. VLDB 12(5):461–474

    Google Scholar 

  20. Ge T, He K, Ke Q, Sun J (2013) Optimized product quantization. IEEE Trans Pattern Anal Mach Intell 36(4):744–755

    Article  Google Scholar 

  21. Gupta S, Roy PP, Dogra DP, Kim B (2020) Retrieval of colour and texture images using local directional peak valley binary pattern. Pattern Anal Appl 23(4):1569–1585. https://doi.org/10.1007/s10044-020-00879-4

    Article  Google Scholar 

  22. Harwood B, Drummond T (2016) Fanng: fast approximate nearest neighbour graphs. In: CVPR, pp 5713–5722

  23. He R, Cai Y, Tan T, Davis LS (2015) Learning predictable binary codes for face indexing. Pattern Recogn 48(10):3160–3168

    Article  Google Scholar 

  24. Jegou H, Douze M, Schmid C (2010) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33(1):117–128

    Article  Google Scholar 

  25. Jégou H, Tavenard R, Douze M, Amsaleg L (2011) Searching in one billion vectors: re-rank with source coding. In: ICASSP, pp 861–864. IEEE

  26. Jiang Z, Xie L, Deng X, Xu W, Wang J (2016) Fast nearest neighbor search in the hamming space. In: International conference on multimedia modeling, pp 325–336. Springer

  27. Jin L, Li Z, Tang J (2020) Deep semantic multimodal hashing network for scalable image-text and video-text retrievals. IEEE Transactions on Neural Networks and Learning Systems, pp 1–14. https://doi.org/10.1109/TNNLS.2020.2997020

  28. Kalantidis Y, Avrithis Y (2014) Locally optimized product quantization for approximate nearest neighbor search. In: CVPR, pp 2321–2328

  29. Li Z, Tang J, Zhang L, Yang J (2020) Weakly-supervised semantic guided hashing for social image retrieval. Int J Comput Vis 128(8):2265–2278. https://doi.org/10.1007/s11263-020-01331-0

    Article  MathSciNet  MATH  Google Scholar 

  30. Li W, Zhang Y, Sun Y, Wang W, Li M, Zhang W, Lin X (2020) Approximate nearest neighbor search on high dimensional data - experiments, analyses, and improvement. IEEE Trans Knowl Data Eng 32(8):1475–1488

    Article  Google Scholar 

  31. Lin P, Zhao W (2019) A comparative study on hierarchical navigable small world graphs. CoRR arXiv:1904.02077

  32. Liu Y, Cheng H, Cui J (2017) PQBF: i/o-efficient approximate nearest neighbor search by product quantization. In: CIKM, pp 667–676

  33. Liu S, Shao J, Lu H (2017) Generalized residual vector quantization and aggregating tree for large scale search. IEEE Trans Multimedia PP(8):1–1

    Google Scholar 

  34. Lv Q, Charikar M, Li K (2004) Image similarity search with compact data structures. In: CIKM, pp 208–217

  35. Malkov YA, Yashunin DA (2018) Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs IEEE. Transactions on Pattern Analysis and Machine Intelligence

  36. Matsui Y, Uchida Y, Jégou H, Satoh S (2018) A survey of product quantization. ITE Transactions on Media Technology & Applications

  37. Matsui Y, Yamasaki T, Aizawa K (2018) Pqtable: Nonexhaustive fast search for product-quantized codes using hash tables. IEEE Trans Multim 20 (7):1809–1822

    Article  Google Scholar 

  38. Muja M, Lowe DG (2009) Fast approximate nearest neighbors with automatic algorithm configuration. In: VISAPP, pp 331–340

  39. Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. In: CVPR

  40. Shakhnarovich G, Darrell T, Indyk P (2006) Nearest-neighbor methods in learning and vision: theory and practice (neural Information Processing). The MIT press

  41. Sivic Z (2003) Video google: a text retrieval approach to object matching in videos. In: ICCV

  42. Teodoro G, Valle E, Mariano N, Torres R, Meira W, Saltz JH (2014) Approximate similarity search for online multimedia services on distributed cpu–gpu platforms. VLDB J 23(3):427–448

    Article  Google Scholar 

  43. Weber R, Schek H-J, Blott S (1998) A quantitative analysis and performance study for similarity-search methods in high-dimensional spaces. In: VLDB, vol 98, pp 194–205

  44. Zhang T, Du C, Wang J (2014) Composite quantization for approximate nearest neighbor search. In: ICML, vol 2, p 3

  45. Zhao K, Pan P, Zheng Y, Zhang Y, Wang C, Zhang Y, Xu Y, Jin R (2019) Large-scale visual search with binary distributed graph at Alibaba. In: CIKM, pp 2567–2575

Download references

Acknowledgements

The work reported in this paper is partially supported by NSF of Shanghai under grant number 22ZR1402000, the Fundamental Research Funds for the Central Universities under grant number 2232021A-08, State Key Laboratory of Computer Architecture (ICT,CAS) under Grant No. CARCHB 202118, Information Development Project of Shanghai Economic and Information Commission (202002009) and National Natural Science Foundation of China (No.61906035).

Funding

The work reported in this paper is partially supported by NSF of Shanghai under grant number 22ZR1402000, the Fundamental Research Funds for the Central Universities under grant number 2232021A-08, State Key Laboratory of Computer Architecture (ICT,CAS) under Grant No. CARCHB 202118, Information Development Project of Shanghai Economic and Information Commission (202002009) and National Natural Science Foundation of China (No.61906035).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Donghua University, Shanghai, China

    Kaixiang Yang, Hongya Wang, Ming Du, Zhizheng Wang & Zongyuan Tan

  2. State Key Laboratory of Computer Architecture, ICT, CAS, Beijing, China

    Kaixiang Yang & Hongya Wang

  3. Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China

    Kaixiang Yang & Hongya Wang

  4. Institute of Artificial Intelligence, Donghua University, Shanghai, China

    Jie Zhang

  5. School of CSE, Tianjin University of Technology, Tianjin, China

    Yingyuan Xiao

Authors
  1. Kaixiang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongya Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhizheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zongyuan Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yingyuan Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K.Y. performed the experiments and manuscript preparation.

K.Y., H.W., conceived the conception of the study and wrote the manuscript.

M.D., Z.W., Z.T. performed the data analysis.

J.Z., Y.X. disscussed with constructive discussions.

Corresponding author

Correspondence to Hongya Wang.

Ethics declarations

Conflict of Interests

No

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Portions of this work were presented at the 29th Pacific Conference on Computer Graphics and Applications

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, K., Wang, H., Du, M. et al. An efficient indexing technique for billion-scale nearest neighbor search. Multimed Tools Appl 82, 31673–31689 (2023). https://doi.org/10.1007/s11042-023-14825-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14825-z

Keywords

Search

Navigation