Zepu Lu
Enhong Chen
ABSTRACT
Vector quantization techniques, such as Product Quantization (PQ), play a vital role in approximate nearest neighbor search (ANNs) and maximum inner product search (MIPS) owing to their remarkable search and storage efficiency. However, the indexes in vector quantization cannot be trained together with the inference models since data indexing is not differentiable. To this end, differentiable vector quantization approaches, such as DiffPQ and DeepPQ, have been recently proposed, but existing methods have two drawbacks. First, they do not impose any constraints on codebooks, such that the resultant codebooks lack diversity, leading to limited retrieval performance. Second, since data indexing resorts to operator, differentiability is usually achieved by either relaxation or Straight-Through Estimation (STE), which leads to biased gradient and slow convergence. To address these problems, we propose a Differentiable Optimized Product Quantization method (DOPQ) and beyond in this paper. Particularly, each data is projected into multiple orthogonal spaces, to generate multiple views of data. Thus, each codebook is learned with one view of data, guaranteeing the diversity of codebooks. Moreover, instead of simple differentiable relaxation, DOPQ optimizes the loss based on direct loss minimization, significantly reducing the gradient bias problem. Finally, DOPQ is evaluated with seven datasets of both recommendation and image search tasks. Extensive experimental results show that DOPQ outperforms state-of-the-art baselines by a large margin.
- Artem Babenko and Victor Lempitsky. 2014. Additive quantization for extreme vector compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 931–938.Google Scholar
Digital Library
- Moses S Charikar. 2002. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing. 380–388.Google ScholarDigital Library
- Lin Chen, Hossein Esfandiari, Gang Fu, and Vahab Mirrokni. 2019. Locality-sensitive hashing for f-divergences: Mutual information loss and beyond. Advances in Neural Information Processing Systems 32 (2019).Google Scholar
- Qi Chen, Haidong Wang, Mingqin Li, Gang Ren, Scarlett Li, Jeffery Zhu, Jason Li, Chuanjie Liu, Lintao Zhang, and Jingdong Wang. 2018. SPTAG: A library for fast approximate nearest neighbor search.Google Scholar
- Ting Chen, Lala Li, and Yizhou Sun. 2020. Differentiable product quantization for end-to-end embedding compression. In International Conference on Machine Learning. PMLR, 1617–1626.Google Scholar
- Ting Chen, Martin Renqiang Min, and Yizhou Sun. 2018. Learning k-way d-dimensional discrete codes for compact embedding representations. In International Conference on Machine Learning. PMLR, 854–863.Google Scholar
- Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, and Le Song. 2017. Stochastic generative hashing. In International Conference on Machine Learning. PMLR, 913–922.Google Scholar
- Xinyan Dai, Xiao Yan, Kelvin KW Ng, Jiu Liu, and James Cheng. 2020. Norm-explicit quantization: Improving vector quantization for maximum inner product search. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 51–58.Google ScholarCross Ref
- Sanjoy Dasgupta and Yoav Freund. 2008. Random projection trees and low dimensional manifolds. In Proceedings of the fortieth annual ACM symposium on Theory of computing. 537–546.Google ScholarDigital Library
- Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. 2004. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry. 253–262.Google ScholarDigital Library
- Venice Erin Liong, Jiwen Lu, Gang Wang, Pierre Moulin, and Jie Zhou. 2015. Deep hashing for compact binary codes learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2475–2483.Google Scholar
- Chao Feng, Wuchao Li, Defu Lian, Zheng Liu, and Enhong Chen. 2022. Recommender Forest for Efficient Retrieval. In Advances in Neural Information Processing Systems.Google Scholar
- Chao Feng, Defu Lian, Xiting Wang, Zheng Liu, Xing Xie, and Enhong Chen. 2023. Reinforcement Routing on Proximity Graph for Efficient Recommendation. ACM Transactions on Information Systems 41, 1 (2023), 1–27.Google ScholarDigital Library
- Lianli Gao, Xiaosu Zhu, Jingkuan Song, Zhou Zhao, and Heng Tao Shen. 2019. Beyond product quantization: Deep progressive quantization for image retrieval. arXiv preprint arXiv:1906.06698 (2019).Google Scholar
- Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2013. Optimized product quantization. IEEE transactions on pattern analysis and machine intelligence 36, 4 (2013), 744–755.Google Scholar
- Aristides Gionis, Piotr Indyk, Rajeev Motwani, 1999. Similarity search in high dimensions via hashing. In Vldb, Vol. 99. 518–529.Google Scholar
- Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha. 2016. Quantization based fast inner product search. In Artificial intelligence and statistics. PMLR, 482–490.Google Scholar
- Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning. PMLR, 3887–3896.Google Scholar
- Kaiming He, Fang Wen, and Jian Sun. 2013. K-means hashing: An affinity-preserving quantization method for learning binary compact codes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2938–2945.Google ScholarDigital Library
- Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence 33, 1 (2010), 117–128.Google Scholar
- Yunjiang Jiang, Han Zhang, Yiming Qiu, Yun Xiao, Bo Long, and Wen-Yun Yang. 2022. Givens Coordinate Descent Methods for Rotation Matrix Learning in Trainable Embedding Indexes. arXiv preprint arXiv:2203.05082 (2022).Google Scholar
- Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data 7, 3 (2019), 535–547.Google ScholarCross Ref
- Jon Kleinberg. 2000. The small-world phenomenon: An algorithmic perspective. In Proceedings of the thirty-second annual ACM symposium on Theory of computing. 163–170.Google ScholarDigital Library
- Brian Kulis and Trevor Darrell. 2009. Learning to hash with binary reconstructive embeddings. Advances in neural information processing systems 22 (2009).Google Scholar
- Hanjiang Lai, Yan Pan, Ye Liu, and Shuicheng Yan. 2015. Simultaneous feature learning and hash coding with deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3270–3278.Google ScholarCross Ref
- Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. 2019. Approximate nearest neighbor search on high dimensional data—experiments, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering 32, 8 (2019), 1475–1488.Google ScholarCross Ref
- Xiaoyun Li and Ping Li. 2019. Random projections with asymmetric quantization. Advances in Neural Information Processing Systems 32 (2019).Google Scholar
- Defu Lian, Yong Ge, Fuzheng Zhang, Nicholas Jing Yuan, Xing Xie, Tao Zhou, and Yong Rui. 2015. Content-aware collaborative filtering for location recommendation based on human mobility data. In 2015 IEEE international conference on data mining. IEEE, 261–270.Google ScholarDigital Library
- Defu Lian, Haoyu Wang, Zheng Liu, Jianxun Lian, Enhong Chen, and Xing Xie. 2020. Lightrec: A memory and search-efficient recommender system. In Proceedings of The Web Conference 2020. 695–705.Google ScholarDigital Library
- Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2016. Deep supervised hashing for fast image retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2064–2072.Google ScholarCross Ref
- Jie Liu, Xiao Yan, Xinyan Dai, Zhirong Li, James Cheng, and Ming-Chang Yang. 2020. Understanding and improving proximity graph based maximum inner product search. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 139–146.Google ScholarCross Ref
- Qi Liu, Jin Zhang, Defu Lian, Yong Ge, Jianhui Ma, and Enhong Chen. 2021. Online Additive Quantization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1098–1108.Google ScholarDigital Library
- Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang, and Shih-Fu Chang. 2012. Supervised hashing with kernels. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2074–2081.Google ScholarDigital Library
- Guy Lorberbom, Andreea Gane, Tommi Jaakkola, and Tamir Hazan. 2019. Direct Optimization through argmax for Discrete Variational Auto-Encoder. Advances in neural information processing systems 32 (2019).Google Scholar
- Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824–836.Google Scholar
- Stanislav Morozov and Artem Babenko. 2018. Non-metric similarity graphs for maximum inner product search. Advances in Neural Information Processing Systems 31 (2018).Google Scholar
- Marius Muja and David G Lowe. 2014. Scalable nearest neighbor algorithms for high dimensional data. IEEE transactions on pattern analysis and machine intelligence 36, 11 (2014), 2227–2240.Google ScholarCross Ref
- Behnam Neyshabur and Nathan Srebro. 2015. On symmetric and asymmetric lshs for inner product search. In International Conference on Machine Learning. PMLR, 1926–1934.Google Scholar
- Uri Shalit and Gal Chechik. 2014. Coordinate-descent for learning orthogonal matrices through Givens rotations. In International Conference on Machine Learning. PMLR, 548–556.Google Scholar
- Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. 2015. Supervised discrete hashing. In Proceedings of the IEEE conference on computer vision and pattern recognition. 37–45.Google ScholarCross Ref
- Ron Shepard, Scott R Brozell, and Gergely Gidofalvi. 2015. The representation and parametrization of orthogonal matrices. The Journal of Physical Chemistry A 119, 28 (2015), 7924–7939.Google ScholarCross Ref
- Anshumali Shrivastava and Ping Li. 2014. Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). Advances in neural information processing systems 27 (2014).Google Scholar
- Yang Song, Alexander Schwing, Raquel Urtasun, 2016. Training deep neural networks via direct loss minimization. In International conference on machine learning. PMLR, 2169–2177.Google Scholar
- Jun Wang, Wei Liu, Sanjiv Kumar, and Shih-Fu Chang. 2015. Learning to hash for indexing big data—A survey. Proc. IEEE 104, 1 (2015), 34–57.Google ScholarCross Ref
- Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, 2017. A survey on learning to hash. IEEE transactions on pattern analysis and machine intelligence 40, 4 (2017), 769–790.Google Scholar
- Hanwei Wu and Markus Flierl. 2019. Learning product codebooks using vector-quantized autoencoders for image retrieval. In 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP). IEEE, 1–5.Google ScholarCross Ref
- Xiang Wu, Ruiqi Guo, Ananda Theertha Suresh, Sanjiv Kumar, Daniel N Holtmann-Rice, David Simcha, and Felix Yu. 2017. Multiscale quantization for fast similarity search. Advances in neural information processing systems 30 (2017).Google Scholar
- Yongji Wu, Defu Lian, Neil Zhenqiang Gong, Lu Yin, Mingyang Yin, Jingren Zhou, and Hongxia Yang. 2021. Linear-time self attention with codeword histogram for efficient recommendation. In Proceedings of the Web Conference 2021. 1262–1273.Google ScholarDigital Library
- Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. 2014. Supervised hashing for image retrieval via image representation learning. In Twenty-eighth AAAI conference on artificial intelligence.Google ScholarDigital Library
- Tan Yu, Junsong Yuan, Chen Fang, and Hailin Jin. 2018. Product quantization network for fast image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV). 186–201.Google ScholarDigital Library
- Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2022. Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 1328–1336.Google ScholarDigital Library
- Jin Zhang, Qi Liu, Defu Lian, Zheng Liu, Le Wu, and Enhong Chen. 2022. Anisotropic Additive Quantization for Fast Inner Product Search. In Proceedings of the AAAI conference on Artificial Intelligence.Google ScholarCross Ref
- Ting Zhang, Chao Du, and Jingdong Wang. 2014. Composite quantization for approximate nearest neighbor search. In International Conference on Machine Learning. PMLR, 838–846.Google Scholar
- Han Zhu, Mingsheng Long, Jianmin Wang, and Yue Cao. 2016. Deep hashing network for efficient similarity retrieval. In Proceedings of the AAAI conference on Artificial Intelligence, Vol. 30.Google ScholarCross Ref
Index Terms
- Differentiable Optimized Product Quantization and Beyond
Recommendations
Adaptive bit allocation product quantization
Product quantization (PQ) is a popular vector quantization method for approximate nearest neighbor search. The key idea of PQ is to decompose the original data space into the Cartesian product of some low-dimensional subspaces and then every subspace is ...
Codebook-softened product quantization for high accuracy approximate nearest neighbor search
AbstractProduct quantization (PQ) is a fundamental technique for approximate nearest neighbor (ANN) search in many applications such as information retrieval, computer vision and pattern recognition. In the existing PQ-based methods for ...
Locally Optimized Product Quantization for Approximate Nearest Neighbor Search
CVPR '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern RecognitionWe present a simple vector quantizer that combines low distortion with fast search and apply it to approximate nearest neighbor (ANN) search in high dimensional spaces. Leveraging the very same data structure that is used to provide non-exhaustive ...
Comments