Abstract
User relevance feedback (URF) is emerging as an important component of the multimedia analytics toolbox. State-of-the-art URF systems employ high-dimensional vectors of semantic features and train linear-SVM classifiers in each round of interaction. In a round, they present the user with the most confident media items, which lie furthest from the SVM plane. Due to the scale of current media collections, URF systems must be supported by a high-dimensional index. Usually, these indexes are designed for nearest-neighbour point queries, and it is not known how well they support the URF process. In this paper, we study the performance of four state-of-the-art high-dimensional indexes in the URF context. We analyse the quality of query results, compared to a sequential analysis of the collection, over a range of classifiers, showing that result quality depends (i) heavily on the quality of the SVM classifier and (ii) the index structure itself. We also consider a search-oriented workload, where the goal is to find the first relevant item for a task. The results show that the indexes perform similarly overall, despite differences in their paths to the solution. Interestingly, worse recall can lead to better application-specific performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
This experiment was also conducted using Annoy, HNSW, and IVF built using inner product instead of Euclidean distance. In all cases, average recall @1000 was lower, while for HNSW recall @25 was improved.
- 3.
Similar results are observed when (roughly) targeting a certain number of distance computations across all indexes.
- 4.
The 0-valued outliers for HNSW stem from URF sessions stopping early, as everything returned has already been seen, while the actual minimum was around 4700.
References
Aumüller, M., Bernhardsson, E., Faithfull, A.J.: ANN-benchmarks: a benchmarking tool for approximate nearest neighbor algorithms. Inf. Syst. 87, 101374 (2020)
Bachrach, Y., et al.: Speeding up the Xbox recommender system using a Euclidean transformation for inner-product spaces. In: RecSys, pp. 257–264. ACM (2014)
Barraco, M., Cornia, M., Cascianelli, S., Baraldi, L., Cucchiara, R.: The unreasonable effectiveness of CLIP features for image captioning: an experimental analysis. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4662–4670 (2022)
Bartolini, I., Ciaccia, P., Waas, F.: FeedbackBypass: a new approach to interactive similarity query processing. In: Very Large Data Bases Conference (VLDB), pp. 201–210 (2001)
Bernhardsson, E.: Annoy, github.com/spotify/annoy
Dubey, S.R.: A decade survey of content based image retrieval using deep learning. IEEE TCSVT 32(5), 2687–2704 (2021)
Gurrin, C., et al.: Comparing approaches to interactive lifelog search at the lifelog search challenge (LSC2018). ITE TMTA 7(2), 46–59 (2019)
Huang, Q., Lei, Y., Tung, A.K.H.: Point-to-hyperplane nearest neighbor search beyond the unit hypersphere. In: SIGMOD, pp. 777–789. ACM (2021)
Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing (STOC), pp. 604–613 (1998)
Iwasaki, M., Miyazaki, D.: Optimization of Indexing Based on k-Nearest Neighbor Graph for Proximity Search in High-dimensional Data. ArXiv e-prints (2018)
Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE TBD 7(3), 535–547 (2021)
Khan, O.S., et al.: Interactive learning for multimedia at large. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12035, pp. 495–510. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45439-5_33
Khan, O.S., Jónsson, B.Þ., Zahálka, J., Rudinac, S., Worring, M.: Impact of interaction strategies on user relevance feedback. In: International Conference on Multimedia Retrieval (ICMR), pp. 590–598. ICMR 2021, ACM (2021)
Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE TPAMI 42(4), 824–836 (2020)
Moise, D., Shestakov, D., Gudmundsson, G., Amsaleg, L.: Indexing and searching 100M images with map-reduce. In: Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval (ICMR), pp. 17–24. ICMR 2013, ACM (2013)
Morozov, S., Babenko, A.: Non-metric similarity graphs for maximum inner product search. In: Neural Information Processing Systems (NeurIPS), pp. 4726–4735 (2018)
Rui, Y., Huang, T.: Optimizing learning in image retrieval. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR). vol. 1, pp. 236–243 (2000)
Rui, Y., Huang, T.S., Ortega, M., Mehrotra, S.: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE TCSVT 8, 644–655 (1998)
Schoeffmann, K.: A user-centric media retrieval competition: the video browser showdown 2012–2014. IEEE Multimedia 21(4), 8–13 (2014)
Shrivastava, A., Li, P.: Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS). In: Advances in Neural Information Processing Systems (NeurIPS), pp. 2321–2329 (2014)
Subramanya, S.J., Devvrit, F., Simhadri, H.V., Krishnaswamy, R., Kadekodi, R.: DiskANN: fast accurate billion-point nearest neighbor search on a single node. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 13748–13758 (2019)
Zahálka, J., Worring, M.: Towards interactive, intelligent, and integrated multimedia analytics. In: 2014 IEEE Conference on Visual Analytics Science and Technology (IEEE VAST), pp. 3–12. IEEE (2014)
Zhou, X., Huang, T.: Relevance feedback in image retrieval: a comprehensive review. Multimedia Syst. 8, 536–544 (2003)
Acknowledgements
This work was supported by Icelandic Research Fund grant 239772-051 and by the Innovation Fund Denmark for the project DIREC (9142-00001B).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Khan, O.S., Aumüller, M., Jónsson, B.Þ. (2023). Suitability of Nearest Neighbour Indexes for Multimedia Relevance Feedback. In: Pedreira, O., Estivill-Castro, V. (eds) Similarity Search and Applications. SISAP 2023. Lecture Notes in Computer Science, vol 14289. Springer, Cham. https://doi.org/10.1007/978-3-031-46994-7_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-46994-7_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46993-0
Online ISBN: 978-3-031-46994-7
eBook Packages: Computer ScienceComputer Science (R0)