skip to main content
10.1145/3488560.3498462acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
research-article
Open access

Retrieving Black-box Optimal Images from External Databases

Published: 15 February 2022 Publication History

Abstract

Suppose we have a black-box function (e.g., deep neural network) that takes an image as input and outputs a value that indicates preference. How can we retrieve optimal images with respect to this function from an external database on the Internet? Standard retrieval problems in the literature (e.g., item recommendations) assume that an algorithm has full access to the set of items. In other words, such algorithms are designed for service providers. In this paper, we consider the retrieval problem under different assumptions. Specifically, we consider how users with limited access to an image database can retrieve images using their own black-box functions. This formulation enables a flexible and finer-grained image search defined by each user. We assume the user can access the database through a search query with tight API limits. Therefore, a user needs to efficiently retrieve optimal images in terms of the number of queries. We propose an efficient retrieval algorithm Tiara for this problem. In the experiments, we confirm that our proposed method performs better than several baselines under various settings.

Supplementary Material

MP4 File (WSDM22-fp474.mp4)
Presentation video for "Retrieving Black-box Optimal Images from External Databases" (10 min)

References

[1]
anchi, Gentile, and Mansour]alon2013fromNoga Alon, Nicolò Cesa-Bianchi, Claudio Gentile, and Yishay Mansour. From bandits to experts: A tale of domination and independence. In NeurIPS, pages 1610--1618, 2013.
[2]
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings. In ICLR, 2017.
[3]
anchi, and Fischer]auer2002finitePeter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn., 47 (2--3): 235--256, 2002.
[4]
Artem Babenko and Victor S. Lempitsky. Aggregating local deep features for image retrieval. In ICCV, pages 1269--1277, 2015.
[5]
tes et al.(2005)Baeza-Yates, Castillo, Mar'i n, and Rodr'i guez]baezayates2005crawlingRicardo A. Baeza-Yates, Carlos Castillo, Mauricio Mar'i n, and M. Andrea Rodr'i guez. Crawling a country: better strategies than breadth-first for web page ordering. In WWW, pages 864--872, 2005.
[6]
Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures. In ICML, pages 471--480, 2018.
[7]
Luciano Barbosa and Juliana Freire. An adaptive crawler for locating hidden-web entry points. In WWW, pages 441--450, 2007.
[8]
Sean Bell and Kavita Bala. Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph., 34 (4): 98:1--98:10, 2015.
[9]
Yue Cao, Mingsheng Long, Jianmin Wang, Qiang Yang, and Philip S. Yu. Deep visual-semantic hashing for cross-modal retrieval. In KDD, pages 1445--1454, 2016.
[10]
Alexandra Carpentier and Michal Valko. Extreme bandits. In NeurIPS, pages 1089--1097, 2014.
[11]
Carlos Castillo. Effective web crawling. SIGIR Forum, 39 (1): 55--56, 2005.
[12]
Soumen Chakrabarti, Martin van den Berg, and Byron Dom. Focused crawling: A new approach to topic-specific web resource discovery. Comput. Networks, 31 (11--16): 1623--1640, 1999.
[13]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597--1607, 2020.
[14]
lina(2003)]cho2003effectiveJunghoo Cho and Hector Garcia-Molina. Effective page refresh policies for web crawlers. ACM Trans. Database Syst., 28 (4): 390--426, 2003.
[15]
lina, and Page]cho1998efficientJunghoo Cho, Hector Garcia-Molina, and Lawrence Page. Efficient crawling through URL ordering. Comput. Networks, 30 (1--7): 161--172, 1998.
[16]
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. Towards conversational recommender systems. In KDD, pages 815--824, 2016.
[17]
Vincent A. Cicirello and Stephen F. Smith. The max K-armed bandit: A new model of exploration applied to search heuristic selection. In AAAI, pages 1355--1361, 2005.
[18]
Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi. Deep filter banks for texture recognition and segmentation. In CVPR, pages 3828--3836, 2015.
[19]
Thomas Desautels, Andreas Krause, and Joel W. Burdick. Parallelizing exploration-exploitation tradeoffs with gaussian process bandit optimization. In ICML, 2012.
[20]
Michelangelo Diligenti, Frans Coetzee, Steve Lawrence, C. Lee Giles, and Marco Gori. Focused crawling using context graphs. In VLDB, pages 527--534, 2000.
[21]
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341 (3): 1, 2009.
[22]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126--1135, 2017.
[23]
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2414--2423, 2016.
[24]
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[25]
, Revaud, and Larlus]gordo2016deepAlbert Gordo, Jon Almazá n, Jé rô me Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. In ECCV, pages 241--257, 2016.
[26]
Ziyu Guan, Can Wang, Chun Chen, Jiajun Bu, and Junfeng Wang. Guide focused crawler efficiently and effectively using on-line topical importance estimation. In SIGIR, pages 757--758, 2008.
[27]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770--778, 2016.
[28]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726--9735, 2020.
[29]
Judy Johnson, Kostas Tsioutsiouliklis, and C. Lee Giles. Evolving strategies for focused web crawling. In ICML, pages 298--305, 2003.
[30]
Makoto P. Kato, Hiroaki Ohshima, Satoshi Oyama, and Katsumi Tanaka. Can social tagging improve web image search? In WISE, pages 235--249, 2008.
[31]
et al.(2014)Kocá k, Neu, Valko, and Munos]kocak2014efficientTomá vs Kocá k, Gergely Neu, Michal Valko, and Ré mi Munos. Efficient learning by implicit exploration in bandit problems with side observations. In NeurIPS, pages 613--621, 2014.
[32]
Saeid Balaneshin Kordan and Alexander Kotov. Deep neural architecture for multi-modal retrieval based on joint embedding space for text and images. In WSDM, pages 28--36, 2018.
[33]
]krishnamurthy2016contextualAkshay Krishnamurthy, Alekh Agarwal, and Miroslav Dud'i k. Contextual semibandits via supervised learning oracles. In NeurIPS, pages 2388--2396, 2016.
[34]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1106--1114, 2012.
[35]
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[36]
Hanjiang Lai, Yan Pan, Ye Liu, and Shuicheng Yan. Simultaneous feature learning and hash coding with deep neural networks. In CVPR, pages 3270--3278, 2015.
[37]
(2020)]lattimore2020banditTor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[38]
Luis A. Leiva, Mauricio Villegas, and Roberto Paredes. Query refinement suggestion in multimodal image retrieval with relevance feedback. In ICMI, pages 311--314, 2011.
[39]
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In WWW, pages 661--670, 2010.
[40]
Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Deep supervised hashing for fast image retrieval. In CVPR, pages 2064--2072, 2016.
[41]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[42]
Shie Mannor and Ohad Shamir. From bandits to experts: On the value of side-observations. In NeurIPS, pages 684--692, 2011.
[43]
Andrew McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Inf. Retr., 3 (2): 127--163, 2000.
[44]
Robert Meusel, Peter Mika, and Roi Blanco. Focused crawling for structured data. In CIKM, pages 1039--1048, 2014.
[45]
Tomá s Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
[46]
Anh Mai Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In NeurIPS, pages 3387--3395, 2016.
[47]
Liqiang Nie, Shuicheng Yan, Meng Wang, Richang Hong, and Tat-Seng Chua. Harvesting visual concepts for image search with complex queries. In MM, pages 59--68, 2012.
[48]
Wei Niu, James Caverlee, and Haokai Lu. Neural personalized ranking for image recommendation. In WSDM, pages 423--431, 2018.
[49]
ChangYong Oh, Jakub M. Tomczak, Efstratios Gavves, and Max Welling. Combinatorial bayesian optimization using the graph cartesian product. In NeurIPS, pages 2910--2920, 2019.
[50]
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532--1543, 2014.
[51]
Kien Pham, Aé cio S. R. Santos, and Juliana Freire. Bootstrapping domain-specific content discovery on the web. In WWW, pages 1476--1486, 2019.
[52]
Masaki Saito and Yusuke Matsui. Illustration2vec: a semantic vector representation of illustrations. In SIGGRAPH Asia Technical Briefs, pages 5:1--5:4, 2015.
[53]
Ryoma Sato. Retrieving black-box optimal images from external databases. arXiv, abs/2112.14921, 2021.
[54]
Ryoma Sato. Private recommender systems: How can users build their own fair recommender systems without log data? In Proceedings of the 2022 SIAM International Conference on Data Mining, SDM, 2022.
[55]
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. In ACL, pages 440--450, 2018.
[56]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR workshop, 2014.
[57]
Ashudeep Singh and Thorsten Joachims. Fairness of exposure in rankings. In KDD, pages 2219--2228, 2018.
[58]
Aleksandrs Slivkins. Introduction to multi-armed bandits. Found. Trends Mach. Learn., 12 (1--2): 1--286, 2019.
[59]
Yueming Sun and Yi Zhang. Conversational recommender system. In SIGIR, pages 235--244, 2018.
[60]
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, ICML, pages 3319--3328, 2017.
[61]
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25 (3/4): 285--294, 1933.
[62]
Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Maarten de Rijke, Qingyao Ai, Yufei Huang, Min Zhang, and Shaoping Ma. Improving web image search with contextual information. In CIKM, pages 1683--1692, 2019.
[63]
Hao Yuan, Jiliang Tang, Xia Hu, and Shuiwang Ji. XGNN: towards model-level explanations of graph neural networks. In KDD, pages 430--438, 2020.

Cited By

View all
  • (2022)Word Tour: One-dimensional Word Embeddings via the Traveling Salesman ProblemWord Tour: One-dimensional Word Embeddings via the Traveling Salesman ProblemJournal of Natural Language Processing10.5715/jnlp.29.129729:4(1297-1301)Online publication date: 2022
  • (2022)Towards Principled User-side Recommender SystemsProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557476(1757-1766)Online publication date: 17-Oct-2022
  • (2022)CLEAR: A Fully User-side Image Search SystemProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557172(4970-4974)Online publication date: 17-Oct-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining
February 2022
1690 pages
ISBN:9781450391320
DOI:10.1145/3488560
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2022

Check for updates

Author Tags

  1. information retrieval
  2. linear bandits
  3. private recommender systems
  4. web searching

Qualifiers

  • Research-article

Funding Sources

Conference

WSDM '22

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)90
  • Downloads (Last 6 weeks)10
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Word Tour: One-dimensional Word Embeddings via the Traveling Salesman ProblemWord Tour: One-dimensional Word Embeddings via the Traveling Salesman ProblemJournal of Natural Language Processing10.5715/jnlp.29.129729:4(1297-1301)Online publication date: 2022
  • (2022)Towards Principled User-side Recommender SystemsProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557476(1757-1766)Online publication date: 17-Oct-2022
  • (2022)CLEAR: A Fully User-side Image Search SystemProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557172(4970-4974)Online publication date: 17-Oct-2022

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media