skip to main content
10.1145/3404835.3462950acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

An Image is Worth a Thousand Terms? Analysis of Visual E-Commerce Search

Published: 11 July 2021 Publication History

Abstract

Visual search has become popular in recent years, allowing users to search by an image they are taking using their mobile device or uploading from their photo library. One domain in which visual search is especially valuable is electronic commerce, where users seek for items to purchase. In this work, we present an in-depth comprehensive study of visual e-commerce search. We perform query log analysis of one of the largest e-commerce platforms' mobile search application. We compare visual and textual search by a variety of characteristics, with special focus on the retrieved results and user interaction with them. We also examine image query characteristics, refinement by attributes, and performance prediction for visual search queries. Our analysis points out a variety of differences between visual and textual e-commerce search. We discuss the implications of these differences for the design of future e-commerce search systems.

Supplementary Material

MP4 File (SIGIR'21 talk - Image Paper.mp4)
Visual search has become popular in recent years, allowing users to search by an image they are taking using their mobile device or uploading from their photo library. One domain in which visual search is especially valuable is electronic commerce, where users seek for items to purchase. In this work, we present an in-depth comprehensive study of visual e-commerce search. We perform query log analysis of one of the largest e-commerce platforms? mobile search application. We compare visual and textual search by a variety of characteristics, with special focus on the retrieved results and user interaction with them. We also examine image query characteristics, refinement by attributes, and performance prediction for visual search queries. Our analysis points out a vari- ety of differences between visual and textual e-commerce search. We discuss the implications of these differences for the design of future e-commerce search systems.

References

[1]
2018. Fashion Product Images Dataset. https://www.kaggle.com/paramaggarwal/fashion-product-images-dataset/, last accessed on 05/13/21.
[2]
2019. How Visual Search has transformed the modern shopping experience. https://www.visenze.com/blog/how-visual-search-has-transformed-the-modern-shopping-experience/, last accessed on 05/13/21.
[3]
2019. Visual Search Wins Over Text as Consumers' Most Trusted InformationSource. https://www.businesswire.com/news/home/20190204005613/en/Visual-Search-Wins-Text-Consumers%E2%80%99-Trusted-Information, last accessedon 05/13/21.
[4]
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In Proc. of ICLR.
[5]
Sean Bell, Yiqun Liu, Sami Alsheikh, Yina Tang, Edward Pizzi, M. Henning, Karun Singh, Omkar Parkhi, and Fedor Borisyuk. 2020. GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce. In Proc. of KDD. 2608--2616.
[6]
Adam Berger and John Lafferty. 2017. Information Retrieval as Statistical Translation. SIGIR Forum, Vol. 51, 2 (2017), 219--226.
[7]
Sergey Bezryadin, Pavel Bourov, and Dmitry Ilinih. 2007. Brightness calculation in digital image processing. In Proc. of TDPF. 10--15.
[8]
Anurag Bhardwaj, Atish Das Sarma, Wei Di, Raffay Hamid, Robinson Piramuthu, and Neel Sundaresan. 2013. Palette Power: Enabling Visual Search through Colors. In Proc. of KDD. 1321--1329.
[9]
Indrani Bhattacharya, Arkabandhu Chowdhury, and Vikas C. Raykar. 2019. Multimodal Dialog for Browsing Large Visual Catalogs Using Exploration-Exploitation Paradigm in a Joint Embedding Space. In Proc. of ICMR. 187--191.
[10]
Yiltan Bitirim, Selin Bitirim, Duygu Celik Ertugrul, and Onsen Toygar. 2020. an Evaluation of reverse Image search performance of Google. In Proc. of COMPSAC. 1368--1372.
[11]
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. JMLR, Vol. 3 (2003), 993--1022.
[12]
Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum, Vol. 36. 3--10.
[13]
SM Aqil Burney and Humera Tariq. 2014. K-means cluster analysis for image segmentation. International Journal of Computer Applications, Vol. 96, 4 (2014).
[14]
David Carmel, Elad Haramaty, Arnon Lazerson, Liane Lewin-Eytan, and Yoelle Maarek. 2020. Why do people buy seemingly irrelevant items in voice product search? On the relation between product relevance and customer satisfaction in ecommerce. In Proc. of WSDM. 79--87.
[15]
David Carmel, Erel Uziel, Ido Guy, Yosi Mass, and Haggai Roitman. 2012. Folksonomy-Based Term Extraction for Word Cloud Generation. ACM Trans. Intell. Syst. Technol., Vol. 3, 4, Article 60 (2012), 20 pages.
[16]
David Carmel and Elad Yom-Tov. 2010. Estimating the query difficulty for information retrieval. Morgan & Claypool Publishers.
[17]
Abon Chaudhuri, Paolo Messina, Samrat Kokkula, Aditya Subramanian, Abhinandan Krishnan, Shreyansh Gandhi, Alessandro Magnani, and Venkatesh Kandaswamy. 2018. A smart system for selection of optimal product images in e-commerce. In Proc. of Big Data. 1728--1736.
[18]
Steve Cronen-Townsend, Yun Zhou, and W Bruce Croft. 2002. Predicting query performance. In Proc. of SIGIR. 299--306.
[19]
Ronan Cummins, Joemon M. Jose, and Colm O'Riordan. 2011. Improved query performance prediction using standard deviation. In Proc. of SIGIR. 1089--1090.
[20]
Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2008. Image Retrieval: Ideas, Influences, and Trends of the New Age. ACM Comput. Surv., Vol. 40, 2, Article 5 (2008), 60 pages.
[21]
Ritendra Datta, Jia Li, and James Z. Wang. 2005. Content-Based Image Retrieval: Approaches and Trends of the New Age. In Proc. of MIR. 253--262.
[22]
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proc. of TREC-3.
[23]
Sarah Elkasrawi, Andreas Dengel, Ahmed Abdelsamad, and Syed Saqib Bukhari. 2016. What you see is what you get? Automatic Image Verification for Online News Content. In Proc. of DAS. 114--119.
[24]
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, Vol. 76, 5 (1971), 378--382.
[25]
Ziba Gandomkar and Claudia Mello-Thoms. 2019. Visual search in breast imaging. The British journal of radiology, Vol. 92, 1102 (2019), 20190057.
[26]
Abby Goodrum and Amanda Spink. 2001. Image searching on the Excite Web search engine. Information Processing & Management, Vol. 37, 2 (2001), 295--311.
[27]
Ido Guy. 2016. Searching by Talking: Analysis of Voice Queries on Mobile Web Search. In Proc. of SIGIR. 35--44.
[28]
Ido Guy. 2018. The Characteristics of Voice Search: Comparing Spoken with Typed-in Mobile Web Search Queries. ACM Trans. Inf. Syst., Vol. 36, 3, Article 30 (2018), 28 pages.
[29]
Benjamin Hadwiger and Christian Riess. 2020. The Forchheim Image Database for Camera Identification in the Wild. arXiv preprint, Vol. abs/2011.02241 (2020).
[30]
Claudia Hauff, Leif Azzopardi, and Djoerd Hiemstra. 2009. The combination and evaluation of query performance prediction methods. In Proc. of ECIR. 301--312.
[31]
Claudia Hauff, Djoerd Hiemstra, and Franciska de Jong. 2008. A survey of pre-retrieval query performance predictors. In Proc. of CIKM. 1419--1420.
[32]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proc. of CVPR. 770--778.
[33]
Narayan Hegde, Jason D Hipp, Yun Liu, Michael Emmert-Buck, Emily Reif, Daniel Smilkov, Michael Terry, Carrie J Cai, Mahul B Amin, Craig H Mermel, et al. 2019. Similar image search for histopathology: SMILY. NPJ digital medicine, Vol. 2, 1 (2019), 1--9.
[34]
Sharon Hirsch, Ido Guy, Alexander Nus, Arnon Dagan, and Oren Kurland. 2020. Query Reformulation in E-Commerce Search. In Proc. of SIGIR. 1319--1328.
[35]
Jen-Hao Hsiao and Li-Jia Li. 2014. On Visual Similarity based Interactive Product Recommendation for Online Shopping. In Proc. of ICIP. 3038--3041.
[36]
Houdong Hu, Yan Wang, Linjun Yang, Pavel Komlev, Li Huang, Xi (Stephen) Chen, Jiapei Huang, Ye Wu, Meenaz Merchant, and Arun Sacheti. 2018. Web-Scale Responsive Visual Search at Bing. In Proc. of KDD. 359--367.
[37]
Hao Jiang, Aakash Sabharwal, Adam Henderson, Diane Hu, and Liangjie Hong. 2019. Understanding the Role of Style in E-Commerce Shopping. In Proc. of KDD. 3112--3120.
[38]
Yushi Jing, David Liu, Dmitry Kislyuk, Andrew Zhai, Jiajing Xu, Jeff Donahue, and Sarah Tavel. 2015. Visual Search at Pinterest. In Proc. of KDD. 1889--1898.
[39]
Rosie Jones and Kristina Lisa Klinkner. 2008. Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs. In Proc. of CIKM. 699--708.
[40]
Wang-Cheng Kang, Eric Kim, Jure Leskovec, Charles Rosenberg, and Julian McAuley. 2019. Complete the look: Scene-based complementary product recommendation. In Proc. of CVPR. 10532--10541.
[41]
Taewan Kim, Seyeong Kim, Sangil Na, Hayoon Kim, Moonki Kim, and Byoung-Ki Jeon. 2016. Visual Fashion-Product Search at SK Planet. arXiv preprint, Vol. abs/1609.07859 (2016).
[42]
Oren Kurland, Anna Shtok, David Carmel, and Shay Hummel. 2011. A unified framework for post-retrieval query-performance prediction. In Proc. of ICTIR. 15--26.
[43]
Katrien Laenen, Susana Zoghbi, and Marie-Francine Moens. 2018. Web Search of Fashion Items with Multimodal Querying. In Proc. of WSDM. 342--350.
[44]
Bing Li, Ling-Yu Duan, Yiming Chen, Rongrong Ji, and Wen Gao. 2012. Predicting the effectiveness of queries for visual search. In Proc. of ICASSP. 2361--2364.
[45]
Eileen Li, Eric Kim, Andrew Zhai, Josh Beal, and Kunlong Gu. 2020 b. Bootstrapping Complete The Look at Pinterest. In Proc. of KDD. 3299--3307.
[46]
Fengzi Li, Shashi Kant, Shunichi Araki, Sumer Bangera, and Swapna Samir Shukla. 2020 a. Neural Networks for Fashion Image Classification and Visual Search. arXiv preprint, Vol. abs/2005.08170 (2020).
[47]
Jie Li, Haifeng Liu, Chuanghua Gui, Jianyu Chen, Zhenyuan Ni, Ning Wang, and Yuan Chen. 2018. The Design and Implementation of a Real Time Visual Search System on JD E-Commerce Platform. In Proc. of Middleware. 9--16.
[48]
Lizi Liao, Xiangnan He, Bo Zhao, Chong-Wah Ngo, and Tat-Seng Chua. 2018. Interpretable Multimodal Retrieval for Fashion Products. In Proc. of MM. 1571--1579.
[49]
Yen-Chieh Lien, Hamed Zamani, and W. Bruce Croft. 2020. Recipe Retrieval with Visual Query of Ingredients. In Proc. of SIGIR. 1565--1568.
[50]
Kevin Lin, Fan Yang, Qiaosong Wang, and Robinson Piramuthu. 2019. Adversarial Learning for Fine-Grained Image Search. In Proc. of ICME. 490--495.
[51]
David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision, Vol. 60, 2 (2004), 91--110.
[52]
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-Based Recommendations on Styles and Substitutes. In Proc. of SIGIR. 43--52.
[53]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proc. of NIPS. 3111--3119.
[54]
Aashish Kumar Misraa, Ajinkya Kale, Pranav Aggarwal, and Ali Aminian. 2020. Multi-Modal Retrieval using Graph Neural Networks. arXiv preprint, Vol. abs/2010.01666 (2020).
[55]
Viken Parikh, Madhura Keskar, Dhwanil Dharia, and Pradnya Gotmare. 2018. A Tourist Place Recommendation and Recognition System. In Proc. of ICICCT. 218--222.
[56]
Jay M. Ponte and W. Bruce Croft. 1998. A language modeling approach to information retrieval. In Proc. of SIGIR. 275--281.
[57]
Haggai Roitman, Shai Erera, Oren Sar Shalom, and Bar Weiner. 2017. Enhanced mean retrieval score estimation for query performance prediction. In Proc. of ICTIR. 35--42.
[58]
Diego Saez-Trumper. 2014. Fake Tweet Buster: A Webtool to Identify Users Promoting Fake News Ontwitter. In Proc. of HT. 316--317.
[59]
Yevhenii B. Shapovalov, Zhanna I. Bilyk, Artem I. Atamas, Viktor B. Shapovalov, and Aleksandr D. Uchitel. 2018. The Potential of Using Google Expeditions and Google Lens Tools under STEM-education in Ukraine. arXiv preprint, Vol. abs/1808.06465 (2018).
[60]
Raymond Shiau, Hao-Yu Wu, Eric Kim, Yue Li Du, Anqi Guo, Zhiyuan Zhang, Eileen Li, Kunlong Gu, Charles Rosenberg, and Andrew Zhai. 2020. Shop The Look: Building a Large Scale Visual Shopping System at Pinterest. In Proc. of KDD. 3203--3212.
[61]
Anna Shtok, Oren Kurland, David Carmel, Fiana Raiber, and Gad Markovits. 2012. Predicting query performance by query-drift estimation. ACM TOIS, Vol. 30, 2 (2012), 11.
[62]
Gyanit Singh, Nish Parikh, and Neel Sundaresan. 2012. Rewriting Null E-Commerce Queries to Recommend Products. In Proc. of WWW Companion. 73--82.
[63]
Parikshit Sondhi, Mohit Sharma, Pranam Kolari, and ChengXiang Zhai. 2018. A Taxonomy of Queries for E-Commerce Search. In Proc. of SIGIR. 1245--1248.
[64]
Ning Su, Jiyin He, Yiqun Liu, Min Zhang, and Shaoping Ma. 2018. User Intent, Behaviour, and Perceived Satisfaction in Product Search. In Proc. of WSDM. 547--555.
[65]
Hossein Talebi and Peyman Milanfar. 2018. NIMA: Neural image assessment. IEEE Transactions on Image Processing, Vol. 27, 8 (2018), 3998--4011.
[66]
Yongquan Tao and Shengli Wu. 2014. Query performance prediction by considering score magnitude and variance together. In Proc. of CIKM. 1891--1894.
[67]
Huawei Tian, Yanhui Xiao, Gang Cao, Yongsheng Zhang, Zhiyin Xu, and Yao Zhao. 2019. Daxing Smartphone Identification Dataset. IEEE Access, Vol. 7 (2019), 101046--101053.
[68]
Riku Togashi and Tetsuya Sakai. 2020. Visual Intents vs. Clicks, Likes, and Purchases in E-Commerce. In Proc. of SIGIR. 1869--1872.
[69]
Manos Tsagkias, Tracy Holloway King, Surya Kallumadi, Vanessa Murdock, and Maarten de Rijke. 2020. Challenges and research opportunities in ecommerce search and recommendations. SIGIR Forum, Vol. 54, 1 (2020).
[70]
Daniel Tunkelang. 2009. Faceted search. Synthesis lectures on information concepts, retrieval, and services, Vol. 1, 1 (2009), 1--80.
[71]
Hen Tzaban, Ido Guy, Asnat Greenstein-Messica, Arnon Dagan, Lior Rokach, and Bracha Shapira. 2020. Product Bundle Identification Using Semi-Supervised Learning. In Proc. of SIGIR. 791--800.
[72]
Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li. 2014. Deep Learning for Content-Based Image Retrieval: A Comprehensive Study. In Proc. of MM. 157--166.
[73]
Anna Wróblewska and Łukasz Rka czkowski. 2016. Visual Recommendation Use Case for an Online Marketplace Platform: Allegro.Pl. In Proc. of SIGIR. 591--594.
[74]
Fan Yang, Ajinkya Kale, Yury Bubnov, Leon Stein, Qiaosong Wang, Hadi Kiapour, and Robinson Piramuthu. 2017. Visual Search at EBay. In Proc. of KDD. 2101--2110.
[75]
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, Vol. 2 (2014), 67--78.
[76]
Andrew Zhai, Dmitry Kislyuk, Yushi Jing, Michael Feng, Eric Tzeng, Jeff Donahue, Yue Li Du, and Trevor Darrell. 2017. Visual Discovery at Pinterest. In Proc. of WWW Companion. 515--524.
[77]
Andrew Zhai, Hao-Yu Wu, Eric Tzeng, Dong Huk Park, and Charles Rosenberg. 2019. Learning a Unified Embedding for Visual Search at Pinterest. In Proc. of KDD. 2412--2420.
[78]
Yanhao Zhang, Pan Pan, Yun Zheng, Kang Zhao, Jianmin Wu, Yinghui Xu, and Rong Jin. 2019. Virtual ID Discovery from E-Commerce Media at Alibaba: Exploiting Richness of User Click Behavior for Visual Search Relevance. In Proc. of CIKM. 2489--2497.
[79]
Yanhao Zhang, Pan Pan, Yun Zheng, Kang Zhao, Yingya Zhang, Xiaofeng Ren, and Rong Jin. 2018. Visual Search at Alibaba. In Proc. of KDD. 993--1001.
[80]
Ying Zhao, Falk Scholer, and Yohannes Tsegay. 2008. Effective pre-retrieval query performance prediction using similarity and variability evidence. In Proc. of ECIR. 52--64.
[81]
Yun Zhou and W. Bruce Croft. 2007. Query performance prediction in web search environments. In Proc. of SIGIR. 543--550.
[82]
Bin Zhu, Chong-Wah Ngo, Jingjing Chen, and Yanbin Hao. 2019. R2GAN: Cross-modal recipe retrieval with generative adversarial network. In Proc. of CVPR. 11477--11486.

Cited By

View all
  • (2024)Applications of Artificial Intelligence in E-Commerce – From Clicks to ConvictionsAsian Journal of Management10.52711/2321-5763.2024.00032(205-210)Online publication date: 21-Jun-2024
  • (2024)AI-Powered PersonalizationLeveraging AI for Effective Digital Relationship Marketing10.4018/979-8-3693-5340-0.ch006(163-198)Online publication date: 18-Oct-2024
  • (2024)CMLsearch: Semantic visual search and simulation through segmented colour, material, and lighting in interior imageJournal of Computational Design and Engineering10.1093/jcde/qwae11412:1(179-299)Online publication date: 30-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 2021
2998 pages
ISBN:9781450380379
DOI:10.1145/3404835
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. e-commerce search
  2. product search
  3. query log analysis
  4. query performance prediction
  5. search by image
  6. visual search

Qualifiers

  • Research-article

Conference

SIGIR '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)96
  • Downloads (Last 6 weeks)13
Reflects downloads up to 01 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Applications of Artificial Intelligence in E-Commerce – From Clicks to ConvictionsAsian Journal of Management10.52711/2321-5763.2024.00032(205-210)Online publication date: 21-Jun-2024
  • (2024)AI-Powered PersonalizationLeveraging AI for Effective Digital Relationship Marketing10.4018/979-8-3693-5340-0.ch006(163-198)Online publication date: 18-Oct-2024
  • (2024)CMLsearch: Semantic visual search and simulation through segmented colour, material, and lighting in interior imageJournal of Computational Design and Engineering10.1093/jcde/qwae11412:1(179-299)Online publication date: 30-Dec-2024
  • (2023)E-commerce Search via Content Collaborative Graph Neural NetworkProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599320(2885-2897)Online publication date: 6-Aug-2023
  • (2023)Promoting Tail Item Recommendations in E-CommerceProceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3565472.3592968(194-203)Online publication date: 18-Jun-2023
  • (2023)Exploring the Use of the Delphi Method in Visual Search Service Effectiveness Measurement Research in e-Commerce Platform2023 International Conference on Software and System Engineering (ICoSSE)10.1109/ICoSSE58936.2023.00017(52-56)Online publication date: Apr-2023
  • (2023)Shop by image: characterizing visual search in e-commerceInformation Retrieval10.1007/s10791-023-09418-126:1-2Online publication date: 3-Mar-2023
  • (2022)PHOcusProceedings of the VLDB Endowment10.14778/3554821.355486115:12(3630-3633)Online publication date: 1-Aug-2022
  • (2022)A comparison of dataset search behaviour of internal versus search engine referred sessionsProceedings of the 2022 Conference on Human Information Interaction and Retrieval10.1145/3498366.3505821(158-168)Online publication date: 14-Mar-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media