Junshi Huang
Shuicheng Yan
Abstract
Taking the shoe as a concrete example, we present an innovative product retrieval system that leverages object detection and retrieval techniques to support a brand-new online shopping experience in this article. The system, called Circle & Search, enables users to naturally indicate any preferred product by simply circling the product in images as the visual query, and then returns visually and semantically similar products to the users. The system is characterized by introducing attributes in both the detection and retrieval of the shoe. Specifically, we first develop an attribute-aware part-based shoe detection model. By maintaining the consistency between shoe parts and attributes, this shoe detector has the ability to model high-order relations between parts and thus the detection performance can be enhanced. Meanwhile, the attributes of this detected shoe can also be predicted as the semantic relations between parts. Based on the result of shoe detection, the system ranks all the shoes in the repository using an attribute refinement retrieval model that takes advantage of query-specific information and attribute correlation to provide an accurate and robust shoe retrieval. To evaluate this retrieval system, we build a large dataset with 17,151 shoe images, in which each shoe is annotated with 10 shoe attributes e.g., heel height, heel shape, sole shape, etc.). According to the experimental result and the user study, our Circle & Search system achieves promising shoe retrieval performance and thus significantly improves the users' online shopping experience.
- Relja Arandjelovic and Andrew Zisserman. 2011. Smooth object retrieval using a bag of boundaries. In Proceedings of the International Conference on Computer Vision (ICCV'11). IEEE, 375--382. Google Scholar
Digital Library
- Tamara L. Berg, Alexander C. Berg, and Jonathan Shih. 2010. Automatic attribute discovery and characterization from noisy web data. In Proceedings of the European Conference on Computer Vision (ECCV'10). Springer, 663--676. Google ScholarDigital Library
- Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. 2011. Describing people: A poselet-based approach to attribute classification. In Proceedings of the International Conference on Computer Vision (ICCV'11). IEEE, 1543--1550. Google ScholarDigital Library
- Huizhong Chen, Andrew Gallagher, and Bernd Girod. 2012. Describing clothing by semantic attributes. In Proceedings of the European Conference on Computer Vision (ECCV'12). Springer, 609--623. Google ScholarDigital Library
- Navneet Dalal and Bill Triggs. 2005. INRIA person dataset. http://pascal.inrialpes.fr/data/human.Google Scholar
- Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09). 1778--1785.Google ScholarCross Ref
- Pedro Felzenszwalb and Daniel Huttenlocher. 2004. Distance transforms of sampled functions. Tech. rep., Department of Computing and Information Science, Cornell. http://www.cs.cornell.edu/~dph/papers/dt.pdf.Google Scholar
- Pedro Felzenszwalb, Ross B. Girshick, David Mcallester, and Deva Ramanan. 2010. Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intel. 32, 9, 1627--1645. Google ScholarDigital Library
- Vittorio Ferrari and Andrew Zisserman. 2008. Learning visual attributes. In Proceedings of the Neural Information Processing Systems Conference (NIPS'08).Google Scholar
- Junfeng He, Jinyuan Feng, Xianglong Liu, Tao Cheng, Tai-Hsu Lin, Hyunjin Chung, and Shih-Fu Chang. 2012. Mobile product search with bag of hash bits and boundary reranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 3005--3012. Google ScholarDigital Library
- Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2011. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intel. 33, 1, 117--128. Google ScholarDigital Library
- Hongwen Kang, Martial Hebert, Alexei A. Efros, and Takeo Kanade. 2012. Connecting missing links: Object discovery from sparse observations using 5 million product images. In Proceedings of the European Conference on Computer Vision (ECCV'12). Springer, 794--807. Google ScholarDigital Library
- Adriana Kovashka, Devi Parikh, and Kristen Grauman. 2012. WhittleSearch: Image search with relative attribute feedback. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 2973--2980. Google ScholarDigital Library
- Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. 2009. Attribute and simile classifiers for face verification. In Proceedings of the International Conference on Computer Vision (ICCV'09). IEEE, 365--372.Google Scholar
- Si Liu, Zheng Song, Guangcan Liu, Changsheng Xu, Hanqing Lu, and Shuicheng Yan. 2012. Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 3330--3337. Google ScholarDigital Library
- Shiyang Lu, Tao Mei, Jingdong Wang, Jian Zhang, Zhiyong Wang, David Dagan Feng, Jian-Tao Sun, and Shipeng Li. 2012. Browse-to-search. In Proceedings of the 20th ACM International Conference on Multimedia (ACM-MM'12). ACM Press, New York, 1323--1324. Google ScholarDigital Library
- Devi Parikh and Kristen Grauman. 2011. Interactively building a discriminative vocabulary of nameable attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 1681--1688. Google ScholarDigital Library
- Xiaohui Shen, Zhe Lin, Jonathan Brandt, and Ying Wu. 2012. Mobile product image search by automatic query object extraction. In Proceedings of the European Conference on Computer Vision (ECCV'12). Springer, 114--127. Google ScholarDigital Library
- Behjat Siddiquie, Rogerio Schmidt Feris, and Larry S. Davis. 2011. Image ranking and retrieval based on multi-attribute queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 801--808. Google ScholarDigital Library
- Yang Wang and Greg Mori. 2010. A discriminative latent model of object classes and attributes. In Proceedings of the European Conference on Computer Vision (ECCV'10). Springer, 155--168. Google ScholarDigital Library
- Yi Yang and Deva Ramanan. 2011. Articulated pose estimation with flexible mixtures-of-parts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 1385--1392. Google ScholarDigital Library
Index Terms
- Circle & Search: Attribute-Aware Shoe Retrieval
Recommendations
DeepShoe: An improved Multi-Task View-invariant CNN for street-to-shop shoe retrieval
AbstractThe difficulty of describing a shoe item seeing on street with text for online shopping demands an image-based retrieval solution. We call this problem street-to-shop shoe retrieval, whose goal is to find exactly the same shoe in the ...
Possibility of guiding arm movement in circle drawing
SMC'09: Proceedings of the 2009 IEEE international conference on Systems, Man and CyberneticsWe tried to guide human action using galvanic vestibular stimulation (GVS). GVS has a possibility of human behavior guidance without any attention. We tried to guide the trajectory of the subjects' hands when as the continuously drew circles. Previous ...
Detecting heel strikes for gait analysis through acceleration flow
In some forms of gait analysis, it is important to be able to capture when the heel strikes occur. In addition, in terms of video analysis of gait, it is important to be able to localise the heel where it strikes on the floor. In this study, a new motion ...
Comments