ABSTRACT
Understanding the mix-and-match relationships of fashion items receives increasing attention in fashion industry. Existing methods have primarily utilized the visual content to learn the visual compatibility and performed matching in a latent space. Despite their effectiveness, these methods work like a black box and cannot reveal the reasons that two items match well. The rich attributes associated with fashion items, e.g.,off-shoulder dress and black skinny jean, which describe the semantics of items in a human-interpretable way, have largely been ignored.
This work tackles the interpretable fashion matching task, aiming to inject interpretability into the compatibility modeling of items. Specifically, given a corpus of matched pairs of items, we not only can predict the compatibility score of unseen pairs, but also learn the interpretable patterns that lead to a good match, e.g., white T-shirt matches with black trouser. We propose a new solution named A ttribute-based I nterpretable C ompatibility (AIC) method, which consists of three modules: 1) a tree-based module that extracts decision rules on matching prediction; 2) an embedding module that learns vector representation for a rule by accounting for the attribute semantics; and 3) a joint modeling module that unifies the visual embedding and rule embedding to predict the matching score. To justify our proposal, we contribute a new Lookastic dataset with fashion attributes available. Extensive experiments show that AIC not only outperforms several state-of-the-art methods, but also provides good interpretability on matching decisions.
- Ziad Al-Halah, Rainer Stiefelhagen, and Kristen Grauman. 2017. Fashion Forward: Forecasting Visual Style in Fashion. In ICCV. 388--397.Google Scholar
- Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS. 2787--2795. Google ScholarDigital Library
- Leo Breiman. 2017. Classification and regression trees. Routledge.Google Scholar
- Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018. Neural Collective Entity Linking. In COLING. 675--686.Google Scholar
- Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Chengjiang Li, Xu Chen, and Tiansi Dong. 2018. Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervision. In EMNLP. 227--237.Google Scholar
- Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In SIGIR. ACM, 335--344. Google ScholarDigital Library
- Long Chen and Yuhang He. 2018. Dress Fashionably: Learn Fashion Collocation With Deep Mixed-Category Metric Learning. In AAAI. 2103--2110.Google Scholar
- Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In SIGKDD. ACM, 785--794. Google ScholarDigital Library
- Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM, 7--10. Google ScholarDigital Library
- Jerome H. Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189--1232.Google Scholar
- Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogerio Feris. 2018. Dialog-based interactive image retrieval. In Advances in Neural Information Processing Systems. 678--688. Google ScholarDigital Library
- Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. 2017. Automatic spatially-aware fashion concept discovery. In ICCV. IEEE, 1472--1480.Google Scholar
- Xintong Han, Zuxuan Wu, Yu-Gang Jiang, and Larry S. Davis. 2017. Learning fashion compatibility with bidirectional lstms. In ACM MM. ACM, 1078--1086. Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.Google Scholar
- Ruining He and Julian McAuley. 2016. VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback. In AAAI. AAAI Press, 144--150. Google ScholarDigital Library
- Ruining He, Charles Packer, and Julian McAuley. 2016. Learning compatibility across categories for heterogeneous item recommendation. In ICDM. IEEE, 937--942.Google Scholar
- Xiangnan He and Tat-Seng Chua. 2017. Neural factorization machines for sparse predictive analytics. In SIGIR. ACM, 355--364. Google ScholarDigital Library
- Xiangnan He, Zhenkui He, Jingkuan Song, Zhenguang Liu, Yu-Gang Jiang, and Tat-Seng Chua. 2018. NAIS: Neural Attentive Item Similarity Model for Recommendation. EEE Transactions on Knowledge and Data Engineering, Vol. 30, 12 (2018), 2354--2366.Google ScholarDigital Library
- Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In WWW. 173--182. Google ScholarDigital Library
- R. Hong, L. Li, J. Cai, D. Tao, M. Wang, and Q. Tian. 2017. Coherent Semantic-Visual Indexing for Large-Scale Image Retrieval in the Cloud. IEEE Transactions on Image Processing, Vol. 26, 9 (2017), 4128--4138.Google ScholarDigital Library
- Wei-Lin Hsiao and Kristen Grauman. 2017. Learning the latent ?look": Unsupervised discovery of a style-coherent embedding from fashion images. In ICCV.Google Scholar
- Wei-Lin Hsiao and Kristen Grauman. 2018. Creating capsule wardrobes from fashion images. In CVPR. 7161--7170.Google Scholar
- Lizi Liao, Xiangnan He, Bo Zhao, Chong-Wah Ngo, and Tat-Seng Chua. 2018. Interpretable Multimodal Retrieval for Fashion Products. In ACM MM. ACM, 1571--1579. Google ScholarDigital Library
- Lizi Liao, Yunshan Ma, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2018. Knowledge-aware Multimodal Dialogue Systems. In ACM MM. ACM, 801--809. Google ScholarDigital Library
- Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR. 1096--1104.Google Scholar
- Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In SIGIR. ACM, 43--52. Google ScholarDigital Library
- Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI. AUAI Press, 452--461. Google ScholarDigital Library
- Amrita Saha, Mitesh M Khapra, and Karthik Sankaranarayanan. 2018. Towards Building Large Scale Multimodal Domain-Aware Conversation Systems. In AAAI.Google Scholar
- Edgar Simo-Serra and Hiroshi Ishikawa. 2016. Fashion style in 128 floats: joint ranking and classification using weak data for feature extraction. In CVPR. 298--307.Google Scholar
- Xuemeng Song, Fuli Feng, Xianjing Han, Xin Yang, Wei Liu, and Liqiang Nie. 2018. Neural Compatibility Modeling with Attentive Knowledge Distillation. In SIGIR. New York, USA, 5--14. Google ScholarDigital Library
- Xuemeng Song, Fuli Feng, Jinhuan Liu, Zekun Li, Liqiang Nie, and Jun Ma. 2017. Neurostylist: Neural compatibility modeling for clothing matching. In ACM MM. ACM, 753--761. Google ScholarDigital Library
- Mariya I. Vasileva, Bryan A. Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David Forsyth. 2018. Learning Type-Aware Embeddings for Fashion Compatibility. In ECCV. 390--405.Google Scholar
- Andreas Veit, Balazs Kovacs, Sean Bell, Julian McAuley, Kavita Bala, and Serge Belongie. 2015. Learning visual clothing style with heterogeneous dyadic co-occurrences. In ICCV. IEEE, 4642--4650. Google ScholarDigital Library
- Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. KGAT: Knowledge Graph Attention Network for Recommendation. In KDD. Google ScholarDigital Library
- Xiang Wang, Xiangnan He, Fuli Feng, Liqiang Nie, and Tat-Seng Chua. 2018. Tem: Tree-enhanced embedding model for explainable recommendation. In WWW. 1543--1552. Google ScholarDigital Library
- Xiang Wang, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2017. Item silk road: Recommending items from information domains to social users. In SIGIR. ACM, 185--194. Google ScholarDigital Library
- Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural Graph Collaborative Filtering. In SIGIR. ACM. Google ScholarDigital Library
- Qidi Xu, Fumin Shen, Li Liu, and Heng Tao Shen. 2018. GraphCAR: Content-aware Multimedia Recommendation with Graph Autoencoder. In SIGIR. ACM, 981--984. Google ScholarDigital Library
- Xun Yang, Yunshan Ma, Lizi Liao, Meng Wang, and Tat-Seng Chua. 2019. TransNFCM: Translation-Based Neural Fashion Compatibility Modeling. In AAAI.Google Scholar
- Xun Yang, Meng Wang, and Dacheng Tao. 2018. Person Re-Identification With Metric Learning Using Privileged Information. IEEE Transactions on Image Processing, Vol. 27, 2 (2018), 791--805.Google ScholarCross Ref
- Xun Yang, Peicheng Zhou, and Meng Wang. 2018. Person Reidentification via Structural Deep Metric Learning. IEEE Transactions on Neural Networks and Learning Systems 99 (2018), 1--12.Google ScholarCross Ref
- Wenhui Yu, Huidi Zhang, Xiangnan He, Xu Chen, Li Xiong, and Zheng Qin. 2018. Aesthetic-based clothing recommendation. In WWW. 649--658. Google ScholarDigital Library
- Xishan Zhang, Jia Jia, Ke Gao, Yongdong Zhang, Dongming Zhang, Jintao Li, and Qi Tian. 2017. Trip outfits advisor: Location-oriented clothing recommendation. IEEE Transactions on Multimedia, Vol. 19, 11 (2017), 2533--2544.Google ScholarCross Ref
- Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. 2017. Memory-augmented attribute manipulation networks for interactive fashion search. In CVPR. 1520--1528.Google Scholar
- Qian Zhao, Yue Shi, and Liangjie Hong. 2017. Gb-cent: Gradient boosted categorical embedding and numerical trees. In WWW. 1311--1319. Google ScholarDigital Library
Index Terms
- Interpretable Fashion Matching with Rich Attributes
Recommendations
Learning compatibility knowledge for outfit recommendation with complementary clothing matching
AbstractWith the rapid development of mobile networks and e-commerce, clothing recommendation has achieved considerable success in recent years. Fashion outfit matching has become an essential component to users while shopping, which helps ...
FCBoost-Net: A Generative Network for Synthesizing Multiple Collocated Outfits via Fashion Compatibility Boosting
MM '23: Proceedings of the 31st ACM International Conference on MultimediaOutfit generation is a challenging task in the field of fashion technology, in which the aim is to create a collocated set of fashion items that complement a given set of items. Previous studies in this area have been limited to generating a unique set ...
Learning Tuple Compatibility for Conditional Outfit Recommendation
MM '20: Proceedings of the 28th ACM International Conference on MultimediaOutfit recommendation requires the answers of some challenging outfit compatibility questions such as 'Which pair of boots and school bag go well with my jeans and sweater?'. It is more complicated than conventional similarity search, and needs to ...
Comments