skip to main content
10.1145/3552468.3555363acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

CI-OCM: Counterfactural Inference towards Unbiased Outfit Compatibility Modeling

Published: 10 October 2022 Publication History

Abstract

As a key task to support intelligent fashion shop construction, outfit compatibility modeling, which aims to estimate whether the given set of fashion items makes a compatible outfit, has attracted much research attention. Although previous efforts have achieved compelling success, they still suffer from the spurious correlation between the category matching and outfit compatibility, which hurts the generalization of the model and misleads the model to be biased. To tackle this problem, we introduce the causal graph tool to analyze the causal relationship among variables of outfit compatibility modeling. In particular, we find that the spurious correlation is attributed to the direct effect of the category information on outfit compatibility prediction by the causal graph. To remove this bad effect from the category information, we present a novel counterfactual inference framework for outfit compatibility modeling, dubbed as CI-OCM. Thereinto, we capture the direct effect of the category information on model prediction in the training phase and then subtract it from the total effect in the testing phase to achieve debiased prediction. Extensive experiments on two splits of a widely-used dataset~(\ie under the independent identically distribution and out-of-distribution assumptions) clearly demonstrate that our CI-OCM can achieve significant improvement over the existing baselines. In addition, we released our code to facilitate the research community.

Supplementary Material

MP4 File (MCFR22-fp014.mp4)
Presentation video.

References

[1]
Ré mi Cadè ne, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2019. RUBi: Reducing Unimodal Biases for Visual Question Answering. In Annual Conference on Neural Information Processing Systems. 839--850.
[2]
Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. 2020. Counterfactual Samples Synthesizing for Robust Visual Question Answering. In Conference on Computer Vision and Pattern Recognition. IEEE/CVF, 10797--10806.
[3]
Konstantina Christakopoulou, Madeleine Traverse, Trevor Potter, Emma Marriott, Daniel Li, Chris Haulk, Ed H. Chi, and Minmin Chen. 2020. Deconfounding User Satisfaction Estimation from Response Rate Bias. In Conference on Recommender Systems. ACM, 450--455.
[4]
Aaron Clauset. 2011. A brief primer on probability distributions. In Santa Fe Institute.
[5]
Guillem Cucurull, Perouz Taslakian, and David Vá zquez. 2019. Context-Aware Visual Compatibility Prediction. In Conference on Computer Vision and Pattern Recognition. IEEE/CVF, 12617--12626.
[6]
Zeyu Cui, Zekun Li, Shu Wu, Xiaoyu Zhang, and Liang Wang. 2019. Dressing as a Whole: Outfit Compatibility Learning Based on Node-wise Graph Neural Networks. In The World Wide Web Conference. ACM, 307--317.
[7]
Fuli Feng, Jizhi Zhang, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. 2021. Empowering Language Understanding with Counterfactual Reasoning. In Findings of the Association for Computational Linguistics. ACL, 2226--2236.
[8]
Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. 2016. Causal inference in statistics: A primer. John Wiley & Sons.
[9]
Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks. Studies in Computational Intelligence, Vol. 385. Springer.
[10]
Weili Guan, Haokun Wen, Xuemeng Song, Chung-Hsing Yeh, Xiaojun Chang, and Liqiang Nie. 2021. Multimodal Compatibility Modeling via Exploring the Consistent and Complementary Correlations. In Proceedings of the International Conference on Multimedia. ACM, 2299--2307.
[11]
Xintong Han, Zuxuan Wu, Yu-Gang Jiang, and Larry S. Davis. 2017. Learning Fashion Compatibility with Bidirectional LSTMs. In Proceedings of the International Conference on Multimedia. ACM, 1078--1086.
[12]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.
[13]
Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations.
[14]
Xingchen Li, Xiang Wang, Xiangnan He, Long Chen, Jun Xiao, and Tat-Seng Chua. 2020. Hierarchical Fashion Graph Network for Personalized Outfit Recommendation. In Proceedings of the International Conference on Research and Development in Information Retrieval. ACM, 159--168.
[15]
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016. Gated Graph Sequence Neural Networks. In International Conference on Learning Representations.
[16]
Rishabh Mehrotra, Prasanta Bhattacharya, and Mounia Lalmas. 2020. Inferring the Causal Impact of New Track Releases on Music Recommendation Platforms through Counterfactual Predictions. In Conference on Recommender Systems. ACM, 687--691.
[17]
Takuma Nakamura and Ryosuke Goto. 2018. Outfit Generation and Style Extraction via Bidirectional LS™ and Autoencoder. CoRR, Vol. abs/1807.03133 (2018).
[18]
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counterfactual VQA: A Cause-Effect Look at Language Bias. In Conference on Computer Vision and Pattern Recognition. IEEE/CVF, 12700--12710.
[19]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kö pf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Annual Conference on Neural Information Processing Systems. 8024--8035.
[20]
Judea Pearl. 2001. Direct and Indirect Effects. In Proceedings of the Conference in Uncertainty in Artificial Intelligence. Morgan Kaufmann, 411--420.
[21]
Judea Pearl. 2009. Causality. Cambridge university press.
[22]
Judea Pearl et al. 2000. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, Vol. 19, 2 (2000).
[23]
Judea Pearl and Dana Mackenzie. 2018. The book of why: the new science of cause and effect. Basic books.
[24]
Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, and Pengjun Xie. 2021. Counterfactual Inference for Text Classification Debiasing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing. ACL, 5434--5445.
[25]
James Robins. 1986. A new approach to causal inference in mortality studies with a sustained exposure period-application to control of the healthy worker survivor effect. Mathematical modelling, Vol. 7, 9--12 (1986), 1393--1512.
[26]
James M Robins. 2003. Semantics of causal DAG models and the identification of direct and indirect effects. Oxford Statistical Science Series (2003), 70--82.
[27]
Donald B Rubin. 1978. Bayesian inference for causal effects: The role of randomization. The Annals of Statistics (1978), 34--58.
[28]
Masahiro Sato, Sho Takemori, Janmajay Singh, and Tomoko Ohkuma. 2020. Unbiased Learning for the Causal Effect of Recommendation. In Conference on Recommender Systems. ACM, 378--387.
[29]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The Graph Neural Network Model. Trans. Neural Networks, Vol. 20, 1 (2009), 61--80.
[30]
Xuemeng Song, Fuli Feng, Jinhuan Liu, Zekun Li, Liqiang Nie, and Jun Ma. 2017. NeuroStylist: Neural Compatibility Modeling for Clothing Matching. In Proceedings of the International Conference on Multimedia. ACM, 753--761.
[31]
Xuemeng Song, Xianjing Han, Yunkai Li, Jingyuan Chen, Xin-Shun Xu, and Liqiang Nie. 2019. GP-BPR: Personalized Compatibility Modeling for Clothing Matching. In Proceedings of the International Conference on Multimedia. ACM, 320--328.
[32]
Xuemeng Song, Liqiang Jing, Dengtian Lin, Zhongzhou Zhao, Haiqing Chen, and Liqiang Nie. 2022. V2P: Vision-to-Prompt based Multi-Modal Product Summary Generation. In The International Conference on Research and Development in Information Retrieval. ACM, 992--1001.
[33]
Tianyu Su, Xuemeng Song, Na Zheng, Weili Guan, Yan Li, and Liqiang Nie. 2021. Complementary Factorization towards Outfit Compatibility Modeling. In Proceedings of the International Conference on Multimedia. ACM, 4073--4081.
[34]
Teng Sun, Wenjie Wang, Liqiang Jing, Yiran Cui, Xuemeng Song, and Liqiang Nie. 2022. Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis. CoRR, Vol. abs/2207.11652 (2022).
[35]
Reuben Tan, Mariya I. Vasileva, Kate Saenko, and Bryan A. Plummer. 2019. Learning Similarity Conditions Without Explicit Supervision. In International Conference on Computer Vision. IEEE/CVF, 10372--10381.
[36]
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased Scene Graph Generation From Biased Training. In Conference on Computer Vision and Pattern Recognition. IEEE/CVF, 3713--3722.
[37]
Mariya I. Vasileva, Bryan A. Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David A. Forsyth. 2018. Learning Type-Aware Embeddings for Fashion Compatibility. In European Conference on Computer Vision (Lecture Notes in Computer Science, Vol. 11220). Springer, 405--421.
[38]
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. 2020. Visual Commonsense R-CNN. In Conference on Computer Vision and Pattern Recognition. IEEE/CVF, 10757--10767.
[39]
Wenjie Wang, Fuli Feng, Xiangnan He, Xiang Wang, and Tat-Seng Chua. 2021a. Deconfounded Recommendation for Alleviating Bias Amplification. In The Conference on Knowledge Discovery and Data Mining. ACM, 1717--1725.
[40]
Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. 2021b. Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue. In The International Conference on Research and Development in Information Retrieval. ACM, 1288--1297.
[41]
Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. 2021. Deconfounded Video Moment Retrieval with Causal Intervention. In The International Conference on Research and Development in Information Retrieval. ACM, 1--10.
[42]
Hanwang Zhang, Zheng-Jun Zha, Yang Yang, Shuicheng Yan, Yue Gao, and Tat-Seng Chua. 2013. Attribute-augmented semantic hierarchy: towards bridging semantic gap and intention gap in image retrieval. In Proceedings of the International Conference on Multimedia. ACM, 33--42.
[43]
Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal Intervention for Leveraging Popularity Bias in Recommendation. In The International Conference on Research and Development in Information Retrieval. ACM, 11--20.

Cited By

View all
  • (2024)Multimodal High-Order Relationship Inference Network for Fashion Compatibility Modeling in Internet of Multimedia ThingsIEEE Internet of Things Journal10.1109/JIOT.2023.328560111:1(353-365)Online publication date: 1-Jan-2024
  • (2024)GCN-ICD: A Graph Convolutional Network for Icing Cover DetectionMultidimensional Signal Processing: Methods and Applications10.1007/978-981-97-5181-5_29(355-368)Online publication date: 11-Dec-2024
  • (2023)Multimodal Cross-Attention Graph Network for Desire DetectionArtificial Neural Networks and Machine Learning – ICANN 202310.1007/978-3-031-44216-2_42(512-523)Online publication date: 26-Sep-2023

Index Terms

  1. CI-OCM: Counterfactural Inference towards Unbiased Outfit Compatibility Modeling

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      MCFR '22: Proceedings of the 1st Workshop on Multimedia Computing towards Fashion Recommendation
      October 2022
      44 pages
      ISBN:9781450394987
      DOI:10.1145/3552468
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 10 October 2022

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. counterfactual inference
      2. fashion analysis
      3. outfit compatibility modeling

      Qualifiers

      • Research-article

      Conference

      MM '22
      Sponsor:

      Acceptance Rates

      MCFR '22 Paper Acceptance Rate 4 of 8 submissions, 50%;
      Overall Acceptance Rate 4 of 8 submissions, 50%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)31
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 19 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Multimodal High-Order Relationship Inference Network for Fashion Compatibility Modeling in Internet of Multimedia ThingsIEEE Internet of Things Journal10.1109/JIOT.2023.328560111:1(353-365)Online publication date: 1-Jan-2024
      • (2024)GCN-ICD: A Graph Convolutional Network for Icing Cover DetectionMultidimensional Signal Processing: Methods and Applications10.1007/978-981-97-5181-5_29(355-368)Online publication date: 11-Dec-2024
      • (2023)Multimodal Cross-Attention Graph Network for Desire DetectionArtificial Neural Networks and Machine Learning – ICANN 202310.1007/978-3-031-44216-2_42(512-523)Online publication date: 26-Sep-2023

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media