Fashion analysis and understanding with artificial intelligence

doi:10.1016/j.ipm.2020.102276

Information Processing & Management

Volume 57, Issue 5, September 2020, 102276

https://doi.org/10.1016/j.ipm.2020.102276 Get rights and content

Highlights

•
We introduce the progress in fashion research in recent years.
•
We provide a taxonomy of these fashion studies which include low-level fashion recognition, middle-level fashion understanding and high-level fashion applications.
•
We discuss the challenges that when the fashion industry faces AI technologies.

Abstract

As handling fashion big data with Artificial Intelligence (AI) has become exciting challenges for computer scientists, fashion studies have received increasing attention in computer vision, machine learning and multimedia communities in the past few years. In this paper, introduce the progress in fashion research and provide a taxonomy of these fashion studies that include low-level fashion recognition, middle-level fashion understanding and high-level fashion applications. Finally, we discuss the challenges that when the fashion industry faces AI technologies.

Section snippets

Introdution

From time immemorial, fashion has been intimately associated with being a human as evidenced by beads and jewelry found even in most ancient cultures. In contemporary society, fashion has had a significant effect on every aspect of social life, causing and reflecting changes in social, economic, political, and cultural landscapes. The fashion industry has become one of the biggest segments of the economy in the world, estimated at 3 trillion dollars as of 2018, representing two percent of

Low-level fashion recognition

Fashion recognition focuses on pixel level computation of fashion images, which includes clothing parsing (or human parsing) and landmark detection. Clothing parsing predicts pixel-wise labeling for garment items (e.g., hair, head, upper clothes and pants), which builds a foundation for other fashion understanding tasks. Human parsing further partitions the human body along with clothing items into semantic regions. Clothing/human parsing is extremely challenging due to the wide variety of

Middle-level fashion understanding

Clothing attributes are an informative and compact representation for describing people. As illustrated in Fig. 1, beyond color and pattern, clothing attributes include other important features such as material, collar, length, cut and fastener. Fine-grained clothing attributes recognition can be used for fashion retrieval, fashion recommendation and fashion analysis. Different from clothing attributes, fashion styles emerge organically from how people assemble outfits of clothing, serving as

High-level fashion applications

Supported by the low-level fashion recognition and middle-level fashion understanding techniques, high-level fashion applications blossom in fashion retrieval, fashion recommendation, fashion compatibility, fashion image synthesis and fashion data mining. In the fashion domain, fashion retrieval focuses on identifying clothing items from an image database based on an input query, while fashion recommendation emphasizes recommending clothing items or outfits under certain conditions such as

Fashion benchmark datasets

A variety of benchmark datasets have been introduced and contributed to a comprehensive understanding of fashion. Some datasets are specifically tailored for a particular task such as clothing parsing, style prediction, fashion recommendation, fashion compatibility and fashion trends analysis, while some are designed to evaluate multiple tasks of fashion understanding and analysis simultaneously. Table 2 summarizes the comparison among the most representative fashion datasets.

Discussion and future directions

Currently, some promising results have been achieved in fashion studies including fashion recognition, fashion understanding and fashion applications. Several new techniques have been seamlessly embedded in the products for customers. For instance, fashion recognition and fashion recommendation algorithms have been applied to the world’s largest eCommerce website, Taobao.

However, from the perspective of the fashion industry, the biggest problem is the huge gap between design, manufacturing and

CRediT authorship contribution statement

Xiaoling Gu: Conceptualization, Methodology, Writing - original draft. Fei Gao: Writing - review & editing. Min Tan: Software, Visualization. Pai Peng: Supervision.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants 61802100, 61971172, 61972119, 61702145 and 61971339. This work was also supported by the China Post-Doctoral Science Foundation under Grant 2019M653563.

References (150)

K. Abe et al.
Changing fashion cultures
CoRR
(2017)
K.E. Ak et al.
Learning attribute representations with localization for flexible fashion search
CVPR
(2018)
Z. Al-Halah et al.
Fashion forward: Forecasting visual style in fashion
ICCV
(2017)
G. Balakrishnan et al.
Synthesizing images of humans in unseen poses
CVPR
(2018)
L. Bossard et al.
Apparel classification with style
ACCV
(2012)
C. Bracher et al.
Fashion dna: Merging content and sales data for recommendation and article mapping
CoRR
(2016)
Y.-T. Chang et al.
Fashion world map: Understanding cities through streetwear fashion
ACM multimedia
(2017)
H. Chen et al.
Describing clothing by semantic attributes
ECCV (3)
(2012)
K. Chen et al.
Who are the devils wearing Prada in New York City?
ACM multimedia
(2015)
K.-T. Chen et al.
When fashion meets big data: Discriminative mining of best selling clothing features
WWW (companion volume)
(2017)

L. Chen et al.

Dress fashionably: Learn fashion collocation with deep mixed-category metric learning

AAAI

(2018)

Q. Chen et al.

Deep domain adaptation for describing people based on fine-grained clothing attributes

CVPR

(2015)

W. Chen et al.

Pog: Personalized outfit generation for fashion recommendation at alibaba ifashion

CoRR

(2019)

X. Chen et al.

Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation

SIGIR

(2019)

C.-T. Chou et al.

Pivtons: Pose invariant virtual try-on shoe with conditional image completion

ACCV (6)

(2018)

C. Corbire et al.

Leveraging weakly annotated data for fashion image retrieval and label prediction

ICCV Workshops

(2017)

Y.R. Cui et al.

Fashiongan: Display your fashion design using conditional generative adversarial nets

Computer Graphics Forum : Journal of the European Association for Computer Graphics

(2018)

Z. Cui et al.

Dressing as a whole: Outfit compatibility learning based on node-wise graph neural networks

WWW

(2019)

W. Di et al.

Style finder: Fine-grained clothing style detection and retrieval

CVPR Workshops

(2013)

H. Dong et al.

Soft-gated warping-gan for pose-guided person image synthesis

NEURIPS

(2018)

H. Dong et al.

Towards multi-pose guided virtual try-on network

CoRR

(2019)

J. Dong et al.

Towards unified human parsing and pose estimation

CVPR

(2014)

J. Dong et al.

A deformable mixture parsing model with parselets

ICCV

(2013)

Q. Dong et al.

Multi-task curriculum transfer deep learning of clothing attributes

WACV

(2017)

P. Esser et al.

A variational u-net for conditional appearance and shape generation

CVPR

(2018)

C. Farabet et al.

Learning hierarchical features for scene labeling

IEEE Transactions on Pattern Analysis and Machine Intelligence

(2013)

Z. Feng et al.

Interpretable partitioned embedding for customized multi-item fashion outfit composition

ICMR

(2018)

Y. Ge et al.

Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images

CoRR

(2019)

K. Gong et al.

Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing

CVPR

(2017)

I.J. Goodfellow et al.

Generative adversarial nets

NIPS

(2014)

X. Gu et al.

Understanding fashion trends from street photos via neighbor-constrained embedding learning

ACM multimedia

(2017)

X. Gu et al.

Multi-modal and multi-domain embedding learning for fashion retrieval and analysis

IEEE Transactions on Multimedia

(2019)

X. Guo et al.

Dialog-based interactive image retrieval

NEURIPS

(2018)

M. Gnel et al.

Language guided fashion image manipulation with feature-wise transformations

CoRR

(2018)

X. Han et al.

Automatic spatially-aware fashion concept discovery

ICCV

(2017)

X. Han et al.

Compatible and diverse fashion image inpainting

CoRR

(2019)

X. Han et al.

Learning fashion compatibility with bidirectional lstms

ACM multimedia

(2017)

X. Han et al.

Viton: An image-based virtual try-on network

CVPR

(2018)

R. He et al.

Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering

WWW

(2016)

R. He et al.

Vbpr: Visual Bayesian personalized ranking from implicit feedback

AAAI

(2016)

R. He et al.

Learning compatibility across categories for heterogeneous item recommendation

Icdm

(2016)

S.C. Hidayati et al.

What dress fits me best?: Fashion recommendation on the clothing style for personal body shape

ACM multimedia

(2018)

M. Hou et al.

Explainable fashion recommendation: A semantic attribute region guided approach

CoRR

(2019)

W.-L. Hsiao et al.

Learning the latent ”look”: Unsupervised discovery of a style-coherent embedding from fashion images

ICCV

(2017)

W.-L. Hsiao et al.

Creating capsule wardrobes from fashion images

CVPR

(2018)

W.-L. Hsiao et al.

Fashion++: Minimal edits for outfit improvement

CoRR

(2019)

Y. Hu et al.

Collaborative fashion recommendation: A functional tensor factorization approach

ACM multimedia

(2015)

Z. Hu et al.

Deep generative models with learnable knowledge constraints

NEURIPS

(2018)

J. Huang et al.

Cross-domain image retrieval with a dual attribute-aware ranking network

ICCV

(2015)

C.P. Huynh et al.

Craft: Complementary recommendations using adversarial feature transformer

CoRR

(2018)

Cited by (49)

PFNet: Attribute-aware personalized fashion editing with explainable fashion compatibility analysis
2024, Information Processing and Management
Attribute-aware editing provides a feasible way for users to participate in fashion design. In a sense, an explainable model of fashion compatibility can assist users to perceive the fashionability of their design. However, previous fashion editing researches are opaque in compatibility and rely on labels to manipulate the editing of specific coarse-grained attributes. Consequently, we propose a novel attribute-aware personalized fashion editing network with explainable fashion compatibility modeling, named PFNet, which can simultaneously decouple entangled attributes to make them editable and generate an attribute-wise compatibility explanation for fashion design. In PFNet, we propose an unsupervised garment attribute decoupling network, which independently encodes attributes by hierarchical style control and minimizing mutual correlation. Besides, we develop an attribute compatibility-aware attention network to deeply explore compatible interactions of attributes and visualize their internal decisions. The empirical experiments and user study on the FashionVC and Polyvore datasets reveal that the decoupling accuracy of PFNet for multiple clothing attributes is increased by an average of 22% compared to the state-of-the-art method, and provides more popular compatibility insights with an accuracy rate of 75.5%.
Personalized fashion outfit generation with user coordination preference learning
2023, Information Processing and Management
This paper focuses on personalized outfit generation, aiming to generate compatible fashion outfits catering to given users. Personalized recommendation by generating outfits of compatible items is an emerging task in the recommendation community with great commercial value but less explored. The task requires to explore both user-outfit personalization and outfit compatibility, any of which is challenging due to the huge learning space resulted from large number of items, users, and possible outfit options. To specify the user preference on outfits and regulate the outfit compatibility modeling, we propose to incorporate coordination knowledge in fashion. Inspired by the fact that users might have coordination preference in terms of category combination, we first define category combinations as templates and propose to model user-template relationship to capture users’ coordination preferences. Moreover, since a small number of templates can cover the majority of fashion outfits, leveraging templates is also promising to guide the outfit generation process. In this paper, we propose Template-guided Outfit Generation (TOG) framework, which unifies the learning of user-template interaction, user–item interaction and outfit compatibility modeling. The personal preference modeling and outfit generation are organically blended together in our problem formulation, and therefore can be achieved simultaneously. Furthermore, we propose new evaluation protocols to evaluate different models from both the personalization and compatibility perspectives. Extensive experiments on two public datasets have demonstrated that the proposed TOG can achieve preferable performance in both evaluation perspectives, namely outperforming the most competitive baseline BGN by 7.8% and 10.3% in terms of personalization precision on iFashion and Polyvore datasets, respectively, and improving the compatibility of the generated outfits by over 2%.
Exploring commonly used terms from online reviews in the fashion field to predict review helpfulness
2023, International Journal of Information Management Data Insights
Online shopping for fashion products is a challenging process for consumers. Although customers can facilitate purchasing, review content and helpful voting systems can be unreliable. This study aims to apply linguistic approaches on term recognition to identify and extract frequent terms in fashion reviews and predict their helpfulness. Features are chosen using the latent Dirichlet allocation (LDA) model for topics, bi-grams using the term frequency- inverse document frequency (TF-IDF) vectorizer and topics plus bi-grams using the TF-IDF vectorizer. The feature sets are then used to train four supervised algorithms on an imbalanced dataset to highlight the model performance. Models are validated using a dataset of 828,700 customer reviews collected from Amazon Fashion platform. The experimental results show that choosing LDA plus n-grams using the TF-IDF vectorizer for a random forest classifier outperforms the other models, with an accuracy of 0.81 and an F1-score of 0.78. Furthermore, the study indicates that reviews describing fabric quality, trend and fashion aesthetics, size details, price, and return experience are more helpful. Using the results, customers are made aware of how to narrow their search terms and retailers can optimize their review system more intelligently, especially on the first page of a product's description.
Diagnosing fashion outfit compatibility with deep learning techniques
2023, Expert Systems with Applications
Citation Excerpt :
There are a lot of research fields, such as FRS, facial beauty, clothing detection, landmark detection and others. Gu et al., 2020, present recent progress of artificial intelligence studies in the fashion industry. The researchers also prepare a taxonomy of fashion studies, including compatibility studies and benchmark datasets which are used in the fashion field.
Fashion image understanding is a popular research field with many different machine learning applications. There have been many studies regarding outfit prediction and outfit composition in the field of fashion. However, there are few works that explain the prediction. This paper addresses a method of diagnosing outfit compatibility through clothing images. The proposed system not only predicts compatibility, but also diagnoses incompatible clothing items in outfits. First, a new dataset named ModAI, which has clothing images and compatibility comments from different users was created. After this, a common compatibility comment was created according to user comments for each clothing image. Lastly, image captioning techniques were used to generate compatibility suggestion texts from clothing images. Different segmentation techniques were also used to improve captioning capabilities. The model achieves a 0.62 BLEU-4 score. Experiments show that image captioning techniques can also be used to diagnose outfit compatibility.
Dual Preference Perception Network for Fashion Recommendation in Social Internet of Things
2024, IEEE Internet of Things Journal
A Survey on Fashion Image Retrieval
2024, ACM Computing Surveys

View all citing articles on Scopus

View full text

Fashion analysis and understanding with artificial intelligence

Highlights

Abstract

Section snippets

Introdution

Low-level fashion recognition

Middle-level fashion understanding

High-level fashion applications

Fashion benchmark datasets

Discussion and future directions

CRediT authorship contribution statement

Acknowledgements

Changing fashion cultures

CoRR

Learning attribute representations with localization for flexible fashion search

CVPR

Fashion forward: Forecasting visual style in fashion

ICCV

Synthesizing images of humans in unseen poses

CVPR

Apparel classification with style

ACCV

Fashion dna: Merging content and sales data for recommendation and article mapping

CoRR

Fashion world map: Understanding cities through streetwear fashion

ACM multimedia

Describing clothing by semantic attributes

ECCV (3)

Who are the devils wearing Prada in New York City?

ACM multimedia

When fashion meets big data: Discriminative mining of best selling clothing features

WWW (companion volume)

Dress fashionably: Learn fashion collocation with deep mixed-category metric learning

AAAI

Deep domain adaptation for describing people based on fine-grained clothing attributes

CVPR

Pog: Personalized outfit generation for fashion recommendation at alibaba ifashion

CoRR

Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation

SIGIR

Pivtons: Pose invariant virtual try-on shoe with conditional image completion

ACCV (6)

Leveraging weakly annotated data for fashion image retrieval and label prediction

ICCV Workshops

Fashiongan: Display your fashion design using conditional generative adversarial nets

Computer Graphics Forum : Journal of the European Association for Computer Graphics

Dressing as a whole: Outfit compatibility learning based on node-wise graph neural networks

WWW

Style finder: Fine-grained clothing style detection and retrieval

CVPR Workshops

Soft-gated warping-gan for pose-guided person image synthesis

NEURIPS

Towards multi-pose guided virtual try-on network

CoRR

Towards unified human parsing and pose estimation

CVPR

A deformable mixture parsing model with parselets

ICCV

Multi-task curriculum transfer deep learning of clothing attributes

WACV

A variational u-net for conditional appearance and shape generation

CVPR

Learning hierarchical features for scene labeling

IEEE Transactions on Pattern Analysis and Machine Intelligence

Interpretable partitioned embedding for customized multi-item fashion outfit composition

ICMR

Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images

CoRR

Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing

CVPR

Generative adversarial nets

NIPS

Understanding fashion trends from street photos via neighbor-constrained embedding learning

ACM multimedia

Multi-modal and multi-domain embedding learning for fashion retrieval and analysis

IEEE Transactions on Multimedia

Dialog-based interactive image retrieval

NEURIPS

Language guided fashion image manipulation with feature-wise transformations

CoRR

Computer Graphics Forum : Journal of the European Association for Computer Graphics