short-paper

Automatic Dataset Creation from User-generated Recipes for Ingredient-centric Food Image Analysis

Authors:

Kiyoharu AizawaAuthors Info & Claims

MMAsia '23: Proceedings of the 5th ACM International Conference on Multimedia in Asia

Article No.: 101, Pages 1 - 5

https://doi.org/10.1145/3595916.3626748

Published: 01 January 2024 Publication History

Abstract

We aim to develop an application that automatically creates a nutrition facts label from food images for precise dietary control. Firstly, we constructed a new dataset with food category labels and a list of ingredients in a nutritionally calculable format using an image classification model and BERT for 1.6 million recipes accompanied by images. The nutritional value of the recipe can be calculated using a conversion table consisting of the food item number and unit class. Next, using deep learning techniques, we built models that estimate the list of food item numbers from food images. While the multi-task model that identifies the food category label and the ingredient list simultaneously is only effective within a limited number of recipes, the single-task model that only identified the ingredient list achieved a Micro-F1 of 53.32% in total.

References

[1]

Kiyoharu Aizawa. 2019. FoodLog: Multimedia Food Recording Platform and its Application. MADiMa ’19: Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management, 32–32. https://doi.org/10.1145/3347448.3352809

Digital Library

[2]

Oscar Beijbom, Neel Joshi, Dan Morris, Scott Saponas, and Siddharth Khullar. 2015. Menu-Match: Restaurant-Specific Food Logging from Images. In 2015 IEEE Winter Conference on Applications of Computer Vision. 844–851. https://doi.org/10.1109/WACV.2015.117

Digital Library

[3]

Abhijit Bendale and Terrance Boult. 2015. Towards Open World Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]

Vinay Bettadapura, Edison Thomaz, Aman Parnami, Gregory Abowd, and Irfan Essa. 2015. Leveraging Context to Support Automated Food Recognition in Restaurants. (10 2015). https://doi.org/10.1109/WACV.2015.83

Digital Library

[5]

Marc Bolaños, Aina Ferrà, and Petia Radeva. 2017. Food Ingredients Recognition Through Multi-label Learning. In New Trends in Image Analysis and Processing – ICIAP 2017. Springer International Publishing, Cham, 394–402.

[6]

Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, and Matthieu Cord. 2018. Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR ’18). Association for Computing Machinery, New York, NY, USA, 35–44. https://doi.org/10.1145/3209978.3210036

Digital Library

[7]

Jingjing Chen, Bin Zhu, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. 2021. A Study of Multi-Task and Region-Wise Deep Learning for Food Ingredient Recognition. IEEE Transactions on Image Processing 30 (2021), 1514–1526. https://doi.org/10.1109/TIP.2020.3045639

[8]

Yu Chung Chooi, Cherlyn Ding, and Faidon Magkos. 2019. The epidemiology of obesity. Metabolism 92 (2019), 6–10. https://doi.org/10.1016/j.metabol.2018.09.005

[9]

Mikhail Fain, Andrey Ponikar, Ryan Fox, and Danushka Bollegala. 2019. Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA. CoRR abs/1911.12763 (2019). arXiv:1911.12763http://arxiv.org/abs/1911.12763

[10]

Jun Harashima, Michiaki Ariga, Kenta Murata, and Masayuki Ioki. 2016. A Large-scale Recipe and Meal Data Collection as Infrastructure for Food Research. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) (23-28).

[11]

Jun Harashima, Makoto Hiramatsu, and Satoshi Sanjo. 2020. Calorie Estimation in a Real-World Recipe Service. Proceedings of the AAAI Conference on Artificial Intelligence 34, 08 (Apr. 2020), 13306–13313. https://doi.org/10.1609/aaai.v34i08.7041

[12]

Cookpad Inc.2015. Cookpad data. https://doi.org/10.32130/idr.5.1. dataset.

[13]

Akihisa Ishino, Yoko Yamakata, Hiroaki Karasawa, and Kiyoharu Aizawa. 2021. RecipeLog: Recipe Authoring App for Accurate Food Recording. In Proceedings of the 29th ACM International Conference on Multimedia (Virtual Event, China) (MM ’21). Association for Computing Machinery, New York, NY, USA, 2798–2800. https://doi.org/10.1145/3474085.3478563

Digital Library

[14]

Asker E. Jeukendrup. 2017. Periodized Nutrition for Athletes. Sports medicine (Auckland, N.Z.) 47, Suppl 1 (2017), 51–63. https://doi.org/10.1007/s40279-017-0694-2

[15]

Akiko Kamei. 2015. Nutritional management of athletes in Japan- In the case of Japan Institute of Sports Sciences (in Japanese). Japanese Journal of Physical Fitness and Sports Medicine 64, 1 (2015), 69–69. https://doi.org/10.7600/jspfsm.64.69

[16]

Yoshiyuki Kawano and Keiji Yanai. 2013. Real-Time Mobile Food Recognition System. In 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1–7. https://doi.org/10.1109/CVPRW.2013.5

Digital Library

[17]

Yoshiyuki Kawano and Keiji Yanai. 2014. Food Image Recognition with Deep Convolutional Features. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication(UbiComp ’14 Adjunct). 589–593.

Digital Library

[18]

Haozan Liang, Guihua Wen, Yang Hu, Mingnan Luo, Pei Yang, and Yingxue Xu. 2021. MVANet: Multi-Task Guided Multi-View Attention Network for Chinese Food Recognition. IEEE Transactions on Multimedia 23 (2021), 3551–3561. https://doi.org/10.1109/TMM.2020.3028478

Digital Library

[19]

Niki Martinel, Gian Luca Foresti, and Christian Micheloni. 2016. Wide-Slice Residual Networks for Food Recognition. CoRR abs/1612.06543 (2016). arXiv:1612.06543http://arxiv.org/abs/1612.06543

[20]

Javier Marín, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba. 2021. Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 1 (2021), 187–203. https://doi.org/10.1109/TPAMI.2019.2927476

Digital Library

[21]

Yuji Matsuda and Keiji Yanai. 2012. Multiple-food recognition considering co-occurrence employing manifold ranking. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). 2017–2020.

[22]

Zhao-Yan Ming, Jingjing Chen, Yu Cao, Ciarán Forde, Chong-Wah Ngo, and Tat Seng Chua. 2018. Food Photo Recognition for Dietary Tracking: System and Experiment. In MultiMedia Modeling. Springer International Publishing, Cham, 129–141.

[23]

Ministry of Health, Labour Standards, Japan. 2020. Overview of the Dietary Reference Intakes for Japanese. Retrieved 13 May 2022 from https://www.mhlw.go.jp/content/10900000/000862500.pdf

[24]

K Murakami, S Sasaki, Y Takahashi, K Uenishi, M Yamasaki, Hitomi Hayabuchi, Toshinao Goda, J Oka, K Baba, K Ohki, T Kohri, R Watanabe, and Y Sugiyama. 2008. Misreporting of dietary energy, protein, potassium and sodium in relation to body mass index in young Japanese women. European journal of clinical nutrition 62 (2008), 111–118. https://doi.org/10.1038/sj.ejcn.1602683

[25]

Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. 2017. Learning Cross-modal Embeddings for Cooking Recipes and Food Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[26]

Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. 2017. Learning Cross-Modal Embeddings for Cooking Recipes and Food Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]

The Japanese Ministry of Education, Culture, Sports, Science and Technology. 2015. The Japanese Standard Tables of Food Composition (in English). Retrieved 21 July 2022 from https://www.mext.go.jp/en/policy/science_technology/policy

[28]

Liangyu Wang, Yoko Yamakata, Akiko Sunto, and Kiyoharu Aizawa. 2023. Semi-automatic dataset construction from user-generated recipes for image recognition based dietary assessment. IEICE technical report IE2022-65 122, 385 (02 2023), 35–40.

[29]

Zhiling Wang, Weiqing Min, Zhuo Li, Liping Kang, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. 2022. Ingredient-Guided Region Discovery and Relationship Modeling for Food Category-Ingredient Prediction. IEEE Transactions on Image Processing 31, 5214–5226. https://doi.org/10.1109/TIP.2022.3193763

[30]

Yoko Yamakata, Akihisa Ishino, Akiko Sunto, Sosuke Amano, and Kiyoharu Aizawa. 2022. Recipe-Oriented Food Logging for Nutritional Management. In Proceedings of the 30th ACM International Conference on Multimedia (Lisboa, Portugal) (MM ’22). Association for Computing Machinery, New York, NY, USA, 6898–6904. https://doi.org/10.1145/3503161.3549203

Digital Library

[31]

Qing Yu, Masashi Anzawa, Sosuke Amano, Makoto Ogawa, and Kiyoharu Aizawa. 2018. Food Image Recognition by Personalized Classifier. In 2018 25th IEEE International Conference on Image Processing (ICIP). 171–175. https://doi.org/10.1109/ICIP.2018.8451422

[32]

Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. 2020. Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Cited By

Imajuku YYamakata YAizawa K(2025)FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe GenerationMultiMedia Modeling10.1007/978-981-96-2054-8_30(401-414)Online publication date: 3-Jan-2025
https://doi.org/10.1007/978-981-96-2054-8_30
Wang LYamakata YMaeda RAizawa KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Measure and Improve Your Food: Ingredient Estimation Based Nutrition CalculatorProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3684997(11273-11275)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3684997
Daud NKurnianingtyas D(2024)Food Image Classification for Maternal Nutritional Fulfillment Using MobileNet2024 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT)10.1109/COMNETSAT63286.2024.10862862(529-535)Online publication date: 28-Nov-2024
https://doi.org/10.1109/COMNETSAT63286.2024.10862862

Index Terms

Automatic Dataset Creation from User-generated Recipes for Ingredient-centric Food Image Analysis
1. Applied computing
  1. Life and medical sciences
    1. Health care information systems
2. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia content creation

Recommendations

Food category transfer with conditional cycleGAN and a large-scale food image dataset
CEA/MADiMa '18: Proceedings of the Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management

This paper describes "food image transformation" based on a conditional cycleGAN[A3] (cCycleGAN) with a large-scale food image data collected from the Twitter stream. A cCycleGAN is an extension of CycleGAN, which enables "food category transfer" among ...
Digital Food Sensing and Ingredient Analysis Techniques to Facilitate Human-Food Interface Designs
Interactive technologies that shape the traditional human-food experiences are being explored under the emerging field of Human-Food Interaction (HFI). A key challenge in developing HFI technologies is the digital sensing of food, beverages, and their ...
Measure and Improve Your Food: Ingredient Estimation Based Nutrition Calculator
MM '24: Proceedings of the 32nd ACM International Conference on Multimedia

We developed an application that can easily calculate the nutritional content of a meal by utilizing our multimedia recipe dataset tied to the Nutrition Facts table and an ingredient estimation model. A CLIP-based image recognition model and an ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MMAsia '23: Proceedings of the 5th ACM International Conference on Multimedia in Asia

December 2023

745 pages

ISBN:9798400702051

DOI:10.1145/3595916

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

JSPS KAKENHI Grant
JST AIP

Conference

MMAsia '23

Sponsor:

SIGMM

MMAsia '23: ACM Multimedia Asia

December 6 - 8, 2023

Tainan, Taiwan

Acceptance Rates

Overall Acceptance Rate 59 of 204 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
82
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)4

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Imajuku YYamakata YAizawa K(2025)FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe GenerationMultiMedia Modeling10.1007/978-981-96-2054-8_30(401-414)Online publication date: 3-Jan-2025
https://doi.org/10.1007/978-981-96-2054-8_30
Wang LYamakata YMaeda RAizawa KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Measure and Improve Your Food: Ingredient Estimation Based Nutrition CalculatorProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3684997(11273-11275)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3684997
Daud NKurnianingtyas D(2024)Food Image Classification for Maternal Nutritional Fulfillment Using MobileNet2024 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT)10.1109/COMNETSAT63286.2024.10862862(529-535)Online publication date: 28-Nov-2024
https://doi.org/10.1109/COMNETSAT63286.2024.10862862

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten