skip to main content
10.1145/3512527.3531426acmconferencesArticle/Chapter ViewAbstractPublication PagesicmrConference Proceedingsconference-collections
short-paper

Unseen Food Segmentation

Published: 27 June 2022 Publication History

Abstract

Food image segmentation is important for detailed analysis on food images, especially for classification of multiple food items and calorie amount estimation. However, there is a costly problem in training a semantic segmentation model because it requires a large number of images with pixel-level annotations. In addition, the existence of a myriad of food categories causes the problem of insufficient data in each category. Although several food segmentation datasets such as the UEC-FoodPix Complete has been released so far, the number of food categories is still limited to a small number.
In this study, we propose an unseen class segmentation method with high accuracy by using both zero-shot and few-shot segmentation methods for any unseen classes. we make the following contributions: (1) we propose a UnSeen Food Segmentation method (USFoodSeg) that uses the zero-shot model to infer the segmentation mask from the class label words of unseen classes and those images, and uses the few-shot model to refine the segmentation masks. (2) We generate segmentation masks for 156 categories of the unseen class UEC-Food256, totaling 17,000 images, and 85 categories in the Food-101 dataset, totaling 85,000 images, with an accuracy of over 90%. Our proposed method is able to solve the problem of insufficient food segmentation data.

Supplementary Material

MP4 File (ICMR_video.mp4)
Presentation video

References

[1]
L. Bossard, M. Guillaumin, and L. Van Gool. 2014. Food-101 Mining Discrimi-native Components with Random Forest. In Proc.of European Conferenceon Computer Vision (ECCV).
[2]
T. Ege and K. Yanai. 2017a. Estimating Food Calories for Multiple-dish Food Photos. In Proc.of the Asian Conference on Pattern Recognition.
[3]
T. Ege and K. Yanai. 2017b. Imag-Based Food Calorie Estimation Using Knowledge on Food Categories, Ingredients and Cooking Directions. In Proc.of the ACM Multimedia Thematic Workshop.
[4]
T. Ege and K. Yanai. 2018a. Image-Based Food Calorie Estimation Using Recipe Information. IEICE Transactions on Information and Systems, Vol. E101.D, 5 (2018), 1333--1341.
[5]
T. Ege and K. Yanai. 2018b. Multi-task Learning of Dish Detection and Calorie Estimation. In Proc.of the IJCAI and ECAI Workshop on Multimedia Assisted Dietary Management.
[6]
T. Ege and K. Yanai. 2019. A New Large-scale Food Image Segmentation Dataset and Its Application to Food Calorie Estimation Based on Grains of Rice. In Proc. of ACMMM Workshop on Multimedia Assisted Dietary Management(MADiMa).
[7]
J. Gao, W. Tan, L. Ma, Y. Wang, and W. Tang. 2019. MUSEFood: Multi-sensor-based Food Volume Estimation on Smartphones. arXiv preprint arXiv:1903.07437 (2019).
[8]
K. He, X. Zhang, S. Ren, and J. Sun. 2015. Deep Residual Learning for Image Recognition. In Proc.of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[9]
Y. Honbu and K. Yanai. 2021. Few-shot and zero-shot semantic segmentation for food images. In Proc. of CEA Workshop on Multimedia for Cooking and Eating Activities (CEA).
[10]
Y. Kawano and K Yanai. 2014. Automatic expansion of a food image dataset leveraging existing categories with domain adaptation. In ECCV Workshop on Transferring and Adapting Source Knowledge in Computer Vision (TASK-CV).
[11]
P. Kingma and M. Welling. 2014. Auto-Encoding Variational Bayes. In Proc.of International Conference on Machine Learning (ICML).
[12]
Y. Li, K. Swersky, and R. Zemel. 2015. Generative Moment Matching Networks. In Proc.of Proceedings of International Conference on Machine Learning (ICML).
[13]
Y. Matsuda, H. Hoashi, and K. Yanai. 2012. Recognition of Multiple-Food Images by Detecting Candidate Regions. In Proc. of IEEE International Conference on Multimedia and Expo (ICME).
[14]
K. Okamoto and K. Yanai. 2021. UEC-FoodPIX Complete: A Large-scale Food Image Segmentation Dataset. In Proc.of ICPR Workshop on Multimedia Assisted Dietary Management(MADiMa).
[15]
C. Rother, V. Kolmogorov, and A. Blake. 2004. "GrabCut": interactive foreground extraction using iterated graph cuts. Proc.of ACM Trans. Graph., Vol. 23, 3 (2004), 309--314.
[16]
A. Salvador, N. Hynes, Y. Aytar, J. Marin, F. Ofli, I. Weber, and A. Torralba. 2017. Learning Cross-Modal Embeddings for Cooking Recipes and Food Images. In Proc.of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[17]
Z. Tian, H. Zhao, M. Shu, Z. Yang, R. Li, and J. Jia. 2020. Prior Guided Feature Enrichment Network for Few-Shot Segmentation. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 44, 2 (2020), 1050--1065.
[18]
W. Xiongwei, F. Xin, L. Ying, L. Ee-Peng, H. Steven, and S. Qianru. 2021. A Large-Scale Benchmark for Food Image Segmentation. In Proc. of ACM Multimedia.

Cited By

View all
  • (2024)Navigating Weight Prediction with Diet DiaryProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680977(127-136)Online publication date: 28-Oct-2024
  • (2024)Multimodal Temporal Fusion Transformers are Good Product Demand ForecastersIEEE MultiMedia10.1109/MMUL.2024.337382731:2(48-60)Online publication date: 1-Apr-2024
  • (2023)A Study on Food Value Estimation From Images: Taxonomies, Datasets, and TechniquesIEEE Access10.1109/ACCESS.2023.327447511(45910-45935)Online publication date: 2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMR '22: Proceedings of the 2022 International Conference on Multimedia Retrieval
June 2022
714 pages
ISBN:9781450392389
DOI:10.1145/3512527
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. few-shot segmentation
  2. food image segmentation
  3. zero-shot segmentation

Qualifiers

  • Short-paper

Funding Sources

  • JSPS KAKENHI Grant

Conference

ICMR '22
Sponsor:

Acceptance Rates

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)52
  • Downloads (Last 6 weeks)12
Reflects downloads up to 01 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Navigating Weight Prediction with Diet DiaryProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680977(127-136)Online publication date: 28-Oct-2024
  • (2024)Multimodal Temporal Fusion Transformers are Good Product Demand ForecastersIEEE MultiMedia10.1109/MMUL.2024.337382731:2(48-60)Online publication date: 1-Apr-2024
  • (2023)A Study on Food Value Estimation From Images: Taxonomies, Datasets, and TechniquesIEEE Access10.1109/ACCESS.2023.327447511(45910-45935)Online publication date: 2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media