Abstract
The management of daily food intake aids to preserve a healthy body, minimize the risk of many diseases, and monitor chronic diseases, such as diabetes and heart problems. To ensure a healthy food intake, artificial intelligence has been widely used for food image recognition and nutrition analysis. Several approaches have been generated using a powerful type of machine learning: deep learning. In this paper, a systematic review is presented for the application of deep learning in food image recognition and nutrition analysis. A methodology of systematic research has been adopted resulting in three main fields: food image classification, food image segmentation and volume estimation of food items providing nutritional information.“57” original articles were selected and synthesized based on the use case of the approach, the employed model, the used data set, the experiment process and finally the main results. In addition, articles of public and private food data sets are presented. It is noted that among the literature review, several deep learning-based studies have shown great results and outperform the conventional methods. However, certain challenges and limitations are presented. Hence, some research directions are proposed to apply in the future to improve the food recognition systems for dietary assessment.
Similar content being viewed by others
References
Frankenfeld CL, Leslie TF, Makara MA. Diabetes, obesity, and recommended fruit and vegetable consumption in relation to food environment sub-types: a cross-sectional analysis of behavioral risk factor surveillance system, united states census, and food establishment data. BMC Public Health. 2015;15(1):1–9.
Alzubaidi L, Zhang J, Humaidi AJ, Al-Dujaili A, Duan Y, Al-Shamma O, Santamaría J, Fadhel MA, Al-Amidie M, Farhan L. Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J Big Data. 2021;8(1):1–74.
Min W, Jiang S, Liu L, Rui Y, Jain R. A survey on food computing. ACM Comput Surv (CSUR). 2019;52(5):1–36.
Dhillon A, Verma GK. Convolutional neural network: a review of models, methodologies and applications to object detection. Progress Artif Intell. 2020;9(2):85–112.
Lo FPW, Sun Y, Qiu J, Lo B. Image-based food classification and volume estimation for dietary assessment: a review. IEEE J Biomed Health Inform. 2020;24(7):1926–39.
Chopra M, Purwar A. Recent studies on segmentation techniques for food recognition: a survey. Arch Comput Methods Eng 2021;1–14.
Jiang L, Qiu B, Liu X, Huang C, Lin K. Deepfood: food image analysis and dietary assessment via deep model. IEEE Access. 2020;8:47477–89.
Ege T, Ando Y, Tanno R, Shimoda W, Yanai K. Image-based estimation of real food size for accurate food calorie estimation. In: 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2019; p. 274–9. IEEE.
Chiang M-L, Wu C-A, Feng J-K, Fang C-Y, Chen S-W. Food calorie and nutrition analysis system based on mask r-CNN. In: 2019 IEEE 5th International Conference on Computer and Communications (ICCC), 2019;p. 1721–8. IEEE.
Ambadkar A, Chaudhari C, Ghadage M, Bhalekar M. A model for automated food logging through food recognition and attribute estimation using deep learning. In: ICT Analysis and Applications, 2021;p. 583–92 . Springer.
Latif G, Alsalem B, Mubarky W, Mohammad N, Alghazo J. Automatic fruits calories estimation through convolutional neural networks. In: Proceedings of the 2020 6th International Conference on Computer and Technology Applications, 2020;p. 17–21.
Goodfellow I, Bengio Y, Courville A. Deep learning. MIT Press; 2016.
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012;25:1097–105.
Simonyan, K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014; arXiv preprint arXiv:1409.1556.
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015;p. 1–9
Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI Conference on Artificial Intelligence 2017.
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016;p. 770–8
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017;p. 4700–8
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. Mobilenets: efficient convolutional neural networks for mobile vision applications. 2017; arXiv preprint arXiv:1704.04861
Tan M, Le Q. Efficientnet: rethinking model scaling for convolutional neural networks. In:International Conference on Machine Learning, 2019;p. 6105–14 . PMLR
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009;p. 248–55. Ieee
Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C. A survey on deep transfer learning. In:International Conference on Artificial Neural Networks, 2018;p. 270–9. Springer
Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014;p. 580–7
Girshick R. Fast r-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, 2015;p. 1440–8
Ren S, He K, Girshick R, Sun J. Faster r-CNN: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst. 2015;28
Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC. Ssd: single shot multibox detector. In: European Conference on Computer Vision, 2016;p. 21–37. Springer
Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016;p. 779–88
He K, Gkioxari G, Dollár P, Girshick R. Mask r-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, 2017;p. 2961–9
Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, 2015;p. 234–41. Springer
Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell. 2017;40(4):834–48.
Medsker L, Jain LC. Recurrent neural networks: design and applications. CRC Press; 1999.
Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Adv Neural Inf Process Syst. 2017;30
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. Adv Neural Inf Process Syst. 2014;27
Bank D, Koenigstein N, Giryes, R. Autoencoders. 2020; arXiv preprint arXiv:2003.05991
Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–33.
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M. \(\{\)TensorFlow\(\}\): a system for \(\{\)Large-Scale\(\}\) machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), 2016;p. 265–83.
Chollet F, et al. Keras. GitHub 2015.
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, et al. Pytorch: An imperative style, high-performance deep learning library. Adv Neural Inf Process Syst. 2019;32.
Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T. Caffe: convolutional architecture for fast feature embedding. In:Proceedings of the 22nd ACM International Conference on Multimedia, 2014;p. 675–8.
Bergstra J, Breuleux O, Bastien F, Lamblin P, Pascanu R, Desjardins G, Turian J, Warde-Farley D, Bengio Y. Theano: a cpu and gpu math compiler in python. In: Proceeding of 9th Python in Science Conf, 2010;vol. 1, p. 3–10
Ran H, Gao W, Mi J, Zhao Z. Fine-grained recognition of Chinese food image based on densenet with attention mechanism. In: Twelfth International Conference on Graphics and Image Processing (ICGIP 2020), 2021;vol. 11720, p. 117201. International Society for Optics and Photonics
Ye H, Zou Q. Food recognition and dietary assessment for healthcare system at mobile device end using mask r-CNN. In: International Conference on Testbeds and Research Infrastructures, 2019;p. 18–35. Springer
Şengür A, Akbulut Y, Budak Ü. Food image classification with deep features. In: 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), 2019;p. 1–6. IEEE
Zhang W, Wu J, Yang Y. Wi-hsnn: a subnetwork-based encoding structure for dimension reduction and food classification via harnessing multi-cnn model high-level features. Neurocomputing. 2020;414:57–66.
Chakrabarty A, Doyle FJ, Dassau E. Deep learning assisted macronutrient estimation for feedforward-feedback control in artificial pancreas systems. In:2018 Annual American Control Conference (ACC), (2018); p. 3564–70. IEEE.
Qiu J, Lo FPW, Sun Y, Wang S, Lo B. Mining discriminative food regions for accurate food recognition 2019.
Deng L, Chen J, Sun Q, He X, Tang S, Ming Z, Zhang Y, Chua TS. Mixed-dish recognition with contextual relation networks. In: Proceedings of the 27th ACM International Conference on Multimedia, 2019;p. 112–20
Mao R, He J, Shao Z, Yarlagadda SK, Zhu F. Visual aware hierarchy based food recognition. In: International Conference on Pattern Recognition, 2021;p. 571–98. Springer.
Tan RZ, Chew X, Khaw KW. Quantized deep residual convolutional neural network for image-based dietary assessment. IEEE Access. 2020;8:111875–88.
Xue G, Liu S, Ma Y. A hybrid deep learning-based fruit classification using attention model and convolution autoencoder. Complex Intell Syst. 2020;1–11
Mandal B, Puhan NB, Verma A. Deep convolutional generative adversarial network-based food recognition using partially labeled data. IEEE Sens Lett. 2018;3(2):1–4.
Minija SJ, Emmanuel WS. Food recognition using neural network classifier and multiple hypotheses image segmentation. Imaging Sci J. 2020;68(2):100–13.
Lu Y, Stathopoulou T, Vasiloglou MF, Pinault LF, Kiley C, Spanakis EK, Mougiakakou S. gofoodtm: an artificial intelligence system for dietary assessment. Sensors. 2020;20(15):4283.
Freitas CN, Cordeiro FR, Macario V. Myfood: a food segmentation and classification system to aid nutritional monitoring. In:2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 2020;p. 234–9 . IEEE
Siemon MS, Shihavuddin A, Ravn-Haren G. Sequential transfer learning based on hierarchical clustering for improved performance in deep learning based food segmentation. Sci Rep. 2021;11(1):1–14.
Pinzón-Arenas JO, Jiménez-Moreno R, Pachón-Suescún CG. Resseg: residual encoder-decoder convolutional neural network for food segmentation. Int J Elect Comput Eng. 2020;10(1):1017.
Abdullahi S, Muangchoo K. Semantic parsing for automatic retail food image recognition. Int J Adv Trend Comput Sci Eng 2020;7808–7816
Shimoda W, Yanai K. Weakly-supervised plate and food region segmentation. In:2020 IEEE International Conference on Multimedia and Expo (ICME), 2020;p. 1–6 . IEEE.
Pfisterer KJ, Amelard R, Chung AG, Syrnyk B, MacLean A, Keller HH, Wong A. When segmentation is not enough: rectifying visual-volume discordance through multisensor depth-refined semantic segmentation for food intake tracking in long-term care. 2019; arXiv preprint arXiv:1910.11250.
Lu Y, Stathopoulou T, Vasiloglou MF, Christodoulidis S, Stanga Z, Mougiakakou S. An artificial intelligence-based system to assess nutrient intake for hospitalised patients. IEEE Trans Multimed. 2020;23:1136–47.
Sharma U, Artacho B, Savakis A. Gourmetnet: food segmentation using multi-scale waterfall features with spatial and channel attention. Sensors. 2021;21(22):7504.
Nguyen H-T, Ngo C-W, Chan W-K. Sibnet: food instance counting and segmentation. Pattern Recogn. 2022;124: 108470.
Park D, Lee J, Lee J, Lee K. Deep learning based food instance segmentation using synthetic data. In: 2021 18th International Conference on Ubiquitous Robots (UR), 2021;p. 499–505. IEEE.
Gao J, Tan W, Ma L, Wang Y, Tang W. Musefood: Multi-sensor-based food volume estimation on smartphones. In: 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), 2019;p. 899–906 . IEEE
Priyaa PK, Sathyapriya S, Arockiam L. Nutrition monitoring and calorie estimation using internet of things (IoT). Int J Innov Technol Explor Eng. 2019;8(11):2669–72.
Jia W, Ren Y, Li B, Beatrice B, Que J, Cao S, Wu Z, Mao Z-H, Lo B, Anderson AK. A novel approach to dining bowl reconstruction for image-based food volume estimation. Sensors. 2022;22(4):1493.
He J, Shao Z, Wright J, Kerr D, Boushey C, Zhu F. Multi-task image-based dietary assessment for food recognition and portion size estimation. In: 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2020;p. 49–54. IEEE.
Hafiz R, Haque MR, Rakshit A, Uddin MS. Image-based soft drink type classification and dietary assessment system using deep convolutional neural network with transfer learning. J King Saud Univ Comput Inf Sci 2020.
Yang Z, Yu H, Cao S, Xu Q, Yuan D, Zhang H, Jia W, Mao Z-H, Sun M. Human-mimetic estimation of food volume from a single-view rgb image using an AI system. Electronics. 2021;10(13):1556.
Yang Y, Jia W, Bucher T, Zhang H, Sun M. Image-based food portion size estimation using a smartphone without a fiducial marker. Public Health Nutr. 2019;22(7):1180–92.
Fang S, Shao Z, Mao R, Fu C, Delp EJ, Zhu F, Kerr DA, Boushey CJ. Single-view food portion estimation: learning image-to-energy mappings using generative adversarial networks. In: 2018 25th IEEE International Conference on Image Processing (ICIP), 2018;p. 251–5. IEEE
Graikos A, Charisis V, Iakovakis D, Hadjidimitriou S, Hadjileontiadis L. Single image-based food volume estimation using monocular depth-prediction networks. In:International Conference on Human-Computer Interaction, 2020;p. 532–43. Springer.
Makhsous S, Bharadwaj M, Atkinson BE, Novosselov IV, Mamishev AV. Dietsensor: automatic dietary intake measurement using mobile 3d scanning sensor for diabetic patients. Sensors. 2020;20(12):3380.
Muralidhar ES, Gowtham TS, Jain A, Padmaveni K. Development of health monitoring application using machine learning on android platform. In:2020 5th International Conference on Communication and Electronics Systems (ICCES), 2020;p. 1076–85. IEEE.
Lo FP-W, Sun Y, Qiu J, Lo BP. Point2volume: a vision-based dietary assessment approach using view synthesis. IEEE Trans Industr Inf. 2019;16(1):577–86.
Rhyner D, Loher H, Dehais J, Anthimopoulos M, Shevchik S, Botwey RH, Duke D, Stettler C, Diem P, Mougiakakou S. Carbohydrate estimation by a mobile phone-based system versus self-estimations of individuals with type 1 diabetes mellitus: a comparative study. J Med Internet Res. 2016;18(5):5567.
Hou S, Feng Y, Wang Z. Vegfru: a domain-specific dataset for fine-grained visual categorization. In: Proceedings of the IEEE International Conference on Computer Vision, 2017;p. 541–9.
Mureşan H, Oltean M. Fruit recognition from images using deep learning. 2017; arXiv preprint arXiv:1712.00580 .
Waltner G, Schwarz M, Ladstatter S, Weber A, Luley P, Lindschinger M, Schmid I, Scheitz W, Bischof H, Paletta L. Personalized dietary self-management using mobile vision-based assistance. In: Proceeding of ICIAP Workshop on Multimedia Assisted Dietary Management (MADIMA) 2017.
Matsuda Y, Hoashi H, Yanai K. Recognition of multiple-food images by detecting candidate regions. In: 2012 IEEE International Conference on Multimedia and Expo, 2012;p. 25–30 . IEEE.
Kawano Y, Yanai K. Automatic expansion of a food image dataset leveraging existing categories with domain adaptation. In: European Conference on Computer Vision, 2014;p. 3–17. Springer.
Bossard L, Guillaumin M, Van Gool L. Food-101–mining discriminative components with random forests. In: European Conference on Computer Vision, 2014;p. 446–61. Springer.
Farinella GM, Allegra D, Stanco F. A benchmark dataset to study the representation of food images. In: European Conference on Computer Vision, 2014;p. 584–99. Springer.
Pouladzadeh P, Yassine A, Shirmohammadi S. Foodd: food detection dataset for calorie measurement using food images. In: International Conference on Image Analysis and Processing, 2015;p. 441–8. Springer
Kaur P, Sikka K, Wang W, Belongie S, Divakaran A. Foodx-251: a dataset for fine-grained food classification. 2019; arXiv preprint arXiv:1907.06167
Ciocca G, Napoletano P, Schettini R. Learning CNN-based features for retrieval of food images. In: International Conference on Image Analysis and Processing, 2017;p. 426–34 . Springer.
Min W, Liu L, Wang Z, Luo Z, Wei X, Wei X, Jiang S. Isia food-500: a dataset for large-scale food recognition via stacked global-local attention network. In: Proceedings of the 28th ACM International Conference on Multimedia, 2020;p. 393–401.
Chen J, Ngo C-W. Deep-based ingredient recognition for cooking recipe retrieval. In: Proceedings of the 24th ACM International Conference on Multimedia, 2016;p. 32–41.
Chen X, Zhu Y, Zhou H, Diao L, Wang D. Chinesefoodnet: a large-scale image dataset for Chinese food recognition. 2017; arXiv preprint arXiv:1705.02743
Jalal M, Wang K, Jefferson S, Zheng Y, Nsoesie EO, Betke M. Scraping social media photos posted in Kenya and elsewhere to detect and analyze food types. In: Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management, 2019;pp. 50–9.
Chen M, Dhingra K, Wu W, Yang L, Sukthankar R, Yang J. Pfid: pittsburgh fast-food image dataset. In: 2009 16th IEEE International Conference on Image Processing (ICIP), 2009;p. 289–92. IEEE
Ciocca G, Napoletano P, Schettini R. Food recognition: a new dataset, experiments and results. IEEE J Biomed Health Inform. 2017;21(3):588–98. https://doi.org/10.1109/JBHI.2016.2636441.
Termritthikun C, Muneesawang P, Kanprachar S. Nu-innet: Thai food image recognition using convolutional neural networks on smartphone. J Telecommun Electron Comput Eng (JTEC). 2017;9(2–6):63–7.
Min W, Wang Z, Liu Y, Luo M, Kang L, Wei X, Wei X, Jiang S. Large scale visual food recognition. arXiv preprint 2021; arXiv:2103.16107.
Tran AC, Tran NC, Duong-Trung N. Recognition and quantity estimation of pastry images using pre-training deep convolutional networks. In: International Conference on Future Data and Security Engineering, 2020;p. 200–14. Springer.
Cai Q, Li J, Li H, Weng Y. Btbufood-60: dataset for object detection in food field. In: 2019 IEEE International Conference on Big Data and Smart Computing (BigComp), 2019;p. 1–4 . IEEE.
Lee GG, Huang C-W, Chen J-H, Chen S-Y, Chen H-L. Aifood: a large scale food images dataset for ingredient recognition. In: TENCON 2019-2019 IEEE Region 10 Conference (TENCON), 2019;p. 802–5. IEEE.
Güngör C, Baltacı F, Erdem A, Erdem E. Turkish cuisine: a benchmark dataset with Turkish meals for food recognition. In: 2017 25th Signal Processing and Communications Applications Conference (SIU), 2017;p. 1–4. IEEE.
Dadgar M, Joshi KD. The role of information and communication technology in self-management of chronic diseases: an empirical investigation through value sensitive design. J Assoc Inf Syst. 2018;19(2):2.
Sami W, Ansari T, Butt NS, Ab Hamid MR. Effect of diet on type 2 diabetes mellitus: a review. Int J Health Sci. 2017;11(2):65.
Chauveau P, Aparicio M, Bellizzi V, Campbell K, Hong X, Johansson L, Kolko A, Molina P, Sezer S, Wanner C. Mediterranean diet as the diet of choice for patients with chronic kidney disease. Nephrol Dial Transplant. 2018;33(5):725–35.
Ahmed A, Boopathy P. Artificial intelligence for the novel corona virus (Covid-19) pandemic: opportunities, challenges, and future directions. Int J E-Health Med Commun (IJEHMC). 2021;13(2):1–21.
Raju VB, Sazonov E. Detection of oil-containing dressing on salad leaves using multispectral imaging. IEEE Access. 2020;8:86196–206.
Sari YA, Dewi RK, Maligan JM, Ananta AS, Adinugroho S. Automatic food leftover estimation in tray box using image segmentation. In: 2019 International Conference on Sustainable Information Engineering and Technology (SIET), 2019;p. 212–6. IEEE.
Monsat, C. In the secrets of Moroccan gastronomy. https://www.lefigaro.fr/gastronomie/2015/02/09/30005-20150209ARTFIG00282-dans-les-secrets-de-la-gastronomie-marocaine.php
Adams WK. Martinez: summary health statistics for the us population: national health interview survey, 2012. Vital and Health statistics. Series 10, Data From the National Health Survey, 2013;1–95
Yoshimura Y, Zaima N. Application of mass spectrometry imaging for visualizing food components. Foods. 2020;9(5):575.
Huang L, Luo R, Liu X, Hao X. Spectral imaging with deep learning. Light Sci Appl. 2022;11(1):1–19.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Mansouri, M., Benabdellah Chaouni, S., Jai Andaloussi, S. et al. Deep Learning for Food Image Recognition and Nutrition Analysis Towards Chronic Diseases Monitoring: A Systematic Review. SN COMPUT. SCI. 4, 513 (2023). https://doi.org/10.1007/s42979-023-01972-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42979-023-01972-1