Abstract
Food monitoring has become an indispensable practice for personal health management in increasingly growing populations. To facilitate this process, advanced image processing and AI technology have empowered automated recognition of food items and nutrients using food images taken by smart mobile devices. However, precision is often compromised for convenience, which is also applicable in food logging. In this study, we have explored new solutions that can help improve food recognition accuracy with a particular focus on domestic cooking, by leveraging advanced machine learning and natural language processing techniques, in conjunction with comprehensive food nutrient profiles in the knowledge base, as well as contextual ingredient information parsed from publicly available recipes. The optimized models were proved to be effective and have been integrated into an Android app named “FoodInsight”.
This research is supported by the NIH funded IDeA-CTR center at University of Nebraska Medical Center.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – mining discriminative components with random forests. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 446–461. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_29
Ciocca, G., Napoletano, P., Schettini, R.: Food recognition: a new dataset, experiments, and results. IEEE J. Biomed. Health Inform. 21(3), 588–598 (2016)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Fang, S., Liu, C., Zhu, F., Delp, E.J., Boushey, C.J.: Single-view food portion estimation based on geometric models. In: 2015 IEEE International Symposium on Multimedia (ISM), pp. 385–390. IEEE (2015)
Greene, E.: Extracting structured data from recipes using conditional random fields, April 2015. https://open.blogs.nytimes.com/2015/04/09/extracting-structured-data-from-recipes-using-conditional-random-fields/
Hassannejad, H., Matrella, G., Ciampolini, P., De Munari, I., Mordonini, M., Cagnoni, S.: Food image recognition using very deep convolutional networks. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, pp. 41–49 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, Y., Xu, C., Khanna, N., Boushey, C.J., Delp, E.J.: Context based food image analysis. In: 2013 IEEE International Conference on Image Processing, pp. 2748–2752. IEEE (2013)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Kagaya, H., Aizawa, K., Ogawa, M.: Food detection and recognition using convolutional neural network. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 1085–1088 (2014)
Kawano, Y., Yanai, K.: Automatic expansion of a food image dataset leveraging existing categories with domain adaptation. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8927, pp. 3–17. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16199-0_1
Majumder, B.P., Li, S., Ni, J., McAuley, J.: Generating personalized recipes from historical user preferences. arXiv preprint arXiv:1909.00105 (2019)
Matsuda, Y., Hoashi, H., Yanai, K.: Recognition of multiple-food images by detecting candidate regions. In: 2012 IEEE International Conference on Multimedia and Expo, pp. 25–30. IEEE (2012)
Meyers, A., et al.: Im2calories: towards an automated mobile vision food diary. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1233–1241 (2015)
Pandey, P., Deepthi, A., Mandal, B., Puhan, N.B.: Foodnet: recognizing foods using ensemble of deep networks. IEEE Signal Process. Lett. 24(12), 1758–1762 (2017)
Ramshaw, L., Marcus, M.: Text chunking using transformation-based learning (1999)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Singla, A., Yuan, L., Ebrahimi, T.: Food/non-food image classification and food categorization using pre-trained googlenet model. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, pp. 3–11 (2016)
Sun, J., Radecka, K., Zilic, Z.: Foodtracker: a real-time food detection mobile application by deep convolutional neural networks. arXiv preprint arXiv:1909.05994 (2019)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Tan, M., Le, Q.: Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Tanno, R., Okamoto, K., Yanai, K.: Deepfoodcam: A dcnn-based real-time mobile food recognition system. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, pp. 89–89 (2016)
Yanai, K., Kawano, Y.: Food image recognition using deep convolutional network with pre-training and fine-tuning. In: 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1–6. IEEE (2015)
Acknowledgment
The project described is supported by the National Institute of General Medical Sciences, 1U54GM115458, which funds the Great Plains IDeA-CTR Network. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
He, Y., Hakguder, Z., Shi, X. (2022). Smart Diet Management Through Food Image and Cooking Recipe Analysis. In: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (eds) Image Analysis and Processing. ICIAP 2022 Workshops. ICIAP 2022. Lecture Notes in Computer Science, vol 13373. Springer, Cham. https://doi.org/10.1007/978-3-031-13321-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-13321-3_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-13320-6
Online ISBN: 978-3-031-13321-3
eBook Packages: Computer ScienceComputer Science (R0)