Lei Meng
Xiangnan He
ABSTRACT
Visually-aware food recommendation recommends food items based on their visual features. Existing methods typically use the pre-extracted visual features from food classification models, which mainly encode the visual content with limited semantic information, such as the classes and ingredients. Therefore, such features may not cover the personalized visual preferences of users, termed collaborative information, e.g. users may attend to different colors and textures of food based on their preferred ingredients and cooking methods. To address this problem, this paper presents a heterogeneous multi-task learning framework, termed privileged-channel infused network (PiNet). It learns the visual features that contain both the semantic and collaborative information by training the image encoder to simultaneously fulfill the ingredient prediction and food recommendation tasks. However, the heterogeneity between the two tasks may lead to different visual information in need and different directions in model parameter optimization. To handle these challenges, PiNet first employs a dual-gating module (DGM) to enable the encoding and passing of different visual information from the image encoder to individual tasks. Secondly, PiNet adopts a two-phase training strategy and two prior knowledge incorporation methods to ensure an effective model training. Experimental results from two real-world datasets show that the visual features generated by PiNet better attend to the informative image regions, yielding superior performance.
Supplemental Material
- Marc Bola nos, Aina Ferrà, and Petia Radeva. 2017. Food ingredients recognition through multi-label learning. In International Conference on Image Analysis and Processing. Springer, 394--402.Google Scholar
- Chih-Ming Chen, Chuan-Ju Wang, Ming-Feng Tsai, and Yi-Hsuan Yang. 2019 b. Collaborative Similarity Embedding for Recommender Systems. In The World Wide Web Conference. ACM, 2637--2643.Google Scholar
- Jingjing Chen and Chong-Wah Ngo. 2016. Deep-based ingredient recognition for cooking recipe retrieval. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 32--41.Google ScholarDigital Library
- Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Hongyuan Zha, and Zheng Qin. 2019 a. Personalized Fashion Recommendation with Visual Explanations based on Multi-model Attention Network. In SIGIR. 1--10.Google Scholar
- Zhiyong Cheng, Xiaojun Chang, Lei Zhu, Rose C. Kanjirathinkal, and Mohan Kankanhalli. 2019. MMALFM: Explainable Recommendation by Leveraging Reviews and Images. ACM Transactions on Information Systems, Vol. 37, 2 (2019), 16:1--16:28.Google ScholarDigital Library
- Wei-Ta Chu and Ya-Lun Tsai. 2017. A hybrid recommendation system considering visual information for predicting favorite restaurants. World Wide Web, Vol. 20, 6 (2017), 1313--1331.Google ScholarDigital Library
- David Elsweiler, Christoph Trattner, and Morgan Harvey. 2017. Exploiting food choice biases for healthier recipe recommendation. In Proceedings of the 40th international acm sigir conference on research and development in information retrieval. ACM, 575--584.Google ScholarDigital Library
- Jill Freyne and Shlomo Berkovsky. 2010. Intelligent food planning: personalized recipe recommendation. In Proceedings of the 15th international conference on Intelligent user interfaces. ACM, 321--324.Google ScholarDigital Library
- Xiaoyan Gao, Fuli Feng, Xiangnan He, Heyan Huang, Xinyu Guan, Chong Feng, Zhaoyan Ming, and Tat-Seng Chua. 2019. Hierarchical Attention Network for Visually-aware Food Recommendation. IEEE Transactions on Multimedia, Vol. In press (2019), 1--12.Google Scholar
- Mouzhi Ge, Mehdi Elahi, Ignacio Fernaández-Tob'ias, Francesco Ricci, and David Massimo. 2015a. Using tags and latent factors in a food recommender system. In Proceedings of the 5th International Conference on Digital Health. ACM, 105--112.Google ScholarDigital Library
- Mouzhi Ge, Francesco Ricci, and David Massimo. 2015b. Health-aware food recommender system. In Proceedings of the 9th ACM Conference on Recommender Systems. ACM, 333--334.Google ScholarDigital Library
- Ruining He and Julian McAuley. 2016. VBPR: visual bayesian personalized ranking from implicit feedback. In Thirtieth AAAI Conference on Artificial Intelligence .Google ScholarCross Ref
- Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (2020), 639--648.Google ScholarDigital Library
- W. Kang, C. Fang, Z. Wang, and J. McAuley. 2017. Visually-Aware Fashion Recommendation and Design with Generative Image Models. In 2017 IEEE International Conference on Data Mining (ICDM). 207--216.Google Scholar
- Deborah A Kerr et al. 2016. The connecting health and technology study: a 6-month randomized controlled trial to improve nutrition behaviours using a mobile food record and text messaging support in young adults. International Journal of Behavioral Nutrition and Physical Activity, Vol. 13, 1 (2016).Google ScholarCross Ref
- Fang-Fei Kuo, Cheng-Te Li, Man-Kwan Shan, and Suh-Yin Lee. 2012. Intelligent menu planning: Recommending set of recipes by ingredients. In Proceedings of the ACM multimedia 2012 workshop on Multimedia for cooking and eating activities. ACM, 1--6.Google ScholarDigital Library
- Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020. Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining. 304--312.Google ScholarDigital Library
- Chia-Jen Lin, Tsung-Ting Kuo, and Shou-De Lin. 2014. A content-based matrix factorization model for recipe recommendation. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 560--571.Google ScholarCross Ref
- Niki Martinel, Gian Luca Foresti, and Christian Micheloni. 2018. Wide-slice residual networks for food recognition. In IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 567--576.Google ScholarCross Ref
- Lei Meng, Long Chen, Xun Yang, Dacheng Tao, Hanwang Zhang, Chunyan Miao, and Tat-Seng Chua. 2019. Learning using privileged information for food recognition. In ACM international conference on Multimedia. ACM, 1--9.Google ScholarDigital Library
- Michele Merler, Hui Wu, Rosario Uceda-Sosa, Quoc-Bao Nguyen, and John R Smith. 2016. Snap, Eat, RepEat: a food recognition engine for dietary logging. In Proceedings of the 2nd international workshop on multimedia assisted dietary management. ACM, 31--40.Google ScholarDigital Library
- Austin Meyers, Nick Johnston, Vivek Rathod, Anoop Korattikara, Alex Gorban, Nathan Silberman, Sergio Guadarrama, George Papandreou, Jonathan Huang, and Kevin P Murphy. 2015. Im2Calories: towards an automated mobile vision food diary. In Proceedings of the IEEE International Conference on Computer Vision. 1233--1241.Google ScholarDigital Library
- Zhaoyan Ming, Jingjing Chen, Yu Cao, Ciarán Forde, Chong-Wah Ngo, and Tat Seng Chua. 2018. Food Photo Recognition for Dietary Tracking: System and Experiment. In International Conference on Multimedia Modeling. Springer, 129--141.Google Scholar
- Charles Packer, Julian McAuley, and Arnau Ramisa. 2018. Visually-Aware Personalized Recommendation using Interpretable Image Representations. In AI for Fashion workshop, held in conjunction with KDD. 1--4.Google Scholar
- Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09). AUAI Press, 452--461.Google ScholarDigital Library
- Doyen Sahoo, Wang Hao, Shu Ke, Wu Xiongwei, Hung Le, Palakorn Achananuparp, Ee-Peng Lim, and Steven CH Hoi. 2019. FoodAI: Food Image Recognition via Deep Learning for Smart Food Logging. In KDD. 2260--2268.Google Scholar
- Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision. 618--626.Google ScholarCross Ref
- Chun-Yuen Teng, Yu-Ru Lin, and Lada A Adamic. 2012. Recipe recommendation using ingredient networks. In Proceedings of the 4th Annual ACM Web Science Conference. ACM, 298--307.Google ScholarDigital Library
- Christoph Trattner and David Elsweiler. 2017. Investigating the healthiness of internet-sourced recipes: implications for meal planning and recommender systems. In Proceedings of the 26th international conference on world wide web. International World Wide Web Conferences Steering Committee, 489--498.Google ScholarDigital Library
- Mayumi Ueda, Mari Takahata, and Shinsuke Nakajima. 2011. User's food preference extraction for personalized cooking recipe recommendation. In Workshop of ISWC. 98--105.Google Scholar
- Youri van Pinxteren, Gijs Geleijnse, and Paul Kamsteeg. 2011. Deriving a recipe similarity measure for recommending healthful meals. In Proceedings of the 16th international conference on Intelligent user interfaces. ACM, 105--114.Google ScholarDigital Library
- Vladimir Vapnik and Rauf Izmailov. 2015. Learning using privileged information: similarity control and knowledge transfer. Journal of machine learning research, Vol. 16, 2023--2049 (2015), 2.Google ScholarDigital Library
- Vladimir Vapnik and Akshay Vashist. 2009. A new learning paradigm: Learning using privileged information. Neural networks, Vol. 22, 5--6 (2009), 544--557.Google Scholar
- Liping Wang, Qing Li, Na Li, Guozhu Dong, and Yu Yang. 2008. Substructure similarity measurement in chinese recipes. In Proceedings of the 17th international conference on World Wide Web. ACM, 979--988.Google ScholarDigital Library
- Suhang Wang, Yilin Wang, Jiliang Tang, Kai Shu, Suhas Ranganath, and Huan Liu. 2017. What your images reveal: Exploiting visual contents for point-of-interest recommendation. In Proceedings of the 26th International Conference on World Wide Web. 391--400.Google ScholarDigital Library
- Longqi Yang, Yin Cui, Fan Zhang, John P. Pollak, Serge Belongie, and Deborah Estrin. 2015. PlateClick: Bootstrapping Food Preferences Through an Adaptive Visual Interface. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (Melbourne, Australia) (CIKM'15). ACM, New York, NY, USA, 183--192.Google ScholarDigital Library
- Longqi Yang, Cheng-Kang Hsieh, Hongjian Yang, John P. Pollak, Nicola Dell, Serge Belongie, Curtis Cole, and Deborah Estrin. 2017a. Yum-Me: A Personalized Nutrient-Based Meal Recommender System. ACM Transactions on Information Systems, Vol. 36, 1, Article 7 (2017), 31 pages.Google ScholarDigital Library
- Xun Yang, Meng Wang, Richang Hong, Qi Tian, and Yong Rui. 2017c. Enhancing person re-identification in a self-trained subspace. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 13, 3 (2017), 1--23.Google ScholarDigital Library
- Xun Yang, Meng Wang, and Dacheng Tao. 2017b. Person re-identification with metric learning using privileged information. IEEE Transactions on Image Processing, Vol. 27, 2 (2017), 791--805.Google ScholarCross Ref
- Yongfeng Zhang, Qingyao Ai, Xu Chen, and W. Bruce Croft. 2017. Joint Representation Learning for Top-N Recommendation with Heterogeneous Information Sources. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 1449--1458.Google Scholar
Index Terms
- Heterogeneous Fusion of Semantic and Collaborative Information for Visually-Aware Food Recommendation
Recommendations
Self-supervised Calorie-aware Heterogeneous Graph Networks for Food Recommendation
With the rapid development of online recipe sharing platforms, food recommendation is emerging as an important application. Although recent studies have made great progress on food recommendation, they have two shortcomings that are likely to affect the ...
Collaborative information and semantic information Fusion over Heterogeneous information Network for Top-N Recommendation System
CCRIS '23: Proceedings of the 2023 4th International Conference on Control, Robotics and Intelligent SystemIn recent years, additional information in the side of both users and items are more and more common in the natural world. Traditional recommendation methods mainly utilize the the user-item rating information. These information, if properly used, can ...
Meta-Graph Based Recommendation Fusion over Heterogeneous Information Networks
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningHeterogeneous Information Network (HIN) is a natural and general representation of data in modern large commercial recommender systems which involve heterogeneous types of data. HIN based recommenders face two problems: how to represent the high-level ...
Comments