Kento Terauchi
ABSTRACT
In recent years, as people become more health-conscious, dietary management has become increasingly important. Existing methods record only one person’s meals or eating movements, but cannot record the meals of multiple people at the same time. Therefore, we aim to record the meals of all people around a dining table using an omnidirectional camera simultaneously.
In this study, we propose CalorieCam360, a system that records the entire dining table using only an omnidirectional camera and a smartphone. Note that all the processing is done inside the smartphone application without using any external servers. Since the images from the omnidirectional camera are distorted and cannot be used for detection as they are, the distortion is corrected using plane projection. The corrected images are used to detect rectangular objects that serve as references for object size, and the area is calculated by combining object detection and region segmentation to estimate the amount of calories from the area. The system then uses person detection and region segmentation to track the person and the food and records the amount of food consumed and its calorie content for each person. We demonstrate that CalorieCam360 can record an entire meal at once for multiple people around the table.
Supplemental Material
- Kento Adachi and Keiji Yanai. 2022. DepthGrillCam: A Mobile Application for Real-Time Eating Action Recording Using RGB-D Images. In Proc. of the 7th International Workshop on Multimedia Assisted Dietary Management.Google Scholar
Digital Library
- Yoshikazu Ando, Takumi Ege, Jaehyeong Cho, and Keiji Yanai. 2019. Depthcaloriecam: A mobile application for volume-based food calorie estimation using depth cameras. In Proc. of International Workshop on Multimedia Assisted Dietary Management.Google ScholarDigital Library
- Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In Proc. of European Conference on Computer Vision.Google ScholarDigital Library
- Shih-Han Chou, Cheng Sun, Wen-Yen Chang, Wan-Ting Hsu, Min Sun, and Jianlong Fu. 2020. 360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images. In Proc. of IEEE Winter Conference on Applications of Computer Vision.Google ScholarCross Ref
- MMSegmentation Contributors. 2020. MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark. https://github.com/open-mmlab/mmsegmentation.Google Scholar
- Takumi Ege, Yoshikazu Ando, Ryosuke Tanno, Wataru Shimoda, and Keiji Yanai. 2019. Image-Based Estimation of Real Food Size for Accurate Food Calorie Estimation. In Proc. of IEEE International Conference on Multimedia Information Processing and Retrieval.Google ScholarCross Ref
- Takumi Ege, Wataru Shimoda, and Keiji Yanai. 2019. A New Large-scale Food Image Segmentation Dataset and Its Application to Food Calorie Estimation Based on Grains of Rice. In Proc. of ACMMM Workshop on Multimedia Assisted Dietary Management.Google ScholarDigital Library
- Takumi Ege and Keiji Yanai. 2017. Estimating Food Calories for Multiple-dish Food Photos. In Proc. of Asian Conference on Pattern Recognition.Google ScholarCross Ref
- Takumi Ege and Keiji Yanai. 2017. Image-Based Food Calorie Estimation Using Knowledge on Food Categories, Ingredients and Cooking Directions. In Proc. of the on Thematic Workshops of ACM Multimedia.Google ScholarDigital Library
- Takumi Ege and Keiji Yanai. 2018. Multi-task Learning of Dish Detection and Calorie Estimation. In Proc. of International Workshop on Multimedia Assisted Dietary Management.Google ScholarDigital Library
- Apple Inc.2023. Vision | Apple Developer Documentation. https://developer.apple.com/documentation/vision.Google Scholar
- Yuji Matsuda, Hajime Hoashi, and Keiji Yanai. 2012. Recognition of Multiple-Food Images by Detecting Candidate Regions. In Proc. of IEEE International Conference on Multimedia and Expo.Google ScholarDigital Library
- Shu Naritomi and Keiji Yanai. 2021. Hungry Networks: 3D Mesh Reconstruction of a Dish and a Plate from a Single Dish Image for Estimating Food Volume. In Proc. of ACM International Conference on Multimedia in Asia.Google ScholarDigital Library
- Koichi Okamoto and Keiji Yanai. 2014. Realtime eating action recognition system on a smartphone. In Proc. of IEEE International Conference on Multimedia and Expo Workshops.Google Scholar
- Koichi Okamoto and Keiji Yanai. 2016. An Automatic Calorie Estimation System of Food Images on a Smartphone. In Proc. of International Workshop on Multimedia Assisted Dietary Management.Google ScholarDigital Library
- Kaimu Okamoto and Keiji Yanai. 2021. UEC-FoodPIX Complete: A Large-scale Food Image Segmentation Dataset. In Proc. of ICPR Workshop on Multimedia Assisted Dietary Management.Google ScholarDigital Library
- Ryosuke Tanno, Takumi Ege, and Keiji Yanai. 2018. AR DeepCalorieCam: An iOS App for Food Calorie Estimation with Augmented Reality. In Proc. of International Conference on Multimedia Modeling.Google ScholarCross Ref
- Ryosuke Tanno, Takumi Ege, and Keiji Yanai. 2018. AR DeepCalorieCam V2: Food Calorie Estimation with CNN and AR-Based Actual Size Estimation. In Proc. of ACM Symposium on Virtual Reality Software and Technology.Google ScholarDigital Library
- Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2022. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696 (2022).Google Scholar
- Wenyan Yang, Yanlin Qian, Joni-Kristian Kämäräinen, Francesco Cricri, and Lixin Fan. 2018. Object Detection in Equirectangular Panorama. In Proc. of International Conference on Pattern Recognition.Google ScholarCross Ref
- Yiming Zhang, Xiangyun Xiao, and Xubo Yang. 2017. Real-Time Object Detection for 360-Degree Panoramic Image Using CNN. In Proc. of International Conference on Virtual Reality and Visualization.Google ScholarCross Ref
Index Terms
- CalorieCam360: Simultaneous Eating Action Recognition of Multiple People Using an Omnidirectional Camera
Recommendations
Generation Method for Immersive Bullet-Time Video Using an Omnidirectional Camera in VR Platform
AVSU'18: Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive MultimediaThis paper proposes a generation method of immersive bullet-time video that continuously switches the images captured by multi-viewpoint omnidirectional cameras arranged around the subject. In ordinary bullet-time processing, it is possible to observe a ...
A simplified two-view geometry based external calibration method for omnidirectional and PTZ camera pairs
We present a new external calibration method for an omnidirectional-PTZ camera pair.The method is based on two-view geometry and requires only two scene points.Unlike alternative methods, it does not make very restrictive assumptions on camera ...
Camera handoff and placement for automated tracking systems with multiple omnidirectional cameras
In a multi-camera surveillance system, both camera handoff and placement play an important role in generating an automated and persistent object tracking, typical of most surveillance requirements. Camera handoff should comprise three fundamental ...
Comments