Abstract
The world is full of objects with different properties, such as volume, density, mass, material, etc. These properties are crucial for meaningful interaction with these objects. However, none of these properties is explicitly known. As humans rely on perception to interact with the world, robots need to understand the properties of objects to function effectively. This includes inferring properties that may not be explicitly stated, such as an object’s mass. A single robot might not be sufficient to execute complex tasks in dynamic environments. In such cases, an agent can assemble a team of robots at runtime to complete the task. The agent must estimate the object’s mass to make this decision effectively. For example, if a team of robots needs to move a heavy object, accurately estimating its mass is essential. This information allows the agent to determine the appropriate team size, ensuring enough collective strength to complete the task successfully. In this work, we describe techniques that enable robots to learn to fill these knowledge gaps (missing properties) before attempting to interact with these objects. The salient features of our work lie in our observation that the properties of an object are not entirely independent of each other, and learning to predict/infer one can give clues regarding predicting/inferring the other. Our evaluation indicates that the proposed solutions outperform the existing state-of-the-art method. We also simulated estimating the object’s properties in the ROS environment.
















Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The dataset used for this reserach is available at the following link. https://github.com/tstandley/image2mass.
References
Agmon N, Stone P. Leading ad hoc agents in joint action settings with multiple teammates. In: AAMAS; 2012. pp. 341–348.
Aujeszky T, Korres G, Eid M, Khorrami F. Estimating weight of unknown objects using active thermography. Robotics. 2019;8(4):1–13.
Balaban MO, Ünal Şengör GF, Soriano MG, Ruiz EG. Using image analysis to predict the weight of Alaskan salmon of different species. J Food Sci. 2010;75(3):E157–62.
Bell S, Upchurch P, Snavely N, Bala K. Opensurfaces: a richly annotated catalog of surface appearance. ACM Trans Gr. 2013;32(4):1–17.
Bell S, Upchurch P, Snavely N, Bala K. Material recognition in the wild with the materials in context database. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. pp. 3479–3487.
Bozkurt Y, Aktan S, Ozkaya S. Body weight prediction using digital image analysis for slaughtered beef cattle. J Appl Anim Res. 2007;32(2):195–8.
Cieslak MC, Castelfranco AM, Roncalli V, Lenz PH, Hartline DK. t-distributed stochastic neighbor embedding (t-sne): a tool for eco-physiological transcriptomic analysis. Mar Genom. 2020;51: 100723.
Dohmen R, Catal C, Liu Q. Image-based body mass prediction of heifers using deep neural networks. Biosys Eng. 2021;204:283–93.
Flanagan JR, Vetter P, Johansson RS, Wolpert DM. Prediction precedes control in motor learning. Curr Biol. 2003;13(2):146–50.
Hamdan M, Rover D, Darr M, Just J. Mass estimation from images using deep neural network and sparse ground truth. In: 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA). IEEE; 2019. pp. 1987–1992.
Jiang Y, Lim M, Zheng C, Saxena A. Learning to place new objects in a scene. Int J Robot Res. 2012;31(9):1021–43.
Kingma DP, Ba J. Adam: a method for stochastic optimization. 2014. arXiv preprint arXiv:1412.6980
Konovalov DA, Saleh A, Efremova DB, Domingos JA, Jerry DR (2019) Automatic weight estimation of harvested fish from images. In: 2019 digital image computing: techniques and applications (DICTA). IEEE. p. 1–7.
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012;25:1097–105.
Krotkov E. Robotic perception of material. In: IJCAI. 1995. pp. 88–95.
Le S, Lee M, Fang A. Non-linear image-based regression of body segment parameters. In: 13th International conference on biomedical engineering. Springer; 2009. pp. 2038–2042.
Mavrakis N, Stolkin R. Estimation and exploitation of objects’ inertial parameters in robotic grasping and manipulation: A survey. Robot Auton Syst. 2020;124: 103374.
Nath A, Arun AR, Niyogi R. A distributed approach for road clearance with multi-robot in urban search and rescue environment. Int J Intell Robot Appl. 2019;3(4):392–406.
Nath A, Arun AR, Niyogi R. A distributed approach for autonomous cooperative transportation in a dynamic multi-robot environment. In: Proceedings of the 35th annual ACM symposium on applied Computing; 2020. pp. 792–799.
Nof SY. Handbook of industrial robotics. Oxford: Wiley; 1999.
Patel D, Nath A, Niyogi R. Adding material embedding to the image2mass problem. In: Computational science and its applications—ICCSA 2022 workshops: Malaga, Spain, July 4–7, 2022, Proceedings, Part I. Springer; 2022. pp. 77–90.
Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng AY, et al. Ros: an open-source robot operating system. In: ICRA workshop on open source software, Vol. 3. Kobe, Japan; 2009. p. 5.
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. arXiv preprint arXiv:1409.1556.
Singh SK, Vidyarthi SK, Tiwari R. Machine learnt image processing to predict weight and size of rice kernels. J Food Eng. 2020;274: 109828.
Standley T, Sener O, Chen D, Savarese S. image2mass: estimating the mass of an object from its image. In: Conference on Robot Learning; 2017. pp. 324–333.
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2015. pp. 1–9.
Tadokoro S, Kitano H, Takahashi T, Noda I, Matsubara H, Shinjoh A, Koto T, Takeuchi I, Takahashi H, Matsuno F, et al. The robocup-rescue project: a robotic approach to the disaster mitigation problem. In: Proceedings ICRA, Vol. 4. IEEE; 2000. pp. 4089–4094.
Vidyarthi SK, Tiwari R, Singh SK, Xiao HW. Prediction of size and mass of pistachio kernels using random forest machine learning. J Food Process Eng. 2020;43(9): e13473.
Funding
The research of the Amar Nath was partially funded by a grant from the Department of Science and Technology: Science and Engineering Research Board (DST-SERB), grant number EEQ/2023/000792. The third author was in part supported by a research grant from Google.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no Conflict of interest.
Ethical approval
No human or animal subjects were involved in this study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Nath, A., Patel, D. & Niyogi, R. Mass Prediction and Analysis of an Object’s Mass from Its Image Using Deep Learning. SN COMPUT. SCI. 5, 711 (2024). https://doi.org/10.1007/s42979-024-03050-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42979-024-03050-6