Abstract
In retail field, customer culture is shifting towards in-store researching, and retailers need to re-evaluate their location services to better assist customer. In-store mapping help retailers learn how their employees are interacting and it satisfies user intent to search for products, something that is often ignored by retailers especially for the secondary placement, which contains offers and promotions that change very often. In this paper, we describe a retail robot that moves autonomously inside a store and gathers points cloud data for a semantic store mapping. With all the data collected, it is possible to build a 3D map of the store with the exact product locations. This retail robot combines the features of both Robotics and Artificial Intelligence. Three classification approach have been compared in order to achieve the best performances: a machine learning technique, PointNet++ and a novel Reflectance PointNet++ especially designed for this task. Experiments are performed in a real retail environment that is an Italian supermarket, during business hours. A dataset has been built and made publicly available. The application of our approach yields good results in terms of precision, recall and F1-score and demonstrates the effectiveness of the proposed approach.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
References
Capobianco, R., Serafin, J., Dichtl, J., Grisetti, G., Iocchi, L., Nardi, D.: A proposal for semantic map representation and evaluation. In: 2015 European Conference on Mobile Robots (ECMR), pp. 1–6. IEEE (2015)
Engelmann, F., Kontogianni, T., Schult, J., Leibe, B.: Know what your neighbors do: 3D semantic segmentation of point clouds. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Goerke, N., Braun, S.: Building semantic annotated maps by mobile robots. In: Proceedings of the Conference Towards Autonomous Robotic Systems, pp. 149–156 (2009)
Günther, M., Wiemann, T., Albrecht, S., Hertzberg, J.: Building semantic object maps from sparse and noisy 3D data. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2228–2233. IEEE (2013)
Landrieu, L., Simonovsky, M.: Large-scale point cloud semantic segmentation with superpoint graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4558–4567 (2018)
Liciotti, D., Paolanti, M., Pietrini, R., Frontoni, E., Zingaretti, P.: Convolutional networks for semantic heads segmentation using top-view depth data in crowded environment. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 1384–1389. IEEE (2018)
Mozos, O.M., Mizutani, H., Kurazume, R., Hasegawa, T.: Categorization of indoor places using the kinect sensor. Sensors 12(5), 6695–6711 (2012)
Nüchter, A., Wulf, O., Lingemann, K., Hertzberg, J., Wagner, B., Surmann, H.: 3D mapping with semantic knowledge. In: Bredenfeld, A., Jacoff, A., Noda, I., Takahashi, Y. (eds.) RoboCup 2005. LNCS (LNAI), vol. 4020, pp. 335–346. Springer, Heidelberg (2006). https://doi.org/10.1007/11780519_30
Paolanti, M., Romeo, L., Martini, M., Mancini, A., Frontoni, E., Zingaretti, P.: Robotic retail surveying by deep learning visual and textual data. Robot. Auton. Syst. 118, 179–188 (2019)
Paolanti, M., Frontoni, E., Mancini, A., Pierdicca, R., Zingaretti, P.: Automatic classification for anti mixup events in advanced manufacturing system. In: ASME 2015 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, p. V009T07A061. American Society of Mechanical Engineers (2015)
Paolanti, M., Liciotti, D., Pietrini, R., Mancini, A., Frontoni, E.: Modelling and forecasting customer navigation in intelligent retail environments. J. Intell. Robot. Syst. 91(2), 165–180 (2018)
Paolanti, M., Romeo, L., Felicetti, A., Mancini, A., Frontoni, E., Loncarski, J.: Machine learning approach for predictive maintenance in industry 4.0. In: 2018 14th IEEE/ASME International Conference on Mechatronic and Embedded Systems and Applications (MESA), pp. 1–6. IEEE (2018)
Paolanti, M., Sturari, M., Mancini, A., Zingaretti, P., Frontoni, E.: Mobile robot for retail surveying and inventory using visual and textual analysis of monocular pictures based on deep learning. In: 2017 European Conference on Mobile Robots (ECMR), pp. 1–6. IEEE (2017)
Pierdicca, R., Malinverni, E., Piccinini, F., Paolanti, M., Felicetti, A., Zingaretti, P.: Deep convolutional neural network for automatic detection of damaged photovoltaic cells. In: International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, vol. 42, no. 2 (2018)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, pp. 5099–5108 (2017)
Ranganathan, A., Dellaert, F.: Semantic modeling of places using objects. In: Proceedings of the 2007 Robotics: Science and Systems Conference, vol. 3, pp. 27–30. Georgia Institute of Technology (2007)
Rusu, R.B., Marton, Z.C., Blodow, N., Dolha, M., Beetz, M.: Towards 3D point cloud based object maps for household environments. Robot. Auton. Syst. 56(11), 927–941 (2008)
Rusu, R.B., Marton, Z.C., Blodow, N., Holzbach, A., Beetz, M.: Model-based and learned semantic object labeling in 3D point cloud maps of kitchen environments. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3601–3608. IEEE (2009)
Sturari, M., et al.: Robust and affordable retail customer profiling by vision and radio beacon sensor fusion. Pattern Recogn. Lett. 81, 30–40 (2016)
Trevor, A.J., Rogers, J.G., Nieto-Granda, C., Christensen, H.I.: Tables, counters, and shelves: semantic mapping of surfaces in 3D. Georgia Institute of Technology (2010)
Weinmann, M., Jutzi, B., Hinz, S., Mallet, C.: Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers. ISPRS J. Photogram. Remote Sens. 105, 286–304 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Paolanti, M. et al. (2019). Semantic 3D Object Maps for Everyday Robotic Retail Inspection. In: Cristani, M., Prati, A., Lanz, O., Messelodi, S., Sebe, N. (eds) New Trends in Image Analysis and Processing – ICIAP 2019. ICIAP 2019. Lecture Notes in Computer Science(), vol 11808. Springer, Cham. https://doi.org/10.1007/978-3-030-30754-7_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-30754-7_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30753-0
Online ISBN: 978-3-030-30754-7
eBook Packages: Computer ScienceComputer Science (R0)
