Abstract
This work proposes 3D body landmarks, location, and identification in multioccupancy contexts from vision sensors. Methods include high-performance vision tools, such as Yolo, DeepFace, and MediaPipe, to estimate 3D body landmarks and identification. First, we sense a smart space where a vision sensor is deployed to collect the activities of inhabitants. Our proposed model computes, identifies, tracks and obtains 3D body landmarks in multi-occupancy contexts. Third, 2D location over the floor is estimated based on homography projection, enabling fusing multiple vision sensors’ information. Third, tracking and face recognition are integrated with non-supervised tracking to identify the inhabitants in the smart environment and relate the landmarks to them. A case study is presented to illustrate the proposal with an encouraging performance (f1-score: 0.98) in tracking multi-occupancy of two inhabitants with five scenes in two rooms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Castro, J., Delgado, M., Medina, J., Ruiz-Lozano, M.: An expert fuzzy system for predicting object collisions: its application for avoiding pedestrian accidents. Expert Syst. Appl. 38(1), 486–494 (2011)
Castro, L.A., Bravo, J.: Modeling interactions in ambient intelligence. Pers. Ubiq. Comput. 26(6), 1333–1335 (2022)
Chen, W., Huang, H., Peng, S., Zhou, C., Zhang, C.: Yolo-face: a real-time face detector. Vis. Comput. 37, 805–813 (2021)
Cheng, S., Sun, J.X., Cao, Y.G., Zhao, L.R., et al.: Target tracking based on incremental deep learning (2015)
Chum, O., Matas, J., Kittler, J.: Locally optimized RANSAC. In: Michaelis, B., Krell, G. (eds.) DAGM 2003. LNCS, vol. 2781, pp. 236–243. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-45243-0_31
Gopinath, V., Srija, A., Sravanthi, C.N.: Re-design of smart homes with digital twins. In: Journal of Physics: Conference Series, vol. 1228, p. 012031. IOP Publishing (2019)
Jian, S., Kaiming, H., Shaoqing, R., Xiangyu, Z.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision & Pattern Recognition, pp. 770–778 (2016)
Jiang, P., Ergu, D., Liu, F., Cai, Y., Ma, B.: A review of yolo algorithm developments. Procedia Comput. Sci. 199, 1066–1073 (2022)
Jiang, T., Zhang, Q., Yuan, J., Wang, C., Li, C.: Multi-type object tracking based on residual neural network model. Symmetry 14(8), 1689 (2022)
Kim, J.W., Choi, J.Y., Ha, E.J., Choi, J.H.: Human pose estimation using mediapipe pose and optimization method based on a humanoid model. Appl. Sci. 13(4), 2700 (2023)
Lin, Y., Jiao, X., Zhao, L.: Detection of 3d human posture based on improved mediapipe. J. Comput. Commun. 11(2), 102–121 (2023)
Liu, P.L., Chang, C.C.: Simple method integrating openpose and rgb-d camera for identifying 3d body landmark locations in various postures. Int. J. Ind. Ergon. 91, 103354 (2022)
Masi, I., Wu, Y., Hassner, T., Natarajan, P.: Deep face recognition: a survey. In: 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 471–478. IEEE (2018)
Nikolakis, N., Alexopoulos, K., Xanthakis, E., Chryssolouris, G.: The digital twin implementation for linking the virtual representation of human-based production tasks to their physical counterpart in the factory-floor. Int. J. Comput. Integr. Manuf. 32(1), 1–12 (2019)
Parkhi, O., Vedaldi, A., Zisserman, A.: Deep face recognition. In: BMVC 2015-Proceedings of the British Machine Vision Conference 2015. British Machine Vision Association (2015)
Pereira, A., Carvalho, P., Pereira, N., Viana, P., Côrte-Real, L.: From a visual scene to a virtual representation: a cross-domain review. IEEE Access 11, 57916–57933 (2023)
Saraee, E., Jalal, M., Betke, M.: Visual complexity analysis using deep intermediate-layer features. Comput. Vis. Image Underst. 195, 102949 (2020)
Serengil, S.: Deepface (2020). https://github.com/serengil/deepface
Singh, A.K., Kumbhare, V.A., Arthi, K.: Real-time human pose detection and recognition using mediapipe. In: eddy, V.S., Prasad, V.K., Wang, J., Reddy, K. (eds.) International Conference on Soft Computing and Signal Processing, vol. 1413, pp. 145–154. Springer, Heidelberg (2021). https://doi.org/10.1007/978-981-16-7088-6_12
Song, L., Yu, G., Yuan, J., Liu, Z.: Human pose estimation and its application to action recognition: a survey. J. Vis. Commun. Image Represent. 76, 103055 (2021)
Tian, Y., Zhang, H., Liu, Y., Wang, L.: Recovering 3D human mesh from monocular images: a survey. arXiv preprint arXiv:2203.01923 (2022)
Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: Yolov7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7464–7475 (2023)
Yang, F., Zhang, X., Liu, B.: Video object tracking based on yolov7 and deepsort. arXiv preprint arXiv:2207.12202 (2022)
Zago, M., Luzzago, M., Marangoni, T., De Cecco, M., Tarabini, M., Galli, M.: 3d tracking of human motion using visual skeletonization and stereoscopic vision. Front. Bioeng. Biotechnol. 8, 181 (2020)
Zhu, C.: Video object tracking using sift and mean shift (2011)
Acknowledgements
This contribution has been supported by the Spanish Institute of Health ISCIII through the project DTS21-00047. Moreover, this research has received funding by EU Horizon 2020 Pharaon Project ‘Pilots for Healthy and Active Ageing’, Grant agreement no. 857188.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Polo-Rodriguez, A., Burns, M., Nugent, C., Florez-Revuelta, F., Medina-Quero, J. (2023). Non-invasive Synthesis from Vision Sensors for the Generation of 3D Body Landmarks, Locations and Identification in Smart Environments. In: Bravo, J., Urzáiz, G. (eds) Proceedings of the 15th International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2023). UCAmI 2023. Lecture Notes in Networks and Systems, vol 842. Springer, Cham. https://doi.org/10.1007/978-3-031-48642-5_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-48642-5_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-48641-8
Online ISBN: 978-3-031-48642-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)