Skip to main content
SpringerLink
Log in

Robotic sensing and object recognition from thermal-mapped point clouds

  • Regular Paper
  • Published:
International Journal of Intelligent Robotics and Applications Aims and scope Submit manuscript

Abstract

Many of the civil structures are more than half way through or nearing their intended service life; frequently assessing and maintaining structural integrity is a top maintenance priority. Robotic inspection technologies using ground and aerial robots with 3D scanning and imaging capabilities have the potential to improve safety and efficiency of infrastructure management. To provide more valuable information to inspectors and agency decision makers, automatic environment sensing and semantic information extraction are fundamental issues in this field. This paper introduces an innovative method for generating thermal-mapped point clouds of a robot’s work environment and performing automatic object recognition with the aid of thermal data fused to 3D point clouds. The laser scanned point cloud and thermal data were collected using a custom-designed mobile robot. The multimodal data was combined with a data fusion process based on texture mapping. The automatic object recognition was performed by two processes: segmentation with thermal data and classification with scanned geometric features. The proposed method was validated with the scan data collected in an entire building floor. Experimental results show that the thermal integrated object recognition approach achieved better performance than a geometry only-based approach, with an average recognition accuracy of 93%, precision of 83%, and recall rate of 86% for objects in the tested environment including humans, display monitors and light fixtures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Anguelov, D., Koller, D., Parker, E., Thrun, S.: Detecting and modeling doors with mobile robots. Proc. ICRA’04 4, 3777–3784 (2004)

  • Anil, E.B., Tang, P., Akinci, B., Huber, D.: Deviation analysis method for the assessment of the quality of the as-is building information models generated from point cloud data. Autom. Constr. 35, 507–516 (2013). doi:10.1016/j.autcon.2013.06.003

    Article  Google Scholar 

  • Borrmann, D., Nuchter, A., Dakulovic, M., Maurovic, I., Petrovic, I., Osmankovic, D., Velagic, J.: A mobile robot based system for fully automated thermal 3D mapping. Adv. Eng. Informatics. 28, 425–440 (2014). doi:10.1016/j.aei.2014.06.002

    Article  Google Scholar 

  • Bosche, F., Haas, C.T., Akinci, B.: Automated recognition of 3D CAD objects in site laser scans for project 3D status visualization and performance control. J. Comput. Civ. Eng. 23, 311–318 (2009). doi:10.1061/(ASCE)0887-3801(2009)23:6(311)

    Article  Google Scholar 

  • Casper, J., Murphy, R.R.: Human–robot interactions during the robot-assisted urban search and rescue response at the world trade center. IEEE Trans. Syst. Man Cybern. 33, 367–385 (2003). doi:10.1109/TSMCB.2003.811794

    Article  Google Scholar 

  • Chen, H., Wulf, O., Wagner, B.: Object detection for a mobile robot using mixed reality. In: 12th International Conference on VSMM 2006, pp. 466–475 (2006)

  • Chi, S., Caldas, C.H.: Automated object identification using optical video cameras on construction sites. Comput. Civ. Infrastruct. Eng. 26, 368–380 (2011). doi:10.1111/j.1467-8667.2010.00690.x

    Article  Google Scholar 

  • Cho, Y., Gai, M.: Projection–recognition–projection method for automatic object recognition and registration for dynamic heavy equipment operations. ASCE J. Comput. Civ. Eng. 28, A4014002 (2014). doi:10.1061/(ASCE)CP.1943-5487.0000332

    Article  Google Scholar 

  • Cho, Y., Wang, C., Gai, M., Park, J.W.: Rapid dynamic target surface modeling for crane operation using hybrid LADAR system. Constr. Res. Congr. (2014). doi:10.1061/9780784413517.0108

  • Cho, Y., Wang, C., Tang, P., Haas, C.T.: Target-focused local workspace modeling for construction automation applications. J. Comput. Civ. Eng. 26, 661–670 (2012). doi:10.1061/(ASCE)CP.1943-5487.0000166

    Article  Google Scholar 

  • Dalal, N., Triggs, B.: (HOG) Histograms of oriented gradients for human detection. In: Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005.vol. I, pp. 886–893 (2005). doi:10.1109/CVPR.2005.177

  • Davids, A.: Urban search and rescue robots: from tragedy to technology. IEEE Intell. Syst. Appl. 17, 81–83 (2002). doi:10.1109/5254.999224

    Google Scholar 

  • Davis, J.W., Sharma, V.: Background-subtraction in thermal imagery using contour saliency. Int. J. Comput. Vis. 71, 161–181 (2007). doi:10.1007/s11263-006-4121-7

    Article  Google Scholar 

  • Gatzke, T., Louis, S., Grimm, C., Louis, S., Zelinka, S.: Curvature maps for local shape comparison. (2005)

  • Golparvar-Fard, M., Peña-Mora, F., Arboleda, C.A., Lee, S.: Visualization of construction progress monitoring with 4D simulation model overlaid on time-lapsed photographs. J. Comput. Civ. Eng. 23, 391–404 (2009). doi:10.1061/(ASCE)0887-3801(2009)23:6(391)

    Article  Google Scholar 

  • Golparvar-Fard, M., Heydarian, A., Niebles, J.C.: Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers. Adv. Eng. Inform. 27, 652–663 (2013). doi:10.1016/j.aei.2013.09.001

    Article  Google Scholar 

  • Gould, S., Baumstarck, P., Quigley, M., Ng, A.Y., Koller, D.: Integrating visual and range data for robotic object detection. Work, multi-camera multi-modal sens, fusion algorithms. Appl. M2SFA2 2008 (2008)

  • Guo, Y., Bennamoun, M., Sohel, F., Lu, M., Wan, J.: 3D object recognition in cluttered scenes with local surface features: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 36, 2270–2287 (2014). doi:10.1109/TPAMI.2014.2316828

    Article  Google Scholar 

  • Han, S., Lee, S.: A vision-based motion capture and recognition framework for behavior-based safety management. Autom. Constr. 35, 131–141 (2013). doi:10.1016/j.autcon.2013.05.001

    Article  Google Scholar 

  • Hausamann, D., Zirnig, W., Schreier, G., Strobl, P.: Monitoring of gas pipelines—a civil UAV application. Aircr. Eng. Aerosp. Technol. 77, 352–360 (2005). doi:10.1108/00022660510617077

    Article  Google Scholar 

  • Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21, 433–449 (1999). doi:10.1109/34.765655

    Article  Google Scholar 

  • Kang, Z., Li, J., Zhang, L., Zhao, Q., Zlatanova, S.: Automatic registration of terrestrial laser scanning point clouds using panoramic reflectance images. Sensors 9, 2621–2646 (2009). doi:10.3390/s90402621

    Article  Google Scholar 

  • Kim, P., Cho, Y., Chen, J.: Target-free automatic registration of point clouds. In: 33rd International on Symposium on Automation and Robotics in Construction (ISARC 2016), pp. 686–693 (2016)

  • Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. (2012). doi:10.1016/j.protcy.2014.09.007

  • Locantore, N., Marron, J.S., Simpson, D.G., Tripoli, N., Zhang, J.T., Cohen, K.L., Boente, G., Fraiman, R., Brumback, B., Croux, C., Fan, J., Kneip, A., Marden, J.I., Peña, D., Prieto, J., Ramsay, J.O., Valderrama, M.J., Aguilera, A.M., Locantore, N., Marron, J.S., Simpson, D.G., Tripoli, N., Zhang, J.T., Cohen, K.L.: Robust principal component analysis for functional data. Test 8, 1–73 (1999). doi:10.1007/BF02595862

    Article  MathSciNet  Google Scholar 

  • Lowe, D.G.: Distinctive image features from scale invariant keypoints. Int. J. Comput. Vis. 60, 91–11020042 (2004). doi:10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  • Memarzadeh, M., Golparvar-Fard, M., Niebles, J.C.: Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors. Autom. Constr. 32, 24–37 (2013). doi:10.1016/j.autcon.2012.12.002

    Article  Google Scholar 

  • Park, M.W., Makhmalbaf, A., Brilakis, I.: Comparative study of vision tracking methods for tracking of construction site resources. Autom. Constr. 20, 905–915 (2011). doi:10.1016/j.autcon.2011.03.007

    Article  Google Scholar 

  • Park, M.-W., Koch, C., Brilakis, I.: Three-dimensional tracking of construction resources using an on-site camera system. J. Comput. Civ. Eng. 26, 541–549 (2012). doi:10.1061/(ASCE)CP.1943-5487.0000168

    Article  Google Scholar 

  • Roca, D., Lagüela, S., Díaz-Vilariño, L., Armesto, J., Arias, P.: Low-cost aerial unit for outdoor inspection of building facades. Autom. Constr. 36, 128–135 (2013). doi:10.1016/j.autcon.2013.08.020

    Article  Google Scholar 

  • Rusu, R.B., Bradski, G., Thibaux, R., Hsu, J.: Fast 3D recognition and pose using the viewpoint feature histogram. Intell. Robot. Syst. pp. 2155–2162 (2010)

  • Rusu, R.B.: Semantic 3D object maps for everyday manipulation in human living environments. 2009

  • Salti, S., Tombari, F., Di Stefano, L.: A performance evaluation of 3D keypoint detectors, 3D imaging, model. Process. Vis. Transm. (3DIMPVT), 2011 Int. Conf. pp. 282–289 (2011). doi:10.1109/3DIMPVT.2011.62

  • Seo, J., Han, S., Lee, S., Kim, H.: Computer vision techniques for construction safety and health monitoring. Adv. Eng. Inform. 29, 239–251 (2015). doi:10.1016/j.aei.2015.02.001

    Article  Google Scholar 

  • Socolinsky, D., Wolff, L., Neuheisel, J.: Illumination invariant face recognition using thermal infrared imagery. IEEE Comput. Vis. Pattern Recognit. 1, 527–534 (2001). doi:10.1109/CVPR.2001.990519

    Google Scholar 

  • Tang, P., Huber, D., Akinci, B., Lipman, R., Lytle, A.: Automatic reconstruction of as-built building information models from laser-scanned point clouds: a review of related techniques. Autom. Constr. 19, 829–843 (2010). doi:10.1016/j.autcon.2010.06.007

    Article  Google Scholar 

  • Thrun, S., Martin, C., Liu, Y., Hahnel, D.: A real-time expectation maximization algorithms for acquiring multi-planar maps of indoor environments with mobile robots. IEEE Trans. Robot. 20, 433–443 (2004). doi:10.1109/TRA.2004.825520

    Article  Google Scholar 

  • Treptow, A., Cielniak, G., Duckett, T.: Active people recognition using thermal and grey images on a mobile security robot. IEEE/RSJ Int Conf. Intell. Robot. Syst. IROS. 2005, 3610–3615 (2005). doi:10.1109/IROS.2005.1545530

    Google Scholar 

  • Wang, C., Cho, Y.: Smart scanning and near real-time 3D surface modeling of dynamic construction equipment from a point cloud. Autom. Constr. 49, 239–249 (2015). doi:10.1016/j.autcon.2014.06.003

    Article  Google Scholar 

  • Wang, C., Cho, Y., Gai, M.: As-is 3D thermal modeling for existing building envelopes using a hybrid LIDAR system. J. Comput. Civ. Eng. 27, 645–656 (2013). doi:10.1061/(ASCE)CP.1943-5487.0000273

    Article  Google Scholar 

  • Wang, C., Cho, Y., Kim, C.: Automatic BIM component extraction from point clouds of existing buildings for sustainability applications. Autom. Constr. 56, 1–13 (2015a). doi:10.1016/j.autcon.2015.04.001

    Article  Google Scholar 

  • Wang, C., Cho, Y., Kim, C., Component, A.B.: Automatic BIM component extraction from point clouds of existing buildings for sustainability applications. Autom. Constr. 56(2015), 1–13 (2015b). doi:10.1016/j.autcon.2015.04.001

    Article  Google Scholar 

  • Wohlkinger, W., Vincze, M.: Ensemble of shape functions for 3D object classification. IEEE Int Conf. Robot. Biomimetics. 2011, 2987–2992 (2011). doi:10.1109/ROBIO.2011.6181760

    Google Scholar 

  • Xue, Z., Ming, D., Song, W., Wan, B., Jin, S.: Infrared gait recognition based on wavelet transform and support vector machine. Pattern Recognit. 43, 2904–2910 (2010). doi:10.1016/j.patcog.2010.03.011

    Article  MATH  Google Scholar 

  • Yang, J., Park, M.-W., Vela, P. A., Golparvar-Fard, M.: Construction performance monitoring via still images, time-lapse photos, and video streams: now, tomorrow, and the future. Adv. Eng. Inform. 29, 211–224 (2015). doi:10.1016/j.aei.2015.01.011

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundation (CMMI-1358176). Any opinions, findings, and conclusions or recommendations expressed on this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. School of Civil and Environmental Engineering, Georgia Institute of Technology, 777 Atlantic Dr. N.W., Atlanta, GA, 30332-0355, USA

    Pileun Kim

  2. Institute for Robotics and Intelligent Machines, Georgia Institute of Technology, 777 Atlantic Dr. N.W., Atlanta, GA, 30332-0355, USA

    Jingdao Chen

  3. School of Civil and Environmental Engineering, Georgia Institute of Technology, 790 Atlantic Dr. N.W., Atlanta, GA, 30332-0355, USA

    Yong K. Cho

Authors
  1. Pileun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingdao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong K. Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong K. Cho.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, P., Chen, J. & Cho, Y.K. Robotic sensing and object recognition from thermal-mapped point clouds. Int J Intell Robot Appl 1, 243–254 (2017). https://doi.org/10.1007/s41315-017-0023-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41315-017-0023-9

Keywords

Search

Navigation