Abstract
With the continued advancement of autonomous vehicles and their implementation in public roads, accurate detection of vulnerable road users (VRUs) is vital for ensuring safety. To provide higher levels of safety for these VRUs, an effective detection system should be employed that can correctly identify VRUs in all types of environments (e.g. VRU appearance, crowded scenes) and conditions (e.g. fog, rain, night-time). This paper presents optimal methods of sensor fusion for pedestrian and cyclist detection using Deep Neural Networks (DNNs) for higher levels of feature abstraction. Typically, visible sensors have been utilized for this purpose. Recently, thermal sensors system or combination of visual and thermal sensors have been employed for pedestrian detection with advanced detection algorithm. DNNs have provided promising results for improving the accuracy of pedestrian and cyclist detection. This is because they are able to extract features at higher levels than typical hand-crafted detectors. Previous studies have shown that amongst the several sensor fusion techniques that exist, Halfway Fusion has provided the best results in terms of accuracy and robustness. Although sensor fusion and DNN implementation have been used for pedestrian detection, there is considerably less research undertaken for cyclist detection.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Angelova, A., Krizhevsky, A., Vanhoucke, V., Ogale, A., Ferguson, D.: Real-time pedestrian detection with deep network cascades. In: Proceedings of the British Machine Vision Conference, pp. 1–12 (2015). https://doi.org/10.5244/C.29.32
Baek, J., Hong, S., Kim, J., Kim, E.: Efficient pedestrian detection at nighttime using a thermal camera. Sens. (Switz.) 17(8), 1850 (2017). https://doi.org/10.3390/s17081850
Bertozzi, M., Broggi, A., Caraffi, C., Del Rose, M., Felisa, M., Vezzoni, G.: Pedestrian detection by means of far-infrared stereo vision. Comput. Vis. Image Underst. 106(2–3), 194–204 (2007). https://doi.org/10.1016/j.cviu.2006.07.016
Biswas, S.K., Milanfar, P.: Linear support tensor machine with LSK channels: pedestrian detection in thermal infrared images. IEEE Trans. Image Proc. 26(9), 4229–4242 (2017). https://doi.org/10.1109/TIP.2017.2705426
Chang, S.L., Yang, F.T., Wu, W.P., Cho, Y.A., Chen, S.W.: Nighttime pedestrian detection using thermal imaging based on HOG feature. In: Proceedings 2011 International Conference on System Science and Engineering, pp. 694–698. IEEE (2011). https://doi.org/10.1109/ICSSE.2011.5961992
Dai, C., Zheng, Y., Li, X.: Layered representation for pedestrian detection and tracking in infrared imagery. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005) - Workshops. vol. 3, pp. 13–13. IEEE (2005). https://doi.org/10.1109/CVPR.2005.483
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 886–893. IEEE (2005). https://doi.org/10.1109/CVPR.2005.177
Davis, J., Sharma, V.: Robust detection of people in thermal imagery. In: 2004 Proceedings of the 17th International Conference on Pattern Recognition, ICPR, pp. 713–716. IEEE (2004). https://doi.org/10.1109/ICPR.2004.1333872
European Road Safety Observatory: Traffic Safety Basic Facts 2012. Technical report, European Road Safety Observatory (2012)
Gandhi, T., Trivedi, M.M.: Pedestrian protection systems: issues, survey, and challenges. IEEE Trans. Intell. Transp. Syst. 8(3), 413–430 (2007). https://doi.org/10.1109/TITS.2007.903444
Gerónimo, D., López, A.M., Sappa, A.D., Graf, T.: Survey of pedestrian detection for advanced driver assistance systems. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1239–1258 (2010). https://doi.org/10.1109/TPAMI.2009.122
Gilmore, E.T., Frazier, P.D, Chouikha, M.F.: Improved human detection using image fusion. In: Proceedings of the IEEE ICRA (2009)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580–587 (2014)
González, A., et al.: Pedestrian detection at day/night time with visible and FIR cameras: a comparison. Sens. (Switz.) 16(6), 1–11 (2016). https://doi.org/10.3390/s16060820
Hurney, P., Jones, E., Waldron, P., Glavin, M., Morgan, F.: Night-time pedestrian classification with histograms of oriented gradients-local binary patterns vectors. IET Intell. Transp. Syst. 9(1), 75–85 (2015). https://doi.org/10.1049/iet-its.2013.0163
Hwang, S., Park, J., Kim, N., Choi, Y., So, I.: Multispectral pedestrian detection: benchmark dataset and baseline. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1037–1045 (2015)
Zhao, J., Cheung, S.C.S.: Human segmentation by fusing visible-light and thermal imaginary. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 1185–1192. IEEE (2009). https://doi.org/10.1109/ICCVW.2009.5457476, http://ieeexplore.ieee.org/document/5457476/
Kocic, J., Jovicic, N., Drndarevic, V.: Sensors and sensor fusion in autonomous vehicles. In: 2018 26th Telecommunications Forum (TELFOR), pp. 420–425. IEEE (2018). https://doi.org/10.1109/TELFOR.2018.8612054, https://ieeexplore.ieee.org/document/8612054/
Lee, J.H., et al.: Robust pedestrian detection by combining visible and thermal infrared cameras. Sens. (Switz.) 15(5), 10580–10615 (2015). https://doi.org/10.3390/s150510580
Li, W., Zheng, D., Zhao, T., Yang, M.: An effective approach to pedestrian detection in thermal imagery. In: 2012 8th International Conference on Natural Computation, pp. 325–329. IEEE (2012). https://doi.org/10.1109/ICNC.2012.6234621
Li, X., et al.: A new benchmark for vision-based cyclist detection. In: Proceedings of IEEE Intelligent Vehicles Symposium, pp. 1028–1033 (2016)
Li, X., et al.: A unified framework for concurrent pedestrian and cyclist detection. IEEE Trans. Intell. Transp. Syst. 18(2), 269–281 (2017). https://doi.org/10.1109/TITS.2016.2567418
Li, Z., Zhang, J., Wu, Q., Geers, G.: Feature enhancement using gradient salience on thermal image. In: 2010 International Conference on Digital Image Computing: Techniques and Applications, pp. 556–562. IEEE (2010). https://doi.org/10.1109/DICTA.2010.99, http://ieeexplore.ieee.org/document/5692620/
Liu, J., Zhang, S., Wang, S., Metaxas, D.N.: Multispectral deep neural networks for pedestrian detection. In: British Machine Vision Conference, pp. 1–13 (2016)
Mikolajczyk, K., Schmid, C., Zisserman, A.: Human detection based on a probabilistic assembly of robust part detectors. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3021, pp. 69–82. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24670-1_6
Neagoe, V.E., Ciotec, A.D., Bărar, A.P.: A concurrent neural network approach to pedestrian detection in thermal imagery. In: 2012 9th International Conference on Communications (COMM), pp. 133–136 (2012). https://doi.org/10.1109/ICComm.2012.6262539
Olmeda, D., Armingol, J.M., de la Escalera, A.: Discrete features for rapid pedestrian detection in infrared images. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3067–3072. IEEE (2012). https://doi.org/10.1109/IROS.2012.6385928, http://ieeexplore.ieee.org/document/6385928/
O’Malley, R., Jones, E., Glavin, M.: Detection of pedestrians in far-infrared automotive night vision using region-growing and clothing distortion compensation. Infrared Phys. Technol. 53(6), 439–449 (2010). https://doi.org/10.1016/J.INFRARED.2010.09.006
Tian, W., Lauer, M.: Fast and robust cyclist detection for monocular camera systems. In: International joint Conference on Computer Vision Imaging and Computer Graphics Theory and Applications (VISIGRAPP) (2015)
Tian, W., Lauer, M.: Detection and orientation estimation for cyclists by max pooled features. In: Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, SCITEPRESS - Science and Technology Publications, pp. 17–26 (2017). https://doi.org/10.5220/0006085500170026
Uijlings, J.R.R., van de Sande, K.E.A., Gevers, T., Smeulders, A.W.M.: Selective search for object recognition. Int. J. Comput. Vision 104(2), 154–171 (2013). https://doi.org/10.1007/s11263-013-0620-5
Viola, P., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. In: Proceedings of the 9th IEEE International Conference on Computer Vision, vol. 1, no. 9, pp. 734–741 (2003). https://doi.org/10.1109/ICCV.2003.1238422
Wagner, J., Fischer, V., Herman, M.: Multispectral pedestrian detection using deep fusion convolutional neural networks. In: European Symposium on Artificial Neural Networks (2016)
World Health Organisation: Global Status Report on Road Safety 2015 - Summary (2015)
Wu, T.E., Tsai, C.C., Guo, J.I.: LiDAR/camera sensor fusion technology for pedestrian detection. In: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1675–1678. IEEE (2017). https://doi.org/10.1109/APSIPA.2017.8282301
Xu, D., Ouyang, W., Ricci, E., Wang, X., Sebe, N.: Learning cross-modal deep representations for robust pedestrian detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4236–4244. IEEE (2017). https://doi.org/10.1109/CVPR.2017.451, http://ieeexplore.ieee.org/document/8099934/
Xu, F., Liu, X., Fujimura, K.: Pedestrian detection and tracking with night vision. IEEE Trans. Intell. Transp. Syst. 6(1), 63–71 (2005). https://doi.org/10.1109/TITS.2004.838222
Chen, Y., Han, C.: Night-time pedestrian detection by visual-infrared video fusion. In: 2008 7th World Congress on Intelligent Control and Automation, pp. 5079–5084. IEEE (2008). https://doi.org/10.1109/WCICA.2008.4593753
Zhang, S., Benenson, R., Omran, M., Hosang, J., Schiele, B.: How far are we from solving pedestrian detection?. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1259–1267 (2016). https://doi.org/10.1109/CVPR.2016.141
Zhao, X., He, Z., Zhang, S., Liang, D.: Robust pedestrian detection in thermal infrared imagery using a shape distribution histogram feature and modified sparse representation classification. Pattern Recogn. 48(6), 1947–1960 (2015). https://doi.org/10.1016/J.PATCOG.2014.12.013
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ahmed, S., Huda, M.N., Rajbhandari, S., Saha, C., Elshaw, M., Kanarachos, S. (2019). Visual and Thermal Data for Pedestrian and Cyclist Detection. In: Althoefer, K., Konstantinova, J., Zhang, K. (eds) Towards Autonomous Robotic Systems. TAROS 2019. Lecture Notes in Computer Science(), vol 11650. Springer, Cham. https://doi.org/10.1007/978-3-030-25332-5_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-25332-5_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-25331-8
Online ISBN: 978-3-030-25332-5
eBook Packages: Computer ScienceComputer Science (R0)