Skip to main content
SpringerLink
Log in

Nighttime object detection system with lightweight deep network for internet of vehicles

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Autonomous driving systems in internet of vehicles (IoV) applications usually adopt a cloud computing mode. In these systems, information got at the edge of the cloud computing center for data analysis and situation response. However, the conventional IoV face enormous challenges to meet the requirements in terms of storage, communication, and computing problems because of the considerable amount of information on the traffic environment. The environment perception during the nighttime is poorer than that during the daytime that this problem also requires addressing. To solve these problems, we propose a nighttime object detection scheme based on a lightweight deep learning model in the edge computing mode. First, the pedestrian detection and the vehicle detection algorithm that using the thermal images based on the YOLO architecture. We can implement the model on edge devices that can achieve real-time detection through the designed lightweight strategy. Next, a spatial prior information and temporal prior information into the detection algorithm and divide the frames into key and non-key frames to increase the performance and speed of the system simultaneously. Finally, we implemented the detection network for performance and feasibility verification on the Jetson TX2 edge device. The experimental results show that the proposed system can achieve real-time and high-accuracy object detection on edge devices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Zhang, J., Wang, F., Wang, K., Lin, W., Xu, X., Chen, C.: Data-driven intelligent transportation systems: a survey. Trans. Intell. Transp. Syst. 12(4), 1624–1639 (2011)

    Article  Google Scholar 

  2. Guo, J., Hsia, C., Wong, K., Wu, J., Wu, Y., Wang, N.: Nighttime vehicle detection and tracking with adaptive mask training. IEEE Trans. Veh. Technol. 65(6), 4023–4032 (2016)

    Article  Google Scholar 

  3. Boden, M.A.: Creativity and artificial intelligence. Artif. Intell. 103, 347–356 (1998)

    Article  MathSciNet  Google Scholar 

  4. Hsia, C., Yen, S., Jang, J.: An intelligent IoT-based vision system for nighttime vehicle detection and energy saving. Sens. Mater. 31(6), 1803–1814 (2019)

    Google Scholar 

  5. Yang, F., Wang, S., Li, J., Liu, Z., Sun, Q.: An overview of internet of vehicles. China Commun. 11(10), 1–15 (2014)

    Article  Google Scholar 

  6. Liu, Q., Kumar, S., Mago, V.: SafeRNet: safe transportation routing in the era of internet of vehicles and mobile crowd sensing. In: IEEE Annual Consumer Communications & Networking Conference, pp. 299–304 (2017)

  7. Sun, Y., Wang, B., Li, S., Sun, Z., Nguyen, H., Duong, T.Q.: Manipulation with domino effect for cache- and buffer-enabled social IIoT: preserving stability in tripartite graphs. IEEE Trans. Ind. Inf. 16(8), 5389–5400 (2020)

    Article  Google Scholar 

  8. Cao, H., Wu, S., Aujla, G., Wang, Q., Yang, L., Zhu, H.: Dynamic embedding and quality of service-driven adjustment for cloud networks. IEEE Trans. Ind. Inf. 16(2), 1406–1416 (2020)

    Article  Google Scholar 

  9. Cheng, J., Yuan, G., Zhou, M., Gao, S., Liu, C., Duan, H., Zeng, Q.: Accessibility analysis and modeling for IoV in an urban scene. IEEE Trans. Veh. Technol. 69(4), 4246–4256 (2020)

    Article  Google Scholar 

  10. Wang, B., Sun, Y., Li, S., Cao, Q.: Hierarchical matching with peer effect for low-latency and high-reliable caching in social IoT. IEEE Internet Things J. 6(1), 1193–1209 (2019)

    Article  Google Scholar 

  11. Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2189–2202 (2012)

    Article  Google Scholar 

  12. Zhu, J., Zeng, H., Huang, J., Liao, S., Lei, Z., Cai, C., Zheng, L.: Vehicle re-identification using quadruple directional deep learning features. IEEE Trans. Intell. Transp. Syst. 21(1), 410–420 (2020)

    Article  Google Scholar 

  13. Wang, Y., Piao, Y.: Enhancement system of nighttime infrared video image and visible video image. In: Selected Proceedings from CSOE (2016)

  14. Bhowmik, M., Saha, K., Majumder, S., Majumder, G., Saha, A., Sarma, A., Bhattacharjee, D., Basu, D., Nasipuri, M.: Thermal infrared face recognition—a biometric identification technique for robust security system. In: Reviews Refinements and New Ideas in Face Recognition (2011)

  15. Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3962–3971 (2019)

  16. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.: MobileNetV2: inverted residuals and linear bottlenecks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp/ 4510–4520 (2018)

  17. Amert, T., Otterness, N., Yang, M., Anderson, J. H., Smith, F. D.: GPU scheduling on the NVIDIA TX2: hidden details revealed. In: IEEE Real-Time Systems Symposium, pp. 104–115 (2017)

  18. Bourlai, T., Cukic, B.: Multi-spectral face recognition: identification of people in difficult environments. In: IEEE International Conference on Intelligence and Security Informatics, pp. 196–201 (2012)

  19. Fernández-Caballero, A., Castillo, J.C., Martínez-Cantos, J., Martínez-Tomás, R.: Optical flow or image subtraction in human detection from infrared camera on mobile robot. Robot. Auton. Syst. 58(12), 1273–1281 (2010)

    Article  Google Scholar 

  20. Davis, J., Sharma, V.: Fusion-based background-subtraction using contour saliency. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, p. 11 (2005)

  21. Jeon, E.S., Kim, J.H., Hong, H.G., Batchuluun, G., Park, K.R.: Human detection based on the generation of a background image and fuzzy system by using a thermal camera. Sensors 16, 453 (2016)

    Article  Google Scholar 

  22. Lin, C., Chen, C., Hwang, W., Hwang, C., Chang, C.: Novel outline features for pedestrian detection system with thermal images. Pattern Recogn. 48(11), 3440–3450 (2015)

    Article  Google Scholar 

  23. Zhao, X., He, Z., Zhang, S., Liang, D.: Robust pedestrian detection in thermal infrared imagery using a shape distribution histogram feature and modified sparse representation classification. Pattern Recogn. 48(6), 1947–1960 (2015)

    Article  Google Scholar 

  24. Qi, Y., An, G.: Infrared moving targets detection based on optical flow estimation. Int. Conf. Comput. Sci. Netw. Technol. 4, 2452–2455 (2011)

    Google Scholar 

  25. Gilmore, E.T., Ugbome, C., Kim, C.: An IR-based pedestrian detection system implemented with Matlab-equipped laptop and low-cost microcontroller. Int. J. Comput. Sci. Inf. Technol. 3(5), 79–87 (2015)

    Google Scholar 

  26. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 886–893 (2005)

    Google Scholar 

  27. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)

    Article  Google Scholar 

  28. Xu, Y., Yu, G., Wang, Y., Wu, X., Ma, Y.: A hybrid vehicle detection method based on Viola-Jones and HOG + SVM from UAV images. Sensors 16(8), 1325 (2016)

    Article  Google Scholar 

  29. Cilulko, J., Janiszewski, P., Bogdaszewski, M., Szczygielska, E.: Infrared thermal imaging in studies of wild animals. Eur. J. Wildl. Res. 59, 17–23 (2012)

    Article  Google Scholar 

  30. Goodall, T., Bovik, A., Paulter, N.: Tasking on natural statistics of infrared images. IEEE Trans. Image Process. 25, 65–79 (2016)

    Article  MathSciNet  Google Scholar 

  31. Lee, E., Ko, B., Nam, J.: Recognizing pedestrian’s unsafe behaviors in far-infrared imagery at night. Infrared Phys. Technol. 76, 261–270 (2016)

    Article  Google Scholar 

  32. Rodger, I., Connor, B., Robertson, N. M.: Classifying objects in LWIR imagery via CNNS. Secur. Defence 9987, 99870–99884 (2016)

  33. Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)

    Article  Google Scholar 

  34. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6517–6525 (2017)

  35. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., Berg, A.: SSD: single shot multibox detector. In: European Conference on Computer Vision (2016)

  36. Zhang, H., Luo, C., Wang, Q., Kitchin, M., Parmley, A., Monge-Álvarez, J., Casaseca-de-la-Higuera, P.: A novel infrared video surveillance system using deep learning based techniques. Multim. Tools Appl. 77, 26657–26676 (2018)

    Article  Google Scholar 

  37. Ghose, D., Desai, S.M., Bhattacharya, S., Chakraborty, D., Fiterau, M., Rahman, T.: Pedestrian detection in thermal images using saliency maps. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 988–997 (2019)

  38. Chen, Y., Li, G., Jhong, S., Chen, P., Tsai, C., Chen, P.: Nighttime pedestrian detection based on thermal imaging and convolutional neural networks. Sens. Mater. 32(10), 3157–3167 (2020)

    Google Scholar 

  39. Devaguptapu, C., Akolekar, N., Sharma, M., Balasubramanian, V.: Borrow from anywhere: pseudo multi-modal object detection in thermal imagery. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1029–1038 (2019)

  40. Heo, D., Lee, E., Ko, B.C.: Pedestrian detection at night using deep neural networks and saliency maps. J. Imaging Sci. Technol. 61(6), 060403-1–060403-9 (2017)

  41. Abbott, R., Del Rincon, J. M., Connor, B., Robertson, N.: Deep object classification in low resolution LWIR imagery via transfer learning. In: IMA Conference on Mathematics in Defence (2017)

  42. Dai, X., Yuan, X., Wei, X.: TIRNet: object detection in thermal infrared images for autonomous driving. Appl. Intell. 51, 1–18 (2020)

  43. Krišto, M., Ivasic-Kos, M., Pobar, M.: Thermal object detection in difficult weather conditions using YOLO. IEEE Access 8, 125459–125476 (2020)

    Article  Google Scholar 

  44. Zagoruyko, S., Komodakis, N.: Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: International Conference on Learning Representations (2017)

  45. Chen, Y., Zhang, Y.: Drift-free tracking surveillance based on online latent structured SVM and Kalman filter modules. IEICE Trans. Inf. Syst. 101-D, 491–503 (2018)

    Article  Google Scholar 

  46. Bochkovskiy, A., Wang, C.Y., Liao, H.: Yolov4: optimal speed and accuracy of object detection. arXiv:2004.10934 (2020)

  47. He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. In: IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

  48. Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42, 318–327 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Engineering Science, National Cheng Kung University, Tainan, Taiwan

    Sin-Ye Jhong, Chih-Hsien Hsia & Chin-Feng Lai

  2. Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan

    Yung-Yao Chen

  3. Graduate Institute of Automation Technology, National Taipei University of Technology, Taipei, Taiwan

    Shih-Chang Lin & Kuo-Hua Hsu

Authors
  1. Sin-Ye Jhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yung-Yao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chih-Hsien Hsia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shih-Chang Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kuo-Hua Hsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chin-Feng Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chin-Feng Lai.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jhong, SY., Chen, YY., Hsia, CH. et al. Nighttime object detection system with lightweight deep network for internet of vehicles. J Real-Time Image Proc 18, 1141–1155 (2021). https://doi.org/10.1007/s11554-021-01110-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01110-1

Keywords

Search

Navigation