Skip to main content
SpringerLink
Log in

CCA-Based Fusion of Camera and Radar Features for Target Classification Under Adverse Weather Conditions

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Deep learning models such as deep convolutional neural networks (DCNNs) image classifiers have achieved outstanding performance over the last decade. However, these models are mostly trained with high-quality images drawn from publicly available datasets such as ImageNet. Recently, many researchers have evaluated the impact of low-quality image degradations on the performance of different neural network-based image classifiers. But, most of these studies generate low-quality images by synthetic modification of the high-quality images. Besides, most of the studies employed various image processing techniques to remove the image degradations and trained the DCNNs again to achieve better performance. But it has since been discovered that such methods could not improve the classification accuracy of DCNNs. The robustness of DCNNs based image classifiers trained on low-quality images resulting from natural factors common in autonomous driving and other intelligent system settings was rarely studied over the recent years. In this paper, we proposed a canonical correlation analysis (CCA) based fusion of camera and radar features for improving the performance of DCNNs image classifiers trained on natural adverse weather data. CCA is a statistical approach that creates a highly discriminative feature vector by measuring the linear relationship between the camera and radar features. A spatial attention network was designed to re-weight the camera features before associating them with radar features in the CCA-feature fusion block. Our findings based on experimental evaluations have proven that, indeed, the performance of the DCNN models (i.e., Alex-Net and VGG-16-Net) is heavily affected by degradations arising from natural factors. Specifically, the DCNN models are more affected by the degradations arising from rainfall, foggy and nighttime conditions using Radiate and Carrada datasets. However, the proposed fusion frameworks have improved the performance of the individual sensing modalities significantly. The radar data has helped substantially in enhancing the fusion performance, mainly using rainfall data where the camera data is heavily affected.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances of neural information processing systems, pp 1097–1105

  2. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Proceedings of international conference on learning representations, pp 1–14

  3. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of IEEE conference for computer vision and pattern recognition, pp 770–778

  4. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of IEEE conference for computer vision and pattern recognition, pp 580–587

  5. Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  6. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell 404:834–848

    Article  Google Scholar 

  7. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of IEEE conference for computer vision and pattern recognition, pp 3431–2440

  8. Abdu FJ, Zhang Y, Fu M, Li Y, Deng Z (2021) Application of deep learning on millimeter-wave radar signals: a review. Sensors 21:1951

    Article  Google Scholar 

  9. Li W, Wen S, Shi K, Yang Y, Huang T (2022) Neural architecture search with a lightweight transformer for text-to-image synthesis. IEEE Trans Netw Sci Eng (TNSE) 9(3):1567–1576

    Article  Google Scholar 

  10. Li S et al (2021) Auto-FERNet: a facial expression recognition network with architecture search. IEEE Trans Netw Sci Eng 8(3):2213–2222

    Article  Google Scholar 

  11. Cao Y, Cao Y, Wen S, Huang T, Zeng Z (2019) Passivity analysis of delayed reaction–diffusion memristor-based neural networks. Neural Netw 109:159–167

    Article  MATH  Google Scholar 

  12. Wen S, Xiao S, Yang Y, Yan Z, Zeng Z, Huang T (2019) Adjusting learning rate of memristor-based multilayer neural networks via fuzzy method. Trans Computer Aided Design Integrated Circuit Syst 38(6):1084–1094

    Article  Google Scholar 

  13. Cao Y, Jiang W, Wang J (2021) Anti-synchronization of delayed memristive neural networks with leakage term and reaction–diffusion terms. Knowl Based Syst 233:107539

    Article  Google Scholar 

  14. Schmidhuber J (2014) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  15. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the KITTI vision benchmark suite. In: Proceedings of IEEE conference for computer vision and pattern recognition, pp 3354–3361

  16. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In Proceedings of IEEE conference for computer vision and pattern recognition, pp 3213–3223

  17. Griffin G, Holub A, Perona P ( 2007) Caltech-256 object category dataset

  18. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis 88:303–338

    Article  Google Scholar 

  19. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition, pp 248–255

  20. Zhang J, Cao Y, Fang S, Kang Y, Chen CW (2017) Fast haze removal for nighttime image using maximum reflectance prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7418–7426

  21. Dodge S, Karam L (2017) A Study and comparison of human and deep learning recognition performance under visual distortions. In: IEEE international conference on computer communication and networks (ICCCN), pp 1–7

  22. Azulay A, Weiss Y (2018) Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv:1805.12177

  23. Pei Y, Huang Y, Zou Q et al (2019) Effects of image degradation and degradation removal to CNN-based image classification. IEEE Trans Pattern Anal Mach Intell 43(4):1239–1253

    Article  Google Scholar 

  24. Koziarski M, Cyganek B (2018) Impact of low resolution on image recognition with deep neural networks: an experimental study. Int J Appl Math Comput Sci 28(4):735–744

    Article  MathSciNet  MATH  Google Scholar 

  25. Pei Y, Huang Y, Zou Q et al (2018) Does haze removal help CNN-based image classification? In Proceedings of the European conference on computer vision (ECCV), pp 697–712

  26. Dodge S, Karam L (2016) Understanding how image quality affects deep neural networks. In: International conference on quality of multimedia experience (QoMEX), pp 1–6

  27. Roy P et al (2018) Effects of degradations on deep neural network architectures. ArXiv, pp 1807–10108

  28. Karahan S, Kilinc Yildirum M, Kirtac K, Rende F S, Butun G, Ekenel H K (2016) How image degradations affect deep CNN-based face recognition? In: International conference of the biometrics special interest group (BIOSIG), pp 1–5

  29. Liu D, Cheng B, Wang Z, Zhang H, Huang TS (2019) Enhance visual recognition under adverse conditions via deep networks. IEEE Trans Image Process 28(9):4401–4412

    Article  MathSciNet  MATH  Google Scholar 

  30. Vasiljevic I, Chakrabarti A, Shakhnarovich G (2016) Examining the impact of blur on recognition by convolutional networks. arXiv:1611.05760

  31. Wang Y, Cao Y, Zha ZJ, Zhang J, Xiong Z (2020) Deep degradation prior for low-quality image classification. In: Proceedings of IEEE conference for computer vision pattern recognition, pp 11046–11055

  32. Dodge S, Karam L (2019) Human and DNN classification performance on images with quality distortions: a comparative study. ACM Trans Appl Percept (TAP) 16(7):1–17

    Google Scholar 

  33. Dodge S, Karam L (2017) Quality resilient deep neural networks. arXiv preprint arXiv:1703.08119

  34. Hotelling H (1936) Relations between two sets of variates. Bio-metrika 28:321–377

    MATH  Google Scholar 

  35. Uurtio V, Monteiro JM, Kandola J, Shawe-Taylor J, Fernandez-Reyes D, Rousu J (2018) A tutorial on canonical correlation methods. ACM Comput Surv (CSUR) 50(6):95

    Article  Google Scholar 

  36. Jing X, Li S, Lan C, Zhang D, Yang J, Liu Q (2011) Color image canonical correlation analysis for face feature extraction and recognition. Signal Process 91(8):2132–2140

    Article  MATH  Google Scholar 

  37. Chetana K, Shubhangi D, Suresh G, Aditya A (2019) Novel canonical correlation analysis based feature level fusion algorithm for multimodal recognition. In biometric sensor systems. Sens Lett 17:75–86

    Article  Google Scholar 

  38. Sun QS, Zeng SG, Liu Y, Heng PA, Xia DS (2005) A new method of feature fusion and its application in image recognition. Pattern Recogn 38(12):2437–2448

    Article  Google Scholar 

  39. Ahmad Z, Khan N (2018) Towards improved human action recognition using convolutional neural networks and multimodal fusion of depth and inertial sensor data. In: IEEE international symposium on multimedia (ISM), pp 223–230

  40. Marcel S, Emanuele P, Saptarshi M, Alireza A, Sen W, Andrew W (2020) RADIATE: A radar dataset for automotive perception. arXiv, preprint arXiv

  41. Ouaknine A, Newson A, Rebut J, Tupin F, Pérez P (2021) CARRADA dataset: camera and automotive radar with range-angle-doppler annotations. In: Proceedings of IEEE conference for computer vision and pattern recognition, pp 5068–5075

  42. Navtech. Navtech radar technical specifications. https://navtechradar.com/clearway-technical-specifications/. Accessed 03 June 2021

Download references

Funding

This work was supported in part by the Science and Technology Innovation Project of Xiongan New Area (No. 2022XAGG0181), the Science and Technology Key Project of Fujian Province (No. 2020HZ020005, 2021HZ021004 and 2021H61010115), the National Natural Science Foundation of China (No.U1705263), the President’s Fund of Xiamen University for Undergraduate (No. 20720212006), and the Open Project of State Key Laboratory of Matamaterial Electromagnetic Modulation Technology (No. XM-202204-0024).

Author information

Authors and Affiliations

  1. Department of Information and Communication Engineering, School of Informatics, Xiamen University, 422 Siming South Road, Siming District, Xiamen, 361005, Fujian, China

    Fahad Jibrin Abdu & Zhenmiao Deng

  2. School of Electronic Science and Engineering (National Model Microelectronics College), Xiamen University, Xiamen, 361005, China

    Yixiong Zhang

Authors
  1. Fahad Jibrin Abdu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yixiong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenmiao Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yixiong Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abdu, F.J., Zhang, Y. & Deng, Z. CCA-Based Fusion of Camera and Radar Features for Target Classification Under Adverse Weather Conditions. Neural Process Lett 55, 7293–7319 (2023). https://doi.org/10.1007/s11063-023-11261-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11261-w

Keywords

Search

Navigation