Skip to main content
SpringerLink
Log in

Video surveillance image enhancement via a convolutional neural network and stacked denoising autoencoder

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In an extensive-scale surveillance system, the quality of the surveillance camera installed varies. This variation of surveillance camera produces different image quality in terms of resolution, illumination, and noise. The quality of the captured image depends on the surveillance camera hardware, placement and orientation, and the surrounding light. A pixelated, low illumination and noisy image produced by a low-quality surveillance camera causes critical issues for video surveillance face recognition systems. To address these issues, a deep learning image enhancement (DLIE) model is proposed. By utilizing a deep learning architecture such as a convolutional neural network (CNN) and a denoising autoencoder, the image quality can be enhanced. The DLIE model is able to improve image resolution and illumination and reduce noise in an image. There are two deep learning blocks (DLB) in the DLIE model, which are DLB1 and DLB2. Both DLBs are arranged in parallel so that all the stated problems can be addressed simultaneously. DLB1 is proposed to address the occurrence of pixelated images by reconstructing a low-resolution image into a high-resolution image using a CNN. DLB2 used the capability of a denoising autoencoder to reconstruct the corrupted image into a clean image by enhancing the dark and noisy images. The output of each DLB is fused using image fusion to obtain the optimum image quality. The image is evaluated using the peak to signal noise ratio (PSNR) and structural similarity index (SSIM). The enhanced image from the DLIE model exhibits superior quality compared to the original image ranging from 13.3625 to 22.7728 for PSNR and 0.6207 to 0.8155 for SSIM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Amolins K, Zhang Y, Dare P (2007) Wavelet based image fusion techniques-an introduction, review and comparison. ISPRS J Photogram Remote Sens 62(4):249–263. https://doi.org/10.1016/j.isprsjprs.2007.05.009

    Article  Google Scholar 

  2. Bonetto R, Latzko V (2020) Chapter 8-machine learning. In: Fitzek FH, Granelli F, Seeling P (eds) Computing in communication networks. Academic Press, London

    Google Scholar 

  3. Bovik AC, Acton ST (2009) Chapter 10—basic linear filtering with application to image enhancement. In: Bovik A (ed) The essential guide to image processing. Academic Press, Boston, pp 225–239. https://doi.org/10.1016/B978-0-12-374457-9.00010-X

    Chapter  Google Scholar 

  4. Buber E, Diri B (2018) Performance analysis and cpu vs gpu comparison for deep learning. In: 2018 6th International conference on control engineering information technology (CEIT), pp 1–6

  5. Cao G, Huang L, Tian H, Huang X, Wang Y, Zhi R (2018) Contrast enhancement of brightness-distorted images by improved adaptive gamma correction. Comput Electr Eng 66:569–582. https://doi.org/10.1016/j.compeleceng.2017.09.012

    Article  Google Scholar 

  6. Cermeño E, Pérez A, Sigüenza JA (2018) Intelligent video surveillance beyond robust background modeling. Expert Syst Appl 91:138–149. https://doi.org/10.1016/j.eswa.2017.08.052

    Article  Google Scholar 

  7. Chen CY, Chen CH, Chen CH, Lin KP (2016) An automatic filtering convergence method for iterative impulse noise filters based on psnr checking and filtered pixels detection. Expert Syst Appl 63:198–207. https://doi.org/10.1016/j.eswa.2016.07.003

    Article  Google Scholar 

  8. Dong C, Loy CC, He K, Tang X (2016a) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  9. Dong C, Loy CC, Tang X (2016b) Accelerating the super-resolution convolutional neural network. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9906 LNCS:391–407, 1608.00367

  10. Fan Z, Hu X, Chen C, Wang X, Peng S (2020) Facial image super-resolution guided by adaptive geometric features. Eurasip J Wirel Commun Netw 2020(1). https://doi.org/10.1186/s13638-020-01760-y

    Article  Google Scholar 

  11. Gai S, Bao Z (2019) New image denoising algorithm via improved deep convolutional neural network with perceptive loss. Expert Syst Appl 138:112815. https://doi.org/10.1016/j.eswa.2019.07.032

    Article  Google Scholar 

  12. Gharbi M, Chaurasia G, Paris S, Durand F (2016) Deep joint demosaicking and denoising. ACM Trans Graph 35(6), ACM SIGGRAPH Asia Conference, Macao, 2016. https://doi.org/10.1145/2980179.2982399

  13. Grgic M, Delac K, Grgic S (2011) SCface—surveillance cameras face database. Multimed Tools Appl 51(3):863–879. https://doi.org/10.1007/s11042-009-0417-2

    Article  Google Scholar 

  14. He H, Siu WC (2011) Single image super-resolution using gaussian process regression. CVPR 2011:449–456. https://doi.org/10.1109/CVPR.2011.5995713

    Article  Google Scholar 

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  16. Hore A, Ziou D (2010) Image quality metrics: Psnr vs. ssim. In: 2010 20th International conference on pattern recognition, pp 2366–2369. https://doi.org/10.1109/ICPR.2010.579

  17. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of psnr in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  18. Iizuka S, Simo-Serra E, Ishikawa H (2016) Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans Graph 10(1145/2897824):2925974

    Google Scholar 

  19. Jain V, Seung S (2009) Natural image denoising with convolutional networks. In: Koller D, Schuurmans D, Bengio Y, Bottou L (eds) Advances in neural information processing systems 21. Curran Associates Inc, London, pp 769–776

    Google Scholar 

  20. Jiang Y, Gong X, Liu D, Cheng Y, Fang C, Shen X, Yang J, Zhou P, Wang Z (2021) Enlightengan: deep light enhancement without paired supervision. IEEE Trans Image Process 30:2340–2349. https://doi.org/10.1109/TIP.2021.3051462

    Article  Google Scholar 

  21. Jianji W, Chen P, Zheng N, Chen B, Principe JC, Wang FY (2021) Associations between mse and ssim as cost functions in linear decomposition with application to bit allocation for sparse coding. Neurocomputing 422:139–149. https://doi.org/10.1016/j.neucom.2020.10.018

    Article  Google Scholar 

  22. Kim J, Lee JK, Lee KM (2015) Accurate image super-resolution using very deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  23. Ko KE, Sim KB (2018) Deep convolutional framework for abnormal behavior detection in a smart surveillance system. Eng Appl Artif Intell 67:226–234. https://doi.org/10.1016/j.engappai.2017.10.001

    Article  Google Scholar 

  24. Kwasniewska A, Ruminski J, Szankin M, Kaczmarek M (2020) Super-resolved thermal imagery for high-accuracy facial areas detection and analysis. Eng Appl Artif Intell 87:103263. https://doi.org/10.1016/j.engappai.2019.103263

    Article  Google Scholar 

  25. Lore KG, Akintayo A, Sarkar S (2017) LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recogn 61(Supplement C):650–662(Supplement C):650–652. https://doi.org/10.1016/j.patcog.2016.06.008

    Article  Google Scholar 

  26. Łoza A, Bull DR, Hill PR, Achim AM (2013) Automatic contrast enhancement of low-light images based on local statistics of wavelet coefficients. Digital Signal Process 23(6):1856–1866. https://doi.org/10.1016/j.dsp.2013.06.002

    Article  Google Scholar 

  27. Mafi M, Martin H, Cabrerizo M, Andrian J, Barreto A, Adjouadi M (2019) A comprehensive survey on impulse and gaussian denoising filters for digital images. Signal Process 157:236–260. https://doi.org/10.1016/j.sigpro.2018.12.006

    Article  Google Scholar 

  28. Meena SK, Potnis A, Mishra M, Dwivedy P, Soofi S (2017) Review and application of different contrast enhancement technique on various images. In: 2017 1st International conference on electronics, materials engineering and nano-technology (IEMENTech), pp 1–6. https://doi.org/10.1109/IEMENTECH.2017.8076993

  29. Nixon M, Aguado AS (2012) Feature extraction and image processing for computer vision, 3rd edn. Academic Press, London

    Google Scholar 

  30. Omar N, Sengur A, Al-Ali SGS (2020) Cascaded deep learning-based efficient approach for license plate detection and recognition. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2020.113280

    Article  Google Scholar 

  31. Pang S, [del Coz] JJ, Yu Z, Luaces O, Díez J, (2017) Deep learning to frame objects for visual target tracking. Eng Appl Artif Intell 65:406–420. https://doi.org/10.1016/j.engappai.2017.08.010

    Article  Google Scholar 

  32. Shen X, Tao X, Gao H, Zhou C, Jia J (2016). Deep automatic portrait matting. In: Leibe B, Matas J, Sebe, N, Welling M (eds) Computer vision-ECCV 2016, PT I, Lecture Notes in Computer Science, vol 9905, pp 92–107, 10.1007/978-3-319-46448-0\_6, 14th European conference on computer vision (ECCV), Amsterdam, 2016

  33. Sonali Sahu S, Singh AK, Ghrera S, Elhoseny M (2019) An approach for de-noising and contrast enhancement of retinal fundus image using clahe. Optics Laser Technol 110:87–98. https://doi.org/10.1016/j.optlastec.2018.06.061

    Article  Google Scholar 

  34. Tao L, Zhu C, Xiang G, Li Y, Jia H, Xie X (2017) Llcnn: a convolutional neural network for low-light image enhancement. In: 2017 IEEE visual communications and image processing (VCIP), pp 1–4

  35. Vanmali AV, Kataria T, Kelkar SG, Gadre VM (2020) Ringing artifacts in wavelet based image fusion: analysis, measurement and remedies. Inf Fusion 56:39–69. https://doi.org/10.1016/j.inffus.2019.10.003

    Article  Google Scholar 

  36. Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning-ICML ’08, pp 1096–1103. https://doi.org/10.1145/1390156.1390294, http://portal.acm.org/citation.cfm?doid=1390156.1390294, arXiv:1412.6550v4

  37. Wang X, Tao Q, Wang L, Li D, Zhang M (2015) Deep convolutional architecture for natural image denoising. In: 2015 International conference on wireless communications and signal processing (WCSP), pp 1–4. https://doi.org/10.1109/WCSP.2015.7341021, http://ieeexplore.ieee.org/document/7341021/

  38. Wu S, Zhong S, Liu Y (2017) Deep residual learning for image steganalysis. Multimed Tools Appl. https://doi.org/10.1007/s11042-017-4440-4

    Article  Google Scholar 

  39. Xiao B, Tang H, Jiang Y, Li W, Wang G (2018) Brightness and contrast controllable image enhancement based on histogram specification. Neurocomputing 275:2798–2809. https://doi.org/10.1016/j.neucom.2017.11.057

    Article  Google Scholar 

  40. Zhang K, Zuo W, Chen Y, Meng D, Zhang L (2017) Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Trans Image Process 26(7):3142–3155

    Article  MathSciNet  MATH  Google Scholar 

  41. Zhou W, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

Download references

Acknowledgements

This research is fully supported by Universiti Sains Malaysia Bridging Grant (Bridging Grant) No. 304/PELECT/6316115 and Universiti Sains Malaysia Research University Individual (RUI) Research Grant No. 1001/PELECT/8014056.

Author information

Authors and Affiliations

  1. Intelligent Biometric Group, School of Electrical and Electronic Engineering, USM Engineering Campus, Universiti Sains Malaysia, 14300, Nibong Tebal, Pulau Pinang, Malaysia

    Muhamad Faris Che Aminudin & Shahrel Azmin Suandi

Authors
  1. Muhamad Faris Che Aminudin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shahrel Azmin Suandi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shahrel Azmin Suandi.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research is fully supported by Universiti Sains Malaysia Bridging Grant (Bridging Grant) No. 304/PELECT/6316115 and Universiti Sains Malaysia Research University Individual (RUI) Research Grant No. 1001/PELECT/8014056.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Che Aminudin, M.F., Suandi, S.A. Video surveillance image enhancement via a convolutional neural network and stacked denoising autoencoder. Neural Comput & Applic 34, 3079–3095 (2022). https://doi.org/10.1007/s00521-021-06551-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06551-0

Keywords

Search

Navigation