Skip to main content
SpringerLink
Log in
SpringerLink
Log in

M2M: Learning to Enhance Low-Light Image from Model to Mobile FPGA

  • Conference paper
  • First Online:
Advances in Computer Graphics (CGI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13002))

Included in the following conference series:

Abstract

With the development of convolutional neural networks, the effectiveness of low-light image enhancement techniques have been greatly advanced. However, the calculate-intensive and memory-intensive characteristics of convolutional neural networks make them difficult to be implemented in mobile platform with low power and limited bandwidth. This paper proposes a complete solution for low-light image enhancement from CNN model to mobile (M2M) FPGA. The proposed solution utilizes a pseudo-symmetry quantization method to compress the low-light image enhancement model, and an accelerator to permit the processing ability of the system significantly. We implemented the whole system on a customized FPGA SOC platform (a low-cost chip, Xilinx Inc. ZYNQ\(^{TM}\)XC7Z035). Extensive experiments show that our method achieved competitive results with the other three platforms,  i.e. achieved better speed compare to ARM and CPU; and achieved better power efficiency compared to ARM, CPU, and GPU.

This work was supported by the Natural Science Foundation of China (U1803262, 61602349, and 61440016).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Xu, X., Liu, L., Zhang, X., Guan, W., Hu, R.: Rethinking data collection for person re-identification: active redundancy reduction. Pattern Recogn. 113(4), 107827 (2021)

    Article  Google Scholar 

  2. Xu, X., Wang, S., Wang, Z., Zhang, X., Hu, R.: Exploring image enhancement for salient object detection in low light images. ACM Trans. Multimedia Comput. Commun. Appl. 17(8), 1–19 (2020)

    Google Scholar 

  3. Xie, P., Xu, X., Wang, Z., Yamasaki, T.: Unsupervised video person re-identification via noise and hard frame aware clustering, pp. 1–6 (2021)

    Google Scholar 

  4. Mu, N., Xu, X., Zhang, X., Lin, X.: Discrete stationary wavelet transform based saliency information fusion from frequency and spatial domain in low contrast images. Pattern Recogn. Lett. 115, 84–91 (2018)

    Article  Google Scholar 

  5. Mu, N., Xu, X., Zhang, X.: Salient object detection in low contrast images via global convolution and boundary refinement. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 743–751 (2019)

    Google Scholar 

  6. Hu, J., Zhang, H., Song, L., Han, Z., Poor, H.V.: Reinforcement learning for a cellular internet of UAVs: protocol design, trajectory control, and resource management. IEEE Wirel. Commun. 27(1), 116–123 (2020)

    Article  Google Scholar 

  7. Yu, W., Shaobo, W., Xinting, H.: Cooperative path planning of UAVs & UGVs for a persistent surveillance task in urban environments. IEEE Internet Things J. 8, 4906–4919 (2020)

    Google Scholar 

  8. Wang, Y., Ma, X., Wang, J., Wang, H.: Pseudo-3d vision-inertia based underwater self-localization for AUVs. IEEE Trans. Veh. Technol. 69(7), 7895–7907 (2020)

    Article  Google Scholar 

  9. Guo, C., et al.: Zero-reference deep curve estimation for low-light image enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1780–1789 (2020)

    Google Scholar 

  10. Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. In: BMVC, p. 155 (2018)

    Google Scholar 

  11. Wang, R., Zhang, Q., Fu, C.-W., Shen, X., Zheng, W.-S., Jia, J.: Underexposed photo enhancement using deep illumination estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6849–6857 (2019)

    Google Scholar 

  12. Ren, W., et al.: Low-light image enhancement via a deep hybrid network. IEEE Trans. Image Process. 28(9), 4364–4375 (2019)

    Article  MathSciNet  Google Scholar 

  13. Zhang, Y., Zhang, J., Guo, X.: Kindling the darkness: a practical low-light image enhancer. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 1632–1640 (2019)

    Google Scholar 

  14. Cai, J., Shuhang, G., Zhang, L.: Learning a deep single image contrast enhancer from multi-exposure images. IEEE Trans. Image Process. 27(4), 2049–2062 (2018)

    Article  MathSciNet  Google Scholar 

  15. Zhu, M., Pan, P., Chen, W., Yang, Y.: EEMEFN: low-light image enhancement via edge-enhanced multi-exposure fusion network. Proc. AAAI Conf. Artif. Intell. 34, 13106–13113 (2020)

    Google Scholar 

  16. Yang, K.-F., Zhang, X.-S., Li, Y.-J.: A biological vision inspired framework for image enhancement in poor visibility conditions. IEEE Trans. Image Process. 29, 1493–1506 (2019)

    Article  Google Scholar 

  17. Xu, K., Yang, X., Yin, B., Lau, R.W.H.: Learning to restore low-light images via decomposition-and-enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2281–2290 (2020)

    Google Scholar 

  18. Land, E.H.: The retinex theory of color vision. Sci. Am. 237(6), 108–129 (1977)

    Article  Google Scholar 

  19. Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2016)

    Article  MathSciNet  Google Scholar 

  20. Li, M., Liu, J., Yang, W., Sun, X., Guo, Z.: Structure-revealing low-light image enhancement via robust retinex model. IEEE Trans. Image Process. 27(6), 2828–2841 (2018)

    Article  MathSciNet  Google Scholar 

  21. Kim, G., Kwon, D., Kwon, J.: Low-lightgan: low-light enhancement via advanced generative adversarial network with task-driven training. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 2811–2815. IEEE (2019)

    Google Scholar 

  22. Jiang, Y., et al.: EnlightenGAN: deep light enhancement without paired supervision. IEEE Trans. Image Process. 30, 2340–2349 (2021)

    Article  Google Scholar 

  23. Yang, W., Wang, S., Fang, Y., Wang, Y., Liu, J.: From fidelity to perceptual quality: a semi-supervised approach for low-light image enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3063–3072 (2020)

    Google Scholar 

  24. Canran, X., Peng, Z., Xuanzhen, H., Zhang, W., Chen, L., An, F.: FPGA-based low-visibility enhancement accelerator for video sequence by adaptive histogram equalization with dynamic clip-threshold. IEEE Trans. Circuits Syst. I Regul. Pap. 67(11), 3954–3964 (2020)

    Article  Google Scholar 

  25. Zhu, C., Han, S., Mao, H., Dally, W.J.:L Trained ternary quantization. arXiv preprint arXiv:1612.01064 (2016)

  26. Choi, J., Wang, Z., Venkataramani, S., Chuang, P.I.-J., Srinivasan, V., Gopalakrishnan, K.: PACT: parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 (2018)

  27. Dong, Z., Yao, Z., Arfeen, D., Gholami, A., Mahoney, M.W., Keutzer, K.: HAWQ-V2: Hessian aware trace-weighted quantization of neural networks. In: Advances in Neural Information Processing Systems, pp. 18518–18529 (2020)

    Google Scholar 

  28. Shen, S., et al.: Q-BERT: Hessian based ultra low precision quantization of BERT. Proc. AAAI Conf. Artif. Intell. 34, 8815–8821 (2020)

    Google Scholar 

  29. Hegde, K., Yu, J., Agrawal, R., Yan, M., Pellauer, M., Fletcher, C.: UCNN: exploiting computational reuse in deep neural networks via weight repetition. In: 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 674–687. IEEE (2018)

    Google Scholar 

  30. Albericio, J., et al.: Bit-pragmatic deep neural network computing. In: Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 382–394 (2017)

    Google Scholar 

  31. Kwon, H., Samajdar, A., Krishna, T.: MERI: enabling flexible dataflow mapping over DNN accelerators via reconfigurable interconnects. ACM SIGPLAN Not. 53(2), 461–475 (2018)

    Article  Google Scholar 

  32. Lu, H., Wei, X., Lin, N., Yan, G., Li, X.: Tetris: re-architecting convolutional neural network computation for machine learning accelerators. In: 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–8. IEEE (2018)

    Google Scholar 

  33. Zhang, J., Chen, X., Song, M., Li, T.: Eager pruning: algorithm and architecture support for fast training of deep neural networks. In: 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA), pp. 292–303. IEEE (2019)

    Google Scholar 

  34. Faraone, J., Fraser, N., Blott, M., Leong, P.H.W.: SYQ: learning symmetric quantization for efficient deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4300–4309 (2018)

    Google Scholar 

  35. Han, S., et al.: EIE: efficient inference engine on compressed deep neural network. ACM SIGARCH Comput. Archit. News 44(3), 243–254 (2016)

    Article  Google Scholar 

  36. Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., Cong, J.: Optimizing FGPA-based accelerator design for deep convolutional neural networks. In: Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 161–170 (2015)

    Google Scholar 

  37. Chen, Y.-H., Krishna, T., Emer, J.S., Sze, V.: Eyeriss: an energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE J. Solid-State Circ. 52(1), 127–138 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, Hubei, People’s Republic of China

    Ying Chen, Wei Wang, Wei Hu & Xin Xu

  2. Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, Hubei, People’s Republic of China

    Wei Wang, Wei Hu & Xin Xu

Authors
  1. Ying Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Wang .

Editor information

Editors and Affiliations

  1. University of Geneva, Carouge, Switzerland

    Nadia Magnenat-Thalmann

  2. University of Minnesota, Minneapolis, MN, USA

    Victoria Interrante

  3. EPFL, Lausanne, Switzerland

    Daniel Thalmann

  4. University of Crete, Heraklion, Crete, Greece

    George Papagiannakis

  5. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  6. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  7. University of Calgary, Calgary, AB, Canada

    Marina Gavrilova

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, Y., Wang, W., Hu, W., Xu, X. (2021). M2M: Learning to Enhance Low-Light Image from Model to Mobile FPGA. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2021. Lecture Notes in Computer Science(), vol 13002. Springer, Cham. https://doi.org/10.1007/978-3-030-89029-2_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-89029-2_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-89028-5

  • Online ISBN: 978-3-030-89029-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation