Skip to main content
SpringerLink
Log in
Book cover

European Conference on Computer Vision

ECCV 2020: Computer Vision – ECCV 2020 Workshops pp 647–664Cite as

Joint Demosaicking and Denoising for CFA and MSFA Images Using a Mosaic-Adaptive Dense Residual Network

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12537))

Abstract

Color filter array (CFA) has been a basis for modern photography and recently multispectral filter array (MSFA) has gradually found its wide application. A deep learning network capable of joint demosaicking and denoising for both CFA and MSFA raw images is proposed in this paper. First, a novel dense residual network that includes multiple types of skip connections is introduced to learn features at different resolutions. Then, mosaic adaptive convolution and data augmentation based on mosaic shifting are put forward to fully make use of common characteristics of CFA and MSFA mosaic images. Moreover, an L1 loss function normalized by noise standard deviation is suggested to train the deep residual network so it does not rely on an explicit input of known or estimated noise standard deviation. Extensive experiments using simulated and real mosaic images from CFA cameras demonstrate that the proposed mosaic-adaptive dense residual network (MDRN) outperforms other state-of-the-art deep learning algorithms significantly. For simulated MSFA mosaics and real MSFA raw images, it also shows much improved results compared to other methods.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://github.com/cig-skoltech/deep_demosaick.

  2. 2.

    https://github.com/mgharbi/demosaicnet.

  3. 3.

    http://spectraldevices.com/products/multispectral-snapshot-cameras.

References

  1. Akiyama, H., Tanaka, M., Okutomi, M.: Pseudo four-channel image denoising for noisy CFA raw data. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 4778–4782. IEEE (2015)

    Google Scholar 

  2. Al-khafaji, S.L., Zhou, J., Zia, A., Liew, A.W.C.: Spectral-spatial scale invariant feature transform for hyperspectral images. IEEE Trans. Image Process. 27(2), 837–850 (2018)

    Article  MathSciNet  Google Scholar 

  3. Bayer, B.E.: Color imaging array. United States Patent 3,971,065 (1976)

    Google Scholar 

  4. Brauers, J., Aach, T.: A color filter array based multispectral camera. In: Group, G.C. (ed.) 12. Workshop Farbbildverarbeitung. Ilmenau, 5–6 October 2006 (2006)

    Google Scholar 

  5. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 60–65. IEEE (2005)

    Google Scholar 

  6. Condat, L.: A simple, fast and efficient approach to denoisaicking: joint demosaicking and denoising. In: 2010 IEEE International Conference on Image Processing, pp. 905–908. IEEE (2010)

    Google Scholar 

  7. Condat, L.: A new color filter array with optimal properties for noiseless and noisy color image acquisition. IEEE Trans. Image Process. 20(8), 2200–2210 (2011)

    Article  MathSciNet  Google Scholar 

  8. Condat, L., Mosaddegh, S.: Joint demosaicking and denoising by total variation minimization. In: 2012 19th IEEE International Conference on Image Processing, pp. 2781–2784. IEEE (2012)

    Google Scholar 

  9. Dong, W., Yuan, M., Li, X., Shi, G.: Joint demosaicing and denoising with perceptual optimization on a generative adversarial network. arXiv preprint arXiv:1802.04723 (2018)

  10. Fotiadou, K., Tsagkatakis, G., Tsakalides, P.: Deep convolutional neural networks for the classification of snapshot mosaic hyperspectral imagery. Electron. Imaging 2017(17), 185–190 (2017)

    Article  Google Scholar 

  11. Gao, J., Nuyttens, D., Lootens, P., He, Y., Pieters, J.G.: Recognising weeds in a maize crop using a random forest machine-learning algorithm and near-infrared snapshot mosaic hyperspectral imagery. Biosyst. Eng. 170, 39–50 (2018)

    Article  Google Scholar 

  12. Geelen, B., Tack, N., Lambrechts, A.: A compact snapshot multispectral imager with a monolithically integrated per-pixel filter mosaic. In: Advanced Fabrication Technologies for Micro/Nano Optics and Photonics VII, vol. 8974, p. 89740L. International Society for Optics and Photonics (2014)

    Google Scholar 

  13. Getreuer, P.: Color demosaicing with contour stencils. In: 2011 17th International Conference on Digital Signal Processing (DSP), pp. 1–6. IEEE (2011)

    Google Scholar 

  14. Getreuer, P.: Zhang-Wu directional LMMSE image demosaicking. Image Process. On Line 1, 117–126 (2011)

    Google Scholar 

  15. Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. (TOG) 35(6), 1–12 (2016). https://doi.org/10.1145/2980179.2982399

    Article  Google Scholar 

  16. Habtegebrial, T.A., Reis, G., Stricker, D.: Deep convolutional networks for snapshot hypercpectral demosaicking. In: 2019 10th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. IEEE (2019)

    Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  18. Heide, F., et al.: FlexISP: a flexible camera image processing framework. ACM Trans. Graph. (TOG) 33(6), 231 (2014)

    Article  Google Scholar 

  19. Henz, B., Gastal, E.S., Oliveira, M.M.: Deep joint design of color filter arrays and demosaicing. In: Computer Graphics Forum, vol. 37, pp. 389–399. Wiley Online Library (2018). https://doi.org/10.1111/cgf.13370

  20. Jaiswal, S.P., Fang, L., Jakhetiya, V., Pang, J., Mueller, K., Au, O.C.: Adaptive multispectral demosaicking based on frequency-domain analysis of spectral correlation. IEEE Trans. Image Process. 26(2), 953–968 (2017). https://doi.org/10.1109/tip.2016.2634120

    Article  MathSciNet  MATH  Google Scholar 

  21. Jeon, G., Dubois, E.: Demosaicking of noisy Bayer-sampled color images with least-squares luma-chroma demultiplexing and noise level estimation. IEEE Trans. Image Process. 22(1), 146–156 (2013)

    Article  MathSciNet  Google Scholar 

  22. Kalevo, O., Rantanen, H.: Noise reduction techniques for Bayer-matrix images. In: Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications III, vol. 4669, pp. 348–359. International Society for Optics and Photonics (2002)

    Google Scholar 

  23. Khashabi, D., Nowozin, S., Jancsary, J., Fitzgibbon, A.W.: Joint demosaicing and denoising via learned nonparametric random fields. IEEE Trans. Image Process. 23(12), 4968–4981 (2014)

    Article  MathSciNet  Google Scholar 

  24. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  25. Klatzer, T., Hammernik, K., Knobelreiter, P., Pock, T.: Learning joint demosaicing and denoising based on sequential energy minimization. In: 2016 IEEE International Conference on Computational Photography (ICCP), pp. 1–11. IEEE (2016)

    Google Scholar 

  26. Kokkinos, F., Lefkimmiatis, S.: Deep image demosaicking using a cascade of convolutional residual denoising networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 317–333. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_19

    Chapter  Google Scholar 

  27. Kokkinos, F., Lefkimmiatis, S.: Iterative joint image demosaicking and denoising using a residual denoising network. IEEE Trans. Image Process. 28, 4177–4188 (2019)

    Article  MathSciNet  Google Scholar 

  28. Li, X., Gunturk, B., Zhang, L.: Image demosaicing: a systematic survey. In: Visual Communications and Image Processing 2008, vol. 6822, p. 68221J. International Society for Optics and Photonics (2008)

    Google Scholar 

  29. MacKenzie, L., Choudhary, T., McNaught, A.I., Harvey, A.R.: In vivo oximetry of human bulbar conjunctival and episcleral microvasculature using snapshot multispectral imaging. Exp. Eye Res. 149, 48–58 (2016)

    Article  Google Scholar 

  30. Miao, L., Qi, H., Ramanath, R., Snyder, W.E.: Binary tree-based generic demosaicking algorithm for multispectral filter arrays. IEEE Trans. Image Process. 15(11), 3550–3558 (2006). https://doi.org/10.1109/tip.2006.877476

    Article  Google Scholar 

  31. Mihoubi, S., Losson, O., Mathon, B., Macaire, L.: Multispectral demosaicing using pseudo-panchromatic image. IEEE Trans. Comput. Imaging 3(4), 982–995 (2017). https://doi.org/10.1109/tci.2017.2691553

    Article  MathSciNet  Google Scholar 

  32. Monno, Y., Kiku, D., Kikuchi, S., Tanaka, M., Okutomi, M.: Multispectral demosaicking with novel guide image generation and residual interpolation. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 645–649. IEEE (2014). https://doi.org/10.1109/icip.2014.7025129

  33. Monno, Y., Kiku, D., Tanaka, M., Okutomi, M.: Adaptive residual interpolation for color and multispectral image demosaicking. Sensors 17(12), 2787–1-21 (2017). https://doi.org/10.3390/s17122787

  34. Monno, Y., Tanaka, M., Okutomi, M.: Multispectral demosaicking using adaptive kernel upsampling. In: 2011 18th IEEE International Conference on Image Processing, pp. 3157–3160. IEEE (2011). https://doi.org/10.1109/icip.2011.6116337

  35. Monno, Y., Tanaka, M., Okutomi, M.: Multispectral demosaicking using guided filter. In: Digital Photography VIII, vol. 8299, pp. 82990O-1-7. International Society for Optics and Photonics (2012)

    Google Scholar 

  36. Monno, Y., Teranaka, H., Yoshizaki, K., Tanaka, M., Okutomi, M.: Single-sensor RGB-NIR imaging: high-quality system design and prototype implementation. IEEE Sens. J. 19(2), 497–507 (2019)

    Article  Google Scholar 

  37. Nascimento, S.M., Amano, K., Foster, D.H.: Spatial distributions of local illumination color in natural scenes. Vis. Res. 120, 39–44 (2016). https://doi.org/10.1016/j.visres.2015.07.005

    Article  Google Scholar 

  38. Pan, Z., Li, B., Cheng, H., Bao, Y.: Deep panchromatic image guided residual interpolation for multispectral image demosaicking. In: 2019 10th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. IEEE (2019)

    Google Scholar 

  39. Park, S.H., Kim, H.S., Lansel, S., Parmar, M., Wandell, B.A.: A case for denoising before demosaicking color filter array data. In: 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers, pp. 860–864. IEEE (2009)

    Google Scholar 

  40. Paszke, A., et al.: Automatic differentiation in pytorch. In: NIPS-W (2017)

    Google Scholar 

  41. Shinoda, K., et al.: Multispectral filter array and demosaicking for pathological images. In: 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 697–703. IEEE (2015). https://doi.org/10.1109/apsipa.2015.7415362

  42. Shinoda, K., Yoshiba, S., Hasegawa, M.: Deep demosaicking for multispectral filter arrays. arXiv preprint arXiv:1808.08021 (2018)

  43. Shrestha, R., Pillay, R., George, S., Hardeberg, J.Y.: Quality evaluation in spectral imaging-quality factors and metrics. JAIC-J. Int. Colour Assoc. 12, 22–35 (2014)

    Google Scholar 

  44. Tan, D.S., Chen, W.Y., Hua, K.L.: DeepDemosaicking: adaptive image demosaicking via multiple deep fully convolutional networks. IEEE Trans. Image Process. 27(5), 2408–2419 (2018)

    Article  MathSciNet  Google Scholar 

  45. Tan, H., Zeng, X., Lai, S., Liu, Y., Zhang, M.: Joint demosaicing and denoising of noisy bayer images with ADMM. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 2951–2955. IEEE (2017)

    Google Scholar 

  46. Uemori, T., Ito, A., Moriuchi, Y., Gatto, A., Murayama, J.: Skin-based identification from multispectral image data using CNNs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12349–12358 (2019)

    Google Scholar 

  47. Wang, X., Thomas, J.B., Hardeberg, J.Y., Gouton, P.: Discrete wavelet transform based multispectral filter array demosaicking. In: 2013 Colour and Visual Computing Symposium (CVCS), pp. 1–6. IEEE (2013). https://doi.org/10.1109/cvcs.2013.66262741

  48. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. 19(9), 2241–2253 (2010)

    Article  MathSciNet  Google Scholar 

  49. Zhang, L., Lukac, R., Wu, X., Zhang, D.: PCA-based spatially adaptive denoising of CFA images for single-sensor digital cameras. IEEE Trans. Image Process. 18(4), 797–812 (2009)

    Article  MathSciNet  Google Scholar 

  50. Zhang, L., Wu, X.: Color demosaicking via directional linear minimum mean square-error estimation. IEEE Trans. Image Process. 14(12), 2167–2178 (2005)

    Article  Google Scholar 

  51. Zhang, L., Wu, X., Buades, A., Li, X.: Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. J. Electron. Imaging 20(2), 023016 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Baidu Research, Sunnyvale, CA, 94089, USA

    Zhihong Pan, Baopu Li & Yingze Bao

  2. Baidu Shenzhen R&D, Shenzhen, 518000, China

    Hsuchun Cheng

Authors
  1. Zhihong Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Baopu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hsuchun Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yingze Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhihong Pan .

Editor information

Editors and Affiliations

  1. University of Clermont Auvergne, Clermont Ferrand, France

    Adrien Bartoli

  2. Università degli Studi di Udine, Udine, Italy

    Andrea Fusiello

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 152 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pan, Z., Li, B., Cheng, H., Bao, Y. (2020). Joint Demosaicking and Denoising for CFA and MSFA Images Using a Mosaic-Adaptive Dense Residual Network. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12537. Springer, Cham. https://doi.org/10.1007/978-3-030-67070-2_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67070-2_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67069-6

  • Online ISBN: 978-3-030-67070-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation