Skip to main content
SpringerLink
Log in

MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13678))

Included in the following conference series:

Abstract

The high-resolution screen of edge devices stimulates a strong demand for efficient image super-resolution (SR). An emerging research, SR-LUT, responds to this demand by marrying the look-up table (LUT) with learning-based SR methods. However, the size of a single LUT grows exponentially with the increase of its indexing capacity. Consequently, the receptive field of a single LUT is restricted, resulting in inferior performance. To address this issue, we extend SR-LUT by enabling the cooperation of Multiple LUTs, termed MuLUT. Firstly, we devise two novel complementary indexing patterns and construct multiple LUTs in parallel. Secondly, we propose a re-indexing mechanism to enable the hierarchical indexing between multiple LUTs. In these two ways, the total size of MuLUT is linear to its indexing capacity, yielding a practical method to obtain superior performance. We examine the advantage of MuLUT on five SR benchmarks. MuLUT achieves a significant improvement over SR-LUT, up to 1.1 dB PSNR, while preserving its efficiency. Moreover, we extend MuLUT to address demosaicing of Bayer-patterned images, surpassing SR-LUT on two benchmarks by a large margin.

J. Li and C. Chen—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1122–1131 (2017)

    Google Scholar 

  2. Ahn, N., Kang, B., Sohn, K.-A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 256–272. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_16

    Chapter  Google Scholar 

  3. Buades, A., Coll, B., Morel, J., Sbert, C.: Self-similarity driven color demosaicking. IEEE Trans. Image Process. 18(6), 1192–1202 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  4. Chang, H., Yeung, D., Xiong, Y.: Super-resolution through neighbor embedding. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 275–282 (2004)

    Google Scholar 

  5. Chang, K., Ding, P.L.K., Li, B.: Color image demosaicking using inter-channel correlation and nonlocal self-similarity. Sig. Process. Image Commun. 39, 264–279 (2015)

    Article  Google Scholar 

  6. Chen, C., Xiong, Z., Tian, X., Zha, Z., Wu, F.: Camera lens super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1652–1660 (2019)

    Google Scholar 

  7. Chen, H., et al.: AdderNet: do we really need multiplications in deep learning? In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1465–1474 (2020)

    Google Scholar 

  8. Cheng, Z., Xiong, Z., Chen, C., Liu, D., Zha, Z.: Light field super-resolution with zero-shot learning. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10010–10019 (2021)

    Google Scholar 

  9. Chu, X., Zhang, B., Ma, H., Xu, R., Li, Q.: Fast, accurate and lightweight super-resolution with neural architecture search. In: International Conference on Pattern Recognition (ICPR), pp. 59–64 (2020)

    Google Scholar 

  10. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  11. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  12. Duran, J., Buades, A.: Self-similarity and spectral correlation adaptive algorithm for color demosaicking. IEEE Trans. Image Process. 23(9), 4031–4040 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  13. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)

    Article  Google Scholar 

  14. Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning low-level vision. Int. J. Comput. Vis. 40(1), 25–47 (2000)

    Article  MATH  Google Scholar 

  15. Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. 35(6), 191:1–191:12 (2016)

    Google Scholar 

  16. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: IEEE International Conference on Computer Vision (ICCV), pp. 349–356 (2009)

    Google Scholar 

  17. Gu, J., Dong, C.: Interpreting super-resolution networks with local attribution maps. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9199–9208 (2021)

    Google Scholar 

  18. Hirakawa, K., Parks, T.W.: Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans. Image Process. 14(3), 360–369 (2005)

    Article  Google Scholar 

  19. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269 (2017)

    Google Scholar 

  20. Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206 (2015)

    Google Scholar 

  21. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 723–731 (2018)

    Google Scholar 

  22. Jeon, G., Dubois, E.: Demosaicking of noisy Bayer-sampled color images with least-squares Luma-chroma demultiplexing and noise level estimation. IEEE Trans. Image Process. 22(1), 146–156 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  23. Jo, Y., Kim, S.J.: Practical single-image super-resolution using look-up table. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 691–700 (2021)

    Google Scholar 

  24. Kasson, J.M., Nin, S.I., Plouffe, W., Hafner, J.L.: Performing color space conversions with three-dimensional linear interpolation. J. Electron. Imaging 4(3), 226–250 (1995)

    Article  Google Scholar 

  25. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Sig. Process. 29, 1153–1160 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  26. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2016)

    Google Scholar 

  27. Kim, S.J., Lin, H.T., Lu, Z., Süsstrunk, S., Lin, S., Brown, M.S.: A new in-camera imaging model for color computer vision and its application. IEEE Trans. Pattern Anal. Mach. Intell. 34(12), 2289–2302 (2012)

    Article  Google Scholar 

  28. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)

    Google Scholar 

  29. Kokkinos, F., Lefkimmiatis, S.: Iterative joint image demosaicking and denoising using a residual denoising network. IEEE Trans. Image Process. 28(8), 4177–4188 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  30. Lee, R., et al.: Journey towards tiny perceptual super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12371, pp. 85–102. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58574-7_6

    Chapter  Google Scholar 

  31. Li, H., et al.: PAMS: quantized super-resolution via parameterized max scale. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 564–580. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_34

    Chapter  Google Scholar 

  32. Li, X., Gunturk, B.K., Zhang, L.: Image demosaicing: a systematic survey. In: Electronic Imaging (2008)

    Google Scholar 

  33. Li, Y., Gu, S., Zhang, K., Van Gool, L., Timofte, R.: DHP: differentiable meta pruning via hypernetworks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12353, pp. 608–624. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_36

    Chapter  Google Scholar 

  34. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017)

    Google Scholar 

  35. Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. In: International Conference on Learning Representations (ICLR) (2017)

    Google Scholar 

  36. Luo, X., Xie, Y., Zhang, Y., Qu, Y., Li, C., Fu, Y.: LatticeNet: towards lightweight image super-resolution with lattice block. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12367, pp. 272–289. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58542-6_17

    Chapter  Google Scholar 

  37. Mantiuk, R., Daly, S.J., Kerofsky, L.: Display adaptive tone mapping. ACM Trans. Graph. 27(3), 68 (2008)

    Article  Google Scholar 

  38. Martin, D.R., Fowlkes, C.C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE International Conference on Computer Vision (ICCV), pp. 416–425 (2001)

    Google Scholar 

  39. Matsui, Y., et al.: Sketch-based manga retrieval using manga109 dataset. Multim. Tools Appl. 76(20), 21811–21838 (2017)

    Article  Google Scholar 

  40. Mukherjee, J., Mitra, S.K.: Enhancement of color images by scaling the DCT coefficients. IEEE Trans. Image Process. 17(10), 1783–1794 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  41. Pan, Z., et al.: Towards bidirectional arbitrary image rescaling: joint optimization and cycle idempotence. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17389–17398 (2022)

    Google Scholar 

  42. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Google Scholar 

  43. Qiu, G.: Interresolution look-up table for improved spatial magnification of image. J. Vis. Commun. Image Represent. 11(4), 360–373 (2000)

    Article  Google Scholar 

  44. Romano, Y., Isidoro, J., Milanfar, P.: RAISR: rapid and accurate image super resolution. IEEE Trans. Comput. Imaging 3(1), 110–125 (2017)

    Article  MathSciNet  Google Scholar 

  45. Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3791–3799 (2015)

    Google Scholar 

  46. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883 (2016)

    Google Scholar 

  47. Song, D., Wang, Y., Chen, H., Xu, C., Xu, C., Tao, D.: AdderSR: towards energy efficient image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15648–15657 (2021)

    Google Scholar 

  48. Song, D., Xu, C., Jia, X., Chen, Y., Xu, C., Wang, Y.: Efficient residual dense block search for image super-resolution. In: Conference on Artificial Intelligence (AAAI), pp. 12007–12014 (2020)

    Google Scholar 

  49. Song, Q., Xiong, R., Liu, D., Xiong, Z., Wu, F., Gao, W.: Fast image super-resolution via local adaptive gradient field sharpening transform. IEEE Trans. Image Process. 27(4), 1966–1980 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  50. Timofte, R., Smet, V.D., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International Conference on Computer Vision (ICCV), pp. 1920–1927 (2013)

    Google Scholar 

  51. Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_8

    Chapter  Google Scholar 

  52. Wang, L., Wu, H., Pan, C.: Fast image upsampling via the displacement field. IEEE Trans. Image Process. 23(12), 5123–5135 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  53. Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_5

    Chapter  Google Scholar 

  54. Wang, Y.: A multilayer neural network for image demosaicking. In: IEEE International Conference on Image Processing (ICIP), pp. 1852–1856 (2014)

    Google Scholar 

  55. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  56. Xiao, Z., Fu, X., Huang, J., Cheng, Z., Xiong, Z.: Space-time distillation for video super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2113–2122 (2021)

    Google Scholar 

  57. Xiao, Z., Xiong, Z., Fu, X., Liu, D., Zha, Z.: Space-time video super-resolution using temporal profiles. In: ACM Multimedia Conference, pp. 664–672 (2020)

    Google Scholar 

  58. Xin, J., Wang, N., Jiang, X., Li, J., Huang, H., Gao, X.: Binarized neural network for single image super resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 91–107. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_6

    Chapter  Google Scholar 

  59. Xiong, Z., Sun, X., Wu, F.: Robust web image/video super-resolution. IEEE Trans. Image Process. 19(8), 2017–2028 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  60. Xiong, Z., Xu, D., Sun, X., Wu, F.: Example-based super-resolution with soft information and decision. IEEE Trans. Multim. 15(6), 1458–1465 (2013)

    Article  Google Scholar 

  61. Xu, R., Xiao, Z., Yao, M., Zhang, Y., Xiong, Z.: Stereo video super-resolution via exploiting view-temporal correlations. In: ACM Multimedia Conference, pp. 460–468 (2021)

    Google Scholar 

  62. Yang, J., Lin, Z., Cohen, S.: Fast image super-resolution based on in-place example regression. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1059–1066 (2013)

    Google Scholar 

  63. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  64. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces - 7th International Conference, Avignon, France, 24–30 June 2010, Revised Selected Papers, vol. 6920, pp. 711–730 (2010)

    Google Scholar 

  65. Zhang, H., Liu, D., Xiong, Z.: Two-stream action recognition-oriented video super-resolution. In: IEEE International Conference on Computer Vision (ICCV), pp. 8798–8807 (2019)

    Google Scholar 

  66. Zhang, L., Wu, X.: Color demosaicking via directional linear minimum mean square-error estimation. IEEE Trans. Image Process. 14(12), 2167–2178 (2005)

    Google Scholar 

  67. Zhang, L., Wu, X., Buades, A., Li, X.: Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. J. Electron. Imaging 20(2), 023016 (2011)

    Google Scholar 

  68. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  69. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2472–2481 (2018)

    Google Scholar 

Download references

Acknowledgments

We acknowledge funding from National Key R &D Program of China under Grant 2017YFA0700800, and National Natural Science Foundation of China under Grants 62131003 and 62021001.

Author information

Authors and Affiliations

  1. University of Science and Technology of China, Hefei, China

    Jiacheng Li, Zhen Cheng & Zhiwei Xiong

  2. Huawei Noah’s Ark Lab, Beijing, China

    Chang Chen

Authors
  1. Jiacheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhen Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiwei Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiwei Xiong .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 11283 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, J., Chen, C., Cheng, Z., Xiong, Z. (2022). MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13678. Springer, Cham. https://doi.org/10.1007/978-3-031-19797-0_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19797-0_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19796-3

  • Online ISBN: 978-3-031-19797-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation