Skip to main content
SpringerLink
Log in

Video Enhancement via Super-Resolution Using Deep Quality Transfer Network

  • Conference paper
  • First Online:
Computer Vision – ACCV 2016 (ACCV 2016)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10113))

Included in the following conference series:

Abstract

Streaming low bitrate while preserving high-quality video content is a crucial topic in multimedia and video surveillance. In this work, we explore the problem of spatially and temporally reconstructing high-resolution (HR) frames from a high frame-rate low-resolution (LR) sequence and a few temporally subsampled HR frames. The targeted problem is essentially different from the problems handled by typical super-resolution (SR) methods such as single-image SR and video SR, which attempt to reconstruct HR images using only LR images. To tackle the targeted problem, we propose a deep quality transfer network, based on the convolutional neural network (CNN), which consists of modules including generation and selection of HR pixel candidates, fusion with LR input, residual learning and bidirectional architecture. The proposed CNN model has real-time performance at inference stage. The empirical studies have verified the generality of the proposed CNN model showing significant quality gains for video enhancement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    YUV video dataset: http://www.codersvoice.com/a/webbase/video/08/152014/130.html.

References

  1. Wiegand, T., Sullivan, G.J., Bjøntegaard, G., Luthra, A.: Overview of the h. 264/avc video coding standard. IEEE Trans. Circuits Syst. Video Technol. 13, 560–576 (2003)

    Article  Google Scholar 

  2. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intel. (PAMI) 32, 295–307 (2016)

    Article  Google Scholar 

  3. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  4. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206 (2015)

    Google Scholar 

  5. Huang, Y., Wang, W., Wang, L.: Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Neural Information Processing Systems (NIPS), pp. 235–243 (2015)

    Google Scholar 

  6. Liao, R., Tao, X., Li, R., Ma, Z., Jia, J.: Video super-resolution via deep draft-ensemble learning. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 531–539 (2015)

    Google Scholar 

  7. Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., Wu, E.: Handling motion blur in multi-frame super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5224–5232 (2015)

    Google Scholar 

  8. Shahar, O., Faktor, A., Irani, M.: Space-time super-resolution from a single video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3353–3360 (2011)

    Google Scholar 

  9. Gupta, A., Bhat, P., Dontcheva, M., Deussen, O., Curless, B., Cohen, M.: Enhancing and experiencing spacetime resolution with videos and stills. In: Proceedings of the IEEE International Conference on Computational Photography (ICCP), pp. 1–9 (2009)

    Google Scholar 

  10. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

    Article  Google Scholar 

  11. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Neural Information Processing Systems (NIPS), pp. 1097–1105 (2012)

    Google Scholar 

  12. Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., Brox, T.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 2758–2766 (2015)

    Google Scholar 

  13. Protter, M., Elad, M., Takeda, H., Milanfar, P.: Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans. Image Process. (TIP) 18, 36–51 (2009)

    Article  MathSciNet  Google Scholar 

  14. Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. (TIP) 18, 1958–1975 (2009)

    Article  MathSciNet  Google Scholar 

  15. Yang, C.Y., Ma, C., Yang, M.H.: Single-image super-resolution: a benchmark. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 372–386. Springer, Heidelberg (2014). doi:10.1007/978-3-319-10593-2_25

    Google Scholar 

  16. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  17. Baker, S., Kanade, T.: Super-resolution optical flow. Technical report (1999)

    Google Scholar 

  18. Tarvainen, J., Mikko, N., Pirkko, O.: Spatial and temporal information as camera parameters for super-resolution video. In: Proceedings of the International Symposium on Multimedia (ISM), pp. 302–305 (2012)

    Google Scholar 

  19. Ancuti, C., Ancuti, C.O., Bekaert, P.: A patch-based approach to restore videos using additional stills. In: Proceedings of the International Symposium ELMAR, pp. 143–146 (2010)

    Google Scholar 

  20. Horn, B.K., Schunck, B.G.: Determining optical flow. In: Technical Symposium East. International Society for Optics and Photonics, pp. 319–331 (1981)

    Google Scholar 

  21. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: Deepflow: large displacement optical flow with deep matching. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 1385–1392 (2013)

    Google Scholar 

  22. Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: Epicflow: edge-preserving interpolation of correspondences for optical flow. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1164–1172 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Umbo CV Inc., Taipei City, Taiwan

    Pai-Heng Hsiao & Ping-Lin Chang

Authors
  1. Pai-Heng Hsiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ping-Lin Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pai-Heng Hsiao .

Editor information

Editors and Affiliations

  1. National Tsing Hua University, Hsinchu, Taiwan

    Shang-Hong Lai

  2. Graz University of Technology, Graz, Austria

    Vincent Lepetit

  3. Drexel University, Philadelphia, Pennsylvania, USA

    Ko Nishino

  4. The University of Tokyo , Tokyo, Japan

    Yoichi Sato

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Hsiao, PH., Chang, PL. (2017). Video Enhancement via Super-Resolution Using Deep Quality Transfer Network. In: Lai, SH., Lepetit, V., Nishino, K., Sato, Y. (eds) Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science(), vol 10113. Springer, Cham. https://doi.org/10.1007/978-3-319-54187-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54187-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54186-0

  • Online ISBN: 978-3-319-54187-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation