Abstract
Neural Representation for Videos (NeRV) encodes each video into a network, providing a promising solution to video compression. However, existing NeRV methods are limited to representing single-quality videos with fixed-size models. To accommodate varying quality requirements, NeRV methods need multiple separate networks with different sizes, resulting in additional training and storage costs. To address this, we propose a Quality Scalable Video Coding method based on Neural Representation, in which a hierarchical network consisting of a base layer (BL) and several enhancement layers (ELs) represents the same video with coarse-to-fine qualities. As the smallest subnetwork, the BL represents basic content. The larger subnetworks can be formed by gradually adding the ELs which capture residuals between the lower-quality reconstructed frames and original ones. Since the larger subnetworks share the parameters of the smaller ones, our method saves 40% of storage space. In addition, our structural design and training strategy enable each subnetwork to outperform the baseline on average +0.29 PSNR.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wiegand, T., Sullivan, G.J., Bjontegaard, G., Luthra, A.: Overview of the h. 264/avc video coding standard. IEEE Trans. Circuits Syst. Video Technol. 13(7), 560–576 (2003)
Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22(12), 1649–1668 (2012)
Lu, G., Ouyang, W., Xu, D., Zhang, X., Cai, C., Gao, Z.: DVC: an end-to-end deep video compression framework. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11006–11015 (2019)
Li, J., Li, B., Lu, Y.: Deep contextual video compression. Adv. Neural. Inf. Process. Syst. 34, 18114–18125 (2021)
Dupont, E., Goliński, A., Alizadeh, M., Teh, Y.W., Doucet, A.: COIN: compression with implicit neural representations. arXiv preprint arXiv:2103.03123 (2021)
Dupont, E., Loya, H., Alizadeh, M., Goliński, A., Teh, Y.W., Doucet, A.: COIN++: neural compression across modalities. arXiv preprint arXiv:2201.12904 (2022)
Strümpler, Y., Postels, J., Yang, R., Gool, L.V., Tombari, F.: Implicit neural representations for image compression. In: European Conference on Computer Vision, pp. 74–91. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19809-0_5
Zhang, Y., van Rozendaal, T., Brehmer, J., Nagel, M., Cohen, T.: Implicit neural video compression. arXiv preprint arXiv:2112.11312 (2021)
Rho, D., Cho, J., Ko, J.H., Park, E.: Neural residual flow fields for efficient video representations. In: Proceedings of the Asian Conference on Computer Vision, pp. 3447–3463 (2022)
Chen, H., He, B., Wang, H., Ren, Y., Lim, S.N., Shrivastava, A.: NeRV: neural representations for videos. Adv. Neural. Inf. Process. Syst. 34, 21557–21568 (2021)
Li, Z., Wang, M., Pi, H., Xu, K., Mei, J., Liu, Y.: E-NeRV: expedite neural video representation with disentangled spatial-temporal context. In: European Conference on Computer Vision, pp. 267–284. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19833-5_16
Kim, S., Yu, S., Lee, J., Shin, J.: Scalable neural video representations with learnable positional features. Adv. Neural. Inf. Process. Syst. 35, 12718–12731 (2022)
He, B., et al.: Towards scalable neural representation for diverse videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6132–6142 (2023)
Chen, H., Gwilliam, M., He, B., Lim, S.N., Shrivastava, A.: CNeRV: content-adaptive neural representation for visual data. arXiv preprint arXiv:2211.10421 (2022)
Chen, H., Gwilliam, M., Lim, S.N., Shrivastava, A.: HNeRV: a hybrid neural representation for videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10270–10279 (2023)
Zhao, Q., Asif, M.S., Ma, Z.: DNeRV: modeling inherent dynamics via difference neural representation for videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2031–2040 (2023)
Schwarz, H., Marpe, D., Wiegand, T.: Overview of the scalable video coding extension of the h. 264/avc standard. IEEE Trans. Circuits Syst. Video Technol. 17(9), 1103–1120 (2007)
Boyce, J.M., Ye, Y., Chen, J., Ramasubramonian, A.K.: Overview of SHVC: scalable extensions of the high efficiency video coding standard. IEEE Trans. Circuits Syst. Video Technol. 26(1), 20–34 (2015)
Big buck bunny. http://bbb3d.renderfarming.net/download.html
Mercat, A., Viitanen, M., Vanne, J.: UVG dataset: 50/120fps 4k sequences for video codec analysis and development. In: Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297–302 (2020)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. Adv. Neural. Inf. Process. Syst. 33, 7462–7473 (2020)
Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8628–8638 (2021)
Cho, J., Nam, S., Rho, D., Ko, J.H., Park, E.: Streamable neural fields. In: European Conference on Computer Vision, pp. 595–612. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20044-1_34
Landgraf, Z., Hornung, A.S., Cabral, R.S.: PINs: progressive implicit networks for multi-scale neural representations. arXiv preprint arXiv:2202.04713 (2022)
Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986 (2022)
Rusu, A.A., et al.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cao, Q., Zhang, D., Sun, C. (2024). Quality Scalable Video Coding Based on Neural Representation. In: Rudinac, S., et al. MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14554. Springer, Cham. https://doi.org/10.1007/978-3-031-53305-1_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-53305-1_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53304-4
Online ISBN: 978-3-031-53305-1
eBook Packages: Computer ScienceComputer Science (R0)