skip to main content
10.1145/3493700.3493731acmconferencesArticle/Chapter ViewAbstractPublication PagescomadConference Proceedingsconference-collections
research-article

Cost-Optimized Video Transfer using Real-Time Super Resolution Convolutional Neural Networks

Published: 08 January 2022 Publication History

Abstract

The explosion of video generation and consumption, coupled with an inadequate rise in network bandwidth has led to network delays and decreased Quality of Experience, limiting the opportunities to tap into the full potential of video data. These deficiencies in network resources with a shift to cloud computing models have resulted in the need to revisit the overall mechanism for video transfer and storage of videos between edge devices and the cloud. We propose a novel multi-scale real-time super-resolution convolutional neural network to achieve the composite task of optimizing the entire cost of video transfer with minimal loss of quality that can be used for any application involving the transfer of video data. To achieve this, we develop a cost-optimized video transfer system that optimizes the metrics of video transfer to give the best quality video output, given the user budget. The model makes use of Convolution blocks for extracting features and output creation with multiple sub-pixel convolutions in a novel structure. For upscaling to full High Definition video at 30 fps, the model successfully retained the frame rate while the system achieved savings in transfer time and bandwidth usage. This model has been trained on surveillance videos (VIRAT dataset), but consistent results were obtained during testing even on feature films and sports videos which demonstrates its content invariance. The evaluation of our approach averaged over 376 videos, yielded meager quality losses of 8%, measured by a novel non-referential quality metric, also proposed in this paper. Additionally, average network bandwidth savings of 80% and average video transfer time reduction of 52% were achieved.

References

[1]
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. 2012. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. BMVC (2012), 1–10.
[2]
VNI Cisco. 2019. Cisco visual networking index: Forecast and trends, 2017–2022 white paper. Technical report (2019).
[3]
Jenny Darmody. 2020. How will data be managed and transferred in autonomous cars?https://www.siliconrepublic.com/machines/data-autonomous-cars-florian-baumann.
[4]
C. Dong, C. C. Loy, K. He, and X. Tang. 2016. Image Super-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 2(2016), 295–307.
[5]
S. Farsiu, M.D. Robinson, M. Elad, and P. Milanfar. 2004. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing 13, 10 (2004), 1327–1344.
[6]
K. Fukushima. 2004. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics 36 (2004), 193–202.
[7]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger(Eds.), Vol. 27. Curran Associates, Inc., 2672–2680.
[8]
Lena Griebel, Hans-Ulrich Prokosch, Felix Köpcke, Dennis Toddenroth, Jan Christoph, Ines Leb, Igor Engel, and Martin Sedlmayr. 2015. A scoping review of cloud computing in healthcare. BMC Medical Informatics and Decision Making 15 (03 2015).
[9]
Le Kang, Peng Ye, Yi Li, and David Doermann. 2014. Convolutional Neural Networks for No-Reference Image Quality Assessment. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1. 1733–1740.
[10]
A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos. 2016. Video Super-Resolution With Convolutional Neural Networks. IEEE Transactions on Computational Imaging 2, 2 (2016), 109–122.
[11]
J. Kim, J. K. Lee, and K. M. Lee. 2016. Deeply-Recursive Convolutional Network for Image Super-Resolution. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1637–1645.
[12]
T. Kim, Mehdi S. M. Sajjadi, M. Hirsch, and B. Schölkopf. 2018. Spatio-Temporal Transformer Network for Video Restoration. In ECCV.
[13]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6980
[14]
C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 105–114.
[15]
X. Li, Y. Li, T. Liu, J. Qiu, and F. Wang. 2009. The Method and Tool of Cost Analysis for Cloud Computing. In 2009 IEEE International Conference on Cloud Computing. 93–100.
[16]
Weisi Lin and C. C. Jay Kuo. 2011. Perceptual Visual Quality Metrics: A Survey. J. Vis. Comun. Image Represent. 22, 4 (May 2011), 297–312.
[17]
Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang. 2019. Deep Video Frame Interpolation Using Cyclic Frame Generation. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (Jul. 2019), 8794–8802.
[18]
Alice Lucas, Santiago Lopez-Tapia, Rafael Molina, and Aggelos K. Katsaggelos. 2019. Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution. IEEE Transactions on Image Processing 28, 7 (Jul 2019), 3312–3327.
[19]
D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. 416–423 vol.2.
[20]
Artan Mazrekaj, I. Shabani, and Besmir Sejdiu. 2016. Pricing Schemes in Cloud Computing: An Overview. International Journal of Advanced Computer Science and Applications 7 (2016), 80–86.
[21]
A. Mittal, A. K. Moorthy, and A. C. Bovik. 2012. No-Reference Image Quality Assessment in the Spatial Domain. IEEE Transactions on Image Processing 21, 12 (2012), 4695–4708.
[22]
Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, J. K. Aggarwal, Hyungtae Lee, Larry Davis, Eran Swears, Xioyang Wang, Qiang Ji, Kishore Reddy, Mubarak Shah, Carl Vondrick, Hamed Pirsiavash, Deva Ramanan, Jenny Yuen, Antonio Torralba, Bi Song, Anesco Fong, Amit Roy-Chowdhury, and Mita Desai. 2011. A large-scale benchmark dataset for event recognition in surveillance video. In CVPR 2011. 3153–3160. https://doi.org/10.1109/CVPR.2011.5995586
[23]
Raman B. Paranjape. 2000. Fundamental Enhancement Techniques. Academic Press, Inc., USA, 3–18.
[24]
Vineeth Joel Patel. 2021. Think Your Cellphone Uses a lot of Data? Report Claims Autonomous Cars Will Use 4,000 GB in one Day. https://www.futurecar.com/876/Think-Your-Cellphone-Uses-a-lot-of-Data-Report-Claims-Autonomous-Cars-Will-Use-4000-GB-in-one-Day-.
[25]
Nuno Roma and Leonel Sousa. 2007. Efficient Hybrid DCT-Domain Algorithm for Video Spatial Downscaling. EURASIP J. Adv. Signal Process 2007, 2 (June 2007), 30.
[26]
M. A. Saad, A. C. Bovik, and C. Charrier. 2011. DCT statistics model-based blind image quality assessment. In 2011 18th IEEE International Conference on Image Processing. 3093–3096.
[27]
M. A. Saad, A. C. Bovik, and C. Charrier. 2014. Blind Prediction of Natural Video Quality. IEEE Transactions on Image Processing 23, 3 (2014), 1352–1365.
[28]
W. Shi, J. Caballero, Ferenc Huszár, J. Totz, A. Aitken, R. Bishop, D. Rueckert, and Zehan Wang. 2016. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 1874–1883.
[29]
Jialu Wang, Guowei Teng, and Ping An. 2021. Video Super-Resolution Based on Generative Adversarial Network and Edge Enhancement. Electronics 10, 4 (2021). https://www.mdpi.com/2079-9292/10/4/459
[30]
Nicolas Weber, Michael Waechter, Sandra C. Amend, Stefan Guthe, and Michael Goesele. 2016. Rapid, Detail-Preserving Image Downscaling. ACM Trans. Graph. 35, 6, Article 205 (Nov. 2016), 6 pages.
[31]
Wikipedia. 2021. Surveillance. https://en.wikipedia.org/wiki/Surveillance.
[32]
Chih Yuan Yang, Chao Ma, and Ming Hsuan Yang. 2014. Single-image super-resolution: A benchmark. In Computer Vision, ECCV 2014 - 13th European Conference, Proceedings (part 4 ed.)(Lecture Notes in Computer Science, PART 4). Springer Verlag, Germany, 372–386.
[33]
Chia-Hung Yeh, Ying H. Chen, Ming-Chieh Chi, and Mei-Juan Chen. 2010. Parabolic Motion-Vector Re-estimation Algorithm for Compressed Video Downscaling. J. Signal Process. Syst. 61, 3 (2010), 375–386.
[34]
Hyunho Yeo, Sunghyun Do, and Dongsu Han. 2017. How Will Deep Learning Change Internet Video Delivery?. In Proceedings of the 16th ACM Workshop on Hot Topics in Networks (Palo Alto, CA, USA) (HotNets-XVI). Association for Computing Machinery, New York, NY, USA, 57–64.
[35]
Hyunho Yeo, Youngmok Jung, Jaehong Kim, Jinwoo Shin, and Dongsu Han. 2018. Neural Adaptive Content-aware Internet Video Delivery. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 645–661.
[36]
Roman Zeyde, Michael Elad, and Matan Protter. 2010. On Single Image Scale-up Using Sparse-Representations. In Proceedings of the 7th International Conference on Curves and Surfaces (Avignon, France). Springer-Verlag, Berlin, Heidelberg, 711–730.
[37]
Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.

Cited By

View all
  • (2023)Recent advances in deep learning models: a systematic literature reviewMultimedia Tools and Applications10.1007/s11042-023-15295-z82:29(44977-45060)Online publication date: 25-Apr-2023
  • (2022)Super‐Resolution Swin Transformer and Attention Network for Medical CT ImagingBioMed Research International10.1155/2022/44315362022:1Online publication date: 8-Dec-2022

Index Terms

  1. Cost-Optimized Video Transfer using Real-Time Super Resolution Convolutional Neural Networks
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CODS-COMAD '22: Proceedings of the 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)
        January 2022
        357 pages
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 08 January 2022

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. CNN
        2. GAN
        3. cost optimization
        4. deep neural networks
        5. super resolution
        6. video transfer

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        CODS-COMAD 2022
        Sponsor:

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)7
        • Downloads (Last 6 weeks)1
        Reflects downloads up to 17 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2023)Recent advances in deep learning models: a systematic literature reviewMultimedia Tools and Applications10.1007/s11042-023-15295-z82:29(44977-45060)Online publication date: 25-Apr-2023
        • (2022)Super‐Resolution Swin Transformer and Attention Network for Medical CT ImagingBioMed Research International10.1155/2022/44315362022:1Online publication date: 8-Dec-2022

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media