skip to main content
10.1145/3240508.3240654acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

From Data to Knowledge: Deep Learning Model Compression, Transmission and Communication

Published: 15 October 2018 Publication History

Abstract

With the advances of artificial intelligence, recent years have witnessed a gradual transition from the big data to the big knowledge. Based on the knowledge-powered deep learning models, the big data such as the vast text, images and videos can be efficiently analyzed. As such, in addition to data, the communication of knowledge implied in the deep learning models is also strongly desired. As a specific example regarding the concept of knowledge creation and communication in the context of Knowledge Centric Networking (KCN), we investigate the deep learning model compression and demonstrate its promise use through a set of experiments. In particular, towards future KCN, we introduce efficient transmission of deep learning models in terms of both single model compression and multiple model prediction. The necessity, importance and open problems regarding the standardization of deep learning models, which enables the interoperability with the standardized compact model representation bitstream syntax, are also discussed.

References

[1]
ISO/IEC JTC 1. 2000. Information Technology : Generic Coding of Moving Pictures and Associated Audio Information : Video. ITU-T Rec.H262 (2000).
[2]
ISO/IEC JTC 1. 2003. Advanced video coding for generic audiovisual services. Int. Standards Org./Int. Electrotech. Comm. (ISO/IEC) JTC 1, Rec. H.264 and ISO/IEC 14 496--10 (MPEG-4) AVC, 2003 (2003).
[3]
Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg. 2017. Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee. In Advances in Neural Information Processing Systems. 3180--3189.
[4]
Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. 2016. NetVLAD: CNN architecture for weakly supervised place recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 5297--5307.
[5]
Luigi Atzori, Antonio Iera, and Giacomo Morabito. 2010. The internet of things: A survey. Computer networks, Vol. 54, 15 (2010), 2787--2805.
[6]
AVS2/IEEE. 2014. urlhttp://www.ieee1857.org/1857.4.asp . (2014).
[7]
Artem Babenko and Victor Lempitsky. 2015. Aggregating local deep features for image retrieval. In Proceedings of the IEEE international conference on computer vision. 1269--1277.
[8]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
[9]
Werner Bailer. 2018. Use cases and requirements for coded representation of neural networks. ISO/IEC JTC1/SC29/WG11/N17338, Gwangju, Koera.
[10]
Jianshu Chao and Eckehard Steinbach. 2016. Keypoint encoding for improved feature extraction from compressed video at low bitrates. IEEE Transactions on Multimedia, Vol. 18, 1 (2016), 25--39.
[11]
Xin Dong, Shangyu Chen, and Sinno Pan. 2017a. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems. 4860--4874.
[12]
Xin Dong, Shangyu Chen, and Sinno Pan. 2017b. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems. 4860--4874.
[13]
Lingyu Duan, Yihang Lou, Shiqi Wang, Wen Gao, and Yong Rui. 2017b. AI Oriented Large-Scale Video Management for Smart City: Technologies, Standards and Beyond. arXiv preprint arXiv:1712.01432 (2017).
[14]
Ling-Yu Duan, Vijay Chandrasekhar, Jie Chen, Jie Lin, Zhe Wang, Tiejun Huang, Bernd Girod, and Wen Gao. 2016. Overview of the MPEG-CDVS Standard. IEEE Transactions on Image Processing, Vol. 25, 1 (2016), 179--194.
[15]
Ling-Yu Duan, Vijay Chandrasekhar, Shiqi Wang, Yihang Lou, Jie Lin, Yan Bai, Tiejun Huang, Alex Chichung Kot, and Wen Gao. 2017a. Compact Descriptors for Video Analysis: the Emerging MPEG Standard. arXiv preprint arXiv:1704.08141 (2017).
[16]
Emil Eriksson, György Dán, and Viktoria Fodor. 2016. Predictive distributed visual analysis for video in wireless sensor networks. IEEE Transactions on Mobile Computing, Vol. 15, 7 (2016), 1743--1756.
[17]
Emil Eriksson, György Dán, and Viktoria Fodor. 2017. Coordinating Distributed Algorithms for Feature Extraction Offloading in Multi-Camera Visual Sensor Networks. IEEE Transactions on Circuits and Systems for Video Technology (2017).
[18]
Bernd Girod, Vijay Chandrasekhar, David M. Chen, and Ngai Man Cheung. 2011. Mobile Visual Search. IEEE Signal Processing Magazine, Vol. 28, 4 (2011), 61--76.
[19]
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115 (2014).
[20]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
[21]
Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. 2016. Deep image retrieval: Learning global representations for image search. In European Conference on Computer Vision. Springer, 241--257.
[22]
Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus. 2017. End-to-end learning of deep visual representations for image retrieval. International Journal of Computer Vision, Vol. 124, 2 (2017), 237--254.
[23]
Yiwen Guo, Anbang Yao, and Yurong Chen. 2016. Dynamic network surgery for efficient dnns. In Advances In Neural Information Processing Systems. 1379--1387.
[24]
Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).
[25]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[26]
Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In International Conference on Computer Vision .
[27]
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, Vol. 1. 3.
[28]
Tiejun Huang. 2014. Surveillance video: The biggest big data. Computing Now, Vol. 7, 2 (2014), 82--91.
[29]
Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2008. Hamming embedding and weak geometric consistency for large scale image search. In European conference on computer vision. Springer, 304--317.
[30]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.
[31]
E Jebamalar Leavline and D Asir Antony Gnana Singh. 2013. Hardware implementation of LZMA data compression algorithm. International Journal of Applied Information Systems (IJAIS), Vol. 5, 4 (2013), 51--56.
[32]
Cong Leng, Hao Li, Shenghuo Zhu, and Rong Jin. 2017. Extremely low bit neural network: Squeeze the last bit out with admm. arXiv preprint arXiv:1707.09870 (2017).
[33]
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. 2017b. Runtime Neural Pruning. In Advances in Neural Information Processing Systems. 2178--2188.
[34]
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. 2017a. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. arXiv preprint arXiv:1712.01887 (2017).
[35]
Ingrid Lunden. 2017. (2017). https://techcrunch.com/2017/06/08/cisco-ip-traffic-shoots-up-to-3-zettabytes-by-2021-video-will-be-80-of-it/
[36]
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 2017. Thinet: A filter level pruning method for deep neural network compression. arXiv preprint arXiv:1707.06342 (2017).
[37]
Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, Xiaoou Tang, et almbox. 2016. Face Model Compression by Distilling Knowledge from Neurons. In AAAI. 3560--3566.
[38]
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. 2007. Object retrieval with large vocabularies and fast spatial matching. In Computer Vision and Pattern Recognition, 2007. IEEE.
[39]
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. 2008. Lost in quantization: Improving particular object retrieval in large scale image databases. In Computer Vision and Pattern Recognition, 2008. IEEE, 1--8.
[40]
Filip Radenović, Giorgos Tolias, and Ondvr ej Chum. 2016. CNN image retrieval learns from BoW: Unsupervised fine-tuning with hard examples. In European Conference on Computer Vision. Springer, 3--20.
[41]
N Ranganathan and Selwyn Henriques. 1993. High-speed VLSI designs for Lempel-Ziv-based data compression. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, Vol. 40, 2 (1993), 96--106.
[42]
Brandon Reagen, Udit Gupta, Robert Adolf, Michael M Mitzenmacher, Alexander M Rush, Gu-Yeon Wei, and David Brooks. 2017. Weightless: Lossy Weight Encoding For Deep Neural Network Compression. arXiv preprint arXiv:1711.04686 (2017).
[43]
Alessandro Redondi, Luca Baroffio, Lucio Bianchi, Matteo Cesana, and Marco Tagliasacchi. 2016. Compress-then-Analyze vs Analyze-then-Compress: what is best in Visual Sensor Networks? IEEE Transactions on Mobile Computing, Vol. 15, 12 (2016), 3000--3013.
[44]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[45]
Gary J Sullivan, Jens Ohm, Woo-Jin Han, and Thomas Wiegand. 2012. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on circuits and systems for video technology, Vol. 22, 12 (2012), 1649--1668.
[46]
Handle System. 2017. http://www.handle.net/. (2017).
[47]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, et almbox. 2015. Going deeper with convolutions. Cvpr.
[48]
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. 2015. Particular object retrieval with integral max-pooling of CNN activations. arXiv preprint arXiv:1511.05879 (2015).
[49]
Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. 2017. Beyond filters: Compact feature map for portable deep model. In International Conference on Machine Learning. 3703--3711.
[50]
Dapeng Wu, Zhenjiang Li, Jianping Wang, Yuanqing Zheng, Mo Li, and Qiuyuan Huang. 2017. Vision and Challenges for Knowledge Centric Networking (KCN). arXiv preprint arXiv:1707.00805 (2017).
[51]
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015).
[52]
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 7370--7379.
[53]
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818--833.
[54]
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. 2017. Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044 (2017).
[55]
Hao Zhou, Jose M Alvarez, and Fatih Porikli. 2016. Less is more: Towards compact cnns. In European Conference on Computer Vision. Springer, 662--677.
[56]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593 (2017).

Cited By

View all
  • (2024)QSFL: Two-Level Communication-Efficient Federated Learning on Mobile Edge DevicesIEEE Transactions on Services Computing10.1109/TSC.2024.3455098(1-16)Online publication date: 2024
  • (2024)FedPE: Adaptive Model Pruning-Expanding for Federated Learning on Mobile DevicesIEEE Transactions on Mobile Computing10.1109/TMC.2024.337470623:11(10475-10493)Online publication date: Nov-2024
  • (2024)Sensor-Driven mmWave Beam Selection in Heterogeneous Conditions Using Shared-Specific Pruning-Expanding Federated Learning2024 15th International Conference on Information and Communication Technology Convergence (ICTC)10.1109/ICTC62082.2024.10827423(1925-1930)Online publication date: 16-Oct-2024
  • Show More Cited By

Index Terms

  1. From Data to Knowledge: Deep Learning Model Compression, Transmission and Communication

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '18: Proceedings of the 26th ACM international conference on Multimedia
    October 2018
    2167 pages
    ISBN:9781450356657
    DOI:10.1145/3240508
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 15 October 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. deep learning model compression
    2. knowledge communication
    3. standardization

    Qualifiers

    • Research-article

    Funding Sources

    • PKU-NTU Joint Research Institute
    • National Key Research and Development Program of China
    • National Natural Science Foundation of China

    Conference

    MM '18
    Sponsor:
    MM '18: ACM Multimedia Conference
    October 22 - 26, 2018
    Seoul, Republic of Korea

    Acceptance Rates

    MM '18 Paper Acceptance Rate 209 of 757 submissions, 28%;
    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)43
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)QSFL: Two-Level Communication-Efficient Federated Learning on Mobile Edge DevicesIEEE Transactions on Services Computing10.1109/TSC.2024.3455098(1-16)Online publication date: 2024
    • (2024)FedPE: Adaptive Model Pruning-Expanding for Federated Learning on Mobile DevicesIEEE Transactions on Mobile Computing10.1109/TMC.2024.337470623:11(10475-10493)Online publication date: Nov-2024
    • (2024)Sensor-Driven mmWave Beam Selection in Heterogeneous Conditions Using Shared-Specific Pruning-Expanding Federated Learning2024 15th International Conference on Information and Communication Technology Convergence (ICTC)10.1109/ICTC62082.2024.10827423(1925-1930)Online publication date: 16-Oct-2024
    • (2024)A Efficient DNN Sparsity Framework with Data Pruning and Auxiliary Network2024 5th International Conference on Artificial Intelligence and Computer Engineering (ICAICE)10.1109/ICAICE63571.2024.10864032(685-692)Online publication date: 8-Nov-2024
    • (2023)ConvReLU++: Reference-based Lossless Acceleration of Conv-ReLU Operations on Mobile CPUProceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services10.1145/3581791.3596831(503-515)Online publication date: 18-Jun-2023
    • (2022)Overview of the Neural Network Compression and Representation (NNR) StandardIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2021.309597032:5(3203-3216)Online publication date: May-2022
    • (2022)Discrete cosine transform for filter pruningApplied Intelligence10.1007/s10489-022-03604-253:3(3398-3414)Online publication date: 30-May-2022
    • (2021)Digital Retina: A Way to Make the City Brain More Efficient by Visual CodingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2021.310430531:11(4147-4161)Online publication date: Nov-2021
    • (2021)Progressive Transmission and Inference of Deep Learning Models2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA)10.1109/ICMLA52953.2021.00049(271-277)Online publication date: Dec-2021
    • (2020)Towards Modality Transferable Visual Information Representation with Optimal Model CompressionProceedings of the 28th ACM International Conference on Multimedia10.1145/3394171.3413762(3705-3714)Online publication date: 12-Oct-2020
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media