skip to main content
research-article

DANCE: Distributed Generative Adversarial Networks with Communication Compression

Published: 22 October 2021 Publication History

Abstract

Generative adversarial networks (GANs) have shown great success in deep representations learning, data generation, and security enhancement. With the development of the Internet of Things, 5th generation wireless systems (5G), and other technologies, the large volume of data collected at the edge of networks provides a new way to improve the capabilities of GANs. Due to privacy, bandwidth, and legal constraints, it is not appropriate to upload all the data to the cloud or servers for processing. Therefore, this article focuses on deploying and training GANs at the edge rather than converging edge data to the central node. To address this problem, we designed a novel distributed learning architecture for GANs, called DANCE. DANCE can adaptively perform communication compression based on the available bandwidth, while supporting both data and model parallelism training of GANs. In addition, inspired by the gossip mechanism and Stackelberg game, a compatible algorithm, AC-GAN is proposed. The theoretical analysis guarantees the convergence of the model and the existence of approximate equilibrium in AC-GAN. Both simulation and prototype system experiments show that AC-GAN can achieve better training effectiveness with less communication overhead than the SOTA algorithms, i.e., FL-GAN and MD-GAN.

References

[1]
Z. Akhtar, Y. S. Nam, R. Govindan, S. Rao, J. Chen, E. Katz-Bassett, B. Ribeiro, J. Zhan, and H. Zhang. 2018. Oboe: Auto-tuning video ABR algorithms to network conditions. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 44–58.
[2]
D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. 2017. QSGD: Communication-Efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems, Vol. 30, 1709–1720.
[3]
S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah. 2006. Randomized gossip algorithms. IEEE Trans. Inf. Theory 52, 6 (2006), 2508–2530.
[4]
X. Cao, G. Tang, D. Guo, Y. Li, and W. Zhang. 2020. Edge federation: Towards an integrated service provisioning model. IEEE/ACM Transactions on Networking 28, 3 (2020), 1116–1129.
[5]
W. Dai, J. Doyle, X. Liang, H. Zhang, N. Dong, Y. Li, and E. P. Xing. 2017. Scan: Structure correcting adversarial network for chest x-rays organ segmentation. arXiv:1703.08770. Retrieved from https://arxiv.org/abs/1703.08770.
[6]
M. Dereziński, M. K. Warmuth, and D. Hsu. 2019. Unbiased estimators for random design regression. arXiv:1907.03411. Retrieved from https://arxiv.org/abs/1907.03411.
[7]
I. Durugkar, I. Gemp, and S. Mahadevan. 2017. Generative multi-adversarial networks. In Proceedings of the International Conference on Learning Representations.
[8]
A. Ferdowsi and W. Saad. 2019. Generative adversarial networks for distributed intrusion detection in the internet of things. In Proceedings of the IEEE Global Communications Conference (GLOBECOM’19). IEEE, 1–6.
[9]
A. Ghosh, V. Kulharia, V. P. Namboodiri, P. H. S. Torr, and P. K. Dokania. 2018. Multi-agent diverse generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8513–8521.
[10]
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, Vol. 3, 2672–2680.
[11]
C. Hardy, E. L. Merrer, and B. Sericola. 2018. Gossiping GANs position paper. In Proceedings of the 2nd Workshop on Distributed Infrastructures for Deep Learning, Part of Middleware, 25–28.
[12]
C. Hardy, E. L. Merrer, and B. Sericola. 2019. MD-GAN: Multi-discriminator generative adversarial networks for distributed datasets. In Proceedings of the IEEE 33rd International Parallel and Distributed Processing Symposium. 866–877.
[13]
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems. 6626–6637.
[14]
Q. Hoang, T. D. Nguyen, T. Le, and D. Phung. 2017. Multi-generator generative adversarial nets. arXiv:1708.02556. Retrieved from https://arxiv.org/abs/1708.02556.
[15]
T. Karras, T. Aila, S. Laine, and J. Lehtinen. 2018. Progressive growing of gans for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations, 7354–7363.
[16]
D. P. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https://arxiv.org/abs/1412.6980.
[17]
A. Koloskova, S. U. Stich, and M. Jaggi. 2019. Decentralized stochastic optimization and gossip algorithms with compressed communication. In Proceedings of the International Conference on Machine Learning. 3478–3487.
[18]
J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. 2017. Federated learning: Strategies for improving communication efficiency. arXiv:1610.05492. Retrieved from https://arxiv.org/abs/1610.05492.
[19]
A. Krizhevsky, V. Nair, and G. Hinton. 2014. The cifar-10 Dataset. Retrieved from http://www.cs.toronto.edu/kriz/cifar.html.
[20]
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
[21]
J. Li, X. Cao, D. Guo, J. Xie, and H. Chen. 2020. Task Scheduling with UAV-assisted Vehicular Cloud for Road Detection in Highway Scenario. IEEE Internet of Things Journal 7, 8 (2020), 7702–7713.
[22]
M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B. Su. 2014. Scaling distributed machine learning with the parameter server. In Proceedings of the 11th Symposium on Operating Systems Design and Implementation. 583–598.
[23]
Y. Lin, S. Han, H. Mao, Y. Wang, and B. Dally. 2018. Deep gradient compression: Reducing the communication bandwidth for distributed training. In Proceedings of the International Conference on Learning Representations.
[24]
X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2794–2802.
[25]
X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. 2018. On the effectiveness of least squares generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 41, 12 (2018), 2947–2960.
[26]
H. B. McMahan, E. Moore, D. Ramage, and B. A. Arcas. 2016. Federated learning of deep networks using model averaging. arXiv:1602.05629. Retrieved from https://arxiv.org/abs/1602.05629.
[27]
L. Melis, C. Song, E. D. Cristofaro, and V. Shmatikov. 2019. Exploiting unintended feature leakage in collaborative learning. In Proceedings of the IEEE Symposium on Security and Privacy (SP’19). IEEE, 691–706.
[28]
T. Nguyen, T. Le, H. Vu, and D. Phung. 2017. Dual discriminator generative adversarial nets. In Advances in Neural Information Processing Systems. 2670–2680.
[29]
A. Odena, C. Olah, and J. Shlens. 2017. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70. 2642–2651.
[30]
H. Raei and N. Yazdani. 2017. Performability analysis of cloudlet in mobile cloud computing. Inf. Sci.388 (2017), 99–117.
[31]
H. Robbins and S. Monro. 1951. A stochastic approximation method. The Annals of Mathematical Statistics, 400–407.
[32]
T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. 2016. Improved techniques for training gans. In Advances in Neural Information Processing Systems. 2234–2242.
[33]
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 2014. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Proceedings of the Annual Conference of the International Speech Communication Association (2014), 1058–1062.
[34]
H. Shi, J. Dong, W. Wang, Y. Qian, and X. Zhang. 2017. SSGAN: Secure steganography based on generative adversarial networks. In Proceedings of the Pacific Rim Conference on Multimedia. Springer, 534–544.
[35]
Speedtest.net. 2019. Speedtest Report of the U.S. Retrieved February 14, 2020 fromhttps://www.speedtest.net/reports/united-states/.
[36]
S. U. Stich, J. B. Cordonnier, and M. Jaggi. 2018. Sparsified SGD with memory. In Advances in Neural Information Processing Systems, 4447–4458.
[37]
C. Wang, C. Xu, C. Wang, and D. Tao. 2018. Perceptual adversarial networks for image-to-image transformation. IEEE Trans. Image Process. 27, 8 (2018), 4066–4079.
[38]
H. Xiao, K. Rasul, and R. Vollgraf. 2017. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747. Retrieved from https://arxiv.org/abs/1708.07747.
[39]
R. Yonetani, T. Takahashi, A. Hashimoto, and Y. Ushiku. 2019. Decentralized learning of generative adversarial networks from multi-client non-iid data. arXiv:1905.09684. Retrieved from https://arxiv.org/abs/1905.09684.
[40]
H. Zhang, I. J. Goodfellow, D. Metaxas, and A. Odena. 2019. Self-attention generative adversarial networks. In Proceedings of the International Conference on Machine Learning, 7354–7363.
[41]
H. Zhang, S. Xu, J. Jiao, P. Xie, R. Salakhutdinov, and E. P. Xing. 2018. Stackelberg GAN: Towards provable minimax equilibrium via multi-generator architectures. arXiv:1811.08010. Retrieved from https://arxiv.org/abs/1811.08010.

Cited By

View all
  • (2025)Quantum Computing and Generative Adversarial Networks (GANs)Quantum AI and its Applications in Blockchain Technology10.4018/979-8-3373-1657-4.ch007(111-156)Online publication date: 17-Jan-2025
  • (2024)DRD-GAN: A Novel Distributed Conditional Wasserstein Deep Convolutional Relativistic Discriminator GAN with Improved ConvergenceACM Transactions on Probabilistic Machine Learning10.1145/3655030Online publication date: 29-May-2024
  • (2024)A Novel Federated Learning Scheme for Generative Adversarial NetworksIEEE Transactions on Mobile Computing10.1109/TMC.2023.327866823:5(3633-3649)Online publication date: May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet Technology
ACM Transactions on Internet Technology  Volume 22, Issue 2
May 2022
582 pages
ISSN:1533-5399
EISSN:1557-6051
DOI:10.1145/3490674
  • Editor:
  • Ling Liu
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2021
Accepted: 01 March 2021
Revised: 01 October 2020
Received: 01 May 2020
Published in TOIT Volume 22, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Generative adversarial networks
  2. distributed learning
  3. communication compression

Qualifiers

  • Research-article
  • Refereed

Funding Sources

  • National Natural Science Foundation of China
  • Science Fund for Distinguished Young Scholars in Hunan Province
  • Scientific Research Project of National University of Defense Technology

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)142
  • Downloads (Last 6 weeks)2
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Quantum Computing and Generative Adversarial Networks (GANs)Quantum AI and its Applications in Blockchain Technology10.4018/979-8-3373-1657-4.ch007(111-156)Online publication date: 17-Jan-2025
  • (2024)DRD-GAN: A Novel Distributed Conditional Wasserstein Deep Convolutional Relativistic Discriminator GAN with Improved ConvergenceACM Transactions on Probabilistic Machine Learning10.1145/3655030Online publication date: 29-May-2024
  • (2024)A Novel Federated Learning Scheme for Generative Adversarial NetworksIEEE Transactions on Mobile Computing10.1109/TMC.2023.327866823:5(3633-3649)Online publication date: May-2024
  • (2024)A deep learning model of dance generation for young children based on music rhythm and beatConcurrency and Computation: Practice and Experience10.1002/cpe.805336:13Online publication date: 4-Mar-2024
  • (2023)Empirical Comparison of Energy Efficiency Between 3-Tier and High-Speed 3-Tier Network TopologiesInternational Journal of Cloud Applications and Computing10.4018/IJCAC.33276613:1(1-28)Online publication date: 26-Oct-2023
  • (2023)Resource Optimization for Task Offloading With Real-Time Location Prediction in Pedestrian-Vehicle Interaction ScenariosIEEE Transactions on Wireless Communications10.1109/TWC.2023.325025422:11(7331-7344)Online publication date: 6-Mar-2023
  • (2023)MCLA Task Offloading Framework for 5G-NR-V2X-Based Heterogeneous VECNsIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.329214024:12(14329-14346)Online publication date: 3-Aug-2023
  • (2023)SDWD: Style Diversity Weighted Distance Evaluates the Intra-Class Data Diversity of Distributed GANs2023 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP49359.2023.10222285(1925-1929)Online publication date: 8-Oct-2023

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media