Skip to main content
SpringerLink
Log in

Hybrid MPI/OpenMP parallel asynchronous distributed alternating direction method of multipliers

  • Regular Paper
  • Published:
Computing Aims and scope Submit manuscript

Abstract

The distributed alternating direction method of multipliers (ADMM) is one of the most widely used algorithms to solve large-scale optimization problems. Since the memory consumption, communication cost and convergence of the distributed ADMM are affected by the number of workers, how to improve the scalability of the distributed ADMM is one of the main challenges. To address this challenge, this paper proposes an asynchronous distributed ADMM based on the hybrid parallel model (HPAD-ADMM), which uses OpenMP for parallelization inside the node and MPI for message passing between nodes in the distributed system. Each worker solves sub-problem in parallel by multithreading, which reduces the system time at each iteration without affecting the convergence of the system or increasing the communication cost and memory consumption. Furthermore, this paper designs efficient parallelized algorithms to solve sub-problems for different applications. For the L1-regularized logistic regression problem, the sub-problem is solved by parallel trust region newton method and system time is reduced by adjusting the accuracy of the sub-problem. For the lasso problem, parallel matrix inversion algorithms are selected dynamically to reduce the system time according to the size of the data set. Finally, large-scale data sets are used to test the performance of the HPAD-ADMM. Experimental results show that compared with the state-of-the-art distributed ADMM, the HPAD-ADMM has higher scalability without losing accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets.

References

  1. Balamurugan P, Posinasetty A, Shevade S (2016) ADMM for training sparse structural SVMs with augmented L1 regularizers. In: SIAM international conference on data mining. https://doi.org/10.1137/1.9781611974348.77

  2. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J et al (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends® Mach Learn 3(1):1–122. https://doi.org/10.1561/2200000016

    Article  MATH  Google Scholar 

  3. Chang TH, Hong M, Liao WC, Wang X (2016) Asynchronous distributed ADMM for large-scale optimization-part i: algorithm and convergence analysis. IEEE Trans Signal Process 64(12):3118–3130. https://doi.org/10.1109/TSP.2016.2537271

    Article  MathSciNet  MATH  Google Scholar 

  4. Chang TH, Liao WC, Hong M, Wang X (2016) Asynchronous distributed ADMM for large-scale optimization-part ii: linear convergence analysis and numerical performance. IEEE Trans Signal Process 64(12):3131–3144. https://doi.org/10.1109/TSP.2016.2537261

    Article  MathSciNet  MATH  Google Scholar 

  5. Chorley MJ, Walker DW (2010) Performance analysis of a hybrid MPI/OpenMP application on multi-core clusters. J Comput Sci 1(3):168–174. https://doi.org/10.1016/j.jocs.2010.05.001

    Article  Google Scholar 

  6. Dhar S, Yi C, Ramakrishnan N, Shah M (2015) ADMM based scalable machine learning on spark. In: 2015 IEEE international conference on big data (Big Data), IEEE

  7. Feng S, Hua X, Wang Y, Lan Q, Zhu X (2019) Matrix information geometry for signal detection via hybrid MPI/OpenMP. Entropy. https://doi.org/10.3390/e21121184

    Article  MathSciNet  Google Scholar 

  8. Hsia CY, Zhu Y, Lin CJ (2017) A study on trust region update rules in newton methods for large-scale linear classification. In: Proceedings of machine learning research, PMLR, vol 77, pp 33–48

  9. Ibrahim DS, Hamdy S (2017) Hybrid MPI/OpenMP implementation of PCA. In: 2017 eighth international conference on intelligent computing and information systems (ICICIS), IEEE

  10. Jin H, Jespersen D, Mehrotra P, Biswas R, Huang L, Chapman B (2011) High performance computing using MPI and OpenMP on multi-core parallel systems. Parallel Comput 37(9):562–575. https://doi.org/10.1016/j.parco.2011.02.002

    Article  Google Scholar 

  11. Kwedlo W, Czochanski PJ (2019) A hybrid MPI/OpenMP parallelization of k-means algorithms accelerated using the triangle inequality. IEEE Access 7:42280–42297

    Article  Google Scholar 

  12. Li Y, Wang X, Fang W, Xue F, Li X (2019) A distributed ADMM approach for collaborative regression learning in edge computing. Comput Mater Contin 58(2):493–508

    Article  Google Scholar 

  13. Lubell-Doughtie P, Sondag J (2013) Practical distributed classification using the alternating direction method of multipliers algorithm. In: IEEE international conference on big data, IEEE, vol 1, pp 773–776

  14. Smith L, Bull M (2001) Development of mixed mode MPI/OpenMP. Sci Program 9(2):83–98

    Google Scholar 

  15. Song C, Yoon S, Pavlovic V (2016) Fast ADMM algorithm for distributed optimization with adaptive penalty. In: 30th AAAI conference on artificial intelligence, pp 753–759

  16. Wang D, Lei Y, Xie J, Wang G (2021) HSAC-ALADMM: an asynchronous lazy ADMM algorithm based on hierarchical sparse allreduce communication. J Supercomput. https://doi.org/10.1007/s11227-020-03590-7

    Article  Google Scholar 

  17. Wang H, Gao Y, Shi Y, Wang H (2016) A fast distributed classification algorithm for large-scale imbalanced data. In: 16th IEEE international conference on data mining (ICDM), IEEE, pp 1251–1256

  18. Wang H, Gao Y, Shi Y, Wang R (2017) Group-based alternating direction method of multipliers for distributed linear classification. IEEE Trans Cybern 47(11):3568–3582

    Article  Google Scholar 

  19. Wang S, Lei Y (2018) Fast communication structure for asynchronous distributed ADMM under unbalance process arrival pattern. In: 27th international conference on artificial neural networks, Springer

  20. Woodsend K, Gondzio J (2009) Hybrid MPI/OpenMP parallel linear support vector machine training. J Mach Learn Res 10:1937–1953

    MathSciNet  MATH  Google Scholar 

  21. Xie J, Lei Y (2019) ADMMLIB: A library of communication-efficient AD-ADMM for distributed machine learning. In: FIP international conference on network and parallel computing

  22. Xu Z, Taylor G, Li H, Figueiredo M, Goldstein T (2017) Adaptive consensus ADMM for distributed optimization

  23. Yan B, Regueiro RA (2018) Comparison between pure MPI and hybrid MPI-OpenMP parallelism for discrete element method (DEM) of ellipsoidal and poly-ellipsoidal particles. Comput Particle Mech 6(2):271–295

    Article  Google Scholar 

  24. Yuan GX, Chang KW, Hsieh CJ, Lin CJ (2010) A comparison of optimization methods and software for large-scale L1-regularized linear classification. J Mach Learn Res 11(2):3183–3234

    MathSciNet  MATH  Google Scholar 

  25. Zhang R, Kwok J (2014) Asynchronous distributed ADMM for consensus optimization. In: Proceedings of the 31th international conference on machine learning, JMLR, pp. 1701–1709

  26. Zhang Y, Xiao L (2015) Communication-efficient distributed optimization of self-concordant empirical loss. Mathematics 2227:289–341

    MathSciNet  MATH  Google Scholar 

  27. Zhuang Y, Chin WS, Juan YC, Lin CJ (2015) Distributed newton methods for regularized logistic regression. In: 19th Pacific-Asia conference on knowledge discovery and data mining (PAKDD), vol 9078, pp 690–703

Download references

Author information

Authors and Affiliations

  1. School of Computer Engineering and Science, Shanghai University, Shanghai, China

    Dongxia Wang, Yongmei Lei & Jianhui Zhou

Authors
  1. Dongxia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongmei Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianhui Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongmei Lei.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is supported by the National Natural Science Foundation of China under Grant No. U1811461.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, D., Lei, Y. & Zhou, J. Hybrid MPI/OpenMP parallel asynchronous distributed alternating direction method of multipliers. Computing 103, 2737–2762 (2021). https://doi.org/10.1007/s00607-021-00968-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-021-00968-0

Keywords

Mathematics Subject Classification

Search

Navigation