Skip to main content
SpringerLink
Log in

Containers in HPC: a survey

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

OS-level virtualization (containers) has become a popular alternative to hypervisor-based virtualization. From a system-administration point-of-view, containers enable support for user-defined software stacks, thus freeing users of restrictions imposed by the host’s pre-configured software environment. In high performance computing (HPC), containers inspire special interest due to their potentially low overheads on performance. Moreover, they also bring benefits in portability and scientific reproducibility. Despite the potential advantages, the adoption of containers in HPC has been relatively slow, mainly due to specific requirements of the field. These requirements gave rise to various HPC-focused container implementations. Besides unprivileged container execution, they offer different degrees of automation of system-specific optimizations, which are necessary for optimal performance. When we looked into the scientific literature on containers applied to HPC, we were unable to find an up-to-date overview of the state-of-the-art. For this reason, we developed this extensive survey, including 93 carefully selected works. Overall, based on our survey, we argue that issues related to performance overhead are mostly solved. There is, however, a clear trade-off between performance and portability, since optimal performance often depends on host-specific optimizations. A few works propose solutions to mitigate this issue, but there is still room for improvement. Besides, we found surprisingly few works that deal with portability between dedicated HPC systems and public cloud platforms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Notes

  1. http://linux-vserver.org/ (Acc. in Sept. 2022).

  2. https://www.gnu.org/software/coreutils/manual/html_node/chroot-invocation.html (Acc. in Sept. 2022).

  3. https://man7.org/linux/man-pages/man7/namespaces.7.html (Acc. in Sept. 2022).

  4. https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cgroups.html (Acc. in Sept. 2022).

  5. https://openvz.org (Acc. in Sept. 2022).

  6. https://linuxcontainers.org/ (Acc. in Sept. 2022).

  7. https://www.kernel.org/doc/Documentation/block/cfq-iosched.txt (Acc. in Sept. 2022).

  8. https://linuxcontainers.org/lxd/ (Acc. in Sept. 2022).

  9. https://opencontainers.org/ (Acc. in Sept. 2022).

  10. https://podman.io/ (Acc. in Sept. 2022).

  11. https://docs.docker.com/engine/security/rootless/ (Acc. in Sept. 2022).

References

  1. Abraham S, Paul AK, Khan RIS, Butt AR (2020) On the use of containers in high performance computing environments. In: IEEE 13th International Conference on Cloud Computing (CLOUD). IEEE, Beijing, China, pp 284–293. https://doi.org/10.1109/cloud49709.2020.00048

  2. Aoyama K, Watanabe H, Ohue M, Akiyama Y (2020) Multiple HPC environments-aware container image configuration workflow for large-scale all-to-all protein–protein docking calculations. In: Supercomputing Frontiers. Springer, Cham, pp 23–39. https://doi.org/10.1007/978-3-030-48842-0_2

  3. Arango C, Dernat R, Sanabria J (2017) Performance evaluation of container-based virtualization for high performance computing environments. arXiv:1709.10140. Accessed 13 Sept 2022

  4. Azab A (2017) Enabling Docker containers for high-performance and many-task computing. In: IEEE International Conference on Cloud Engineering, pp 279–285. IEEE, Vancouver, Canada. https://doi.org/10.1109/ic2e.2017.52

  5. Bachiega NG, Souza PSL, Bruschi SM, de Souza SRS (2018) Container-based performance evaluation: a survey and challenges. In: IEEE International Conference on Cloud Engineering (IC2E). IEEE, Vancouver, Canada, pp 398–403. https://doi.org/10.1109/ic2e.2018.00075

  6. Bahls D (2016) Evaluating shifter for HPC applications. In: Cray User Group. CUG, London, UK. https://cug.org/proceedings/cug2016_proceedings/includes/files/pap135s2-file1.pdf. Accessed 13 Sept 2022

  7. Belkin M, Haas R, Arnold GW, Leong HW, Huerta EA, Lesny D, Neubauer M (2018) Container solutions for HPC systems. In: Proceedings of the Practice and Experience on Advanced Research Computing. ACM, New York, NY, USA. https://doi.org/10.1145/3219104.3219145

  8. Beltre AM, Saha P, Govindaraju M, Younge A, Grant RE (2019) Enabling HPC workloads on cloud infrastructure using kubernetes container orchestration mechanisms. In: IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, Denver, CO, USA, pp 11–20. https://doi.org/10.1109/canopie-hpc49598.2019.00007

  9. Benedicic L, Cruz FA, Madonna A, Mariotti K (2019) Sarus: highly scalable Docker containers for HPC systems. In: International Conference on High Performance Computing. LNCS—ISC-HPC. Springer, Cham, pp 46–60. https://doi.org/10.1007/978-3-030-34356-9_5

  10. Bentaleb O, Belloum AS, Sebaa A, El-Maouhab A (2022) Containerization technologies: taxonomies, applications and challenges. J Supercomput 78(1):1144–1181. https://doi.org/10.1007/s11227-021-03914-1

    Article  Google Scholar 

  11. Beserra D, Moreno ED, Endo PT, Barreto J (2016) Performance evaluation of a lightweight virtualization solution for HPC I/O scenarios. In: IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Melbourne, Australia, pp 004681–004686. https://doi.org/10.1109/smc.2016.7844970

  12. Brayford D, Vallecorsa S, Atanasov A, Baruffa F, Riviera W (2019) Deploying AI frameworks on secure HPC systems with containers. In: IEEE High Performance Extreme Computing Conference (HPEC). IEEE, Waltham, MA, USA, pp 1–6. https://doi.org/10.1109/hpec.2019.8916576

  13. Brayford D, Allalen M, Iapichino L, Brennan J, Moran N, Q’Riordan LJ, Hanley K (2021) Deploying containerized QuanEX quantum simulation software on HPC systems. In: 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC), pp 1–9. https://doi.org/10.1109/CANOPIEHPC54579.2021.00005

  14. Canon RS, Younge A (2019) A case for portability and reproducibility of HPC containers. In: IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, Denver, CO, USA, pp 49–54. https://doi.org/10.1109/canopie-hpc49598.2019.00012

  15. Casalicchio E, Iannucci S (2020) The state-of-the-art in container technologies: application, orchestration and security. Concurr Comput 32(17):5668. https://doi.org/10.1002/cpe.5668.e5668cpe.5668

    Article  Google Scholar 

  16. Cavet C, Souchal M, Gadrat S, Grasseau G, Satirana A, Bailly-Reyre A, Dadoun O, Mendoza V, Chamont D, Marchal-Duval G, Medernach E, Pansanel J (2020) ComputeOps: container for high performance computing. EPJ Web Conf 245:07006. https://doi.org/10.1051/epjconf/202024507006

    Article  Google Scholar 

  17. Cerin C, Greneche N, Menouer T (2020) Towards pervasive containerization of HPC job schedulers. In: International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). IEEE, Porto, Portugal, pp 281–288. https://doi.org/10.1109/sbac-pad49847.2020.00046

  18. Chang Y-TS, Heistand S, Hood R, Jin H (2021) Feasibility of running singularity containers with hybrid MPI on NASA high-end computing resources. In: 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC), pp 17–28. https://doi.org/10.1109/CANOPIEHPC54579.2021.00007

  19. Chen J, Guan Q, Liang X, Bryant P, Grubel P, McPherson A, Lo L-T, Randles T, Chen Z, Ahrens JP (2018) Build and execution environment (BEE): an encapsulated environment enabling HPC applications running everywhere. In: IEEE International Conference on Big Data (Big Data). IEEE, Seattle, WA, USA, pp 1737–1746. https://doi.org/10.1109/bigdata.2018.8622572

  20. Chung MT, Le A, Quang-Hung N, Nguyen D-D, Thoai N (2016) Provision of Docker and InfiniBand in high performance computing. In: International Conference on Advanced Computing and Applications (ACOMP). IEEE, Can Tho City, Vietnam, pp 127–134. https://doi.org/10.1109/acomp.2016.027

  21. Chung MT, Quang-Hung N, Nguyen M-T, Thoai N (2016) Using Docker in high performance computing applications. In: IEEE Sixth International Conference on Communications and Electronics (ICCE). IEEE, Ha Long Bay, Vietnan, pp 52–57. https://doi.org/10.1109/cce.2016.7562612

  22. de Bayser M, Cerqueira R (2017) Integrating MPI with Docker for HPC. In: IEEE International Conference on Cloud Engineering (IC2E). IEEE, Vancouver, BC, Canada, pp 259–265. https://doi.org/10.1109/ic2e.2017.40

  23. Freyermuth O, Wienemann P, Bechtle P, Desch K (2021) Operating an HPC/HTC cluster with fully containerized jobs using HTCondor, Singularity, CephFS and CVMFS. Comput Softw Big Sci 5(1):1–20. https://doi.org/10.1007/s41781-020-00050-y

    Article  Google Scholar 

  24. Gantikow H, Walter S, Reich C (2020) Rootless containers with Podman for HPC. In: International Conference on High Performance Computing. LNCS—ISC-HPC. Springer, Cham, pp 343–354. https://doi.org/10.1007/978-3-030-59851-8_23

  25. Gerhardt L, Bhimji W, Canon S, Fasel M, Jacobsen D, Mustafa M, Porter J, Tsulaia V (2017) Shifter: containers for HPC. J Phys Conf Ser 898:082021. https://doi.org/10.1088/1742-6596/898/8/082021

    Article  Google Scholar 

  26. Gomes J, Bagnaschi E, Campos I, David M, Alves L, Martins JA, Pina JA, López-García A, Orviz P (2018) Enabling rootless Linux containers in multi-user environments. The udocker tool. Comput Phys Commun 232:84–97. https://doi.org/10.1016/j.cpc.2018.05.021

    Article  Google Scholar 

  27. Grupp A, Kozlov V, Campos I, David M, Gomes J, López García Á (2019) Benchmarking deep learning infrastructures by means of TensorFlow and containers. In: International Conference on High Performance Computing. LNCS—ISC-HPC, pp. 478–489. Springer, Cham. https://doi.org/10.1007/978-3-030-34356-9_36

  28. Hale JS, Li L, Richardson CN, Wells GN (2017) Containers for portable, productive, and performant scientific computing. Comput Sci Eng 19(6):40–50. https://doi.org/10.1109/mcse.2017.2421459

    Article  Google Scholar 

  29. Heinonen N (2019) ALCF research benefits from singularity. https://www.hpcwire.com/off-the-wire/alcf-research-benefits-from-singularity/. Accessed 13 Feb 2022

  30. Herbein S, Dusia A, Landwehr A, McDaniel S, Monsalve J, Yang Y, Seelam SR, Taufer M (2016) Resource management for running HPC applications in container clouds. In: International Conference on High Performance Computing. LNCS—ISC-HPC, vol. 9697, pp. 261–278. Springer, Frankfurt, Germany. https://doi.org/10.1007/978-3-319-41321-1_14

  31. Higgins J, Holmes V, Venters C (2016) Autonomous discovery and management in virtual container clusters. Comput J 60(2):240–252. https://doi.org/10.1093/comjnl/bxw102

    Article  Google Scholar 

  32. Higgins J, Holmes V, Venters C (2016) Securing user defined containers for scientific computing. In: International Conference on High Performance Computing and Simulation (HPCS). IEEE, Innsbruck, Austria, pp 449–453. https://doi.org/10.1109/hpcsim.2016.7568369

  33. Hisle MS, Meier MS, Toth DM (2018) Accelerating AutoDock vina with containerization. In: Proceedings of the Practice and Experience on Advanced Research Computing. ACM, New York, NY, USA. https://doi.org/10.1145/3219104.3219154

  34. Höb M, Kranzlmüller D (2020) Enabling EASEY deployment of containerized applications for future HPC systems. In: 20th International Conference on Computational Science (ICCS). Springer, Cham, pp 206–219. https://doi.org/10.1007/978-3-030-50371-0_15

  35. Hu G, Zhang Y, Chen W (2019) Exploring the performance of singularity for high performance computing scenarios. In: IEEE 5th International Conference on Data Science and Systems (DSS). IEEE, Zhangjiajie, China, pp 2587–2593. https://doi.org/10.1109/hpcc/smartcity/dss.2019.00362

  36. Huang L, Wang Y, Lu C-Y, Liu S (2021) Best practice of IO workload management in containerized environments on supercomputers. In: Proceedings of Practice and Experience in Advanced Research Computing (PEARC). ACM, New York, NY, USA. https://doi.org/10.1145/3437359.3465561

  37. Hursey J (2020) Design considerations for building and running containerized MPI applications. In: IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, Atlanta, GA, USA, pp 35–44. https://doi.org/10.1109/canopiehpc51917.2020.00010

  38. Jacobsen DM, Canon RS (2015) Contain this: unleashing Docker for HPC. https://cug.org/proceedings/cug2015_proceedings/includes/files/pap157-file2.pdf. Accessed 13 Sept 2022

  39. Jung K, Cho Y-K, Tak Y-J (2021) Containers and orchestration of numerical ocean model for computational reproducibility and portability in public and private clouds: application of ROMS 3.6. Simul Model Pract Theory 109:102305. https://doi.org/10.1016/j.simpat.2021.102305

    Article  Google Scholar 

  40. Kadri S, Sboner A, Sigaras A, Roy S (2022) Containers in bioinformatics: applications, practical considerations, and best practices in molecular pathology. J Molec Diag 24(5):442–454. https://doi.org/10.1016/j.jmoldx.2022.01.006

    Article  Google Scholar 

  41. Khan M, Becker T, Kuppuudaiyar P, Elster AC (2018) Container-based virtualization for heterogeneous HPC clouds: insights from the EU h2020 CloudLightning project. In: 2018 IEEE International Conference on Cloud Engineering (IC2E). IEEE, Orlando, FL, USA, pp 392–397. https://doi.org/10.1109/ic2e.2018.00074

  42. Kovacs A (2017) Comparison of different Linux containers. In: 40th International Conference on Telecommunications and Signal Processing (TSP). IEEE, Barcelona, Spain, pp 47–51. https://doi.org/10.1109/tsp.2017.8075934

  43. Kuity A, Peddoju SK (2017) Performance evaluation of container-based high performance computing ecosystem using OpenPOWER. In: International Conference on High Performance Computing. LNCS—ISC-HPC. Springer, Cham, pp 290–308. https://doi.org/10.1007/978-3-319-67630-2_22

  44. Kumar M, Kaur G (2022) Containerized MPI application on Infiniband based HPC: an empirical study. In: 3rd International Conference for Emerging Technology (INCET), pp 1–6. https://doi.org/10.1109/INCET54531.2022.9824366

  45. Kumar Abhishek M (2020) High performance computing using containers in cloud. Int J Adv Trends Comput Sci Eng 9(4):5686–5690. https://doi.org/10.30534/ijatcse/2020/220942020

    Article  Google Scholar 

  46. Kurtzer GM, Sochat V, Bauer MW (2017) Singularity: scientific containers for mobility of compute. PLoS One 12(5):1–20. https://doi.org/10.1371/journal.pone.0177459

    Article  Google Scholar 

  47. Lahiff A, de Witt S, Caballer M, La Rocca G, Pamela S, Coster D (2020) Running HTC and HPC applications opportunistically across private, academic and public clouds. EPJ Web Conf 245:07032. https://doi.org/10.1051/epjconf/202024507032

    Article  Google Scholar 

  48. Le E, Paz D (2017) Performance analysis of applications using singularity container on SDSC Comet. In: Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact (PEARC). ACM, New York, NY, USA. https://doi.org/10.1145/3093338.3106737

  49. Lee M, Ahn H, Hong C-H, Nikolopoulos DS (2022) gShare: a centralized GPU memory management framework to enable GPU memory sharing for containers. Future Gener Comput Sys 130:181–192. https://doi.org/10.1016/j.future.2021.12.016

    Article  Google Scholar 

  50. Lim SB, Woo J, Li G (2020) Performance analysis of container-based networking solutions for high-performance computing cloud. Int J Electr Comput Eng 10(2):1507. https://doi.org/10.11591/ijece.v10i2.pp1507-1514

    Article  Google Scholar 

  51. Liu P, Guitart J (2020) Performance comparison of multi-container deployment schemes for HPC workloads: an empirical study. J Supercomput 77(6):6273–6312. https://doi.org/10.1007/s11227-020-03518-1

    Article  Google Scholar 

  52. Liu P, Guitart J (2022) Performance characterization of containerization for HPC workloads on Infiniband clusters: an empirical study. Clust Comput 25(2):847–868. https://doi.org/10.1007/s10586-021-03460-8

    Article  Google Scholar 

  53. Ma H, Wang L, Tak BC, Wang L, Tang C (2016) Auto-tuning performance of MPI parallel programs using resource management in container-based virtual cloud. In: IEEE 9th International Conference on Cloud Computing (CLOUD). IEEE, San Francisco, CA, USA, pp 545–552. https://doi.org/10.1109/cloud.2016.0078

  54. Maliszewski AM, Vogel A, Griebler D, Roloff E, Fernandes LG, Navaux Philippe OA (2019) Minimizing communication overheads in container-based clouds for HPC applications. In: IEEE Symposium on Computers and Communications (ISCC). IEEE, Barcelona, Spain, pp 1–6. https://doi.org/10.1109/iscc47284.2019.8969716

  55. Maliszewski AM, Roloff E, Griebler D, Gaspary LP, Navaux POA (2020) Performance impact of IEEE 802.3ad in container-based clouds for HPC applications. In: International Conference on Computational Science and Its Applications (ICCSA). Springer, Cham, pp 158–167. https://doi.org/10.1007/978-3-030-58817-5_13

  56. Manalo K, Baber L, Bradley R, You Z-Q, Zhang N (2019) Community collections. In: Proceedings of Practice and Experience in Advanced Research Computing (PEARC) Rise of the Machines (Learning). ACM, New York, NY, USA. https://doi.org/10.1145/3332186.3332199

  57. Martinasso M, Gila M, Sawyer W, Sarmiento R, Peretti-Pezzi G, Karakasis V (2019) Cray programming environments within containers on cray XC systems. Concurr Comput 32(20):5543. https://doi.org/10.1002/cpe.5543.e5543cpe.5543

    Article  Google Scholar 

  58. Medrano-Jaimes F, Lozano-Rizk JE, Castañeda-Avila S, Rivera-Rodriguez R (2019) Use of containers for high-performance computing. In: International Conference on Supercomputing in Mexico. CCIS—ISUM. Springer, Cham, pp 24–32. https://doi.org/10.1007/978-3-030-10448-1_3

  59. Merkel D (2014) Docker: lightweight Linux containers for consistent development and deployment. https://www.linuxjournal.com/content/docker-lightweight-linux-containers-consistent-development-and-deployment. Accessed 13 Sept 2022

  60. Michel M, Burnett N (2019) Enabling GPU-enhanced computer vision and machine learning research using containers. In: International Conference on High Performance Computing. LNCS—ISC-HPC. Springer, Cham, pp 80–87. https://doi.org/10.1007/978-3-030-34356-9_8

  61. Muhtaroglu N, Ari I, Kolcu B (2018) Democratization of HPC cloud services with automated parallel solvers and application containers. Concurr Comput 30(21):4782. https://doi.org/10.1002/cpe.4782

    Article  Google Scholar 

  62. Muscianisi G, Fiameni G, Azab A (2019) Singularity GPU containers execution on HPC cluster. In: Weiland M, Juckeland G, Alam S, Jagode H (eds) International conference on high performance computing. LNCS—ISC-HPC. Springer, Cham, pp 61–68. https://doi.org/10.1007/978-3-030-34356-9_6

  63. Newlin M, Smathers K, DeYoung ME (2019) ARC containers for AI workloads. In: Proceedings of Practice and Experience in Advanced Research Computing (PEARC) Rise of the Machines (Learning). ACM, New York, NY, USA. https://doi.org/10.1145/3332186.3333048

  64. Nguyen N, Bein D (2017) Distributed MPI cluster with docker swarm mode. In: IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC). IEEE, Las Vegas, NV, USA, pp 1–7. https://doi.org/10.1109/ccwc.2017.7868429

  65. Okuno S, Hirai A, Fukumoto N (2022) Performance analysis of multi-containerized MD simulations for low-level resource allocation. In: IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp 1014–1017. https://doi.org/10.1109/IPDPSW55747.2022.00162

  66. Pahl C, Brogi A, Soldani J, Jamshidi P (2019) Cloud container technologies: a state-of-the-art review. IEEE Trans Cloud Comput 7(3):677–692. https://doi.org/10.1109/tcc.2017.2702586

    Article  Google Scholar 

  67. Peiro Conde K (2020) Containers in HPC: is it worth it? Master’s thesis. http://hdl.handle.net/2117/335295. Accessed 13 Sept 2022

  68. Piras ME, Pireddu L, Moro M, Zanetti G (2019) Container orchestration on HPC clusters. In: International Conference on High Performance Computing. LNCS—ISC-HPC. Springer, Cham, pp 25–35. https://doi.org/10.1007/978-3-030-34356-9_3

  69. Priedhorsky R, Randles T (2017) Charliecloud. In: Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’17. ACM, New York, NY, USA. https://doi.org/10.1145/3126908.3126925

  70. Priedhorsky R, Canon RS, Randles T, Younge AJ (2021) Minimizing privilege for building HPC containers. In: Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis, SC, pp 1–14 https://doi.org/10.1145/3458817.3476187

  71. Ramon-Cortes C, Serven A, Ejarque J, Lezzi D, Badia RM (2018) Transparent orchestration of task-based parallel applications in containers platforms. J Grid Comput 16(1):137–160. https://doi.org/10.1007/s10723-017-9425-z

    Article  Google Scholar 

  72. Rudyy O, Garcia-Gasulla M, Mantovani F, Santiago A, Sirvent R, Vazquez M (2019) Containers in HPC: a scalability and portability study in production biological simulations. In: IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, Rio de Janeiro, RJ, Brazil, pp. 567–577. https://doi.org/10.1109/ipdps.2019.00066

  73. Ruhela A, Vaughn M, Harrell SL, Zynda GJ, Fonner J, Evans RT, Minyard T (2020) Containerization on petascale HPC clusters. In: Supercomp State Pract Arch. SC—SOTP. https://sc20.supercomputing.org/proceedings/sotp/sotp_files/sotp120s2-file1.pdf

  74. Ruhela A, Harrell SL, Evans RT, Zynda GJ, Fonner J, Vaughn M, Minyard T, Cazes J (2021) Characterizing containerized HPC applications performance at petascale on CPU and GPU architectures. In: International Conference on High Performance Computing. LNCS—ISC-HPC. Springer, Cham, pp 411–430. https://doi.org/10.1007/978-3-030-78713-4_22

  75. Saha P, Beltre A, Uminski P, Govindaraju M (2018) Evaluation of docker containers for scientific workloads in the Cloud. In: Proceedings of the Practice and Experience on Advanced Research Computing. ACM, New York, NY, USA. https://doi.org/10.1145/3219104.3229280

  76. Sampedro Z, Holt A, Hauser T (2018) Continuous integration and delivery for HPC. In: Proceedings of the Practice and Experience on Advanced Research Computing (PEARC). ACM, New York, NY, USA. https://doi.org/10.1145/3219104.3219147

  77. Sande Veiga V, Simon M, Azab A, Fernandez C, Muscianisi G, Fiameni G, Marocchi S (2019) Evaluation and benchmarking of Singularity MPI containers on EU research e-infrastructure. In: IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, Denver, CO, USA, pp 1–10. https://doi.org/10.1109/canopie-hpc49598.2019.00006

  78. Simchev T, Atanassov E (2020) Performance effects of running container-based Open-MPI cluster in public cloud. In: International Conference on Large-Scale Scientific Computing (LSSC). Springer, Cham, pp 254–262. https://doi.org/10.1007/978-3-030-41032-2_29

  79. Sochat VV, Prybol CJ, Kurtzer GM (2017) Enhancing reproducibility in scientific computing: metrics and registry for Singularity containers. PLoS One 12(11):0188511. https://doi.org/10.1371/journal.pone.0188511

    Article  Google Scholar 

  80. Sparks J (2017) HPC containers in use. In: Proceedings of the Cray User Group (CUG). https://cug.org/proceedings/cug2017_proceedings/includes/files/pap164s2-file1.pdf. Accessed 13 Sept 2022

  81. Sparks J (2018) Enabling docker for HPC. Concurr Comput 31(16):5018. https://doi.org/10.1002/cpe.5018

    Article  Google Scholar 

  82. Steffenel LA, Charão AS, Alves B, de Araujo LR, da Silva LF (2020) MPI to Go: container clusters for MPI applications. In: International Conference on Cloud Computing and Services Science. Springer, Cham, pp 199–222. https://doi.org/10.1007/978-3-030-49432-2_10

  83. Tippit J, Hodson DD, Grimaila MR (2021) Julia and singularity for high performance computing. In: Advances in Parallel and Distributed Processing, and Applications, pp 3–15. Springer, Cham. https://doi.org/10.1007/978-3-030-69984-0_1

  84. Torrez A, Randles T, Priedhorsky R (2019) HPC container runtimes have minimal or no performance impact. In: IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, Denver, CO, USA, pp 37–42. https://doi.org/10.1109/canopie-hpc49598.2019.00010

  85. Tronge J, Chen J, Grubel P, Randles T, Davis R, Wofford Q, Anaya S, Guan Q (2021) BeeSwarm: enabling parallel scaling performance measurement in continuous integration for HPC applications. In: 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 1136–1140. https://doi.org/10.1109/ASE51524.2021.9678805

  86. Vaillancourt PZ, Coulter JE, Knepper R, Barker B (2020) Self-scaling clusters and reproducible containers to enable scientific computing. In: IEEE High Performance Extreme Computing Conference (HPEC). IEEE, Boston, MA, USA, pp 1–8. https://doi.org/10.1109/hpec43674.2020.9286208

  87. Vaillancourt P, Wineholt B, Barker B, Deliyannis P, Zheng J, Suresh A, Brazier A, Knepper R, Wolski R (2020) Reproducible and portable workflows for scientific computing and HPC in the cloud. In: Practice and Experience in Advanced Research Computing (PEARC). ACM, Portland, OR, USA, pp 311–320. https://doi.org/10.1145/3311790.3396659

  88. Vallee G, Gutierrez CEA, Clerget C (2019) On-node resource manager for containerized HPC workloads. In: IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, Denver, CO, USA, pp 43–48. https://doi.org/10.1109/canopie-hpc49598.2019.00011

  89. Wang Y, Evans RT, Huang L (2019) Performant container support for HPC applications. In: Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning) (PEARC). ACM, Chicago, IL, USA. https://doi.org/10.1145/3332186.3332226

  90. Weidner O, Atkinson M, Barker A, Filgueira Vicente R (2016) Rethinking high performance computing platforms. In: ACM International Workshop on Data-Intensive Distributed Computing. ACM, Kyoto, Japan, pp 19–26. https://doi.org/10.1145/2912152.2912155

  91. Yang X, Kasahara M (2022) LPMX: a pure rootless composable container system. BMC Bioinf 23(1):1–13. https://doi.org/10.1186/s12859-022-04649-3

    Article  Google Scholar 

  92. Youn C, Das AK, Yang S, Kim J (2019) Developing a meta framework for key-value memory networks on HPC clusters. In: Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning) (PEARC). ACM, New York, NY, USA. https://doi.org/10.1145/3332186.3332216

  93. Younge AJ, Pedretti K, Grant RE, Brightwell R (2017) A tale of two systems: Using containers to deploy HPC applications on supercomputers and clouds. In: IEEE International Conference on Cloud Computing Technology and Science (CloudCom). IEEE, Hong Kong, pp 74–81. https://doi.org/10.1109/cloudcom.2017.40

  94. Yuan S, Brandt SR, Chen Q, Zhu L, Salatin R, Dooley R (2020) A sustainable collaboratory for coastal resilience research. Future Gener Comput Syst 111:786–792. https://doi.org/10.1016/j.future.2019.11.002

    Article  Google Scholar 

  95. Zhang J, Lu X, Panda DK (2016) High performance MPI library for container-based HPC cloud on InfiniBand clusters. In: 45th international conference on parallel processing (ICPP). IEEE, Philadelphia, PA, USA, pp 268–277. https://doi.org/10.1109/icpp.2016.38

  96. Zhang J, Lu X, Panda DK (2016) Performance characterization of hypervisor-and container-based virtualization for HPC on SR-IOV enabled InfiniBand clusters. In: IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, Chicago, IL, USA, pp 1777–1784. https://doi.org/10.1109/ipdpsw.2016.178

  97. Zhang J, Lu X, Panda DK (2017) Is singularity-based container technology ready for running MPI applications on HPC clouds? In: Proceedings of the10th International Conference on Utility and Cloud Computing (UCC ’17). ACM, New York, NY, USA, pp 151–160. https://doi.org/10.1145/3147213.3147231

  98. Zhou N, Georgiou Y, Pospieszny M, Zhong L, Zhou H, Niethammer C, Pejak B, Marko O, Hoppe D (2021) Container orchestration on HPC systems through Kubernetes. J Cloud Comput 10(1):1–14. https://doi.org/10.1186/s13677-021-00231-z

    Article  Google Scholar 

Download references

Funding

This work received partial financial support from Petrobras, FAPESP (Grants 2013/08293-7 and 2019/12792-5), and CNPq (Grant 314645/2020-9).

Author information

Author notes

  1. Edson Borin contributed equally to this work.

Authors and Affiliations

  1. Institute of Computing, University of Campinas, Av. Albert Einstein, 1251, Campinas, SP, 13083-852, Brazil

    Rafael Keller Tesser & Edson Borin

  2. Center for Computing in Engineering and Sciences, University of Campinas, R. Josué Castro, s/n, Campinas, SP, 13083-861, Brazil

    Rafael Keller Tesser

Authors
  1. Rafael Keller Tesser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edson Borin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rafael Keller Tesser.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial conflicts of interest to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Useful and commonly repeated acronyms and abbreviations: ABI: Application Binary Interface; AI: Artificial Intelligence; AMG: Algebraic Multigrid method; API: Application Programming Interface; AUFS: Docker’s Union File System; AWS: Amazon Web Services; BEE: Build and Execution Environment; CCE: Cray Computing Environment; CDT: Cray Development Toolkit; CFD: Computational Fluid-Dynamics; CFQ: Completely Fair Queuing; CI: Continuous Integration; CI/CD: Continuous Integration and Continuous Delivery; CLE: Cray Linux Environment; CLI: Command-Line Interface; CMA: Cross Memory Attach; CPE: Cray Processing Environment; CPU: Central Processing Unit; CV: Computer Vision; DASH: a C++ template library for distributed data structures; DFE: Data-Flow Engine; EC2: [Amazon] Elastic Compute Cloud; ELF: Executable and Linkable Format; FPGA: Field-Programmable Gate Array; GID: Group Identification; GPU: Graphics Processing Unit; HPC: High Performance Computing; HPCC: High Performance Computing Challenge; HPL: High Performance LINPACK; HTC: High Throughput Computing; IB: InfiniBand; IMB: Intel MPI Benchmarks; IO: Input and Output; IOR: a parallel IO benchmark; IP: Internet Protocol; IPC: Inter-Process Communication; IPoIB: IP over InfiniBand; IT: Information Technology; KMI: K-mer Matching Interface; KNL: [Intel] Knights Landing; KVM: Kernel Virtual Machine; LXC: Linux Containers; LXD: An interface for LXC; MCDRAM: Multi-Channel Dynamic Random Access Memory; MD: Molecular Dynamics; ML: Machine Learning; MPI: Message Passing Interface; NAS: NASA Advanced Supercomputing Division; NFS: Network File System; NIC: Network Interface Controller; NPB: NAS Parallel Benchmark; NUMA: Non-Uniform Memory Access; OCI: Open Container Initiative; OS: Operating System; OSU: Ohio State University; PCI: Peripheral Component Interconnect; PFS: Parallel File System; PID: Process Identification; RAM: Random-Access Memory; RDMA: Remote Direct Memory Access; RNN: Recursive Neural Network; ROMS: Regional Ocean Modeling System; SDN: Software-Defined Network; SHM: Shared Memory; SMT: Simultaneous Multi-Threading; SR-IOV: Single Root IO Virtualization; SSH: Secure Shell; TCP: Transmission Control Protocol; TF: TensorFlow; UDI: User-Defined Image; UDSS: User-Defined Software Stack; UID: User Identification; VCC: Virtual Container Cluster; VM: Virtual Machine; VXLAN: Virtual Extensible Local Area Network; WRF: Weather Research and Forecasting [model].

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Keller Tesser, R., Borin, E. Containers in HPC: a survey. J Supercomput 79, 5759–5827 (2023). https://doi.org/10.1007/s11227-022-04848-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04848-y

Keywords

Search

Navigation