skip to main content
10.1145/3387902.3392616acmconferencesArticle/Chapter ViewAbstractPublication PagescfConference Proceedingsconference-collections
research-article

Scale-out beam longitudinal dynamics simulations

Published: 23 May 2020 Publication History

Abstract

Excessive studies and simulations are required to plan for the upcoming upgrades of the world's largest particle accelerators, and the design of future machines, given the technological challenges and tight budgetary constraints. The Beam Longitudinal Dynamics (BLonD) simulator suite incorporates the most detailed and complex physics phenomena in the field of longitudinal beam dynamics, required for providing extremely accurate predictions. These predictions are invaluable to the operation of existing accelerators, upcoming upgrades, and future studies. To undertake this agenda, and enable for the first time scale-out beam longitudinal dynamics simulations, we implement Hybrid-BLond, a distributed version of BLonD, that efficiently combines horizontal and vertical scaling. We propose a series of techniques that minimize the inter-node communication overhead and improve scalability. Firstly, we exploit mixed data and task parallelism opportunities. Secondly, we discuss two traffic optimisation techniques motivated by the properties of the simulated physics phenomena. Finally, we build a dynamic load-balancing scheme that coordinates effectively all the above features. We evaluate experimentally Hybrid-BLonD in an HPC cluster built with cutting-edge Intel servers and Infiniband interconnection network. Our fully-optimised implementation demonstrates an average 25.7X speedup over the previous state-of-the-art simulator when run on 32 computing nodes, across three real-world testcases.

References

[1]
Georges Aad, JM Butterworth, J Thion, U Bratzler, PN Ratoff, RB Nickerson, JM Seixas, I Grabowska-Bold, F Meisel, S Lokwitz, et al. 2008. The ATLAS experiment at the CERN large hadron collider. Jinst 3 (2008), S08003.
[2]
Simon Albright, Konstantinos Iliakis, Alexandre Lasheen, Danilo Quartullo, Joel Repond, and Helga Timko. 2014. CERN Beam Longitudinal Dynamics code BLonD. https://blond.web.cern.ch/
[3]
Gene M Amdahl. 1967. Validity of the single processor approach to achieving large scale computing capabilities. In Proceedings of the April 18-20, 1967, spring joint computer conference. ACM, 483--z485.
[4]
A. Ball, M. Benedikt, L. Bottura, O. Dominguez, F. Gianotti, B. Goddard, P. Lebrun, M. Mangano, D. Schulte, E. Shaposhnikova, R. Tomas, and F. Zimmermann. 2014. The Future Circular Collider study. CERN Courier 54, 3 (Apr 2014), 16--18. http://cds.cern.ch/record/2064538
[5]
Charles K. Birdsall and A. Bruce Langdon. 2005. Plasma physics via computer simulation. Taylor and Francis, New York.
[6]
Sergey Blagodurov, Alexandra Fedorova, Sergey Zhuravlev, and Ali Kamali. 2010. A case for NUMA-aware contention management on multicore systems. In 2010 19th International Conference on Parallel Architectures and Compilation Techniques (PACT). IEEE, 557--558.
[7]
Michael Borland. 2000. Elegant: A flexible SDDS-compliant code for accelerator simulation. Technical Report. Argonne National Lab., IL (US).
[8]
CMS Collaboration et al. 2008. The CMS experiment at the CERN LHC.
[9]
H Damerau, A Funken, R Garoby, S Gilardoni, B Goddard, K Hanke, A Lombardi, D Manglunki, M Meddahi, B Mikulec, G Rumolo, E Shaposhnikova, M Vretenar, and J Coupard. 2014. LHC Injectors Upgrade, Technical Design Report, Vol. I: Protons. Technical Report CERN-ACC-2014-0337. CERN. http://cds.cern.ch/record/1976692
[10]
Vincenzo Forte, Elena Benedetto, Alessandra Lombardi, and Danilo Quartullo. 2015. Longitudinal Injection Schemes For the CERN PS Booster at 160 MeV Including Space Charge Effects. In Proceedings, 6th International Particle Accelerator Conference (IPAC 2015): Richmond, Virginia, USA, May 3-8, 2015. MOPJE042.
[11]
Matteo Frigo and Steven G Johnson. 1998. FFTW: An adaptive software architecture for the FFT. In Acoustics, Speech and Signal Processing, 1998. Proceedings of the 1998 IEEE International Conference on, Vol. 3. IEEE, 1381--1384.
[12]
Apollinari G., Bejar Alonso I., Bruning O., Fessia P., Lamont M., Rossi L., and Tavian L. 2017. High-Luminosity Large Hadron Collider (HL-LHC): Technical Design Report V. 0.1. CERN, Geneva. http://cds.cern.ch/record/2284929
[13]
Edgar Gabriel, Graham E Fagg, George Bosilca, Thara Angskun, Jack J Dongarra, Jeffrey M Squyres, Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, Andrew Lumsdaine, et al. 2004. Open MPI: Goals, concept, and design of a next generation MPI implementation. In European Parallel Virtual Machine/Message Passing Interface Users' Group Meeting. Springer, 97--104.
[14]
William Gropp. 2002. MPICH2: A new start for MPI implementations. In European Parallel Virtual Machine/Message Passing Interface Users' Group Meeting. Springer, 7--7.
[15]
Wei Huang, Gopalakrishnan Santhanaraman, H-W Jin, Qi Gao, and Dhabaleswar K Panda. 2006. Design of high performance MVAPICH2: MPI2 over InfiniBand. In Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID'06), Vol. 1. IEEE, 43--48.
[16]
Christopher J Hughes. 2015. Single-instruction multiple-data execution. Synthesis Lectures on Computer Architecture 10, 1 (2015), 1--121.
[17]
Konstantinos Iliakis, Helga Timko, Sotirios Xydis, and Dimitrios Soudris. 2018. BLonD++: performance analysis and optimizations for enabling complex, accurate and fast beam dynamics studies. In Proceedings of the 18th International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation, Pythagorion, Greece, July 15-19, 2018. ACM, 123--130.
[18]
intelmkl 2018. Intel Math Kernel Library. https://software.intel.com/en-us/mkl
[19]
Christoph Lameter et al. 2013. NUMA (Non-Uniform Memory Access): An Overview. Acm queue 11, 7 (2013), 40.
[20]
Alexandre Lasheen, Edgaras Radvilas, Elena Shaposhnikova, Toon Roggen, Thomas Bohl, and Steven Hancock. 2016. Single bunch longitudinal instability in the CERN SPS. (2016).
[21]
Yinan Li, Ippokratis Pandis, Rene Mueller, Vijayshankar Raman, and Guy M Lohman. 2013. NUMA-aware algorithms: the case of data shuffling. In CIDR.
[22]
JA MacLachlan. 1984. ESME: Longitudinal Phase Space Particle Tracking-Program Documentation. Technical Report TM-1274. Fermi National Accelerator Lab., Batavia, IL (United States).
[23]
JA MacLachlan. 1992. Particle tracking in E-φ space for synchrotron design and diagnosis. Technical Report. Fermi National Accelerator Lab., Batavia, IL (United States).
[24]
E Métral, T Argyropoulos, H Bartosik, N Biancacci, Xavier Buffat, JF Esteban Muller, W Herr, G Iadarola, A Lasheen, K Li, et al. 2016. Beam instabilities in hadron synchrotrons. IEEE Transactions on Nuclear Science 63, 2 (2016), 1001--1050.
[25]
Ankireddy Nalamalpu, Nasser Kurd, Anant Deval, Chris Mozak, Jonathan Douglas, Ashish Khanna, Fabrice Paillet, Gerhard Schrom, and Boyd Phelps. 2015. Broadwell: A family of IA 14nm processors. In 2015 Symposium on VLSI Circuits (VLSI Circuits). IEEE, C314--C315.
[26]
Gregory F Pfister. 2001. An introduction to the infiniband architecture. High Performance Mass Storage and Parallel I/O 42 (2001), 617--632.
[27]
Danilo Piparo, Vincenzo Innocente, and Thomas Hauth. 2014. Speeding up HEP experiment software with a library of fast and auto-vectorisable mathematical functions. Journal of Physics: Conference Series (jun 2014).
[28]
Danilo Quartullo, Simon Albright, Elena Shaposhnikova, et al. 2017. Studies of Longitudinal Beam Stability in CERN PS Booster After Upgrade. In 8th Int. Particle Accelerator Conf. (IPAC'17), Copenhagen, Denmark, 14â 19 May, 2017. JACOW, Geneva, Switzerland, 4469--4472.
[29]
Danilo Quartullo, Elena Shaposhnikova, and Helga Timko. 2017. Controlled longitudinal emittance blow-up using band-limited phase noise in CERN PSB. In Journal of Physics: Conference Series, Vol. 874. IOP Publishing, 012066.
[30]
Rolf Rabenseifner, Georg Hager, and Gabriele Jost. 2009. Hybrid MPI/OpenMP parallel programming on clusters of multi-core SMP nodes. In 2009 17th Euromicro international conference on parallel, distributed and network-based processing. IEEE, 427--436.
[31]
Joël Repond, Konstantinos Iliakis, Markus Schwarz, and Elena Shaposhnikova. 2018. Simulations of Longitudinal Beam Stabilisation in the CERN SPS With BLonD. In Proceedings ICAP2018: Key West, FL, USA. TUPAF06.
[32]
Boris Schäling. 2011. The boost C++ libraries. Boris Schäling.
[33]
Markus Schwarz, Konstantinos Iliakis, Alexandre Lasheen, Giulia Papotti, Joël Repond, Elena Shaposhnikova, Helga Timko, et al. 2019. Flat-Bottom Instabilities in the CERN SPS. In 10th Int. Partile Accelerator Conf. (IPAC'19), Melbourne, Australia. JACOW, 3224--3227.
[34]
Elena Shaposhnikova, Joël Repond, Helga Timko, Theodoros Argyropoulos, Thomas Bohl, and Alexandre Lasheen. 2016. Identification and Reduction of the CERN SPS Impedance. In Proceedings of the 57th ICFA Advanced Beam Dynamics Workshop on High-Intensity, High Brightness and High Power Hadron Beams, HB2016.
[35]
Andrei Shishlo, S Cousineau, V Danilov, J Galambos, S Henderson, J Holmes, and M Plum. 2006. The ORBIT simulation code: benchmarking and applications. In Proceedings of ICAP.
[36]
Andrei Shishlo, Sarah Cousineau, Jeffrey Holmes, and Timofey Gorlov. 2015. The particle accelerator simulation code PyORBIT. Procedia Computer Science 51 (2015), 1272--1281.
[37]
H. Timko, S. Albright, T. Argyropoulos, K. Iliakis, I. Karpov, A. Lasheen, D. Quartullo, J. Repond, M. Schwarz, and J. Esteban Müller. 2019. Beam Longitudinal Dynamics Simulation Suite BLonD. Physical Review Accelerators and Beams (to be published) (2019).
[38]
Helga Timko, Danilo Quartullo, Alexandre Lasheen, and Juan Esteban Müller. 2016. Benchmarking the beam longitudinal dynamics code BLonD. In Proceedings of the 7th International Particle Accelerator Conference (IPAC 2016): Busan, Korea.
[39]
Helga Timko, Elena Shaposhnikova, Philippe Baudrenghien, and Themis Mastoridis. 2014. Studies on Controlled RF Noise for the LHC. In Proceedings of the 54th ICFA Advanced Beam Dynamics Workshop on High-Intensity, High Brightness and High Power Hadron Beams, East-Lansing, USA, 10 - 14 Nov 2014.
[40]
Yusong Wang and Michael Borland. 2006. PELEGANT: A parallel accelerator simulation code for electron generation and tracking. In AIP Conference Proceedings, Vol. 877. AIP, 241--247.
[41]
Ahmad Yasin. 2014. A top-down method for performance analysis and counters architecture. In Performance Analysis of Systems and Software (ISPASS), 2014 IEEE International Symposium on. IEEE, 35--44.

Cited By

View all
  • (2022)Enabling Large Scale Simulations for Particle AcceleratorsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.319270733:12(4425-4439)Online publication date: 1-Dec-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CF '20: Proceedings of the 17th ACM International Conference on Computing Frontiers
May 2020
298 pages
ISBN:9781450379564
DOI:10.1145/3387902
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 May 2020

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. accelerator physics
  2. beam dynamics
  3. distributed systems
  4. dynamic load balancing
  5. traffic optimisation

Qualifiers

  • Research-article

Conference

CF '20
Sponsor:
CF '20: Computing Frontiers Conference
May 11 - 13, 2020
Sicily, Catania, Italy

Acceptance Rates

Overall Acceptance Rate 273 of 785 submissions, 35%

Upcoming Conference

CF '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Enabling Large Scale Simulations for Particle AcceleratorsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.319270733:12(4425-4439)Online publication date: 1-Dec-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media