Parallelization, Vectorization and Visualization of Large Scale Plasma Particle Simulations and Its Application to Studies of Intense Laser Interactions

Abstract

The use of a three-dimensional PIC (Particle-in-Cell) simulation is dispensable in the studies of nonlinear plasma physics, such as ultra-intense laser interactions with plasmas. The three-dimensional simulation requires a large number of particles more than 107 particles. It is therefore very important to develop a parallelization and vectorization scheme of the PIC code and a visualization method of huge simulation data. In this paper we present a new parallelization scheme suitable for a present day supercomputer and a construction method of scientific color animations to analyze simulation data. We also discuss the advantage of the Abe-Nishihara vectorization method for a large scale PIC simulations. Most of supercomputers in present day consists of multi nodes and each node has multi processors with a sheared memory. We have developed a new parallelization scheme in which domain decomposition is applied among nodes and particle decomposition is used for processors within a nodes. The domain decomposition in PIC requires the exchange of two kinds of data between neighboring domains. One is particle data, such as particle position and velocity, when a particle crosses the boundary between the neighboring domains. The other is field data, such as electric field intensity and current density in the boundary region. In the three-dimensional Electro-magnetic PIC forty two-dimensional variables are transferred to the neighboring domain for each boundary surface. MPI (Message Passive Interface) has been used for the transmission of these data between the nodes. The particle and field data are respectively stored once in one-dimensional data and they are then sent to the other node. This reduces the number of communication. The particle decomposition is performed by using auto-parallelization of do-loop. We measured the scalability of the layered parallelization scheme for the particle number of 25,600,000 and the mesh number of 128x128x128 with the use of sixteen processors of NEC SX-5. The layered parallelization is shown to provide a scalable acceleration of computation for the large system of PIC simulations. In the PIC code it is necessary to calculate the current and charge densities from particle position and velocity. For a example, the calculation of the charge density of a mesh requires the assignment of particle charge on a mesh from its position. Since more than two particles may locate in the same mesh, the calculation of the assignment can not be vectorized. However for a large system of PIC simulation the possibility that more than two particles within a vector register locate in the same mesh becomes very small. Therefore even if the vector operation is enforced for the calculation of the charge assignment, the correction of the calculation is very seldom. The particle’s number of which charge is overwritten by the forced vector operation can be easily found by checking the particle’s number stored. This is the Abe-Nishihara method. We obtained 99% of vectoraization even for list vector calculation in PIC by using Abe-Nishihara method. As a result 0.73 sec per cycle is achieved for the number of particles, 2.56x10⁷ and meshes with (128)³ by using sixteen processors of NEC SX5. Within our knowledge, this is the fastest speed achieved at present for this size of program. Most of advanced visualization software requires interactive operations that is not suitable for drawing many pictures from simulation data to make scientific animations. It is desirable for drawing pictures to be performed by a batch job. We have developed a program that during the computer simulation geometry objects are constructed at the same time by using AVS library. Digital video files are constructed from the geometry objects with the use of the script function of AVS animation encoder on SGI by a batch job. For modern high power lasers, an intensity of laser radiation can be of the order of 10¹⁹ - 10²² W/cm². In this ultra relativistic regime, when the quiver energy of an electron in the laser light becomes much greater than the electron rest mass, a novel nonlinear physics comes in our grasp. In this paper we present our resent studies on the ultra-intense laser interaction with overdense plasmas with the use of a three-dimensional PIC code, EMPAC-3D (Three-Dimensional Electro-Magnetic Particle Code). The three dimensional regime of the laser-plasma interaction reveals novel features of the laser light.