Gaussian mixture model (GMM) is always used to estimate the underlying density function in many real applications. In this paper, we develop an improved Gaussian mixture model (iGMM) based on least-squares cross-validation (LSCV) and Gaussian PSO with Gaussian jump (GPSO_GJ). According to least-squares cross-validation, a new error measure criterion is derived which is used to evaluate the estimation error between the true density function and the estimated density function. Then, GPSO_GJ is used to find the optimal parameters that can make the estimation error reach the minimum. In our experiments, we compare iGMM with two existing methods as GMM with Parzen window (PGMM) and GMM based on particle swarm optimization (PSOGMM) on four probability distributions: Uniform density, Normal density, Exponential density, and Rayleigh density. The experimental results demonstrate that our strategy can get good estimation performance when the corresponding parameters are optimized with GPSO_GJ.