Extreme learning machine (ELM) is one of suitable base-classifiers for ensemble learning systems because of its fast learning speed, good generalization performance and simple setting. For the ensemble learning, how to select the base classifiers is a key issue which influences the performance of the ensemble system dramatically. To obtain a compact ensemble system with improved generalization performance, a diversity guided ensemble of ELMs based on attractive and repulsive particle swarm optimization (ARPSO) is proposed in this paper. In the proposed method, ARPSO considers both the convergence accuracy on the validation data and the diversity of the ensemble system. To effectively weigh the diversity of the ensemble system, a new diversity based on the Euclidean distance among the candidate ELMs is defined in this study. Experimental results on function approximation and benchmark classification problems verify that the proposed method could build more compact ensemble of ELMs with better generalization performance than some classical ensemble of ELMs.