In this paper, a novel codebook design method for sparse code multiple access (SCMA) is proposed and an analytical framework to evaluate the bit error rate (BER) performance is developed. In particular, to meet the codebook design criteria based on the pairwise error probability, a novel codebook with large minimum Euclidean distance employing the star quadrature amplitude modulation signal constellations is designed. In addition, with the aid of the phase distribution of the presented SCMA constellations, BER of SCMA system over downlink Rayleigh fading channel is obtained in closed-form expression. The simulation and analytical results show that the presented SCMA codebook outperforms the existing codebooks and low-density signature, and the proposed design is more efficient for the SCMA codebook with large size and/or high dimension. Moreover, the derived theoretical BER results match well the simulation results, especially in the high signal-to-noise ratio regimes.