Sparse code multiple access (SCMA) is a non- orthogonal codebook (CB) based multiple access scheme, proposed to cope with the heterogeneous and challenging performance requirements for mission critical communication and massive machine-type communication (MTC) in the fifth Generation (5G) wireless system. In this paper, the performance of SCMA has been studied and analyzed, considering the impact of the energy diversity and minimum Euclidean distance of the mother constellation, overloading of the system and the layer specific operators for codebooks generation. An optimized codebooks generation method for four ring star QAM based signaling constellation is proposed. It is demonstrated that by selecting the optimum design parameters, the bit error rate (BER) can be improved. Moreover, an overloading technique is also proposed to enable higher connectivity at lower decoding complexity.