Without any prior structure information, nuclear norm minimization (NNM), a convex relaxation for rank minimization (RM), is a widespread tool for matrix completion and relevant low-rank approximation problems. Nevertheless, the result derivated by NNM generally deviates the solution we desired, because NNM ignores the difference between different singular values. In this paper, we present a non-convex regularizer and utilize it to construct two matrix completion models. In order to solve the constructed models efficiently, we develop an efficient optimization method with convergence guarantee, which can achieve faster convergence speed compared to conventional approaches. Particularly, we show that the proposed regularizer as well as the optimization method are suitable for other RM problems, such as subspace clustering based on low-rank representation. Extensive experimental results on real images demonstrate that the constructed models provide significant advantages over several state-of-the-art matrix completion algorithms. In addition, we implement numerous experiments to investigate the convergence speed of the developed optimization method.