Top-N recommendation refers to mining a few specific items that are supposed to be most appealing to the user. While relevancy has been the prevailing issue of the recommendation problem for the last decades, diversity, which is associated with increasing user satisfaction with the presented recommendation lists and mitigating the overfitting problem, also plays a central role in the success of predictive models. Existing work applied determinantal point processes (DPP) to provide a favorable trade-off between relevance and diversity. However, the maximum a posteriori (MAP) inference for DPP is generally NP-hard. To attain an approximate solution with sufficient accuracy, popular approximation approaches such as forward and backward greedy algorithms are used. Despite their intuitive manner, they are not adequate and still be too computationally expensive to be used in large-scale domains. Thus, this paper aims to enhance forward greedy algorithms incorporating backward elimination algorithms and accelerate the greedy MAP inference for DPP by introducing the Cholesky decomposition and Givens rotation. Experimental results show that our proposed algorithm is faster than most competitors and ensures a substantial improvement over the accuracy-diversity trade-off on the Netflix Prize dataset.