Although many new methods that aim to improve the performance of link prediction have been proposed in recent years, there is still no widely accepted benchmark for evaluating and comparing these link prediction methods. In this paper, we propose LPBenchmark, a solution towards a fair and effective benchmark for link prediction. LPBenchmark offers a suite of well-selected datasets covering major research fields in link prediction without redundancy. These datasets are selected from widely adopted open access collections of datasets via performing AHC(Adapted Hierarchical Clustering) and DNFS(Deepest Node First Selection) Algorithm. LPBenchmark measures the difficulty of each selected dataset through OSR(Optimal Subset Regression) Algorithm, which makes it possible to fairly compare the experiment performance of two methods operated on different datasets. Moreover, LPBenchmark includes three APIs, allowing researchers to obtain the largest connected components of a dataset, modify a dataset based on node degree and construct subgraphs based on node clustering coefficients. After presenting all the characteristics and functionalities of LPBenchmark, we conduct a comprehensive evaluation on several classic and newly proposed link prediction methods by using LPBenchmark. Results show that LPBenchmark is not only capable of fairly comparing each method's overall performance, but also can reveal each method's advantages and limitations on different types of networks.