A reliable protection system is vital to power system. As the major equipment of protection system, protection relay plays a basilica role in power system. So searching for proper settings of relays to make them operate in a better way is significant. In this paper, a new optimization problem formulation is proposed to search the optimal relay setting of over current relays in power systems. Then, a new hybrid evolutionary algorithm based on tabu search (HEATS) is presented to solve this optimization problem, and results under different algorithm parameters are obtained Finally, comparisons among HEATS, one of particle swarm optimizations(PSO) and test evolutionary algorithm(TEA) shown in other literatures are given. Simulation results show the formulation of protection relay setting is feasible and effective, and the proposed algorithm HEATS exhibits a good performance.