As one of the most threatening cyber-attacks, load redistribution (LR) attacks can stealthily misguide the operation of power system by exploiting the vulnerabilities of power information systems. Even though various defending methods have been studied, it is still challenging to mitigate the damaging impact of LR attacks. In this paper, we propose a phase shifting transformers (PST) based mitigation method utilizing the widely deployed PST devices in optimal power flow (OPF), which can further improve the performance of existing mitigation methods. Specifically, a tri-level optimization is formulated and solved by the column and constraint generation (C&CG) decomposition method in the PST-based mitigation method, which gives the optimal set of PSTs for phase angle adjustment. Since a large amount of binary variables are introduced in the C&CG method, an approximate solution method of the tri-level optimization is presented to reduce computing burden without significantly affecting the optimality in large power systems. Simulations in the PJM 5-bus, IEEE 14-bus, IEEE 30-bus and IEEE 118-bus test systems verify the effectiveness and applicability of the PST-based mitigation method.