A common form of serious defect in video is pixel-shift. It is caused by the consecutive pixels loss introduced by video transmission systems. Pixel-shift means a large amount of pixel shifts one by one due to a small quantity of image data loss. The damaged region in affected frame is usually large, and thus the visual effect is often very disturbing. So far there is no method of automatically treating pixel-shift. This paper focuses on a difficult issue to locate pixel-shift in videos. We propose an original algorithm of automatically detecting and restoring pixel-shift. Pixel-shift frames detection relies on spatio-temporal information and motion estimation. Accurate measure of pixels shift is best achieved based on the analysis of temporal-frequency information. Restoration is accomplished by reversing the pixels shift and spatio-temporal interpolation. Experimental results show that our completely automatic algorithm can achieve very good performances.