Primates demonstrate an outstanding ability of gaining a continuous tracking of target in cluttered environments after target disappears in the scene for a long time while it is still a challenge for artificial visual system to do so. Research in psychology indicates that selective visual attention with two attention selection processes is crucial to visual tracking. This paper presents a novel visual attention shift tracking (VAST) algorithm to solve the difficult problems mentioned above by treating tracking as a kind of shifting visual attention to the target in consequent frames. In VAST, the early attentional selection process extracts a pool of salient objects or regions that have good localization properties from a salient map. Then, by the learned knowledge from historical data on the fly, the late attentional selection process generates a sequence of shifting between those objects and implements a detection of the target in them one by one. Experiments under various conditions show that this algorithm is general, robust and can gain better tracking results as compared to the existing tracking algorithms.