In visual tracking, many methods first sample a set of candidate states and then select the optimal state with the best evaluation value. In this way the tracking avoids trapping in local optimum. However, the obtained state is not accurate when the appearance suffers from large challenges or the sample number is small, while the prediction information provided by the surrounding candidates is useful to improve the robustness of state determination. Thus, in this paper we propose a new object localization method which infers the object state by state regression of surrounding states. By acquiring the state weights according to two constraints, i.e. the constraint of representing the confidence of single state and the constraint of approximate states having approximate weights, the sensitivity to appearance variation in state regression is reduced. Experimental results on a set of benchmark videos demonstrate the robustness of the proposed method.