3D reconstruction has been widely used in many important applications. While extensive research has been done in 3D reconstruction, several key issues are still open and the precision of the recovered regions is still far from satisfaction. In this paper, we propose a novel approach to selecting regions of interest in video frames by analyzing multiple spatio-temporal characteristics and reconstructing 3D objects based on the selected regions. Firstly, the static, location and motion attention are extracted from video frames to generate saliency maps. Then, all the video frames are clustered and a candidate set of key frames is extracted based on the saliency maps, where the key frames are extracted according to the constraints in terms of geometry and visibility. Finally, the 3D structure of the attention region is recovered using the selected key frames and the generated saliency maps. The experiments on real-world indoor and outdoor scenes demonstrate that the proposed approach is both more accurate (better attention regions) and computationally more efficient.