In this paper, we propose an automatic approach to segment object from stereo videos, for which the viewpoints are widely apart. We first present a novel saliency analysis to emphasize the foreground object. The saliency map is estimated by combining the depth information recovered by feature matching and the boundary information revealed by color segmentation. The object mask is extracted initially based on the saliency map and then refined by graph-cut segmentation, where color and motion information are efficiently incorporated in both data and smoothness terms. Moreover, a background image is gradually reconstructed during video segmentation, based on which an additional constraint is imposed on the data term to further improve the video segmentation. The proposed method is tested on stereo videos with widely separated viewpoints and severe background clutters. Good experimental results demonstrate the feasibility of the proposed method.