We propose a novel representation for stereo videos namely 2D-plus-depth-cue. This representation is able to encode stereo videos compactly by leveraging the by-product of a stereo video conversion process. Specifically, the depth cues are derived from an interactive labeling process during 2D-to-stereo video conversion—they are contour points of image regions and their corresponding depth models, and so forth. Using such cues and the image features of 2D video frames, the scene depth can be reliably recovered. Experimental results demonstrate that the bit rate can be saved about 10%–50% in coding a stereo video compared with multiview video coding and the 2D-plus-depth methods. In addition, since the objects are segmented in the conversion process, it is convenient to adopt the region-of-interest (ROI) coding in the proposed stereo video coding system. Experimental results show that using ROI coding, the bit rate is reduced by 30%–40% or the video quality is increased by 1.5–4 dB with the fixed bit rate.