We tackle the multi-view image rectification problem by exploiting inter- and intra-disparity constraints. The interdisparity consistency constraint encourages uniform disparity values for all pairs of neighboring frames, e.g., a 3D point should give the same disparity value for all pairs of stereo images. In addition, the intra-disparity consistency constraint keeps the original shape of the image to avoid a degenerate situation. This is equivalent to transform cameras to equally spaced virtual cameras in which a number of multi-view stereo matching algorithms can solve the depth map estimation problem effectively.