Time of Flight (ToF) and stereo vision are the two major technologies for rapid depth information acquisition. The ToF sensor is able to obtain depth map directly at a high rate of speed, but the resolution and accuracy is relatively low. The binocular stereo vision means can obtain high-accuracy disparity map based on triangulation principle, but its performance depends on the textures of the target surface. Therefore it is an interesting research topic to combine above two techniques. In order to carry out the fusion of the two 3D techniques, an efficient calibration method is investigated in this work. Based on a simple calibration rig, point correspondences between the depth image and color image are extracted and then used to calculate both intrinsic and extrinsic parameters of two devices. The experimental result shows that the proposed method could achieve better accuracy in comparison with classical calibration methods.