When you look at the object, the displacement of object is different in left and right eye. This displacement is called parallax. The farther the object is, the smaller the parallax become. Due to the parallax, people will have the feeling of stereoscopic vision. We develop a depth-image-based rendering (DIBR) algorithm to shift (or move) the position of the object in the image to generate the stereoscopic image with parallax according to its depth information. When the object is shifted (or moved) away from its original position, a hole will occur in the original position. Therefore we develop an image inpainting algorithm to fill the hole. In this paper, a hardware architecture is designed for real-time stereoscopic image generation from a 2D image and its depth map. In order to achieve the real-time application, we design a hardware architecture to accomplish the DIBR and image inpainting algorithm. There are totally 2,849 logic registers in this design. By simulation in the experiment, we can achieve the real-time computing in 150MHz clock (320 × 240 @30 fps).