This paper focuses on the visual fatigue issue while viewing 3D contents. The issue is caused by the distance between the screen and the fused images. A stereo image depth optimization system with disparity map calculation, viewpoint optimization and stereo image synthesis is proposed to solve the issue with the following procedure: first, its disparity map calculation adopts the modified binary window block matching algorithm so that the complex and iterative computations can be accelerated by hardware implementation strategies including parallel color difference calculation, parallel memory banks, window shift, and pipelined architecture; second, the viewpoint optimization modifies disparities to the zone of comfort; third, stereo images are synthesized through Depth-Image-Based-Rendering (DIBR); finally, the stereo image depth optimization system is realized on the FPGA board and video files are shown via the HDMI interface. This hardware implementation turns out to be more cost-efficient to achieve high-speed performance when compared with previous works.