In recent years, the research on motion control of quadruped robot based on deep reinforcement learning has become more and more popular. Relying on the generalization ability of DRL, the robot can make adaptive control policy when facing complex terrain. At present, the normal method uses the proprioceptive information such as joint torque/position and center of mass kinematic information to estimate the surrounding environment information. The quadruped robot usually needs to judge the next control policy by physical collision with the real terrain. This is not friendly for some scenarios that require stable operation of the robot such as inspection. This paper innovatively proposes an end-to-end reinforcement learning training scheme to improve the motion adaptation ability of quadruped robot in complex terrain by combining with multi-mode perception. Firstly, we combine the depth image information with the robot proprioceptive information to perceive the surrounding environment. Then, we design a centralized Critic network to improve the training effect of the reinforcement learning algorithm in the simulation environment. In the experiment, the quadruped robot is employed to perform the task of going up steps, and the robot can identify the steps through visual perception and make foot lifting action in advance. The results show that the policy based on multi-mode perception information can significantly reduce unnecessary collisions between the robot and the environment, and further improve the intelligence and stability of the robot.