Adaptability is one of the most important survival and living abilities in nature and human society, which needs to perceive the environment and regulate self-behaviors. To empower robots in manufacturing with this ability, an observer-based variable impedance control scheme is proposed in this article. It senses changes in environmental properties during contact and utilizes the observations to guide the decisions of the robot controller. Specifically, we developed a real-time moving horizon based robot-environment observer, which uses historical information to predict the current impedance parameters of the external contact environment and robot internal joint disturbances simultaneously. Then, a variable impedance control law is designed to eliminate state tracking errors and decrease machining vibration. The stability of the closed-loop system is theoretically proven. Experiments on machining thin-walled workpieces validate the efficacy of the proposed method, where the robot exhibits compliant force behavior resulting in superior workpiece surface roughness among various comparison methods.