This letter presents a user-adaptive variable damping controller that enhances the overall performance of coupled human-robot systems in terms of stability, agility, user effort, and energy expenditure during physical human-robot interaction. The controller accounts for impedance properties of the human limbs and adaptively changes robotic damping from negative to positive values based on user's intent of motion while minimizing energy of the coupled human-robot system. Bayesian optimization is used to evaluate an unknown objective function and optimize noisy performance, which builds on a Gaussian process to account for the uncertainty of human behaviors and noisy observations. To validate the effectiveness of the presented approach and evaluate its potential applications in real-world scenarios, we performed human experiments using a common robotic arm manipulator. Experimental results from five pilot subjects demonstrated that the controller does not require a long parameter tuning process. Compared to variable damping control without user-adaptive parameter changes, the presented adaptive control strategy could reduce ∼45% energy expenditure and achieve average performance improvement of ∼20% when several performance metrics of stability, agility, and user effort are considered together.