Controlling lower-limb exoskeletons is extremely challenging due to their direct physical interaction with users wearing them which imposes additional safety concerns. Furthermore, the control policy needs to adapt for different users and surfaces the robot is traversing. Hence, it is crucial to design a control framework that can perform robustly in the presence of these variations. In this letter, we propose a sample-efficient method based on Bayesian Optimization (BO) to adapt a model-based walking controller for a lower-limb exoskeleton, XoMotion. In order to mitigate safety risks, we use a set of dummy weights with realistic inertial distributions in the experiments with the robot to find optimal policies. An extensive set of experimental results shows that the proposed controller can successfully adapt for different users and different terrains, in less than 15 real-world trials.