Soft robots have intrinsic advantages in interaction with humans or complex environments for actual applications, during which various external disturbances (e.g., external contact or collision) are inevitable. They show remarkable abilities in complicated tasks due to their easily deformable bodies and compliance characteristic, while also bringing challenges to the modeling, control, and trajectory planning in precise tasks. Thereby, perception and reaction to external disturbances are quite critical. In this paper, we focus on the slowly varying external contact and propose a contact detection method for the fiber-reinforced soft bending actuator (FRSBA), which is based on the system dynamical behavior. When contact is detected, a parameter extension method is introduced to modify the dynamic model. Then, a backstepping-based integrated direct/indirect adaptive robust controller with contact detection and accommodation strategy (CDA-DIARC) is designed to deal with system nonlinearities, uncertainties, and parametric variations caused by the external contact. Theoretical proof and physical experiments validate the convergence and high trajectory tracking performance of the proposed methods under different contact environments.