In this brief, robot-environment interaction control problem of n-link robotic manipulator system without force sensors is investigated. At first, a novel sliding mode (SM) based two-layer adaptive force observer is proposed to estimate the robot-environment interaction force. In comparison with the most existing force observers, the proposed force observer ensures arbitrary-time convergence of the observer errors and relaxes assumptions on the external force. Moreover, an admittance controller is developed to ensure exact trajectory tracking within arbitrary time. It is mathematically proved that both trajectory tracking errors and force estimation errors will converge to zero at any setting time. Finally, numerical simulations and comparisons show that the proposed force observer-based control scheme can provide superior interaction performance.