Aiming at the problem that the ultrashort baseline (USBL) installation error is difficult to accurately calibrate in the complex underwater environment. In this article, based on smooth variable structure and unscented Kalman filter, a novel USBL installation error calibration algorithm is proposed. First, the nonlinear system model of USBL installation error based on the filter estimation principle is analyzed. On this basis, the USBL installation error calibration algorithm based on the principle of unscented Kalman filter is given. Second, in view of the shortcomings of the Unscented Kalman Filter (UKF) calibration algorithm, smooth variable structure is introduced, and a novel algorithm combining a smooth variable structure and UKF is proposed. Finally, the effectiveness of the algorithm proposed in this article is verified by simulation and river test. The results show that the algorithm proposed in this article can accurately and stably estimate the USBL installation error parameters when the underwater acoustic signal is disturbed.