Due to limitations of volume and installation conditions, the sensitive points of accelerometers in high-precision fiber optic gyroscope inertial measurement unit (FOG-IMU) cannot be completely coincide with the center of IMU. This deviation may cause tangential and centripetal acceleration errors when the IMU is rotated. Therefore, the inner lever-arm parameters need to be accurately identified and compensated for high-precision FOG-IMU applications. Firstly, the measurement model of accelerometer inner lever-arm parameters is established in this paper. A 21-dimensional Kalman filter for calibrating the inner lever-arm parameters is proposed. Then, an effective sinusoidal swing scheme is designed to calibrate the parameters of 9-dimensional inner lever-arm parameters. Compared with traditional calibration methods, the proposed method can obtain higher accuracy parameters through theoretical analysis and simulation. Afterwards, the feasibility and accuracy of the proposed method are verified by experiments. Finally, the velocity errors of the actual FOG-IMU system are significantly reduced after compensation, improving its navigation performance.