This article investigates an encoder-based relative attitude observation method for self-calibration in a dual-axis rotational inertial navigation system (RINS). First, the calibration error of the gyroscope parameters caused by the velocity and position observations in systematic calibration is analyzed. Second, a relative attitude observation method utilizing photoelectric encoders for gyroscope parameter estimation is proposed. We apply the proposed method in our designed dual-axis RINS to verify its effectiveness in this article. Experimental results show that the calibration accuracy of the gyroscope can be improved to 0.0005$^ \circ \text{/h}$ in bias, and 0.98 ppm in scale factor using the proposed method. The proposed self-calibration method can be utilized as a high-accuracy self-calibration method in the dual-axis RINS when the vehicle is in static condition.