We propose a fast and accurate calibration method for the optical see-through (OST) head-mounted displays (HMD), taking advantage of a low-cost time-of-flight depth-camera. Recently, affordable OST-HMDs and depth-cameras are widely appearing in the commercial market. In order to correctly reflect the user experience into the calibration process, our method demands a user wearing the HMD to repeatedly point at rendered virtual circles with their fingertips. From the repeated calibration data, we perform two stages of full calibration and simplified calibration, to compute key calibration parameters. The full calibration is required when the depth-camera is first installed to the HMD, and afterwards only the simplified calibration is performed whenever a user wears it again. Our experimental results show that the full and simplified calibration can be achieved with 10 and 5 user's repetitions (theoretically 3 and 2 at minimum), which are significantly less than about 20 of the stereo-SPAAM, one of the most popular existing calibration techniques. We also demonstrate that the 3D position errors of our calibration become much quickly smaller than those of the state-of-the-art method.