In this letter, we present VI-HSO, a hybrid sparse monocular visual-inertial odometry system based on two innovative techniques called adaptive interframe alignment (AIA) and dynamic inverse distance filter (DIDF). Although the sparse image alignment algorithm appears efficient for calculating frame-to-frame motion, it tends to fail in case of significant intensity changes and motion blur. To overcome these limitations, we propose an adaptive interframe alignment method that allows for an adaptive selection between the original Lucas-Kanade (LK) method and the inverse compositional method when constructing photometric errors, along with the addition of inertial information in the process. This approach enables the tracking phase to utilize the full image and inertial information. During intense motion, the inverse distance of the new candidate point often fails to converge, leading to either scale drift or tracking failure. We present a dynamic inverse distance filter that can adjust the convergence range to update candidate points' inverse distance. This adjustment is based on the convergence ratio of the inverse distance of keyframes, which enables more convergent map points aiding in robust tracking in regions lacking texture and during rapid rotation. We evaluate the performance of VI-HSO on public datasets and real-world experiments, and our system outperforms state-of-the-art algorithms.