Visual-inertial navigation system (VINS) is widely used for autonomous platforms but suffers from drifting over a long time. To remedy this situation, a lightweight 3D prior map-aided visual-inertial navigation system is presented in this paper, which tightly couples the visual-inertial data stream with a lightweight prior map involving 3D line information. To fill the gaps between 3D maps and 2D images, the mutual geometric feature of line segments is utilized to connect these two types of information in different dimensions. By detecting and matching line features in two data sources, the line pairs are utilized as constraints in the nonlinear optimization model and added to the existing factor graph framework in a tightly coupled form. Meanwhile, a fast line feature tracking strategy is employed to monitor and remove extreme outliers, which will further improve the reliability of this structural characteristic during the cross-modality localization. The effectiveness of the proposed method is evaluated by public indoor unmanned aerial vehicles (UAV) datasets, and outdoor unmanned ground vehicles (UGV) datasets generated by the CARLA simulator.