This paper presents an incremental structural modeling approach that improves the precision and stability of existing batch based methods for sparse and noisy point clouds from visual SLAM. The main idea is to use the generating process of point clouds on SLAM effectively. First, a batch based method is applied to point clouds that are incrementally generated from SLAM. Then, the temporal history of reconstructed geometric primitives is statistically merged to suppress incorrect reconstruction. The evaluation shows that both precision and stability are improved compared to a batch based method and the proposed method is suitable for real-time structural modeling.