In semantic visual simultaneous localization and mapping (SLAM), accurate object-level reconstruction of the environment based on the deep learning techniques is very crucial for high-level scene recognition and semantic object association. However, existing work handles this problem with the assumption of a simple world. There is still a challenge to improve the accuracy of object reconstruction based on images with a complicated environment background. In this work, we propose an improved object recovery method applying the DBSCAN algorithm based on geometric features. Outlier points and abnormal clusters can be identified by combining the clustering algorithm and the nonparametric test. In addition, we develop an adaptive sampling strategy based on line features with varying-step intervals, which can achieve a more accurate estimation of the object orientation. The proposed method is integrated with the ORB-SLAM2 framework to construct a real-time image-based reconstruction system. The qualitative and quantitative evaluation on public data sets and real-world scenarios demonstrates the robustness and effectiveness of our approach compared to the related work.