The traditional HD map production based on mapping vehicles equipped with expensive sensors (e.g. LiDAR) struggles to keep the frequently changing HD maps up-to-date, thus calling for great research efforts in the HD map reconstruction using low-cost vision sensors. However, most existing works in this area limit their focus on reconstructing the landmarks on the road surface. This letter widens this limited scope by focusing on the roadside object reconstruction with a monocular camera. We propose a novel object-level reconstruction framework that boasts two advantageous features: 1) a novel 3D model estimation method to directly reconstruct the specific roadside objects in a vectorized format using their prior geometric knowledge, and 2) a novel data association method to solve the complex tracking problem of roadside objects that often yield thin and coherent observations. The proposed framework is evaluated on both typical highway and urban scenarios in public KAIST Urban dataset. The results demonstrate that our algorithm can reconstruct the roadside objects with high accuracy and recall, and outperforms classic Structure from Motion (SfM) and deep-learning-based methods by a large margin.