Semantic information has been agreed to be a key complement for more accurate SLAM or navigation and higher-level behavior planning. In dynamic scene, it also has the potential to improve performance. In this paper, we combine monocular SLAM with multiple object tracking to obtain robust static semantic map in complicated environment. This system utilizes up-to-date CNN object detector to detect possible moving objects in current view. Without the features located in these objects, monocular SLAM get consecutive pose transformation between two adjacent frames, which help multiple object tracking. With these trackers, this system can filter out dynamic objects, retain static objects for further sematic data association and graph optimization. We evaluate our approaches on the popular KITTI dataset and high dynamic RobotCar dataset.