Semantic visual simultaneous localization and mapping (SLAM) is a fundamental capability for autonomous driving. The high performance of semantic SLAM algorithm is based on the assumption of correct object association. However, most previous work utilize a single method to associate objects, which cannot remove the ambiguity of object association very well. Incorrect object association can degrade the semantic SLAM performance. In this paper, we propose an ensemble association method to associate objects. We combine geometric features with object appearance information to consider the effect of multiple information on object association results. Additionally, the bag-of-words algorithm is introduced to measure the similarity of object appearance for better association decisions. Experimental results on the KITTI dataset show that the method improves both the accuracy of object association and the performance of semantic SLAM.