Keyframe-based monocular SLAM (Simultaneous Localization and Mapping) is one of the main visual SLAM approaches, used to estimate the camera motion together with the map reconstruction over selected frames. These techniques represent the environment by map points located in the three-dimensional space, that can be recognized and located in the frame. However, these techniques usually cannot decide when a map point is an outlier or obsolete information and can be discarded. Another problem is to decide when combining map points corresponding to the same three-dimensional point. In this paper, we present a robust method to maintain a refined map. This approach uses the covisibility graph and an algorithm based on information fusion to build a probabilistic map, that explicitly models outlier measurements. In addition, we incorporate a pruning mechanism to reduce redundant information and remove outliers. In this way, our approach manages to reduce the map size maintaining essential information of the environment. Finally, in order to evaluate the performance of our method, we incorporate it into an ORB-SLAM system and measure the accuracy achieved on publicly available benchmark datasets which contain indoor images sequences recorded with a hand-held monocular camera.