This paper proposes a sensor selection and optimization approach based on factor graph that meets the demand of environment adapting in multi-sensor fusion. Our approach uses a factor graph framework, which encodes sensor measurements with different frequencies, Iatencies, and noise distributions. A sensor selection that estimates and selects sensors on their original measurements and characteristics is presented. A sensor optimization method is proposed using a consistency check method to update sensors' bias and noise model in factor graph framework during fusion process. These methods, integrating with factor graph, provides a multi-sensor fusion framework. Experimental results demonstrate that our approach is able to select sensors when environment changes, with a more accurate fusion result than the original factor graph algorithm.