In recent years, deep learning models are significantly improving capacity of computer vision. Especially in the field of Chinese character recognition, due to the complex structure of the characters, traditional rule-based or machine learning methods have been no longer competitive in terms of the measures such as recognition precision. Inspired by the actual reading process of human beings, we find that the processing of character recognition involves multiple modal information, while single-modal information of images is not only needed. Thus, we study the multi-modal fusion, and propose a novel multi-network model for Chinese character recognition in this paper. The proposed model consists of CNNs, LSTMs and full connection networks, fusing multi-modal information for classifying input images. After experiments, we found that multi-modal fusion can improve the accuracy of character recognition.