Earthquake magnitude estimation is one of the critical parts of earthquake early warning systems. It uses the first few seconds of a waveform recorded by an earthquake detection station, which is required to be rapid and accurate. In this paper, we propose a novel framework to estimate magnitude integrated with deep learning, consisting of feature stage and regression stage. In the feature stage, we extract temporal & spatial features by deep learning methods, and combine them with hand-crafted features embedded expert domain knowledge. Then, each earthquake can be represented by a hybrid feature. Therefore, magnitude estimation can be modeled as a regression problem to solve. Our framework is evaluated on 5,503 earthquake records collected in Sichuan province, China. It is found that, learning the temporal & spatial features by deep neural networks is critical for magnitude estimation. The results demonstrate the state-of-the-art performance, compared with other approaches.