With the research on mental health, rehabilitation training and other fields, obtaining people's real emotion feelings is frequently required in many fields. Emotion recognition method based on physiological signals can directly obtain people's emotion states and avoid pretending expression and emotional expression disorder. In physiological signals, Electroencephalogram (EEG) signal is commonly used in the emotion evaluation, and Heart Rate Variability (HRV) signal is related to people's excited feeling. This paper proposed an emotion recognition method based on EEG and HRV to do the emotion recognition work. This method aims to solve the accuracy problem of instant emotion recognition, and achieve a higher accuracy. According to Russell's model of emotion, the system in this paper use two dimensions, “valence” and “arousal”, to describe people's emotion. The emotion recognition system we proposed combines more advanced neural network models and eigenvalues closely related to emotional states. This system uses DenseNet as the neural network model for machine learning process, which is more accurate than the general deep neural network. Using differential entropy as the main eigenvalue makes the system's ability to analyze emotions based on EEG more efficient.