Depression is a common mental disorder that affects patients’ daily life. Most existing depression detection methods consume a lot of medical resources and exist at risk of subjective judgment. Therefore, we propose an objective and convenient experimental paradigm. Firstly, it selects emotional images as stimuli and records the subjects’ eye movement data. Secondly, we establish a connection between image processing and subjects’ psychological conditions analysis. Rather than some AI-based methods focus on feature engineering of recorded data, we design the saliency difference detection network and semantic segmentation network to explore the images’ deep semantic features and combine them with the subjects’ gaze pattern. Finally, we train a mental state classifier of Support Vector Machine to detect depression. The experimental results demonstrate that it achieves accuracy up to 90.06%, which outperforms previous methods.