Depression is a common mental disorder which is harmful to our family, economics and society. Many people cannot receive timely mental health services, and the diagnosis process is subjective. A primary way for reducing harm is finding an objective and effective depression detection approach. Speech and video are two promising behavior indicators for depression. In this paper, we proposed a speech and video bimodal fusion model based on time-frequency analysis and convolutional neural network for this goal. For the testing of the proposed method, a speech and video dataset of 292 participants were employed for cross-validation. Compared with the single modal classification results, the classification accuracy and generalization ability of this gender-independent model are further improved, which is helpful for the identification of depression.