Since the human brain functional mechanism has been enabled for investigation by the functional magnetic resonance imaging (fMRI) technology, simultaneous modeling of both the spatial and temporal patterns of brain functional networks from 4-D fMRI data has been a fundamental but still challenging research topic for neuroimaging and medical image analysis fields. Currently, general linear model (GLM), independent component analysis (ICA), sparse dictionary learning, and recently deep learning models, are major methods for fMRI data analysis in either spatial or temporal domains, but there are few joint spatial–temporal methods proposed, as far as we know. As a result, the 4-D nature of fMRI data has not been effectively investigated due to this methodological gap. The recent success of deep learning applications for functional brain decoding and encoding greatly inspired us in this paper to propose a novel framework called spatio–temporal convolutional neural network (ST-CNN) to extract both spatial and temporal characteristics from targeted networks jointly and automatically identify of functional networks. The identification of default mode network (DMN) from fMRI data was used for evaluation of the proposed framework. Results show that only training the framework on one fMRI data set is sufficiently generalizable to identify the DMN from different data sets of different cognitive tasks and resting state. Further investigation of the results shows that the joint-learning scheme can capture the intrinsic relationship between the spatial and temporal characteristics of DMN and thus it ensures the accurate identification of DMN from independent data sets. The ST-CNN model brings new tools and insights for fMRI analysis in cognitive and clinical neuroscience studies.