Recently, sparse representation has been successfully used to identify brain networks from task-based fMRI dataset. However, when using the strategy to analyze resting-state fMRI dataset, it is still a challenge to automatically infer the group-wise brain networks under consideration of group commonalities and subject-specific characteristics. In the paper, a novel method based on dual temporal and spatial sparse representation (DTSSR) is proposed to meet this challenge. First, the brain functional networks with subject-specific characteristics are obtained via sparse representation with online dictionary learning for the fMRI time series (temporal domain) of each subject. Next, based on the current brain science knowledge, a simple mathematical model is proposed to describe the complex nonlinear dynamic coupling mechanism of the brain networks, with which the group-wise intrinsic connectivity networks (ICNs) can be inferred by sparse representation for these brain functional networks (spatial domain) of all subjects. Experiments on Leiden_2180 dataset show that most group-wise ICNs obtained by the proposed DTSSR are interpretable by current brain science knowledge and are consistent with previous literature reports. The robustness of DTSSR and the reproducibility of the results are demonstrated by experiments on three different datasets (Leiden_2180, Leiden_2200, and our own dataset). The present work also shed new light on exploring the coupling mechanism of brain networks from perspective of information science.