In this paper, we propose a foreground-background separation (FBS) method using a novel foreground model based on convolutional sparse representation (CSR). Existing FBS methods are based on the assumption that the foreground component is sparse or continuous in time, resulting in poor separation performance for videos containing sparse noise/fast moving objects or for low frame rate ones. Therefore, we propose to use CSR to characterize the similar structure of the foreground component scattered in each frame. This CSR-based model can adaptively capture the "shape" of the foreground component and thus can accurately separate it even under the above difficulties. We formulate FBS as an optimization problem incorporating the CSR-based model and develope an algorithm to solve it based on alternating minimization. Experiments using an infrared video and an electron microscope video show that the proposed foreground model is superior to existing models.