Cardiac segmentation is the first most important step in assessing cardiac diseases. However, it still remains challenging owing to the complicated information of myocardium's boundary. In this work, we investigate approaches based on deep learning for fully automatic segmentation of the left ventricular (LV) endocardium using cardiac magnetic resonance (CMR) images. The deep convolutional neural network architectures, specifically, GoogleNet and U-Net, are modified and deployed to extract the features and then classify each pixel into either endocardium or background. Since adjacent frames for a given slice are imaged over a short time period across a cardiac cycle, the LV endocardium exhibit strong temporal correlation. To utilize the temporal information of heart motion to assist segmentation, we propose to construct multi-channel cardiac images by combining adjacent frames together with the current frame, which are used as the inputs for deep learning models. This allows the deep learning models to automatically learn spatial and temporal information. The performance of our constructed networks is evaluated by using the Dice metric to compare the segmented areas with the manually segmented ground truth. The experiments show that the multi-channel approaches converge more rapidly and achieve higher segmentation accuracy compared to the single channel approach.