The denoising and deblurring of images are the two essential restoration tasks in the document image processing task. As the preprocessing stages of the processing pipeline, the quality of denoising and deblurring heavily influences the result of subsequent tasks, such as character detection and recognition. In this paper, we propose a novel neural method for restoring document images. We named our network Skip-Connected Deep Convolutional Autoencoder (SCDCA), which is composed of multiple layers of convolution followed by a batch normalization layer and the leaky rectified linear unit (Leaky ReLU) activation function. Inspired by the idea of residual learning, we use two types of skip connections in the network. One is identity mapping between convolution layers and the other is used to connect the input and output. Through these connections, the network learns the residual between the noisy and clean images instead of learning an ordinary transformation function. We empirically evaluate our algorithm on an open and challenging document images dataset. We also assess our restoring results using the optical character recognition (OCR) test. Experimental results have demonstrated the effectiveness and efficiency of our proposed algorithm by comparing with several state-of-the-art methods.