Detecting pavement cracks from images is a complex computer vision task due to their varying shapes, backgrounds, and sizes. We propose CrackF-Net, an end-to-end convolutional neural network for automatic crack detection in road images. We construct the CrackF-Net network using an encoder–decoder architecture to extract image features in convolutional blocks with residuals and fuse the multiscale convolutional features produced by the decoder. Convolutional blocks with residuals are used to capture the strong semantic features of cracks, and an adaptive filter fusion module is proposed to assist the network make a selection of filter fusion features on the channels. CrackF-Net fuses the multiscale features in decoder to improve crack detection performance. The proposed CrackF-Net is compared to other advanced crack detection methods using three public datasets. The experimental results show that CrackF-Net achieves state-of-the-art performance, which obtains F-measures of 0.866, 0.737, and 0.852 on the three datasets.