Regular safety inspections of concrete and steel structures during their serviceability are essential since they directly affect the reliability and structural health. Early detection of cracks helps prevent further damage. Traditional methods involve the detection of cracks by human visual inspection. However, it is difficult to visually find cracks and other defects for extremely large structures because of time and cost constraints. Therefore, the development of smart inspection systems has been given utmost importance. We provide a deep convolutional neural network (DCNN) with transfer learning (TF) technique for crack detection. To reduce false detection rates, the images used to train in the TF technique come from two different datasets (CCIC and SDNET). Moreover, the designed CNN is trained on 3200 images of $256 \times 256$ pixel resolutions. Different deep learning networks are considered and the experiments on test images show that the accuracy of the damage detection is more than 99%. Results illustrate the viability of the suggested approach for crack observation and classification.