Several technologies are available for inspecting the aging of infrastructure to investigate for cracks in concrete structures or flaking of tiles. One such technology is the hammering test. In this study, we developed a test device that captures hammering sounds using a microphone and analyzes the sound. Firstly, we classified the detected data by Convolutional Neural Network (CNN) to detect cracks in concrete structures and the accuracy was 90.2 %. In our previous research, the classification accuracy by K-mean method was about 80%, so that Deep Learning provides better result. However, in the real situation, the condition of concrete structures is very different and need to adjust the classification function in the actual situation, so that we tried to use Transfer Learning (TL) to meet such requirements. We tried to evaluate the potential of TL and generated a learning model by using only 40 training data. The classification accuracy was 90.0%. This result was almost equivalent to that of CNN.