Shadow detection is one of the most challenging issues in computer vision. Inspired by the great success of the convolutional neural network (CNN) for the problem of image restoration, learned features have been widely adopted for shadow detection. However, most existing methods still suffer from ambiguities driven by black-colored objects, which are not actually shaded, as well as the background clutter. In this letter, we propose the attentive feedback feature pyramid network (AFFPN) for shadow detection in a single image. The key idea of the proposed method is to extract shadow-relevant features based on multiple feedback modules, which are defined in the feature pyramid network. Specifically, attentive features extracted from each level of the encoder are progressively refined via connections between feedback modules from high-level to low-level layers for learning properties of shadow more accurately. Experimental results on benchmark datasets show that the proposed method is effective for shadow detection under complicated real-world environments. The code and model are publicly available at: https://github.com/JinheeKIM94/AFFPN_release.