Recent studies have shown that convolutional neural networks (CNNs) can boost the performance of audio steganalysis. In this paper, we propose a well-designed fully CNN architecture for MP3 steganalysis based on rich high-pass filtering (HPF). On the one hand, multi-type HPFs are employed for "residual" extraction to enlarge the traces of the signal in view of the truth that signal introduced by secret messages can be seen as high-pass frequency noise. On the other hand, to utilize the spatial characteristics of feature maps better, fully connected (Fc) layers are replaced with convolutional layers. Moreover, this fully CNN architecture can be applied to the steganalysis of MP3 with size mismatch. The proposed network is evaluated on various MP3 steganographic algorithms, bitrates and relative payloads, and the experimental results demonstrate that our proposed network performs better than state-of-the-art methods.