Despite the potential of deep convolutional neural networks for classification of thorax diseases from chest X-ray images, this task is still challenging as it is categorized as a weakly supervised learning problem, and deep neural networks in general suffer from a lack of interpretability. In this paper, a deep convolutional neural network framework with recurrent attention mechanism was investigated to annotate abnormalities in chest X-ray images. A modified MobileNet architecture was adapted in the framework for classification and the prediction difference analysis method was utilized to visualize the basis of network’s decision on each image. A long short-term memory network was utilized as the attention model to focus on relevant regions of each image for classification. The framework was evaluated on NIH chest X-ray dataset. The attention-guided model versus the model with no attention mechanism could annotate the images in an independent test set with an F1-score of 0.58 versus 0.46, and an AUC of 0.94 versus 0.73. The obtained results implied that the proposed attention-guided model could outperform the other methods investigated previously for annotating the same dataset.