To interpret the decision of Deep Neural Networks (DNNs), explainable artificial intelligence research has been widely investigated. Especially, visualizing the attribution maps is known as one of the efficient ways to provide explanations for the trained networks. Applying existing visualization methods on medical images has significant issues in that the medical images commonly have inherent imbalanced data poses and scarcity. To tackle such issues and provide more accurate explanations in medical images, in this paper, we propose a new explainable framework, Counterfactual Generative Network (CGN). We embed counterfactual lesion prediction of DNNs to our explainable framework as prior conditions and guide to generate various counterfactual lesional images from normal input sources, or vice versa. By doing so, CGN can represent detailed attribution maps and generate corresponding normal images from leisonal inputs. Extensive experiments are conducted on the two chest X-ray datasets to verify the effectiveness of our method.