In this paper, we present an automated procedure to determine the presence of cardiomegaly on chest X-ray image based on deep learning. The proposed algorithm CardioXNet uses deep learning methods U-NET and cardiothoracic ratio for diagnosis of cardiomegaly from chest X-rays. U-NET learns the segmentation task from the ground truth data. OpenCV is used to denoise and maintain the precision of region of interest once minor errors occur. Therefore, Cardiothoracic ratio (CTR) is calculated as a criterion to determine cardiomegaly from U-net segmentations. End-to-end Dense-Net neural network is used as baseline. This study has shown that the feasibility of combing deep learning segmentation and medical criterion to automatically recognize heart disease in medical images with high accuracy and agreement with the clinical results.