We consider the problem of training a convolutional neural network for histological localization of colorectal lesions from imperfectly annotated datasets. Given that we have a colonoscopic image dataset for 4-class histology classification and another dataset originally dedicated to polyp segmentation, we propose a weakly supervised learning approach to histological localization by training with the two different types of datasets. With the classification dataset, we first train a convolutional neural network to classify colonoscopic images into 4 different histology categories. By interpreting the trained classifier, we can extract an attention map corresponding to the predicted class for each colonoscopy image. We further improve the localization accuracy of attention maps by training the model to focus on lesions under the guidance of the polyp segmentation dataset. The experimental results show that the proposed approach simultaneously improves histology classification and lesion localization accuracy.