The performance of building extraction in remotely sensed images has been hugely improved with the development of convolutional neural networks and especially the semantic segmentation field. Due to the rich context of the image scene and the way of labeling (based on the pixel-level predicted probability), the segmentation masks are not always regular or close to the real building boundaries. In order to solve this problem, we propose a simple but effective deep framework based on two stages: the coarse result with an automatic semantic segmentation network and the fine result with an interactive refinement network. By using the binary mask of the initial segmentation and the interactions provided by the users, we obtain the final building extraction result through a deep interactive segmentation network. We evaluate our method on the WHU building dataset, and the results show that the method achieves better performance than the state-of-the-art methods.