Global image dependencies help in full image understanding. Self-attention based methods can map the mutual relationship and dependencies among pixels of an image and thus improve semantic segmentation accuracy. In this paper, we propose two segmentation networks based on a novel baseline self-attention network. Compared to existing self-attention methods we utilize lower level feature maps to generate position attention modules which constitute a baseline network. This baseline network is incorporated with global average pooling and U-Net to create two segmentation schemes. These two segmentation networks are evaluated on a natural disaster dataset and perform excellent in damage assessment with a Mean IoU score of 95.61%.