The aim of scene semantic segmentation is to label each pixel with a class which it belongs to in high level cognition. State-of-art works mainly adapt convolutional neural networks originally designed for image classification to make dense prediction. However the inner structure of scene itself and its stuff is more flexible and variable, which is distinct from the objects in image classification task. Therefore we propose a gate function based structure-aware convolution for deep neural networks with the ability of modeling inner variance in scene. The gate function is a RNN-based learnable function or a handcrafted one, which is applied to distinguish efficient activations from convolution area. It is proved that dilated convolution is a subclass of gate function. As shown in our experiments on scene datasets, the proposed convolution method efficiently improves the accuracy of current semantic segmentation systems by partly replacing original networks' convolution layers with ours.