This letter addresses the problem of weakly supervised semantic segmentation. Given training images with only image level annotations (i.e., tags) where the precise locations of tags are unknown, we simultaneously segment the images and assign tags to image regions. In contrast to previous work which segmented images at a specified scale, in this letter we propose a multiscale model for semantically segmenting images in different granularities and exploiting the long-range contextual information between adjacent scales. Then, to capture the geometric context of semantic labels, we augment the multiscale model by (i) the object spatial prior, e.g., “sky” has high probability on the top of an image, and (ii) the object spatial correlations, e.g., “car” always appears above “road”. Finally, we present an iterative top-down bottom-up method to learn the multiscale model by recovering the pixel labels of training images. Experiments on the benchmark MSRC21 and LMO datasets demonstrate the improved performance of our method over previous weakly supervised methods and even over some fully supervised methods.