Recent research has focused on end-to-end networks for indoor scene semantic labeling. However, in addition to learning bottom-up features, high-level knowledge could be implemented to guide the local classification. In this paper, we take advantage of trained semantic labeling networks by using the intermediate layer output as a per-category local detector and implement the context information in a network structure to boost the semantic segmentation performance. A deep learning-based re-inferencing frame work is proposed to boost any pixel-level labeling outputs using our local collaborative object presence (LoCOP) feature as the global-to-local guidance. Experimental results show that the detection accuracy is improved with our re-inference approach.