In conjunction with the recently developed morphological building index (MBI), the proposed postprocessing framework describes the characteristics of buildings by simultaneously considering the spectral, geometrical, and contextual information, and can be successfully applied to large high-spatial-resolution images. In this way, the proposed framework can alleviate the amount of false alarms to a remarkable extent, which mainly come from the bright soil and vegetation in rural and mountainous areas. Validated on a series of large test images obtained by the widely used commercial satellite sensors, the experiments confirm the promising performance of the proposed framework over various areas, including urban, mountainous, rural, and agricultural areas. Furthermore, the proposed framework increases the quality index by 11% and 9% on average compared to the performance of the original MBI and DMP-SVM, respectively. In addition, the parameter sensitivity is analyzed in detail and appropriate ranges of the parameters are suggested. The proposed building detection framework is designed to be of practical use for building detection from high-resolution imagery.