In this work we present a new approach for learning a layered stacked graphical model for the problem of visual object detection and segmentation. It is obvious that visual objects can be represented by multiple feature cues, such as color, texture, shape. The idea is to treat different feature types in different processes for learning classifiers and then integrate them into a unified model. We employ multiple stacked graphical models in stage-wise manner to exploit the discriminative power of each feature cue and to leverage the performance by using spatial context and inter- feature dependencies. The proposed system provides a simple yet efficient way to model complex object classes, which can be easily applied for many learning tasks. Experiments have been conducted extensively on a real-life problem of building classification from aerial images. Experimental results show a promising and improvement of the proposed model over several traditional stat-of-the-art approaches. Besides, we obtain fast learning and inference for the detection and segmentation of buildings at pixel level on huge aerial images.