The paper deals with robust video segmentation using a hierarchical model. This region based method is designed by human sense. Color and size in our model are two main factors, anything with obvious size and color is easily noticed. The hierarchical model works by examining both spatial and temporal features using a parameter set. The parameter set controls the feature's quality and noise. We chose a multiscale morphology based algorithm for spatial domain feature extraction and a change detection based algorithm for temporal domain feature extraction due to its noise robustness in our framework. Thus, our framework does not need smoothing as preprocessing. Results are shown as subjective views and errors are evaluated. Experimental results show that our proposed algorithm works well in most sequences.