In this paper, a novel unified framework for atlas-based segmentation is proposed, consisting of two main components: forward deformation and label refinement. A newly designed distance constraint on mesh edges is enforced with contrast sensitivity in forward deformation based on Markov random field. With the edge distance constraint, the object shapes in the atlas and the target images can remain similar during deformation. Considering the shape variations caused by individual difference, we then develop a label refinement process embracing patch registration and label fusion to compensate the small variations around the structural surfaces. As the anatomical correspondences determined in forward deformation can differ from that in label refinement, the conventional one-to-one correspondence constraint can be relaxed in our framework. Experiments on two publicly available databases IBSR and LPBA40 demonstrate that our method can obtain better performance as compared with other state-of-the-art methods.