Locating the lip in video sequences is one of the primary steps of the automatic lip reading system. Lip contour extraction was a useful technique for obtaining a mouth shape in an image, and was one of the most important techniques for human-machine interface applications in wheelchair system, such as lip reading and speech recognition. In this paper, a new method to extract lip contour from video was propose based on the fact that the lip color and skin-color was varied in the different color spaces. We first extracted frames from digital video first; then we classified face into lip area and non-lip area by the Red Exclusion method; then we locating the lip shape by the Deformable Template Method; so that we can tracing the lip movement accurately by Powell's Methods in Multi-dimensions. At last we reach the accurately location by using PMM extract the lip shape pronouncing. At the same time we obtained some parameters from the lip area to reconstruct the lip contour. The experiment results proved that the proposed method was accurate and robust.