This article realized a method applying Haar features to train Adaboost classifier, and combined skin and lip color separation algorithm to form a self-adaptive skin and lip color separation model, which can dynamically adjust constant parameters of skin and lip separation algorithm based on the result of applying Haar features to train Adaboost classifier. The model can dynamically acquire distribution range of skin color and lip color, and improve the effectiveness and robustness of lip-reading deletion. Applying the method to deal with 4000 image in GENKI database, it successfully detected lip area and had the smaller deviation. Lastly, curve fitting for the edge of the lip region was made to locate lip. The results showed that the method was more effective.