Abstract
Threshold-based segmentation methods provide a simple and efficient way to implement lip segmentation. However, automatic computation of robust thresholds presents a major challenge. This research proposes an adaptive method for selecting the histogram threshold, based on feedback of shape information. The proposed method reduces unnecessary overhead by first comparing the initial segmentation to a reference lip shape model to decide if optimisation is required. In cases where optimisation is required, the algorithm adjusts the threshold until the segmentation is sufficiently similar to a reference shape model. The algorithm is tested on the AR Face Database by comparing the segmentation accuracy before and after optimisation. The proposed method increases the number of segmentations classified as ‘good’ (overlap above 90 %) by 7.1 % absolute, and significantly improves the segmentation in challenging cases containing facial hair.






Similar content being viewed by others
References
Potamianos, G., Neti, C.: Audio-visual speech recognition in challenging environments. In: European Conference on Speech Communication and Technology (EUROSPEECH 2003), Geneva, Switzerland, pp. 1293–1296 (2003)
Gritzman, A.D., Rubin, D.M., Pantanowitz, A.: Comparison of colour transforms used in lip segmentation algorithms. Signal Image Video Process. 9(4), 168–173 (2014)
Wark, T., Sridharan, S., Chandran, V.: An approach to statistical lip modelling for speaker identification via chromatic feature extraction. In: Fourteenth International Conference on Pattern Recognition (ICPR 1998), vol. 1. IEEE, pp. 123–125 (1998)
Chiou, G.I., Hwang, J.-N.: Lipreading from color video. IEEE Trans. Image Process. 6(8), 1192–1195 (1997)
Coianiz, T., Torresani, L., Caprile, B.: 2D deformable models for visual speech analysis. In: Stork, G., Hennecke, M.E. (eds.) NATO Advanced Study Institute: Speechreading by Man and Machine, pp. 391–398. Springer, Berlin (1996)
Zhang, X., Mersereau, R.: Lip feature extraction towards an automatic speechreading system. In: 2000 International Conference on Image Processing (ICIP 2000), vol. 3. IEEE, pp. 226–229 (2000)
Caplier, A., Stillittano, S., Bouvier, C., Coulon, P.: Lip modelling and segmentation. In: Liew, A. W.-C., Wang, S. (eds.) Visual Speech Recognition: Lip Segmentation and Mapping, pp. 70–127. Information Science Reference (an imprint of IGI Global), USA (2009)
Pardàs, M., Sayrol, E.: Motion estimation based tracking of active contours. Pattern Recogn. Lett. 22(13), 1447–1456 (2001)
Werda, S., Mahdi, W., Hamadou, A. B.: Colour and geometric based model for lip localisation: application for lip-reading system. In: 14th International Conference on Image Analysis and Processing (ICIAP 2007). IEEE, pp. 9–14 (2007)
Eveno, N., Caplier, A., Coulon, P.: Accurate and quasi-automatic lip tracking. IEEE Trans. Circuits Syst. Video Technol. 14(5), 706–715 (2004). iD: 1
Yuille, A.L., Hallinan, P.W., Cohen, D.S.: Feature extraction from faces using deformable templates. Int. J. Comput. Vision 8(2), 99–111 (1992)
Kaucic, R., Dalton, B., Blake, A.: Real-time lip tracking for audio-visual speech recognition applications. Comput. Vis. ECCV 1996, 376–387 (1996)
Luettin, J., Thacker, N.A.: Speechreading using probabilistic models. Comput. Vis. Image Underst. 65(2), 163–178 (1997)
Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comput. Vis. 1(4), 321–331 (1988)
Eveno, N., Caplier, A., Coulon, P.-Y.: Jumping snakes and parametric model for lip segmentation. In: International Conference on Image Processing (ICIP 2003), vol. 2. IEEE, pp. 867–870 (2003)
Matthews, I., Cootes, T.F., Bangham, J.A., Cox, S., Harvey, R.: Extraction of visual features for lipreading. IEEE Trans. Pattern Anal. Mach. Intell. 24(2), 198–213 (2002)
Zheng, Z., Jiong, J., Chunjiang, D., Liu, X., Yang, J.: Facial feature localization based on an improved active shape model. Inf. Sci. 178(9), 2215–2223 (2008)
Wang, S., Lau, W., Leung, S., Yan, H.: A real-time automatic lipreading system. In: International Symposium on Circuits and Systems (ISCAS 2004), vol. 2. IEEE, pp. II-101–II-104 (2004)
Saeed, U., Dugelay, J.L.: Combining edge detection and region segmentation for lip contour extraction. Articulated Motion and Deformable Objects, pp. 11–20 (2010)
Bouvier, C., Coulon, P.Y., Maldague, X.: Unsupervised lips segmentation based on ROI optimisation and parametric model. In: 2007 IEEE International Conference on Image Processing, vol. 4. IEEE, pp. IV-301–IV-304 (2007)
Otsu, N.: A threshold selection method from gray-level histograms. Automatica 11(285–296), 23–27 (1975)
Martinez, A.: The AR face database. CVC Technical Report, vol. 24 (1998)
Ding, L., Martinez, A.: Features versus context: an approach for precise and detailed detection and delineation of faces and facial features. IEEE Trans. Pattern Anal. Mach. Intell. 32(11), 2022–2038 (2010)
Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), vol. 1. IEEE, pp. I–511 (2001)
Wang, S., Leung, S., Lau, W.: Lip segmentation by fuzzy clustering incorporating with shape function. In: 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2002), vol. 1. IEEE, pp. I-1077 (2002)
Liew, A.-C., Leung, S.H., Lau, W.H.: Segmentation of color lip images by spatial fuzzy clustering. IEEE Trans. Fuzzy Syst. 11(4), 542–549 (2003)
Eveno, N., Caplier, A., Coulon, P.Y.: New color transformation for lips segmentation. In: 2001 IEEE Fourth Workshop on Multimedia Signal Processing, pp. 3–8 (2001)
Canzler, U., Dziurzyk, T.: Extraction of non manual features for videobased sign language recognition. In: IAPR Workshop on Machine Vision Applications (IAPR MVA 2002), pp. 318–321 (2002)
Kiefer, J.: Sequential minimax search for a maximum. Proc. Am. Math. Soc. 4(3), 502–506 (1953)
Acknowledgments
The financial assistance of the National Research Foundation (NRF) of South Africa towards this research is hereby acknowledged (Grant No. 97742).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gritzman, A.D., Aharonson, V., Rubin, D.M. et al. Automatic computation of histogram threshold for lip segmentation using feedback of shape information. SIViP 10, 869–876 (2016). https://doi.org/10.1007/s11760-015-0834-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-015-0834-9