Abstract
The orthography of many resource-scarce languages includes diacritically marked characters. Falling outside the scope of the standard Latin encoding, these characters are often represented in digital language resources as their unmarked equivalents. This renders corpus compilation more difficult, as these languages typically do not have the benefit of large electronic dictionaries to perform diacritic restoration. This paper describes experiments with a machine learning approach that is able to automatically restore diacritics on the basis of local graphemic context. We apply the method to the African languages of Cilubà, Gĩkũyũ, Kĩkamba, Maa, Sesotho sa Leboa, Tshivenda and Yoruba and contrast it with experiments on Czech, Dutch, French, German and Romanian, as well as Vietnamese and Chinese Pinyin.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
de Schryver, G.M.: Web for/as corpus: A perspective for the African languages. Nordic Journal of African Studies 11/2, 266–282 (2002)
Wagacha, P., De Pauw, G., Getao, K.: Development of a corpus for Gĩkũyũ using machine learning techniques. In: Proceedings of LREC workshop - Networking the development of language resources for African languages, Genoa, Italy, ELRA, pp. 27–30 (2006)
Hussain, F., Cowell, J.: Amharic character recognition using a fast signature based algorithm. In: Proceedings of the IEEE conference on Image Visualisation 2003, London, UK, pp. 384–389. IEEE Computer Society Press, Los Alamitos (2003)
Bailey, D.: Creating a South African keyboard. In: Afrilex 2006, the user perspective in lexicography, programme and abstracts, Pretoria, South Africa (SF)2 Press, pp. 17–18 (2006)
Yarowsky, D.: A comparison of corpus-based techniques for restoring accents in Spanish and French text. In: Proceedings of the Second Annual Workshop on Very Large Corpora, Kyoto, Japan, pp. 19–32 (1994)
Tufiş, D., Chiţu, A.: Automatic diacritics insertion in Romanian texts. In: Proceedings of the International Conference on Computational Lexicography, Pecs, Hungary, pp. 185–194 (1999)
Simard, M.: Automatic insertion of accents in French text. In: Proceedings of the Third Conference on Empirical Methods in Natural Language Processing, Granada, Spain, pp. 27–35 (1998)
Mihalcea, R.F.: Diacritics restoration: Learning from letters versus learning from words. In: Gelbukh, A. (ed.) CICLing 2002. LNCS, vol. 2276, pp. 339–348. Springer, Heidelberg (2002)
Mihalcea, R.F., Nastase, V.: Letter level learning for language independent diacritics restoration. In: Proceedings of CoNLL-2002, Taipei, Taiwan, pp. 105–111 (2002)
Wagacha, P., De Pauw, G., Githinji, P.: A grapheme-based approach for accent restoration in Gĩkũyũ. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation, Genoa, Italy, ELRA, pp. 1937–1940 (2006)
de Schryver, G.M.: Bantu Lexicography and the Concept of Simultaneous Feedback (MA dissertation). Ghent University, Ghent, Belgium (1999)
de Schryver, G.M.: Corpus-based statements of meaning versus descriptions of actual language use in dictionaries. Culture, Language and Identity (CLIDE) Seminar, University of the Western Cape, Bellville, South Africa (2007)
Pala, K., Rychlý, P., Smrž, P.: DESAM - annotated corpus for Czech. In: Jeffery, K.G. (ed.) SOFSEM 1997. LNCS, vol. 1338, pp. 523–530. Springer, Heidelberg (1997)
Baayen, R.H., Piepenbrock, R., van Rijn, H.: The CELEX lexical data base on CD-ROM. Linguistic Data Consortium, Philadelphia, PA (1993)
Ha, L.: A method for word segmentation in Vietnamese. In: Proceedings of the Corpus Linguistics 2003 Conference, pp. 282–287 (2003)
Daelemans, W., Zavrel, J., van den Bosch, A., van der Sloot, K.: TiMBL: Tilburg Memory Based Learner, version 5.1, Reference Guide. ILK Technical Report 04-02, Tilburg University (2004)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
De Pauw, G., Wagacha, P.W., de Schryver, GM. (2007). Automatic Diacritic Restoration for Resource-Scarce Languages. In: Matoušek, V., Mautner, P. (eds) Text, Speech and Dialogue. TSD 2007. Lecture Notes in Computer Science(), vol 4629. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74628-7_24
Download citation
DOI: https://doi.org/10.1007/978-3-540-74628-7_24
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74627-0
Online ISBN: 978-3-540-74628-7
eBook Packages: Computer ScienceComputer Science (R0)