Abstract
The lexicon is a major part of any Machine Translation (MT) system. If the lexicon of an MT system is not adequate, this will affect the quality of the whole system. Building a comprehensive lexicon, i.e., one with a high lexical coverage, is a major activity in the process of developing a good MT system. As such, the evaluation of the lexicon of an MT system is clearly a pivotal issue for the process of evaluating MT systems. In this paper, we introduce a new methodology that was devised to enable developers and users of MT Systems to evaluate their lexicons semi-automatically. This new methodology is based on the idea of the importance of a specific word or, more precisely, word sense, to a given application domain. This importance, or weight, determines how the presence of such a word in, or its absence from, the lexicon affects the MT system's lexical quality, which in turn will naturally affect the overall output quality. The method, which adopts a black-box approach to evaluation, was implemented and applied to evaluating the lexicons of three commercialEnglish–Arabic MT systems. A specific domain was chosen in which the various word-sense weights were determined by feeding sample texts from the domain into a system developed specifically for that purpose. Once this database of word senses and weights was built, test suites were presented to each of the MT systems under evaluation and their output rated by a human operator as either correct or incorrect. Based on this rating, an overall automated evaluation of the lexicons of the systems was deduced.
Similar content being viewed by others
References
Anon: 1996, [The Machine Translator: Al-Wafy], (Arabuter) 8, 27–28.
Arnold, Doug, R. Lee Humphreys and Louisa Sadler (eds): 1993, ‘Special Issue on Evaluation of MT Systems’, Machine Translation 8(1–2).
Dorr, Bonnie J. and Judith Klavans: 1994/5, ‘Special Issue: Building Lexicons for Machine Translation’, Machine Translation 9(3–4), 10(1–2).
Dyson, Mary C. and Jean Hannah: 1987, ‘Towards a Methodology for the Evaluation of Machine-Assisted Translation Systems’, Computers and Translation 2, 163–176.
Eagles (Expert Advisory Group on Language Engineering Standards): 1995, ‘Evaluation of Natural Language Processing Systems’, Eagles Document EAG-EWG-PR.2.
Eagles: 1998, Proceedings of the Second Eagles II Workshop on Evaluation in Human Language Technology, Geneva.
Hutchins, W. John and Harold L. Somers: 1992, An Introduction to Machine Translation, London: Academic Press.
Jihad, A. : 1996, [Has the Arabic Machine Translation Era Started?], (Byte Middle East), November 1996, pp. 36–48.
King, M. and K. Falkedal: 1990, ‘Using Test Suites in Evaluation of Machine Translation Systems’, in COLING-90: Papers presented to the 13th International Conference on Computational Linguistics, Helsinki, Finland, Vol. 2, pp. 211–216.
Klein, J., S. Lehmann, K. Netter and T. Wegst: 1999, ‘DiET in the Context of MT Evaluation’, in Rita Nübel and Uta Seewald-Heeg (eds), Evaluation of the Linguistic Performance of Machine Translation Systems, Proceedings of the Konvens'98, St. Augustin: Gardezi Verlag, pp. 107–126.
Lehmann, Sabine, Stephan Oepen, Sylvie Regnier-Prost, Klaus Netter, Veronika Lux, Judith Klein, Kirsten Falkedal, Frederik Fouvry, Dominique Estival, Eva Dauphin, Herv, Compagnion, Judith Baur, Lorna Balkan and Doug Arnold: 1996, ‘TSNLP — Test Suites for Natural Language Processing’, COLING-96: The 16th International Conference on Computational Linguistics, Copenhagen, Denmark, pp. 711–716.
Lehrberger, John and Laurent Bourbeau: 1987, Machine Translation: Linguistic Characteristics of MT Systems and General Methodology of Evaluation, Amsterdam: John Benjamins.
Mason Jane and Adriane Rinsche: 1995, Ovum Evaluates: Translation Technology Products, London: OVUM Ltd.
Melby, A. K.: 1988, ‘Lexical Transfer: Between a Source Rock and a Hard Target’, Coling Budapest: Proceedings of the 12th International Conference on Computational Linguistics, Budapest, pp. 411–419.
Nagao, M.: 1985, ‘Evaluation of the Quality of Machine-Translated Sentences and the Control of Language’, Journal of the Information Processing Society of Japan 26, 1197–1202.
Nyberg, Eric H. and Teruko Mitamura: 1992, ‘The KANT System: Fast, Accurate, High-Quality Translation in Practical Domains’, Proceedings of the fifteenth [sic] International Conference on Computational Linguistics, COLING-92, Nantes, pp. 1069–1073.
Nyberg, Eric H., Teruko Mitamura and Jaime G. Carbonell: 1994, ‘Evaluation Metrics for Knowledge-Based Machine Translation’, COLING 94: The 15th International Conference on Computational Linguistics, Kyoto, Japan, pp. 95–99.
Qendelft, G. : 1997, [The Translation Program Al-Wafy Is Useful for Getting a General Understanding of a Letter Written in English], (Al-Hayat) (Saturday, 25 October).
Sinaiko, H. W. and G. R. Klare: 1972, ‘Further Experiments in Language Translation: Readability of Computer Translations’, ITL 15, 1–29.
Sinaiko, H. W. and G. R. Klare: 1973, ‘Further Experiments in Language Translation: A Second Evaluation of the Readability of Computer Translations’, ITL 19, 29–52.
van Slype, G.: 1979, ‘Systran: Evaluation of the 1978 Version of the Systran English-French Automatic System of the Commission of the European Communities’, The Incorporated Linguist 18, 86–89.
Vasconcellos, Muriel (ed.): 1988. Technology as Translation Strategy, Binghamton, NY: State University of New York at Binghamton (SUNY).
White, John S., Theresa O'Connell and Francis O'Mara: 1994, ‘The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Future Approaches’, Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas, Columbia, Maryland, pp. 193–205.
Wilks Y.: 1992, ‘Systran: It Obviously Works, But How Much Can It Be Improved?’, in John Newton (ed.), Computers and Translation: A Practical Appraisal, London: Routledge, pp. 166–188.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Guessoum, A., Zantout, R. A Methodology for a Semi-Automatic Evaluation of the Lexicons of Machine Translation Systems. Machine Translation 16, 127–149 (2001). https://doi.org/10.1023/A:1014504808954
Issue Date:
DOI: https://doi.org/10.1023/A:1014504808954