Abstract
This paper describes an intelligibility snap-judgment test. In this exercise, participants are shown a series of human translations and machine translations and are asked to determine whether the author was human or machine. The experiment shows that snap judgments on intelligibility are made successfully and that system rankings on snap judgments are consistent with more detailed intelligibility measures. In addition to demonstrating a quick intelligibility judgment, representing on a few minutes time of each participant, it details the types of errors which led to the snap judgments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
ALPAC, Language and machines: Computers in Translation and Linguistics. A Report by the Automatic Language Processing Advisory Committee. Division of Behavioral Sciences, National Academy of Sciences, National Research Council, Washington, D.C (1966)
Farwell, D., Helmreich, S., Dorr, B., Habash, N., Reeder, F., Miller, K., Levin, L., Mitamura, T., Hovy, E., Rambow, O., Siddharthan, A.: Interlingual Annotation of Multilingual Text Corpora. In: Proceedings of Workshop on Frontiers in Corpus Annotation. NAACL/HLT (2004)
Flanagan, M.: Error Classification for MT Evaluation. In: Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas, Columbia, MD (1994)
Heift, G.: Designed Intelligence: A Language Teacher Model. Unpublished Ph.D. thesis. Simon Fraser University (1998)
Holland, V.M.: Lessons Learned in Designing Intelligent CALL: Managing Communication across Disciplines. Computer Assisted Language Learning 7(3), 227–256 (1994)
Hovy, E., King, M., Popescu-Belis, A.: Principles of Context-Based Machine Translation Evaluation. Machine Translation 17(1), 43–75 (2002)
Meara, P., Babi, A.: Just a few words: how assessors evaluate minimal texts. Vocabulary Acquisition Research Gp. Virtual Lib (1999), http://www.swan.ac.uk/cals/vlibrary/ab99a.html
Papenini, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a Method for Automatic Evaluation of Machine Translation. In: Proceedings of ACL, Philadelphia, PA (2002)
Papenini, K., Roukos, S., Ward, T., Henderson, J., Reeder, F.: Corpus-based comprehensive and diagnostic MT evaluation: Initial Arabic, Chinese, French, and Spanish results. In: Proceedings of Human Language Technology, San Diego, CA (2002)
Reeder, F., Miller, K., Doyon, J., White, J.:The Naming of Things and the Confusion of Tongues: An MT Metric.In: MT-Summit Workshop on MT Evaluation (September 2001)
Van Slype, G.: Critical Methods for Evaluating the Quality of Machine Translation. Prepared for the European Commission Directorate General Scientific and Technical Information and Information Management. Report BR-19142. Bureau Marcel van Dijk (1979)
White, J.: Toward an Automated, Task-Based MT Evaluation Strategy. In: Maegaard, B. (ed.) Proceedings of the Workshop on Machine Translation Evaluation at LREC 2000 ,Athens, Greece (2000)
White, J., et al.: ARPA Workshops on Machine Translation. Series of 4 workshops on comparative evaluation. PRC Inc. McLean (1992-1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Reeder, F. (2004). Investigation of Intelligibility Judgments. In: Frederking, R.E., Taylor, K.B. (eds) Machine Translation: From Real Users to Research. AMTA 2004. Lecture Notes in Computer Science(), vol 3265. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30194-3_25
Download citation
DOI: https://doi.org/10.1007/978-3-540-30194-3_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23300-8
Online ISBN: 978-3-540-30194-3
eBook Packages: Springer Book Archive