Skip to main content
SpringerLink
Log in

Opera goer or Scrabble player: what makes a good translator?

  • Published:
Machine Translation

Abstract

How should you select a person to carry out a translation? One approach is to request a sample translation and to evaluate it by hand. Quality Estimation addresses the problem of evaluation at least for Machine Translation output as a prediction task. This approach facilitates low-cost evaluation of MT outputs without expensive reference translations. However, the prediction of human translation in this way is difficult due to its subtlety of expression. We aimed to find out whether the qualifications, hobbies or personality traits of a person could predict their proficiency at translation. First, we gathered information about 82 participants; for each one we established the values of 146 attributes via a questionnaire. Second, we asked them to carry out some Japanese-to-English translations which we scored by hand. Third, we used the attributes as input and the translation scores as output to train the J48 decision-tree algorithm in order to predict the score of a translator from their attributes. This was then evaluated using ten-fold cross validation. When limiting to professional translators in Experiment 6, the best F-score was with Wrapper selection (0.775). The result was statistically significant (\(p < 0.05\)). This classifier also showed the second highest Precision on Good (0.813). The second best F-score (0.737) has the highest Precision on Good (0.909), using Manual feature selection. Once again this was significant (\(p < 0.05\)). The results suggest that certain attributes affect the prediction; in addition to experience and qualifications in translating into the target language, interest in going to the Opera, playing Scrabble or Contract Bridge, or enjoyment of cryptic crossword puzzles are important factors as well.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://translate.google.com/.

  2. http://trec.nist.gov/.

References

  • Alpaydin E (2004) Introduction to machine learning. MIT Press, Cambridge

    MATH  Google Scholar 

  • Avramidis E, Popovic M, Vilar D, Burchardt A (2011) Evaluate with confidence estimation: machine ranking of translation outputs using grammatical features. In: The 6th workshop on statistical machine translation, Edinburgh, Scotland, UK, pp 65–70

  • Banerjee S, Lavie A (2005) Meteor: an automatic metric for MT evaluation with improved correlation with human judgments. In: Workshop on intrinsic and extrinsic evaluation measures for MT and/or summarization, Ann Arbor, MI, pp 65–72

  • Biel Ł (2011) Training translators or translation service providers? EN 15038:2006 standard of translation services and its training implications. J Spec Transl 16:61–76

    Google Scholar 

  • Billinger M, Daly I, Kaiser V, Jin J, Allison BZ, Müller-Putz GR, Brunner C (2012) Is it significant? Guidelines for reporting BCI performance. In: Allison B et al (eds) Towards practical brain-computer interfaces. Springer, Berlin, pp 333–354

    Chapter  Google Scholar 

  • Blatz J, Fitzgerald E, Foster G, Gandrabur S, Goutte C, Kulesza A, Sanchis A, Ueffing N (2004) Confidence estimation for machine translation. In: Coling 2004: 20th international conference on computational linguistics proceedings, Geneva, Switzerland, pp 315–321

  • Bojar O, Buck C, Federmann C, Haddow B, Koehn P, Leveling J, Monz C, Pecina P, Post M, Saint-Amand H, Soricut R, Specia L, Tamchyna A (2014) Findings of the 2014 workshop on statistical machine translation. In: Proceedings of the ninth workshop on statistical machine translation, Baltimore, MD, USA, pp 12–58

  • Brau M (2014) The FBI develops a translation aptitude test (TAT), interagency language round table plenary session, language testing and assessment unit, National Foreign Language Cente (NFLC), College Park, MD

  • Callison-Burch C (2009) Fast, cheap, and creative: evaluating translation quality using Amazon’s mechanical turk. In: Proceedings of the 2009 conference on empirical methods in natural language processing, Edinburgh, Scotland, UK, pp 286–295

  • Callison-Burch C, Fordyce C, Koehn P, Monz C, Schroeder J (2007) Meta-evaluation of machine translation. In: Proceedings of ACL workshop on statistical machine translation, Prague, Czech Republic, pp 136–158

  • Callison-Burch C, Koehn P, Monz C, Peterson K, Przybocki M, Zaidan OF (2010) Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In: Proceedings of the joint 5th workshop on statistical machine translation and MetricsMATR, Uppsala, Sweden, pp 17–53

  • Doddington G (2002) Automatic evaluation of machine translation quality using N-gram co-occurrence statistics. In: HLT 2002: human language technology conference: proceedings of the second international conference on human language technology research, San Diego, CA, pp 138–145

  • Fomicheva M, Bel N, Specia L, da Cunha I, Malinovskiy A (2016) CobaltF: a fluent metric for MT evaluation. In: Proceedings of the first conference on machine translation, volume 2: shared task papers, Berlin, Germany, pp 483–490

  • Gile D (1995) Basic concepts and models for interpreter and translator training. John Benjamins Publishing Co, Amsterdam

    Book  Google Scholar 

  • Graham Y, Mathur N, Baldwin T (2015) Accurate evaluation of segment-level machine translation metrics. In: Human language technologies: the 2015 annual conference of the North American chapter of the ACL, Denver, CO, pp 1183–1191

  • Hayes AF, Krippendorff K (2007) Answering the call for a standard reliability measure for coding data. Commun Methods Meas 1(1):77–88

    Article  Google Scholar 

  • Howell DC (2007) Statistical methods for psychology, 6th edn. Thomson, Belmont

    Google Scholar 

  • Hubscher-Davidson SE (2009) Personal diversity and diverse personalities in translation: a study of individual differences. Perspect Stud Translatology 17(3):175–192

    Article  Google Scholar 

  • Koehn P (2004) Statistical significance tests for machine translation evaluation. In: EMNLP-2004: proceedings of the 2004 conference on empirical methods in natural language processing, Barcelona, Spain, 8 pp

  • Koehn P (2010) Statistical machine translation. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Krippendorff K (2011) Computing Krippendorff’s alpha-reliability. Departmental papers (ASC), p 43. Retrieved from http://repository.upenn.edu/asc_papers/43 [accessed 16/April/2015]

  • Leusch G, Ueffing N, Ney H (2006) CDER: efficient MT evaluation using block movements. In: Proceedings of the 11th conference of the European chapter of the association for computational linguistics, Trento, Italy, pp 241–248

  • Mooney CZ, Duval RD (1993) Bootstrapping: a nonparametric approach to statistical inference. Sage, London

    Book  Google Scholar 

  • Myers IB, Myers B (1995) Gifts differing. Understanding personality type. Davies-Black Publishing, Mountain View

    Google Scholar 

  • Nagao M, Tsujii J, Nakamura J (1985) The Japanese government project for machine translation. Comput Linguist 11(2–3):91–110

    Google Scholar 

  • Napthine A (1983) Training of translators. In: Picken C (ed) The translator’s handbook. Aslib, London, pp 21–32

    Google Scholar 

  • Nida E (1981) Translators are born not made. Practical papers for the bible translator 32(4):401–405

    Article  Google Scholar 

  • O’Brien S (2013) The borrowers: researching the cognitive aspects of translation. Target Int J Transl Stud 25(1):5–17

    Google Scholar 

  • Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. In: ACL-2002: 40th annual meeting of the association for computational linguistics, Philadelphia, PA, pp 311–318

  • Popović M, Ney H (2007) Word error rates: decomposition over POS classes and applications for error analysis. In the second workshop on statistical machine translation, Prague, Czech Republic, pp 48–55

  • Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann Publishers, Inc, Burlington

    Google Scholar 

  • Rahemi EF, Jufri M, Ardi H (2013) The correlation between reading comprehension and translation ability: a correlational study on forth year students at English department of UNP. J Engl Lang Teach 1(2):178–186

  • Sakaguchi K, Post M, Van Durme B (2014) Efficient elicitation of annotations for human evaluation of machine translation. In: Proceedings of the ninth workshop on statistical machine translation, Baltimore, MD

  • Schopp JF (2007) The European translation standard EN 15038 and it’s [sic] terminology—a mirror of missing professionalism? In: ELETO: 6th conference Hellenic language and terminology, proceedings, Athens, Greece, 10 pp

  • Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul, J (2006) A study of translation edit rate with targeted human annotation. In: AMTA 2006: proceedings of the 7th conference of the Association for Machine Translation in the Americas, “Visions for the Future of Machine Translation”, Cambridge, MA, USA, pp 223–231

  • Specia L, Cancedda N, Dymentman M, Turchi M, Cristianini N (2009) Estimating the sentence-level quality of machine translation systems. In: Proceedings of the 13th annual conference of the EAMT, Barcelona, Spain, pp 28–35

  • Specia L, Raj D, Turchi M (2010) Machine translation evaluation versus quality estimation. Mach Transl 24:39–50

    Article  Google Scholar 

  • Specia L, Shah K (2014) Predicting human translation quality. In: AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, pp 288–300

  • Sumita E, Sasaki Y, Yamamoto S (2005) The forefront of evaluation methods for machine translation systems. IPSJ Magazine, pp 552–557

  • Suzuki A (1988) Aptitude of translators and interpreters. Meta: Transl J 33(1):108–114

    Article  Google Scholar 

  • Tavakoli M, Shafiei S, Hatam AH (2012) The relationship between translation tests and reading comprehension: a case of Iranian University students. Iranian J Appl Lang Stud 4(1):193–211

    Google Scholar 

  • The European Commission (2015) Translation, [online], available: http://ec.europa.eu/dgs/translation/programmes/emt/index_en.htm [accessed 16/April/2015]

  • The House of Representatives of Japan (2014) The transcript of meetings, [online], available: http://www.shugiin.go.jp/internet/itdb_kaigiroku.nsf/html/kaigiroku/kaigi_l.htm [accessed 16/ April/2015]

  • Tillmann C, Vogel S, Ney H, Zubiaga A, Sawaf H (1997) Accelerated DP based search for statistical translation. In: Proceedings of the Fifth European conference on speech communication and technology, Rhodes, Greece, pp 2667–2670

  • Turian JP, Shen L, Melamed ID (2003) Evaluation of machine translation and its evaluation. In: Proceedings of MT summit IX, New Orleans, LA, pp 386–393

  • Verrinder J (1983) Who are the translators? In: Picken C (ed) The translator’s handbook. Aslib, London, pp 33–37

    Google Scholar 

  • Voorhees EM (2003) Overview of the TREC 2003 question answering track. In: Proceedings of the twelfth text retrieval conference (TREC 2003) Gaithersburg, MD, USA, pp 54–68

  • Witten IH (2013) Data mining with Weka (3.5: Pruning decision trees), [online], available: http://www.youtube.com/watch?v=ncR_6UsuggY [accessed 16/April/2015]

  • Witten IH, Frank E, Hall MA (2011) Data mining—practical machine learning tools and techniques, 3rd edn. Morgan Kaufmann, Burlington

    Google Scholar 

  • Zaidan OF, Callison-Burch C (2011) Crowdsourcing translation: professional quality from non-professionals. In: ACL-HLT 2011: proceedings of the 49th annual meeting of the association for computational linguistics, Portland, Oregon, pp 1220–1229

Download references

Acknowledgements

We would like to thank the 82 participants and the four evaluators in this study. This work could not have been achieved without their generous contributions. We are also indebted to Rebecca Bourke for her help. Finally, we thank the anonymous reviewers for their careful reading and detailed comments.

Author information

Authors and Affiliations

  1. Department of CSIS, University of Limerick, Limerick, Ireland

    Naoto Nishio

  2. School of CSEE, University of Essex, Colchester, UK

    Richard F. E. Sutcliffe

Authors
  1. Naoto Nishio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard F. E. Sutcliffe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Naoto Nishio.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nishio, N., Sutcliffe, R.F.E. Opera goer or Scrabble player: what makes a good translator?. Machine Translation 30, 63–109 (2016). https://doi.org/10.1007/s10590-016-9189-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10590-016-9189-4

Keywords

Search

Navigation