Skip to main content
SpringerLink
Log in

Manual labour: tackling machine translation for sign languages

  • Published:
Machine Translation

Abstract

This article explores the application of data-driven machine translation (MT) to sign languages (SLs). The provision of an SL MT system can facilitate communication between Deaf and hearing people by translating information into the native and preferred language of the individual. In this paper we address data-driven SL MT predominantly for Irish SL (ISL) but also for German SL (DGS/Deutsche Gebärdensprache). We take two different purpose-built corpora to feed our MaTrEx MT system and in a set of experiments translating both to and from the SLs, we investigate the effects of SL data on statistical MT (SMT). Exploiting the bidirectionality of the MaTrEx system, we demonstrate how additional modules, such as recognition and SL animation, can potentially build a full SL MT model for spoken and SL communication in addition to promising evaluation scores. A secondary focus of the article is on the two main issues affecting SL MT, those of transcription and evaluation. We offer a discussion on both these common problems before concluding.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bauer B, Nießen S, Heinz H (1999) Towards an automatic sign language translation system. In: Proceedings of the international workshop on physicality and tangibility in interaction: towards new paradigms for interaction beyond the desktop, Siena, Italy

  • Bungeroth J, Stein D, Ney H, Morrissey S, Way A (2008) The ATIS Sign Language Corpus. In: Proceedings of the 3rd Workshop on the representation and processing of sign languages at the 6th international conference on Language Resources and Evaluation (LREC-08), Marrakech, Morocco

  • Cahill P, Carson-Berndsen J (2006) The Jess Blizzard Challenge 2006 Entry. In: Proceedings of the Blizzard Challenge 2006 Workshop, Interspeech 2006, Pittsburgh, PA

  • Chomsky N (2000) New horizons in the study of language and mind. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Conroy P (2006) Signing in & signing out: the education and employment experiences of deaf adults in Ireland. Research report, Irish Deaf Society, Dublin

  • Conway A, Veale T (1994) A linguistic approach to sign language synthesis. In: Cockton G, Draper SW, Weir GRS (eds) People and computers IX: Proceedings of the Human Computer Interface conference (HCI), Glasgow, Scotland, pp 211–222

  • Crasborn O, Zwitserlood I (2008) The Corpus NGT: an online corpus for professionals and laymen. In: Proceedings of the 3rd workshop on the representation and processing of sign languages at the 6th international conference on Language Resources and Evaluation (LREC-08), Marrakech, Morrocco, pp 44–49

  • Dandapat S, Morrissey S, Naskar SK, Somers H (2010) Statistically motivated example-based machine translation using translation memory. In: Proceedings of the 8th International Conference on Natural Language Processing (ICON 2010), Kharagpur, India

  • Dreuw P, Rybach D, Deselaers T, Zahedi M, Ney H (2007) Speech Recognition techniques for a sign language recognition system. In: Proceedings of Interspeech 2006, Antwerp, Belgium, pp 2513–2516

  • Dreuw P, Ney H, Martinez G, Crasborn O, Piater J, Miguel Moya J, Wheatley M (2010) The signspeak project—bridging the gap between signers and speakers. In: Proceedings of the international conference on language resources and evaluation, Valletta, Malta, pp 476–481

  • Fourie J (2006) The design of a generic signing avatar. Technical Report, University of Stellenbosch, South Africa

    Google Scholar 

  • Grieve-Smith AB (1999) English to American sign language machine translation of weather reports. In: Proceedings of the Second High Desert Student Conference in Linguistics (HDSL2), Albuquerque, NM, pp 13–30

  • Hanke T (2002) ViSiCAST Deliverable D5-1: Interface Definitions. Manuscript, Hamburg, Germany

    Google Scholar 

  • Hanke T (2004) HamNoSys—representing sign language data in language resources and language processing contexts. In: Workshop on the representation and processing of sign languages at the Languages and Resources Evaluation Conference (LREC 04), Lisbon, Portugal, pp 1–6

  • Hanke T, Storz J (2008) iLex—a database tool integrating sign language corpus linguistics and sign language lexicography. In: Proceedings of the 3rd Workshop on the representation and processing of sign languages at the 6th international conference on Language Resources and Evaluation (LREC-08), Marrakech, Morrocco

  • Hassan H, Ma Y, Way A (2007) MaTrEx: the DCU Machine Translation System for IWSLT 2007. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Trento, Italy, pp 69–75

  • Hemphill C, Godfrey J, Doddington G (1990) The ATIS Spoken Language Systems Pilot Corpus. In: Proceedings of the workshop on speech and natural language, Hidden Valley, PA, pp 96–101

  • Huenerfauth M (2004) Spatial and planning models of ASL classifier predicates for machine translation. In: Proceedings of the 10th international conference on Theoretical and Methodological Issues in Machine Translation (TMI-04), Baltimore, MD, pp 65–74

  • Huenerfauth M (2006) Generating American sign language classifier predicates for English-to-ASL machine translation. PhD thesis, University of Pennsylvania, Philadelphia, PA

  • Jones D, Gill P (1998) Breaking down language barriers. Br Med J 316(7127): 1280–1476

    Article  Google Scholar 

  • Kanthak S, Vilar D, Matusov E, Zens R, Ney H (2005) Novel reordering approaches in phrase-based statistical machine translation. In: Proceedings of the ACL workshop on building and using parallel texts at the 43rd annual meeting of the Association of Computational Linguistics (ACL-05), Ann Arbor, MI, pp 167–174

  • Kennaway J, Glauert J, Zwitserlood I (2007) Providing signed content on the Internet by synthesized animation. ACM Trans Comput-Hum Interact 14: 1–29

    Article  Google Scholar 

  • Kneser R, Ney H (1995) Improved backing-off for n-gram language modelling. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, vol 1, Detroit, MI, pp 181–184

  • Koehn P, Och F, Marcu D (2003) Statistical phrase-based translation. In: Proceedings of the combined human language technology conference series and the North American Chapter of the Association for Computational Linguistics Conference Series (HLT-NAACL), Edmonton, Canada, pp 48–54

  • Koehn P, Hoang H, Birch A, Callison-Burch C, Federico M, Bertoldi N, Cowan B, Shen W, Moran C, Zens R, Dyer C, Bojar O, Constantin A, Herbst E (2007) Moses: open source toolkit for statistical machine translation. In: Proceedings of Demonstration and Poster Sessions at the 45th annual meeting of the Association of Computational Linguistics (ACL-07), Prague, Czech Republic

  • Leeson L (2001) Aspects of verbal valency in Irish sign language. PhD thesis, University of Dublin, Trinity College, Dublin, Ireland

  • Leeson L (2003) Sign language interpreters: agents of social change in Ireland?. In: Cronin M, ÓCuilleanáin C (eds) Languages of Ireland. Four Courts Press, Dublin, pp 148–164

  • Leeson L, Saeed J, Macduff A, Byrne-Dunne D, Leonard C (2006) Moving heads and moving hands: developing a digital corpus of Irish sign language. In: Proceedings of information technology and telecommunications conference 2006, Carlow, Ireland

  • Leusch G, Ueffing N, Ney H (2006) CDER: efficient MT evaluation using block movements. In: Proceedings of the 11th European Chapter of the Association for Computational Linguistics (EACL-06), Trento, Italy, pp 241–248

  • Liddell S, Johnson RE (1989) American sign language: the phonological base. Sign Lang Stud 64: 195–277

    Google Scholar 

  • Ma Y, Tinsley J, Hassan H, Du J, Way A (2008) MaTrEx: the DCU system for IWSLT 2008. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT 2008), Honolulu, HI, pp 26–33

  • Marshall I, Sáfár E (2002) Sign language generation using HPSG. In: Proceedings of the 9th international conference on Theoretical and Methodological Issues in Machine Translation (TMI-02), Keihanna, Japan, pp 105–114

  • Marshall I, Sáfár E (2003) A prototype text to British Sign Language (BSL) translation system. In: the 41st annual meeting of the Association of Computational Linguistics (ACL-03) conference, Sapporo, Japan, pp 113–116

  • Matusov E, Zens R, Vilar D, Mauser A, Popovic M, Ney H (2006) The RWTH machine translation system. In: Proceedings of the TC-STAR workshop on speech-to-speech translation, Barcelona, Spain, pp 31–36

  • Matusov E, Leusch G, Banchs RE, Bertoldi N, Dechelotte D, Federico M, Kolss M, Lee Y-S, Marino JB, Paulik M, Roukos S, Schwenk H, Ney H (2008) System combination for machine translation of spoken and written language. IEEE Trans Audio Speech Lang Process 16(7): 1222–1237

    Article  Google Scholar 

  • Morrissey S (2008a) Assistive technology for deaf people: translating into and animating Irish sign language. In: Proceedings of the 11th International Conference on Computers Helping People with Special Needs (ICCHP) Young Researchers’ Consortium, Linz, Austria

  • Morrissey S (2008b) Data-driven machine translation for sign languages. PhD thesis, Dublin City University, Dublin, Ireland

  • Morrissey S (2011) Assessing three representation methods for sign language machine translation and evaluation. In: Proceedings of the 15th annual meeting of the European Association for Machine Translation (EAMT 2011), Leuven, Belgium, pp 137–144

  • Morrissey S, Way A (2005) An example-based approach to translating sign language. In: Proceedings of the workshop in example-based machine translation (MT Summit X), Phuket, Thailand, pp 109–116

  • Morrissey S, Way A (2006) Lost in translation: the problems of using mainstream MT evaluation metrics for sign language translation. In: Proceedings of the 5th SALTMIL workshop on minority languages at the Languages and Resources Evaluation Conference (LREC 2006), Genoa, Italy, pp 91–98

  • Morrissey S, Way A (2007) Joining hands: developing a sign language machine translation system with and for the deaf community. In: Proceedings of the conference and workshop on assistive technologies for people with vision & hearing impairments (CVHI), Granada, Spain

  • Morrissey S, Way A, Cahill P, Carson-Berndsen J (2007a) A complete Irish sign language to speech translation system. In: IBM CASCON Dublin Symposium, Dublin, Ireland

  • Morrissey S, Way A, Stein D, Bungeroth J, Ney H (2007b) Combining data-driven MT systems for improved sign language translation. In: Proceedings of machine translation Summit XI, Copenhagen, Denmark, pp 329–336

  • Naqvi S (2007) End-user involvement in assistive technology design for the deaf–are artificial forms of sign language meeting the needs of the target audience?. In: Proceedings of the conference and workshop on assistive technologies for people with vision & hearing impairments (CVHI), Granada, Spain

  • Neidle C, Sclaroff S, Athitsos V (2001) SignStream TM: a tool for linguistic and computer vision research on visual–gestural language data. Behav Res Methods Instrum Comput 33(3): 311–320

    Article  Google Scholar 

  • Ó’Baoill D, Matthews PA (2000) The Irish Deaf Community (Volume 2): the structure of Irish sign language. The Linguistics Institute of Ireland, Dublin, Ireland

    Google Scholar 

  • Och FJ, Ney H (2000) Improved statistical alignment models. In: The 38th annual meeting of the Association for Computation Linguistics (ACL-00), Hong Kong, China, pp 440–447

  • Och F, Ney H (2003) A systematic comparision of various statistical alignment models. Comput Linguist 29(1): 19–51

    Article  MATH  Google Scholar 

  • Papineni K, Roukos S, Ward T, Zhu W (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL-02), Philadelphia, PA, pp 311–318

  • Penkale S, Haque R, Dandapat S, Banerjee P, Srivastava AK, Du J, Pecina P, Naskar SK, Forcada ML, Way A (2010) MATREX: The DCU MT System for WMT 2010. In: Proceedings of the joint fifth workshop on statistical machine Translation and MetricsMATR, Uppsala, Sweden, pp 149–154

  • Pierce J, Carroll J, Hamp E, Hays D, Hockett C, Oettinger A, Perlis A (1966) Language and machines: computers in translation and linguistics. Technical Report: Automatic Language Processing Committee, National Academy of Sciences, National Research Council. Washington, DC

  • Pinker S (1994) The language instinct: how the mind creates language. Penguin, London

    Google Scholar 

  • Pizzuto E, Rossini P, Russo T (2006) Representing signed languages in written form: questions that need to be posed. In: Proceedings of the 2nd workshop on the representation and processing of sign languages at the Languages and Resources Evaluation Conference (LREC 2006), Genoa, Italy, pp 0–13

  • Prillwitz S, Leven R, Zienert H, Hanke T, Henning J (1989) HamNoSys, Version 2.0. Hamburg notation system for sign languages: an introductory guide. Hamburg Signum, Hamburg

    Google Scholar 

  • Rabiner LR, Juang BH (1989) An introduction to hidden Markov models. IEEE ASSP Mag 4(1): 4–16

    Google Scholar 

  • Sáfár E, Marshall I (2002) The architecture of an English-Text-to-Sign-languages translation system. In: Proceedings of the international conference on Recent Advances in Natural Language Processing (RANLP-02), Tzigov Chark, Bulgaria, pp 223–228

  • San-Segundo R, Montero JM, Macias-Guarasa J, Córdoba R, Ferreiros J, Pardo JM (2007) Proposing a speech to gesture translation architecture for Spanish deaf people. J Vis Lang Comput 19:523–538

    Google Scholar 

  • San-Segundo R, López V, Martín R, Sánchez D, García A (2010) Language resources for Spanish—Spanish sign language (lse) translation. In: Proceedings of the 4th workshop on the representation and processing of sign languages: corpora and sign language technologies at LREC 2010, Valetta, Malta, pp 208–211

  • Shneiderman B (1998) Designing user interfaces. Addison Wesley, Menlo Park, CA

    Google Scholar 

  • Somers H (2006) Language engineering and the pathway to healthcare: a user-oriented view. In: HLT/NAACL-06 medical speech translation, Proceedings of the Workshop, New York, NY, pp 32–39

  • Somers H, Lovel H (2003) Computer-based support for patients with limited English. In: Association for computational linguistics EACL 2003, 10th Conference of the European Chapter, Proceedings of the 7th International EAMT Workshop on MT and other language technology tools: improving MT through other language tools, resources and tools for building MT, Budapest, Hungary, pp 41–49

  • Speers D (2001) Representation of American sign language for machine translation. PhD thesis, Georgetown University, Washington, DC

  • Stein D, Bungeroth J, Ney H (2006) The architecture of an English text-to-sign languages translation system. In: Proceedings of the 11th annual conference of the European Association for Machine Translation (EAMT, ’06), Oslo, Norway, pp 169–177

  • Stein D, Dreuw P, Ney H, Morrissey S, Way A (2007) Hand in hand: automatic sign language to English translation. In: Proceedings of the 11th international conference on Theoretical and Methodological Issues in Machine Translation (TMI-07), Skövde, Sweden, pp 214–220

  • Stein D, Schmidt C, Ney H (2010) Sign language machine translation overkill. In: Federico M, Lane I, Paul M, Yvon F (eds) International workshop on spoken language translation, Paris, France, pp 337–344

  • Stein D, Schmidt C, Ney H (2012) Analysis, preparation and optimization of statistical sign language machine translation. Mach Translat 26:325–357

    Google Scholar 

  • Stokoe WC (1960) Sign language structure: an outline of the visual communication system of the American deaf. Studies in Linguistics, Occasional Paper, 2nd printing 1993. Linstok Press, Burtonsville, MD

  • Stokoe WC (1972) Semiotics and human sign languages. Mouton & Co. N.V. Publishers, The Hague

    Google Scholar 

  • Stroppa N, Way A (2006) MaTrEx: DCU Machine translation system for IWSLT 2006. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Kyoto, Japan, pp 31–36

  • Sutton V (1995) Lessons in sign writing, textbook and workbook (2nd edn). The Center for Sutton Movement Writing Inc., La Jolla, CA

    Google Scholar 

  • Tinsley J, Ma Y, Ozdowska S, Way A (2008) MaTrEx: the DCU MT system for WMT 2008. In: Proceedings of the third workshop on statistical machine translation, ACL 2008, Colombus, OH, pp 171–174

  • Traxler C (2000) The Stanford Achievement Test, 9th Edition: national norming and performance standards for deaf and hard-of-hearing students. J Deaf Stud Deaf Educ 5(4): 337–348

    Article  Google Scholar 

  • van Slype G (1979) Critical methods for evaluating the quality of machine translation. Technical Report: BR-19142, European Commission Directorate General Scientific and Technical Information and Information Management, Bureau Marcel van Dijk, Brussells, Belgium

  • van Zijl L, Combrink A (2006) The South African sign language machine translation project: issues on non-manual sign generation. In: Proceedings of South African Institute for Computer Scientists and Information Technologists Conference (SAICSIT 06), Cape Winelands, South Africa, pp 127–134

  • van Zijl L, Olivrin G (2008) South African sign language assistive translation. In: Proceedings of IASTED international conference on assistive technologies, Baltimore, MD

  • Veale T, Conway A, Collins B (1998) The challenges of cross-modal translation: English to sign language translation in the Zardoz system. Mach Transl 13(1): 81–106

    Article  Google Scholar 

  • Vilar D, Stein D, Huck M, Ney H (2010) Jane: open source hierarchical translation, extended with reordering and Lexicon models. In: ACL 2010 joint fifth workshop on statistical Machine Translation and Metrics (MATR), Uppsala, Sweden, pp 262–270

  • Ward W (1991) Understanding spontaneous speech: the phoenix system. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, Toronto, Canada, pp 365–367

  • Weaver W (1949) Recent contributions to the mathematical theory of communication. In: The mathematical theory of communication. The University of Illinois Press, Urbana IL, pp 94–117

  • Wu C-H, Su H-Y, Chiu Y-H, Lin C-H (2007) Transfer-based statistical translation of Taiwanese sign language using PCFG. In: ACM Transactions on Asian Language Information Processing (TALIP), vol 6, No. 1

  • Zens R, Ney H (2008) Improvements in dynamic programming beam search for phrase-based statistical machine translation. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT 08), Honolulu, Hawaii, pp 195–205

  • Zhao L, Kipper K, Schuler W, Vogler C, Badler N, Palmer M (2000) A machine translation system from English to American sign language. In: Envisioning machine translation in the information future: proceedings of the fourth conference of the Association for Machine Translation (AMTA-00), Cuernavaca, Mexico, pp 293–300

Download references

Author information

Authors and Affiliations

  1. Centre for Next Generation Localisation, School of Computing, Dublin City University, Dublin, Ireland

    Sara Morrissey

  2. Lingo24, Greenfield, Greater Manchester, UK

    Andy Way

Authors
  1. Sara Morrissey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andy Way
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andy Way.

Additional information

The work described in this article was, for the most part, carried out while at Dublin City University.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Morrissey, S., Way, A. Manual labour: tackling machine translation for sign languages. Machine Translation 27, 25–64 (2013). https://doi.org/10.1007/s10590-012-9133-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10590-012-9133-1

Keywords

Search

Navigation