Abstract
Initiated in conjunction with DARPA’s low resource languages for emergent incidents (LORELEI) Program, the NIST LoReHLT (Low Resource Human Language Technology) evaluation series seeks to incubate research on fundamental natural language processing tasks in under-resourced languages. While part of the LORELEI program, LoReHLT is an open evaluation workshop that anyone may participate in, with its first evaluation taking place in July 2016. Eight teams, out of the 21 teams that registered, participated in the evaluation over three tasks—machine translation, named entity recognition, and situation frame—in the surprise language Uyghur.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Annotation guidelines for Lorelei situation frames v2.2 (2016). https://www.ldc.upenn.edu
Broad agency announcement low resource languages for emergent incidents (LORELEI) (2015). https://www.fbo.gov
Doddington G (2002) In: Proceedings of the second international conference on human language technology research, HLT ’02. Morgan Kaufmann Publishers Inc., San Francisco, pp 138–145. http://dl.acm.org/citation.cfm?id=1289189.1289273
Entity disambiguation evaluation and error analysis tool. https://github.com/wikilinks/neleval
Ji H (2015) NIST TAC knowledge base population (KBP2015) entity discovery and linking track. http://nlp.cs.rpi.edu/paper/kbp2015.pdf
Lavie A, Agarwal A (2007) In: Proceedings of the second workshop on statistical machine translation, StatMT ’07. Association for Computational Linguistics, Stroudsburg, pp 228–231. http://dl.acm.org/citation.cfm?id=1626355.1626389
Lorehlt translation guidelines v1.1 (2015). https://www.ldc.upenn.edu
Machine translation bleu/nist scoring script v14 (2016). http://nist.gov/itl/iad/mig/lorehlt16.cfm
Meteor automatic machine translation evaluation system v1.5. https://www.cs.cmu.edu/~alavie/METEOR/index.html
NIST (2009) Open machine translation evaluation (mt09) official release of results. http://www.itl.nist.gov/iad/mig/tests/mt/2009/ResultsRelease/currentUrdu.html
NIST (2015) Machine translation evaluation official results. ftp://jaguar.ncsl.nist.gov/mt/mt2015/openmt15results.html
NIST lorehlt 2016 evaluation plan (2016). http://www.nist.gov/itl/iad/mig/lorehlt16.cfm
Papineni K, Roukos S, Ward T, Zhu WJ (2002) In: Proceedings of the 40th annual meeting on association for computational linguistics, ACL ’02. Association for Computational Linguistics, Stroudsburg, pp 311–318. doi:10.3115/1073083.1073135
Simple named entity guidelines year 1 language (il3) v1.3 (2016). https://www.ldc.upenn.edu
Acknowledgements
These results are not to be construed or represented as endorsements of any participants system, methods, or commercial product, or as official findings on the part of NIST or the U.S. Government. Certain commercial equipment, instruments, software, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the equipment, instruments, software or materials are necessarily the best available for the purpose.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tong, A., Diduch, L., Fiscus, J. et al. Overview of the NIST 2016 LoReHLT evaluation. Machine Translation 32, 11–30 (2018). https://doi.org/10.1007/s10590-017-9200-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10590-017-9200-8