Skip to main content
Springer Nature Link
Log in

HTRDP evaluations on Chinese information processing and intelligent human-machine interface

  • Survey Article
  • Published:
Frontiers of Computer Science in China Aims and scope Submit manuscript

Abstract

From 1991 to 2005, China’s High Technology Research and Development Program (HTRDP) sponsored a series of technology evaluations on Chinese information processing and intelligent human-machine interface, which is called HTRDP evaluations, or “863” evaluations in brief. This paper introduces the HTRDP evaluations in detail. The general information of the HTRDP evaluation is presented first, including the history, the concerned technology categories, the organizer, the participants, and the procedure, etc. Then the evaluations on each technology are described in detail respectively, covering Chinese word segmentation, machine translation, acoustic speech recognition, text to speech, text summarization, text categorization, information retrieval, character recognition, and face detection and recognition. For the evaluations on each technology categories, the history, the evaluation tasks, the data, the evaluation method, etc., are given. The last section concludes the paper and discusses possible future work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Sproat R, Emerson T. The First International Chinese Word Segmentation Bakeoff. In: Proceedings of the Second SIGHAN Workshop on Chinese Language Processing. 2003, 133–143

  2. Levow G-A, The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition. In: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing. 2006, 108–117

  3. Emerson T. The Second International Chinese Word Segmentation Bakeoff. In: Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, 2005. 123–133

  4. ACE evaluations: http://www.nist.gov/speech/tests/ace/index.htm

  5. YU S W. Automatic Evaluation of Output Quality for Machine Translation Systems, Machine Translation. Netherlands: Kluwer Academic publisher, 1993, 8: 117–126

    Google Scholar 

  6. Papineni K A, Roukos S, Ward T, et al. Bleu: a method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022). IBM Research Division, Thomas J. Watson Research Center. 2001

  7. Zhang H P, Yu H K, Xiong D Y, et al. HHMM-based Chinese Lexical Analyzer ICTCLAS, In: Proceedings of 2nd SigHan Workshop, 2003. 184–187

  8. http://www.nist.gov/speech/tests/mt

  9. http://nlp.cs.nyu.edu/GTM/

  10. Och F J. Minimum error rate training in statistical machine translation. In: Proceedings of the 41st ACL, Sapporo, Japan, 2003. 160–167

  11. Och F J. Statistical Machine Translation: From Single-Word Models to Alignment Templates. 38–39

  12. Yasuhiro A, et al. Overview of the IWSLT04 Evaluation Campaign. 2004

  13. Paul M, Nakaiwa H, Federico M. Towards innovative evaluation methodologies for speech translation. Working Notes of the NTCIR-4 2004 Meeting. 2004, 2(Suppl). 17–21

    Google Scholar 

  14. White J S, O’Connell T, O’Mara F. The ARPA MT evaluation methodologies: evolution, lessons, and future approaches. In: Proceedings of the AMTA, 1994, 193–205

  15. Papineni K, Roukos S, Ward T. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th ACL, Philadelphia, USA, 2002. 311–318

  16. Doddington G. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: Proceedings of the HLT 2002, San Diego, USA, 2002. 257–258

  17. Turian J P, Shen L, Melamed I D. Evaluation of machine translation and its evaluation. In Proceedings of the MT Summit IX, New Orleans, USA, 2003. 386–393

  18. Liu Q, Liu Y. Machine translation automatic evaluating method and system thereof, Chinese Patent, CN1641631-A

  19. http://www.nist.gov/speech/history/index.htm

  20. http://www.tc-star.org/

  21. Martin A, Doddington TKG, Ordowski M, et al. The DET curve in assessment of detection task performance. In: Proceedings of EuroSpeech’97. 1997, Vol.4, 1895–1898

    Google Scholar 

  22. Van Santen J P H, Pols L C W, Abe M, et al. Report on the Third ESCA TTS Workshop Evaluation Procedure. Third ESCA TTS Workshop, 1998.

  23. http://www.tc-star.org/

  24. Benoit C, Grice M, Hazan V. The SUS test: a method for the assessment of text-to-speech synthesis intelligibility using semantically unpredictable sentences. Speech Commun. 1996, 18:381–392

    Article  Google Scholar 

  25. Document Understanding Conference, http://duc.nist.gov/

  26. He J, Tan A-H, Tan C-L. A Comparative Study on Chinese Text Categorization Methods. PRICAI 2000 Workshop on Text and Web Mining, 2000.

  27. Lewis D D, Yang Y M, Rose T G, et al. RCV1 A New Benchmark Collection for Text Categorization Research. Journal of Machine Learning Research. 2004, 5: 361–397

    Google Scholar 

  28. Sebastiani F. Machine Learning in Automated Text Categorization. ACM Computing Surveys, 2002

  29. Aas K, Eikvil L. Text Categorisation: A survey. Raport NR 941, 1999

  30. Yang Y M. An evaluation of statistical approaches to text categorization. Information Retrieval, 1999, 1(1–2)

  31. The Text Retrieval Conference, http://trec.nist.gov/

  32. Harman D. The first text retrieval conference (TREC-1). Information Processing and Management, 1993, 29(4): 411–414

    Article  MathSciNet  Google Scholar 

  33. Voorhees E M. Overview of TREC 2005. In: Proceedings of the Text REtrieval Conference (TREC). Gaithersburg, Maryland, 2005

  34. NII Test Collection for IR Systems, http://research.nii.ac.jp/ntcir/

  35. Cross Language Evaluation Forum, http://www.clef-campaign.org/

  36. Zhang J L, et al. Research on the 863 Chinese Information Retrieval Evaluation. Journal of Chinese Information Processing, 2006, 20(suppl): 19–24(in Chinese)

    Google Scholar 

  37. Shah C, Croft W B. Evaluating High Accuracy Retrieval Techniques. In: Proceedings of SIGIR’ 04, 2004

  38. Text Retrieval Conference, http://trec.nist.gov.

  39. Chinese Web Information Retrieval Forum, http://www.cwirf.org/.

  40. Cheng Y X, et al. 863 Web Track Experiments at ICST-PKU. Journal of Chinese Information Processing, 2006, 20(suppl): 102–106(in Chinese)

    Article  Google Scholar 

  41. Zhao L, et al. 2005 THUIR Report for 863 Information Retrieval Evaluation. Journal of Chinese Information Processing, 2006, (suppl): 91–95 (in Chinese)

  42. Zhang Z C, et al. Technology Report of HIT-IRLab for Evaluation 2005 of 863 Information Retrieval. Journal of Chinese Information Processing, 2006, 20(suppl): 83–90 (in Chinese)

    Google Scholar 

  43. Xu W R, et al. PRIS Information Retrieval System Report. Journal of Chinese Information Processing, 2006, 20(suppl): 96–101(in Chinese)

    Google Scholar 

  44. Lv B B, et al. 863 Information Retrieval Evaluation-Institute of Automation. Journal of Chinese Information Processing, 2006, 20(suppl): 78–82 (in Chinese)

    Google Scholar 

  45. Kanungo T, Marton G A, Bulbul O. Performance Evaluation of Two Arabic OCR Products. In: Proceedings of AIPR Workshop on Advances in Computer. Assist Recognition, SPIE. Vol 3584

  46. Liu C L, Jaeger S, Nakagawa M. Online Recognition of Chinese Characters: The State-of-the-Art. IEEE Transaction on Patten Analysis and Machine Intelligence, 2004, 26(2): 198–213

    Article  Google Scholar 

  47. Marti U-V, Bunke H. A full English sentence database for off-line handwriting recognition. In: Proceedings of the 5th Int. Conf. on Document Analysis and Recognition, Bangalore, India, 1999. 705–708

  48. Liu C P, Qian Y L, et al. 863 Testing System on Handwritten Chinese Character Recognition. Journal of Chinese Information Processing, 2000, 14(2): 2–7 (in Chinese)

    Google Scholar 

  49. Guo J, Lin Z Q, Zhang H G. A New Database Model of Off-line Handwritten Chinese Characters and Its Applications. Chinese Journal of Electronics, 2000, 28(5): 115–116 (in chinese)

    Google Scholar 

  50. Requirements and test procedure of on-line handwriting Chinese ideogram recognition. Chinese National Standard GB/T 18790-2002. July, 2002

  51. Chinese Ideograms Coded Character Set for Information Interchange-Basic Set. Chinese National Standard GB 2312-1980, 1980

  52. Information technology-Chinese ideograms coded character set for information interchange-Extension for the basic set. Chinese National Standard GB 18030-2000. March, 2000

  53. Chellappa R, et al. Human and Machine Recognition of Faces: A Survey. In: Proceedings of the IEEE, 1995, 83(5): 705–741

    Article  Google Scholar 

  54. Zhao W Y, Chellappa R, Rosenfeld A, et al. Face Recognition: A Literature Survey. ACM Computing Survey, 2003, 35(4): 399–458

    Article  Google Scholar 

  55. Phillips P J, Moon H, Rizvi S, et al. The FERET Evaluation Methodology for Face-Recognition Algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(10): 1090–1104

    Article  Google Scholar 

  56. Phillips P J, Grother P J, et al. Face Recognition Vendor Test 2002: Evaluation Report, Technical Report. NISTIR 6965, National Institute of Standards and Technology, 2003

Download references

Author information

Authors and Affiliations

  1. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100080, China

    Liu Qun, Wang Xiangdong, Liu Hong, Tang Sheng, Xiong Deyi, Hou Hongxu, Lin Shouxun & Qian Yueliang

  2. Institute of Software, Chinese Academy of Sciences, Beijing, 100085, China

    Sun Le, Lv Yuanhua & Li Wenbo

  3. Graduate University of Chinese Academy of Sciences, Beijing, 100085, China

    Wang Xiangdong, Xiong Deyi & Hou Hongxu

Authors
  1. Liu Qun
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Wang Xiangdong
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Liu Hong
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Sun Le
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Tang Sheng
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Xiong Deyi
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Hou Hongxu
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Lv Yuanhua
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Li Wenbo
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Lin Shouxun
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Qian Yueliang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Liu Qun.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liu, Q., Wang, X., Liu, H. et al. HTRDP evaluations on Chinese information processing and intelligent human-machine interface. Front. Comput. Sc. China 1, 58–93 (2007). https://doi.org/10.1007/s11704-007-0007-2

Download citation

  • Received:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-007-0007-2

Keywords