Skip to main content
SpringerLink
Log in

English Cloze Test Based on BERT

  • Conference paper
  • First Online:
Knowledge Science, Engineering and Management (KSEM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12816))

  • 1914 Accesses

Abstract

Cloze test is a common test in language examinations. It is also a research direction of natural language processing, which is an important field of artificial intelligence. In general, some words in a complete article are hidden, and several candidates are given to let the student choose the correct hidden word. To explore whether machine can do cloze test, we have done some research to build down-stream tasks of BERT for cloze test. In this paper, we consider the compound words in articles and make an improvement to help the model handling these kind of words. The experimental results show that our model performs well on questions of compound words and has better accuracy on CLOTH dataset.

Supported by National Key R&D Program of China (No. 2018YFB 1403400).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  2. Zaremba, W.; Sutskever, I., Vinyals, O.: Recurrent Neural Network Regularization. arxiv:1409.2329 (2014)

  3. Vaswani, A.: Attention is all you need, In: Guyon, I. (ed.) Advances in Neural Information Processing Systems 30, Curran Associates Inc, pp. 5998–6008 (2017)

    Google Scholar 

  4. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arxiv:1810.04805 (2018)

  5. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  6. Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.H.: RACE: Large-scale ReAding comprehension dataset from examinations. In: Martha, P., Rebecca, H., Sebastian, R. (eds.) EMNLP, Association for Computational Linguistics, pp. 785–794 (2017)

    Google Scholar 

  7. Xie, Q., Lai, G., Dai, Z., Hovy, E.H.: Large-scale Cloze Test Dataset Designed by Teachers, CoRR arXiv:1711.03225 (2017)

  8. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  9. Cho, K.: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (2014)

    Google Scholar 

  10. Chen, D., Bolton, J., Manning, C.D.: A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task. In: ACL (1), The Association for Computer Linguistics (2016)

    Google Scholar 

  11. Taylor, W.L.: cloze procedure: a new tool for measuring readability. Journalism Bull. 30(4), 415–433 (1953)

    Article  Google Scholar 

  12. Fotos, S.S.: The cloze test as an integrative measure of efl proficiency: a substitute for essays on college entrance examinations? Lang. Learn. 41(3), 313–336 (1991)

    Article  Google Scholar 

  13. Hermann, K.M.: Teaching machines to read and comprehend. In: NIPS (2015)

    Google Scholar 

  14. Hill, F., Bordes, A., Chopra, S., Weston, J.: The goldilocks principle: reading children’s books with explicit memory representations. In: Bengio, Y., LeCun, Y. (ed.) ICLR (2016)

    Google Scholar 

  15. Paperno, D.: The LAMBADA dataset: word prediction requiring a broad discourse context. In: ACL (1), The Association for Computer Linguistics (2016)

    Google Scholar 

  16. Shibuki, H.: Overview of the NTCIR-11 QA-Lab Task. In: Kando, N., Joho, H., Kishida, K. (ed.) NTCIR, National Institute of Informatics (NII) (2014)

    Google Scholar 

  17. Clark, P.: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, CoRR arXiv:1803.05457 (2018)

  18. Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learning to Align and Translate, arxiv:1409.0473Comment (2014) Accepted at ICLR 2015 as oral presentation

  19. Wang, L., et al.: Multi-Perspective Context Aggregation for Semi-supervised Cloze-style Reading Comprehension, CoRR arXiv:1808.06289 (2018)

  20. McCann, B., Bradbury, J., Xiong, C., Socher, R.: Learned in translation: contextualized word vectors. In: Guyon, I. et al. (eds.) NIPS, pp. 6297–6308 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory of Computer Software Testing and Evaluating, Shanghai Development Center of Computer Software Technology, Shanghai, China

    Minjie Ding, Mingang Chen, Wenjie Chen & Lizhi Cai

Authors
  1. Minjie Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenjie Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lizhi Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingang Chen .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Han Qiu

  2. Ibaraki University, Hitachi, Japan

    Cheng Zhang

  3. University of Kentucky, Lexington, KY, USA

    Zongming Fei

  4. Texas A&M University – Commerce, Commerce, TX, USA

    Meikang Qiu

  5. Princeton University, Princeton, NJ, USA

    Sun-Yuan Kung

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ding, M., Chen, M., Chen, W., Cai, L. (2021). English Cloze Test Based on BERT. In: Qiu, H., Zhang, C., Fei, Z., Qiu, M., Kung, SY. (eds) Knowledge Science, Engineering and Management . KSEM 2021. Lecture Notes in Computer Science(), vol 12816. Springer, Cham. https://doi.org/10.1007/978-3-030-82147-0_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82147-0_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82146-3

  • Online ISBN: 978-3-030-82147-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation