Skip to main content
SpringerLink
Log in

BIT-Event at NLPCC-2021 Task 3: Subevent Identification via Adversarial Training

  • Conference paper
  • First Online:
Natural Language Processing and Chinese Computing (NLPCC 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13029))

Abstract

This paper describes the system proposed by the BIT-Event team for NLPCC 2021 shared task on Subevent Identification. The task includes two settings, and these settings face less reliable labeled data and the dilemma about selecting the most valid data to annotate, respectively. Without the luxury of training data, we propose a hybrid system based on semi-supervised algorithms to enhance the performance by effectively learning from a large amount of unlabeled corpus. In this hybrid model, we first fine-tune the pre-trained model to adapt it to the training data scenario. Besides, Adversarial Training and Virtual Adversarial Training are combined to enhance the effect of a single model with unlabeled in-domain data. The additional information is further captured via retraining using pseudo-labels. On the other hand, we apply Active Learning as an iterative process that starts from a small number of labeled seeding instances. The experimental results suggest that the semi-supervised methods fit the low-resource subevent identification problem well. Our best results were obtained by an ensemble of these methods. According to the official results, our approach proved the best for all the settings in this task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://weibo.com.

  2. 2.

    https://github.com/huggingface/transformers.

  3. 3.

    https://github.com/dbiir/UER-py.

References

  1. Glavas, G., Snajder, J.: Constructing coherent event hierarchies from news stories. In: Proceedings of the 9th Workshop on Graph-based Methods for Natural Language Processing, pp. 34–38 (2014). https://doi.org/10.3115/v1/w14-3705

  2. Narayanan, S., Harabagiu, S.M.: Question answering based on semantic structures. In: Proceedings of the 20th International Conference on Computational Linguistics (2004)

    Google Scholar 

  3. de Marneffe, M., Rafferty, A.N., Manning, C.D.: Finding contradictions in text. In: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics, pp. 1039–1047 (2008)

    Google Scholar 

  4. Reimers, N., Schiller, B., Beck, T., Daxenberger, J., Stab, C., Gurevych, I.: Classification and clustering of arguments with contextualized word embeddings. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 567–578 (2019). https://doi.org/10.18653/v1/p19-1054

  5. Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune BERT for text classification? In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds.) Proceedings of the Chinese Computational Linguistics - 18th China National Conference. Lecture Notes in Computer Science, vol. 11856, pp. 194–206 (2019). https://doi.org/10.1007/978-3-030-32381-3_16

  6. Chi, Z., et al.: InfoXLM: an information-theoretic framework for cross-lingual language model pre-training. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3576–3588 (2021)

    Google Scholar 

  7. Chi, Z., Dong, L., Zheng, B., Huang, S., Mao, X., Huang, H., Wei, F.: Improving pretrained cross-lingual language models via self-labeled word alignment. CoRR abs/2106.06381 (2021)

    Google Scholar 

  8. Chi, Z., Dong, L., Wei, F., Wang, W., Mao, X., Huang, H.: Cross-lingual natural language generation via pre-training. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence. pp. 7570–7577 (2020)

    Google Scholar 

  9. Settles, B., Craven, M.: An analysis of active learning strategies for sequence labeling tasks. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 1070–1079 (2008)

    Google Scholar 

  10. Marcheggiani, D., Artières, T.: An experimental comparison of active learning strategies for partially labeled sequences. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 898–906 (2014). https://doi.org/10.3115/v1/d14-1097

  11. Shen, Y., Yun, H., Lipton, Z.C., Kronrod, Y., Anandkumar, A.: Deep active learning for named entity recognition. In: Proceedings of the 6th International Conference on Learning Representations (2018)

    Google Scholar 

  12. Shelmanov, A., et al.: Active learning for sequence tagging with deep pre-trained models and Bayesian uncertainty estimates. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1698–1712. Association for Computational Linguistics (2021)

    Google Scholar 

  13. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019). https://doi.org/10.18653/v1/n19-1423

  14. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019)

    Google Scholar 

  15. Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., Hu, G.: Revisiting pre-trained models for Chinese natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 657–668 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.58

  16. Yoo, D., Kweon, I.S.: Learning loss for active learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 93–102 (2019). https://doi.org/10.1109/CVPR.2019.00018

  17. Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., Liu, J.: FreeLB: enhanced adversarial training for natural language understanding. In: Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020 (2020)

    Google Scholar 

  18. Araki, J., Liu, Z., Hovy, E.H., Mitamura, T.: Detecting subevent structure for event coreference resolution. In: Proceedings of LREC, pp. 4553–4558 (2014)

    Google Scholar 

  19. Aldawsari, M., Finlayson, M.A.: Detecting subevents using discourse and narrative features. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 4780–4790 (2019). https://doi.org/10.18653/v1/p19-1471

  20. Badgett, A., Huang, R.: Extracting subevents via an effective two-phase approach. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 906–911 (2016). https://doi.org/10.18653/v1/d16-1088

  21. Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., Choi, Y.: COMET: commonsense transformers for automatic knowledge graph construction. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 4762–4779 (2019). https://doi.org/10.18653/v1/p19-1470

  22. Sap, M., et al.: ATOMIC: an atlas of machine commonsense for if-then reasoning. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pp. 3027–3035 (2019). https://doi.org/10.1609/aaai.v33i01.33013027

  23. Sun, Y., et al.: ERNIE 2.0: a continual pre-training framework for language understanding. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence, pp. 8968–8975 (2020)

    Google Scholar 

  24. Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems 32, pp. 5754–5764 (2019)

    Google Scholar 

  25. Jia, R., Liang, P.: Adversarial examples for evaluating reading comprehension systems. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2021–2031 (2017). https://doi.org/10.18653/v1/d17-1215

  26. Zhao, Z., Dua, D., Singh, S.: Generating natural adversarial examples. In: Proceedings of the 6th International Conference on Learning Representations (2018)

    Google Scholar 

  27. Belinkov, Y., Bisk, Y.: Synthetic and natural noise both break neural machine translation. In: Proceedings of the 6th International Conference on Learning Representations (2018)

    Google Scholar 

  28. Iyyer, M., Wieting, J., Gimpel, K., Zettlemoyer, L.: Adversarial example generation with syntactically controlled paraphrase networks. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 1875–1885 (2018). https://doi.org/10.18653/v1/n18-1170

  29. Ebrahimi, J., Rao, A., Lowd, D., Dou, D.: HotFlip: white-box adversarial examples for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 31–36 (2018). https://doi.org/10.18653/v1/P18-2006

  30. Ribeiro, M.T., Singh, S., Guestrin, C.: Semantically equivalent adversarial rules for debugging NLP models. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 856–865 (2018). https://doi.org/10.18653/v1/P18-1079

  31. Cheng, Y., Jiang, L., Macherey, W.: Robust neural machine translation with doubly adversarial inputs. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 4324–4333 (2019). https://doi.org/10.18653/v1/p19-1425

  32. Chapelle, O., Schölkopf, B., Zien, A. (eds.): Semi-Supervised Learning. The MIT Press, Cambridge (2006). https://doi.org/10.7551/mitpress/9780262033589.001.0001

  33. Liu, B., Wu, Z., Hu, H., Lin, S.: Deep metric transfer for label propagation with limited annotated data. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshops, pp. 1317–1326 (2019). https://doi.org/10.1109/ICCVW.2019.00167

  34. Kingma, D.P., Mohamed, S., Rezende, D.J., Welling, M.: Semi-supervised learning with deep generative models. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 3581–3589 (2014)

    Google Scholar 

  35. Pu, Y., et al.: Variational autoencoder for deep learning of images, labels and captions. In: Advances in Neural Information Processing Systems 29, pp. 2352–2360 (2016)

    Google Scholar 

  36. Miyato, T., Maeda, S., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1979–1993 (2019). https://doi.org/10.1109/TPAMI.2018.2858821

    Article  Google Scholar 

Download references

Acknowledgement

This work was supported by the Funds of the Integrated Application Software Project. We would like to thank the anonymous reviewers for their thoughtful and constructive comments. And We would like to thank Zewen Chi for his insightful comments to the improvement in technical contents and paper presentation.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China

    Xiao Liu, Bo Wang, Changsen Yuan, Heyan Huang & Chong Feng

  2. Faculty of Information Technology, Beijing University of Technology, Beijing, China

    Ge Shi & Lifang Wu

  3. Southeast Academy of Information Technology, Beijing Institute of Technology, Beijing, China

    Xiao Liu, Bo Wang, Changsen Yuan, Heyan Huang & Chong Feng

Authors
  1. Xiao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ge Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Changsen Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Heyan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chong Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lifang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Heyan Huang .

Editor information

Editors and Affiliations

  1. University of Michigan, Ann Arbor, MI, USA

    Lu Wang

  2. Peking University, Beijing, China

    Yansong Feng

  3. Soochow University, Suzhou, China

    Yu Hong

  4. Tianjin University, Tianjin, China

    Ruifang He

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, X. et al. (2021). BIT-Event at NLPCC-2021 Task 3: Subevent Identification via Adversarial Training. In: Wang, L., Feng, Y., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2021. Lecture Notes in Computer Science(), vol 13029. Springer, Cham. https://doi.org/10.1007/978-3-030-88483-3_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-88483-3_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88482-6

  • Online ISBN: 978-3-030-88483-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation