Incorporating Constituent Syntax into Grammatical Error Correction with Multi-Task Learning
Authors: 
Chen Chen, Bo He, Jing Yuan, Chunyan Hou, Xiaojie YuanAuthors Info & Claims
Chen Chen, Bo He, Jing Yuan, Chunyan Hou, Xiaojie YuanAuthors Info & ClaimsPages 286 - 295
Abstract
Grammatical Error Correction (GEC) is usually considered as a translation task where an erroneous sentence is treated as the source language and the corrected sentence as the target language. The state-of-the-art GEC models often adopt transformer-based sequence-to-sequence architecture of machine translation. However, most of these approaches ignore the syntactic information because the syntax of an erroneous sentence is also full of errors and not beneficial to GEC. In this paper, we propose a novel Error-Correction Constituent Parsing (ECCP) task which uses the constituent parsing of corrected sentences to avoid the harmful effect of the erroneous sentence. We also propose an architecture that includes one encoder and two decoders. There are millions of parameters in transformer-based GEC models, and the labeled training data is substantially less than synthetic pre-training data. Therefore, adapter layers are added to the proposed architecture, and adapter tuning is used for fine-tuning our model to alleviate the low-resource issue. We conduct experiments on CoNLL-2014, BEA-2019, and JFLEG test datasets in unsupervised and supervised settings. Experimental results show that our method outperforms the-state-of-art baselines and achieves superior performance on all datasets.
References
[1]
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Parallel Iterative Edit Models for Local Sequence Transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Stroudsburg, PA, USA, 4259--4269. https://doi.org/10.18653/v1/D19--1435 ISSN: 23318422.
[2]
Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 Shared Task on Grammatical Error Correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Florence, Italy, 52--75. https://doi.org/10.18653/v1/W19--4406
[3]
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) 1 (2017), 793--805. https://doi.org/10.18653/v1/P17--1074 ISBN: 9781945626753.
[4]
Rich Caruana. 1997. Multitask learning. Machine learning 28, 1 (1997), 41--75.
[5]
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. https://doi.org/10.48550/ARXIV.1312.3005
[6]
Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, and Ming Zhou. 2020. Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7162--7169.
[7]
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 568--572.
[8]
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner English: The NUS corpus of learner English. In Proceedings of the eighth workshop on innovative use of NLP for building educational applications. 22--31.
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171--4186. https://doi.org/10.18653/v1/N19--1423
[10]
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 conference on empirical methods in natural language processing. 866--874.
[11]
Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic Extraction of Learner Errors in ESL Sentences Using Linguistically Enhanced Alignments. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 825--835. https://aclanthology.org/C16--1079
[12]
Tao Ge, Furu Wei, and Ming Zhou. 2018. Fluency boost learning and inference for neural grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 1055--1065.
[13]
Sylviane Granger. 2014. The computer learner corpus: a versatile new source of data for SLA research. In Learner English on computer. Routledge, 3--18.
[14]
Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. 252--263.
[15]
Lin Gui, Jia Leng, Jiyun Zhou, Ruifeng Xu, and Yulan He. 2022. Multi Task Mutual Learning for Joint Sentiment Classification and Topic Detection. IEEE Transactions on Knowledge & Data Engineering 34, 04 (2022), 1915--1927.
[16]
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 687--697.
[17]
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of International Conference on Machine Learning. 2790--2799.
[18]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
[19]
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching Neural Grammatical Error Correction as a Low- Resource Machine Translation Task. In Proceedings of NAACL. 595--606.
[20]
Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsburg, PA, USA, 4248--4254. https://doi.org/10.18653/v1/2020.acl-main.391 ISSN: 23318422 _eprint: 2005.00987.
[21]
Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 1236--1242. https://doi.org/10.18653/v1/D19--1119
[22]
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019).
[23]
Jiquan Li, Junliang Guo, Yongxin Zhu, Xin Sheng, Deqiang Jiang, Bo Ren, and Linli Xu. 2022. Sequence-to-Action: Grammatical Error Correction with Action Guided Sequence Generation. In Proceedings of the AAAI Conference on Artificial Intelligence. 10974--10982.
[24]
Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 4582--4597. https://doi.org/10.18653/v1/2021.acl-long.353
[25]
Jared Lichtarge, Chris Alberti, and Shankar Kumar. 2020. Data weighted training strategies for grammatical error correction. Transactions of the Association for Computational Linguistics 8 (2020), 634--646.
[26]
Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019. Corpora generation for grammatical error correction. arXiv (2019), 3291--3301.
[27]
Haowen Lin, Jinlong Li, Xu Zhang, and Huanhuan Chen. 2021. Grammatical Error Correction with Dependency Distance. In Proceedings of CIKM. 1018--1027.
[28]
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).
[29]
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. 55--60. http://www.aclweb.org/anthology/P/P14/P14--5010
[30]
Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In Proceedings of 5th International Joint Conference on Natural Language Processing. 147--155.
[31]
Andrew Mutton, Mark Dras, Stephen Wan, and Robert Dale. 2007. GLEU: Automatic Evaluation of Sentence-Level Fluency. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, 344--351. https://aclanthology.org/P07--1044
[32]
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. Association for Computational Linguistics, Valencia, Spain, 229--234. https://aclanthology.org/E17--2037
[33]
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. Association for Computational Linguistics, Baltimore, Maryland, 1--14. https://doi.org/10.3115/v1/W14--1701
[34]
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR -- Grammatical Error Correction: Tag, Not Rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Stroudsburg, PA, USA, 163--170. https://doi.org/10.18653/v1/2020.bea-1.16 ISSN: 23318422 Issue: April _eprint: 2005.12592.
[35]
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In 16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021. Association for Computational Linguistics (ACL), 487--503.
[36]
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterHub: A Framework for Adapting Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations. Association for Computational Linguistics, Online, 46--54. https://www.aclweb.org/anthology/2020.emnlp-demos.7
[37]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1 (2014), 1929--1958.
[38]
Felix Stahlberg and Shankar Kumar. 2020. Seq2Edits: Sequence transduction using span-level edit operations. In EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. Association for Computational Linguistics, Stroudsburg, PA, USA, 5147--5159. https://doi.org/10.18653/v1/2020.emnlp-main.418 ISSN: 23318422 _eprint: 2009.11136.
[39]
Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021. Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding. In Proceedings of ACL-IJCNLP. 5937--5947.
[40]
Toshikazu Tajiri, Mamoru Komachi, and Yuji Matsumoto. 2012. Tense and aspect error correction for ESL learners using global context. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 198--202.
[41]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[42]
Lihao Wang and Xiaoqing Zheng. 2020. Improving grammatical error correction models with purpose-built adversarial examples. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2858--2869.
[43]
Shaolei Wang, Yue Zhang, Wanxiang Che, and Ting Liu. 2018. Joint extraction of entities and relations based on a novel graph scheme. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 4461--4467.
[44]
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 180--189.
[45]
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. LM-Critic: Language Models for Unsupervised Grammatical Error Correction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 7752--7763.
[46]
Michihiro Yasunaga and Percy Liang. 2021. Break-It-Fix-It: Unsupervised Learning for Program Repair. In International Conference on Machine Learning. PMLR, 11941--11952.
[47]
Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. 2022. A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods. arXiv preprint arXiv:2204.03508 (2022).
[48]
Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data. In Proceedings of NAACL-HLT. 156--165.
Index Terms
- Incorporating Constituent Syntax into Grammatical Error Correction with Multi-Task Learning
Recommendations
Incorporating rich syntax information in Grammatical Error Correction
AbstractSyntax parse trees are a method of representing sentence structure and are often used to provide models with syntax information and enhance downstream task performance. Because grammar and syntax are inherently linked, the ...
Grammatical Error Correction with Dependency Distance
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge ManagementGrammatical Error Correction (GEC) task is always considered as low resource machine translation task which translates a sentence in an ungrammatical language to a grammatical language. As the state-of-the-art approach to GEC task, transformer-based ...
Faster shift-reduce constituent parsing with a non-binary, bottom-up strategy
Highlights- Bottom-up shift-reduce constituent parsers require prior binarization.
- We ...
AbstractAn increasingly wide range of artificial intelligence applications rely on syntactic information to process and extract meaning from natural language text or speech, with constituent trees being one of the most widely used syntactic ...
Comments
Information & Contributors
Information
Published In
October 2023
5508 pages
ISBN:9798400701245
DOI:10.1145/3583780
- General Chairs:
- Ingo Frommholz,
- Frank Hopfgartner,
- Mark Lee,
- Michael Oakes,
- Program Chairs:
- Mounia Lalmas,
- Min Zhang,
- Rodrygo Santos
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- the Fundamental Research Funds for the Central University Nankai University
- National Natural Science Foundation of China
- NSFC-General Technology Joint Fund for Basic Research
Conference
CIKM '23: The 32nd ACM International Conference on Information and Knowledge Management
October 21 - 25, 2023
Birmingham, United Kingdom
Acceptance Rates
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 138Total Downloads
- Downloads (Last 12 months)45
- Downloads (Last 6 weeks)0
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in