Abstract
To date, the number of studies that address the generalization of argument models is still relatively small. In this study, we extend our stacking model from argument identification to an argument unit classification task. Using this model, and for each of the learned tasks, we address three real-world scenarios concerning the model robustness over multiple datasets, different domains and topics. Consequently, we first compare single-datset learning (SDL) with multi-dataset learning (MDL). Second, we examine the model generalization over completely unseen dataset in our cross-domain experiments. Third, we study the effect of sample and topic sizes on the model performance in our cross-topic experiments. We conclude that, in most cases, the ensemble learning stacking approach is more stable over the generalization tests than a transfer learning DistilBERT model. In addition, the argument identification task seems to be easier to generalize across shifted domains than argument unit classification. This work aims at filling the gap between computational argumentation and applied machine learning with regard to the model generalization.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baker, A.: Simplicity, the Stanford Encyclopedia of Philosophy. Metaphysics Research Lab (2016)
Lawrence, J., Reed, C.: Argument mining: a survey. Comput. Linguist. 45(4), 765–818 (2020)
Palau, R.M., Moens, M.F.: Argumentation mining: the detection, classification and structure of arguments in text. In: Proceedings of the 12th International Conference on Artificial Intelligence and Law, pp. 98–107 (2009)
Toulmin, S.E.: The Uses of Argument. Cambridge University Press, Cambridge (2003)
Stab, C., Gurevych, I.: Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics. Technical papers, pp. 1501–1510 (2014)
Song, Y., Heilman, M., Klebanov, B.B., Deane, P.: Applying argumentation schemes for essay scoring. In: Proceedings of the First Workshop on Argumentation Mining, pp. 69–78 (2014)
Samadi, M., Talukdar, P., Veloso, M., Blum, M.: Claimeval: integrated and flexible framework for claim evaluation using credibility of sources. In: Thirtieth AAAI Conference on Artificial Intelligence (2016)
Alhamzeh, A., Bouhaouel, M., Egyed-Zsigmond, E., Mitrovic, J.: Distilbert-based argumentation retrieval for answering comparative questions. In: Working Notes of CLEF (2021)
Al-Khatib, K., Wachsmuth, H., Hagen, M., Köhler, J., Stein, B.: Cross-domain mining of argumentative text through distant supervision. In: Proceedings of NAACL-HLT, pp. 1395–1404 (2016)
McCoy, R.T., Min, J., Linzen, T.: BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. In: Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 217–227. Association for Computational Linguistics (2020)
Alhamzeh, A., Bouhaouel, M., Egyed-Zsigmond, E., Mitrović, J., Brunie, L., Kosch, H.: A stacking approach for cross-domain argument identification. In: Strauss, C., Kotsis, G., Tjoa, A.M., Khalil, I. (eds.) DEXA 2021. LNCS, vol. 12923, pp. 361–373. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86472-9_33
Liga, D., Palmirani, M.: Transfer learning with sentence embeddings for argumentative evidence classification (2020)
Wambsganss, T., Molyndris, N., Söllner, M.: Unlocking transfer learning in argumentation mining: a domain-independent modelling approach. In: 15th International Conference on Wirtschaftsinformatik (2020)
Zhang, W.E., Sheng, Q.Z., Alhazmi, A., Li, C.: Adversarial attacks on deep-learning models in natural language processing: a survey. ACM Trans. Intell. Syst. Technol. (TIST) 11(3), 1–41 (2020)
Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167–7176 (2017)
Schiller, B., Daxenberger, J., Gurevych, I.: Stance detection benchmark: how robust is your stance detection? KI - Künstl. Intell. 35(3), 329–341 (2021). https://doi.org/10.1007/s13218-021-00714-w
Ajjour, Y., Chen, W.F., Kiesel, J., Wachsmuth, H., Stein, B.: Unit segmentation of argumentative texts. In: Proceedings of the 4th Workshop on Argument Mining, pp. 118–128 (2017)
Al Khatib, K., Wachsmuth, H., Kiesel, J., Hagen, M., Stein, B.: A news editorial corpus for mining argumentation strategies. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 3433–3443 (2016)
Habernal, I., Gurevych, I.: Argumentation mining in user-generated web discourse. Comput. Linguist. 43(1), 125–179 (2017)
Bouslama, R., Ayachi, R., Amor, N.B.: Using convolutional neural network in cross-domain argumentation mining framework. In: Ben Amor, N., Quost, B., Theobald, M. (eds.) SUM 2019. LNCS (LNAI), vol. 11940, pp. 355–367. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-35514-2_26
Elangovan, A., He, J., Verspoor, K.: Memorization vs. generalization: quantifying data leakage in NLP performance evaluation. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1325–1335 (2021)
Huan, X., Mannor, S.: Robustness and generalization. Mach. Learn. 86(3), 391–423 (2012). https://doi.org/10.1007/s10994-011-5268-1
Wang, J.: Generalizing to unseen domains: a survey on domain generalization. IEEE Trans. Knowl. Data Eng. (2022)
Stab, C., Gurevych, I.: Parsing argumentation structures in persuasive essays. Comput. Linguist. 43(3), 619–659 (2017)
Stab, C.: Argumentative Writing Support by Means of Natural Language Processing, p. 208 (2017)
Aharoni, E.: A benchmark dataset for automatic detection of claims and evidence in the context of controversial topics. In: Proceedings of the First Workshop on Argumentation Mining, pp. 64–68 (2014)
Sagi, O., Rokach, L.: Ensemble learning: a survey. Wiley Interdis. Rev.: Data Min. Knowl. Discovery 8(4), e1249 (2018)
Moens, M.F., Boiy, E., Palau, R.M., Reed, C.: Automatic detection of arguments in legal texts. In: Proceedings of the 11th International Conference on Artificial Intelligence and Law, pp. 225–230 (2007)
Stab, C., Gurevych, I.: Identifying argumentative discourse structures in persuasive essays. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 46–56 (2014)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint. arXiv:1910.01108 (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint. arXiv:1810.04805 (2018)
Ryu, M., Lee, K.: Knowledge distillation for bert unsupervised domain adaptation. arXiv preprint. arXiv:2010.11478 (2020)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint. arXiv:1711.05101 (2017)
De Winter, J.C.F.: Using the student’s t-test with extremely small sample sizes. Pract. Assess. Res. Eval. 18(1), 10 (2013)
Zhang, Z., et al.: Semantics-aware BERT for language understanding. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 9628–9635 (2020)
Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in bertology: what we know about how bert works. Trans. Assoc. Comput. Linguist. 8, 842–866 (2020)
Acknowledgements
The project on which this report is based was partly funded by the German Federal Ministry of Education and Research (BMBF) under the funding code 01|S20049. The author is responsible for the content of this publication.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer-Verlag GmbH Germany, part of Springer Nature
About this chapter
Cite this chapter
Alhamzeh, A. et al. (2022). Empirical Study of the Model Generalization for Argument Mining in Cross-Domain and Cross-Topic Settings. In: Hameurlain, A., Tjoa, A.M. (eds) Transactions on Large-Scale Data- and Knowledge-Centered Systems LII. Lecture Notes in Computer Science(), vol 13470. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-66146-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-662-66146-8_5
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-66145-1
Online ISBN: 978-3-662-66146-8
eBook Packages: Computer ScienceComputer Science (R0)