skip to main content
10.1145/3661725.3661729acmotherconferencesArticle/Chapter ViewAbstractPublication PagescmldsConference Proceedingsconference-collections
research-article

Adversarial Analysis and Methods for Math Word Problems

Published: 20 June 2024 Publication History

Abstract

Present-day state of the art models can perform well on most language tasks. Math word problems are at the intersection of linguistic semantics and quantitative logic. Two salient state of the art methods to solve math word problems are evaluated against adversarial examples employing extraneous information, associative reordering and defined relationships. The degradation in models’ performance is presented and analyzed in detail. Additionally, proposed methods using quantity cell filtering and semantic mapping are evaluated against adversarial examples. The severe 30%+ degradation in performance and modest improvements using mitigation methods establish a strong need to both build bigger datasets as well as models that can more robustly handle adversarial inputs.

References

[1]
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. arxiv:1905.13319 [cs.CL]
[2]
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-Box Adversarial Examples for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 31–36. https://doi.org/10.18653/v1/P18-2006
[3]
Ruifang Ge and Raymond Mooney. 2005. A Statistical Semantic Parser that Integrates Syntax and Semantics. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005). Association for Computational Linguistics, Ann Arbor, Michigan, 9–16. https://aclanthology.org/W05-0602
[4]
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to Solve Arithmetic Word Problems with Verb Categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 523–533. https://doi.org/10.3115/v1/D14-1058
[5]
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A Math Word Problem Repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, 1152–1157. https://doi.org/10.18653/v1/N16-1136
[6]
Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. 2021. Adversarial Examples for Evaluating Math Word Problem Solvers. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, Punta Cana, Dominican Republic, 2705–2712. https://doi.org/10.18653/v1/2021.findings-emnlp.230
[7]
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to Automatically Solve Algebra Word Problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Baltimore, Maryland, 271–281. https://doi.org/10.3115/v1/P14-1026
[8]
Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. 2021. MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving. https://doi.org/10.48550/ARXIV.2107.13435
[9]
Christian Liguda and Thies Pfeiffer. 2012. Modeling Math Word Problems with Augmented Semantic Networks. In NLDB.
[10]
Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 975–984. https://doi.org/10.18653/v1/2020.acl-main.92
[11]
NSB. [n. d.]. NSB Science Bowl Middle-School Questions. https://science.osti.gov/wdts/nsb/Regional-Competitions/Resources/MS-Sample-Questions
[12]
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP Models really able to Solve Simple Math Word Problems?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 2080–2094. https://doi.org/10.18653/v1/2021.naacl-main.168
[13]
Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically Solving Number Word Problems by Semantic Parsing and Reasoning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, 1132–1142. https://doi.org/10.18653/v1/D15-1135
[14]
Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a Math Word Problem to a Expression Tree. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 1064–1069. https://doi.org/10.18653/v1/D18-1132
[15]
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep Neural Solver for Math Word Problems., 845–854 pages. https://doi.org/10.18653/v1/D17-1088
[16]
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep Neural Solver for Math Word Problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 845–854. https://doi.org/10.18653/v1/D17-1088
[17]
Zhipeng Xie and Shichao Sun. 2019. A Goal-Driven Tree-Structured Neural Model for Math Word Problems. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 5299–5305. https://doi.org/10.24963/ijcai.2019/736
[18]
Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020. Graph-to-Tree Learning for Solving Math Word Problems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 3928–3937. https://doi.org/10.18653/v1/2020.acl-main.362

Index Terms

  1. Adversarial Analysis and Methods for Math Word Problems

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    CMLDS '24: Proceedings of the International Conference on Computing, Machine Learning and Data Science
    April 2024
    381 pages
    ISBN:9798400716393
    DOI:10.1145/3661725
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 June 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Adversarial testing
    2. Math word problems (MWPs).
    3. Natural language understanding (NLU)

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    CMLDS 2024

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 22
      Total Downloads
    • Downloads (Last 12 months)22
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 13 Jan 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media