Abstract
Research on math word problems has made significant advancements due to the emergence of language models. Large language models have excelled in a variety of reasoning tasks. Still, due to the demand for low costs, research on the upper bound of small language models in reasoning tasks and the limitation of the knowledge they can accommodate has drawn attention. In line with previous work on math word problems, we discover that models that only learned a single solution lacked reasoning ability during the decoding process, further exacerbating the error accumulation caused by exposure bias that will fail generalization. To tackle this problem, we suggest using the commutative property to generate a consistent solution set for each data in the training set. Then, we use it as additional training data to optimize the search space in beam search. On this foundation, we will go into great detail about how consistent solutions training affects the work process of beam search. In addition, we found significant differences between models trained using consistent solutions and those trained without consistent solutions, so the model ensemble technique is applied to improve model performance. In the NLPCC-2023 shared task 3, our model ultimately ranks fourth with an accuracy of 23.66%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
Lee, S., Lee, D.B., Hwang, S.J.: Contrastive learning with adversarial perturbations for conditional text generation. In: International Conference on Learning Representations (2021)
Bakman, Y.: Robust understanding of word problems with extraneous information. arXiv preprint math/0701393 (2007)
Roy, S., Vieira, T., Roth, D.: Reasoning about quantities in natural language. Trans. Assoc. Comput. Linguistics 3, 1–13 (2015)
Liang, C.-C., Hsu, K.-Y., Huang, C.-T., Li, C.-M., Miao, S.-Yu., Su, K.-Y.: A tag-based English math word problem solver with understanding, reasoning and explanation. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pp. 67–71, June 2016
Hosseini, M.J., Hajishirzi, H., Etzioni, O., Kushman, N.: Learning to solve arithmetic word problems with verb categorization. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523–533, October 2014
Wang, Y., Liu, X., Shi, S.: Deep neural solver for math word problems. In: Proceedings of the 2017 Conference on Empirical methods in Natural Language Processing, pp. 845–854, September 2017
Liu, Q., Guan, W., Li, S., Kawahara, D.: Tree-structured decoding for solving math word problems. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2370–2379, November 2019
Wang, Y., Lee, H.-Y., Chen, Y.-N.: Tree transformer: integrating tree structures into self-attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1061–1070, November 2019
Xie, Z., Sun, S.: A goal-driven tree-structured neural model for math word problems. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5299–5305, 7 2019
Lin, X., Huang, Z., Zhao, H., Chen, E., Liu, Q., Wang, H., Wang, S.: Hms: A hierarchical solver with dependency-enhanced understanding for math word problem. Proceedings of the AAAI Conference on Artificial Intelligence 35(5), 4232–4240 (2021)
Kim, B., Ki, K.S., Lee, D., Gweon, G.: Point to the expression: solving algebraic word problems using the expression-pointer transformer model. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3768–3779, November 2020
Zhang, J., et al.: Graph-to-tree learning for solving math word problems. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3928–3937, July 2020
Huang, D., Liu, J., Lin, C.-Y., Yin, J.: Neural math word problem solver with reinforcement learning. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 213–223, August 2018
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, June 2019
Qin, J., Liang, X., Hong, Y., Tang, J., Lin, L.: Neural-symbolic solver for math word problems with auxiliary tasks. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 5870–5881, August 2021
Wu, Q., Zhang, Q., Wei, Z., Huang, X.: Math word problem solving with explicit numerical values. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 5859–5869, August 2021
Liang, Z., Zhang, J., Wang, L., Qin, W., Lan, Y., Shao, J., Zhang, X.: MWP-BERT: Numeracy-augmented pre-training for math word problem solving. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp. 997–1009 (2022)
Li, Z., Zhang, W., Yan, C., Zhou, Q., Li, C., Liu, H., Cao, Y.: Seeking patterns, not just memorizing procedures: Contrastive learning for solving math word problems. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 2486–2496 (2022)
Tan, M., Wang, L., Jiang, L., Jiang, J.: Investigating math word problems using pretrained multilingual language models. In: Proceedings of the 1st Workshop on Mathematical Natural Language Processing, pp. 7–16, December 2022
Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., Liu, Q.: Generate & rank: A multi-task framework for math word problems. In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2269–2279 (2021)
Liang, Z., Zhang, J., Wang, L., Wang, Y., Shao, J., Zhang, X.: Generalizing math word problem solvers via solution diversification. In: Proceedings of the AAAI Conference on Artificial Intelligence, 37, pp. 13183–13191, 06 2023
Jie, Z., Li, J., Lu, W.: Learning to reason deductively: math word problem solving as complex relation extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 5944–5955, May 2022
Zhang, Z., et al.: Towards lightweight yet ingenious pre-trained models for chinese. arXiv preprint arXiv:2110.06696 (2021)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations, May 2019
Sun, Y., et al.: Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137, 2021
Liu, Y., et al.: Multilingual denoising pre-training for neural machine translation. Trans. Assoc. Comput. Linguistics 8, 726–742 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xu, Y., Li, S., Pu, C., Wang, J., Zhou, X. (2023). Consistent Solutions for Optimizing Search Space of Beam Search. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14304. Springer, Cham. https://doi.org/10.1007/978-3-031-44699-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-44699-3_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44698-6
Online ISBN: 978-3-031-44699-3
eBook Packages: Computer ScienceComputer Science (R0)
