ARQMath: a new benchmark for math-aware CQA and math formula retrieval
Article No.: 4, Pages 1 - 9
Abstract
The Answer Retrieval for Questions on Math (ARQMath) evaluation was run for the first time at CLEF 2020. ARQMath is the first Community Question Answering (CQA) shared task for math, retrieving existing answers from Math Stack Exchange (MSE) that can help to answer previously unseen math questions. ARQMath also introduces a new protocol for math formula search, where formulas are evaluated in context using a query formula's associated question post, and posts associated with each retrieved formula. Over 70 topics were annotated for each task by eight undergraduate students supervised by a professor of mathematics. A formula index is provided in three formats: LATEX, Presentation MathML, and Content MathML, avoiding the need for participants to extract these themselves. In addition to detailed relevance judgments, tools are provided to parse MSE data, generate question threads in HTML, and evaluate retrieval results. To make comparisons with participating systems fairer, nDCG' (i.e., nDCG for assessed hits only) is used to compare systems for each task. ARQMath will continue in CLEF 2021, with training data from 2020 and baseline systems for both tasks to reduce barriers to entry for this challenging problem domain.
References
[1]
Akiko Aizawa and Michael Kohlhase. Mathematical information retrieval. In Evaluating Information Retrieval and Access Tasks, pages 169--185. Springer, Singapore, 2020.
[2]
Akiko Aizawa, Michael Kohlhase, and Iadh Ounis. NTCIR-10 math pilot task overview. In NTCIR, 2013.
[3]
Akiko Aizawa, Michael Kohlhase, Iadh Ounis, and Moritz Schubotz. NTCIR-11 Math-2 task overview. In NTCIR, volume 11, pages 88--98, 2014.
[4]
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Trans. Assoc. Comput. Linguistics, 5:135--146, 2017. URL https://transacl.org/ojs/index.php/tacl/article/view/999.
[5]
Kenny Davila and Richard Zanibbi. Layout and semantics: Combining representations for mathematical formula search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1165--1168, 2017.
[6]
Liangcai Gao, Zhuoren Jiang, Yue Yin, Ke Yuan, Zuoyu Yan, and Zhi Tang. Preliminary exploration of formula embedding for mathematical information retrieval: can mathematical formulae be embedded like a natural language? 2017.
[7]
Shahab Kamali and Frank Wm Tompa. Retrieving documents with mathematical content. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 353--362. ACM, 2013.
[8]
Giovanni Yoko Kristianto, Goran Topic, and Akiko Aizawa. MCAT math retrieval system for NTCIR-12 MathIR task. In NTCIR, 2016.
[9]
Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In International Conference on Machine Learning, 2014.
[10]
Behrooz Mansouri, Shaurya Rohatgi, Douglas W Oard, Jian Wu, C Lee Giles, and Richard Zanibbi. Tangent-CFT: An embedding model for mathematical formulas. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR), pages 11--18, 2019a.
[11]
Behrooz Mansouri, Richard Zanibbi, and Douglas W. Oard. Characterizing searches for mathematical concepts. In Joint Conference on Digital Libraries, 2019b.
[12]
Behrooz Mansouri, Douglas W Oard, and Richard Zanibbi. DPRL systems in the CLEF 2020 arqmath lab. In Working Notes of CLEF 2020-Conference and Labs of the Evaluation Forum, 2020.
[13]
Yin Ki Ng, Dallas J. Fraser, Besat Kassaie, George Labahn, Mirette S. Marzouk, Frank Wm. Tompa, and Kevin Wang. Dowsing for math answers with Tangent-L. In Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, 2020.
[14]
Lukas Pfahler and Katharina Morik. Semantic search in millions of equations. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 135--143, 2020.
[15]
Tetsuya Sakai and Noriko Kando. On information retrieval metrics designed for evaluation with incomplete relevance assessments. Information Retrieval, 11(5):447--470, 2008.
[16]
Petr Sojka and Martin Líška. The art of mathematics retrieval. In Proceedings of the 11th ACM Symposium on Document Engineering, 2011.
[17]
Abhinav Thanda, Ankit Agarwal, Kushal Singla, Aditya Prakash, and Abhishek Gupta. A document retrieval system for math queries. In NTCIR, 2016.
[18]
Michihiro Yasunaga and John D Lafferty. Topiceq: A joint topic and mathematical equation model for scientific texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7394--7401, 2019.
[19]
Richard Zanibbi and Dorothea Blostein. Recognition and retrieval of mathematical expressions. International Journal on Document Analysis and Recognition (IJDAR), 15(4):331--357, 2012.
[20]
Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topic, and Kenny Davila. NTCIR-12 MathIR task overview. In NTCIR, 2016a.
[21]
Richard Zanibbi, Kenny Davila, Andrew Kane, and Frank Wm Tompa. Multi-stage math formula search: Using appearance-based similarity metrics at scale. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 145--154, 2016b.
[22]
Richard Zanibbi, Douglas W Oard, Anurag Agarwal, and Behrooz Mansouri. Overview of AR-QMath 2020 (updated working notes version): CLEF lab on answer retrieval for questions on math. In Working Notes of CLEF 2020-Conference and Labs of the Evaluation Forum, 2020.
[23]
Wei Zhong, Shaurya Rohatgi, Jian Wu, C Lee Giles, and Richard Zanibbi. Accelerating substructure similarity search for formula retrieval. In European Conference on Information Retrieval, pages 714--727. Springer, 2020.
Index Terms
- ARQMath: a new benchmark for math-aware CQA and math formula retrieval
Index terms have been assigned to the content through auto-classification.
Recommendations
Overview of ARQMath 2020: CLEF Lab on Answer Retrieval for Questions on Math
Experimental IR Meets Multilinguality, Multimodality, and InteractionAbstractThe ARQMath Lab at CLEF considers finding answers to new mathematical questions among posted answers on a community question answering site (Math Stack Exchange). Queries are question posts held out from the searched collection, each containing ...
Advancing Math-Aware Search: The ARQMath-3 Lab at CLEF 2022
Advances in Information RetrievalAbstractARQMath-3 is the third edition of the Answer Retrieval for Questions on Math lab at CLEF. In addition to the two main tasks from previous years, an interesting new pilot task will also be run. The main tasks include: (1) Answer Retrieval, ...
Advancing Math-Aware Search: The ARQMath-2 Lab at CLEF 2021
Advances in Information RetrievalAbstractARQMath-2 is a continuation of the ARQMath Lab at CLEF 2020, with two main tasks: (1) finding answers to mathematical questions among posted answers on a community question answering site (Math Stack Exchange), and (2) formula retrieval, where ...
Comments
Information & Contributors
Information
Published In
Copyright © 2021 Copyright is held by the owner/author(s).
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 August 2021
Published in SIGIR Volume 54, Issue 2
Check for updates
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 54Total Downloads
- Downloads (Last 12 months)15
- Downloads (Last 6 weeks)3
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in