Math Information Retrieval with Contrastive Learning of Formula Embeddings

Abstract

The core and hard part of Mathematical Information Retrieval (MathIR) is formula retrieval. The datasets used for formula retrieval are usually scientific documents containing formulas. However, there is a lack of labeled datasets specifically for formula similarity. Contrastive learning can learn general features of datasets from unlabeled data and autonomously discover latent structures in the data. Furthermore, dense retrieval methods based on bi-encoders have gained increasing attention. Therefore, we propose CLFE, a simple framework for contrastive learning of formula embeddings. It can learn the latent structure and content information of formulas from unlabeled formulas and generate formula embeddings for formula retrieval. We design two frameworks, Contrastive Presentation MathML-Content MathML Learning and Contrastive LaTeX-MathML Learning, which initialize the encoders using transformer-based models and can produce superior LaTeX, Presentation MathML, and Content MathML embeddings for each formula. The combination of these three embeddings results in the Add embedding and Concat embedding. Finally, the formula retrieval task is achieved by finding the k nearest formulas in the vector space through nearest neighbor search. Experimental results show that when applying our proposed method on the NTCIR-12 dataset to retrieve 20 non-wildcard formulas and scoring the top-10 results, the highest achieved scores for mean P@10, mean nDCG@10, and MAP@10 are 0.6250, 0.9792, and 0.9381.