Regularized Contrastive Learning of Semantic Search

Abstract

Semantic search is an important task which objective is to find the relevant index from a database for query. It requires a retrieval model that can properly learn the semantics of sentences. Transformer-based models are widely used as retrieval models due to their excellent ability to learn semantic representations. in the meantime, many regularization methods suitable for them have also been proposed. In this paper, we propose a new regularization method: Regularized Contrastive Learning, which can help transformer-based models to learn a better representation of sentences. It firstly augments several different semantic representations for every sentence, then take them into the contrastive objective as regulators. These contrastive regulators can overcome overfitting issues and alleviate the anisotropic problem. We firstly evaluate our approach on 7 semantic search benchmarks with the outperforming pre-trained model SRoBERTA. The results show that our method is more effective for learning a superior sentence representation. Then we evaluate our approach on 2 challenging FAQ datasets, Cough and Faqir, which have long query and index. The results of our experiments demonstrate that our method outperforms baseline methods.

A A APPENDIX

1.1 A.1 A Training Details

The Base model is $SRoberta_{base}$ [8]. We download their weights from SentenceTransformers. $SRoberta_{base}$ is trained on MNLI and SNLI by using the entailment pairs as positive instances and the contradiction pairs as hard negative instances.

For STS Tasks. We carry out the grid search of batch size $\in$[32,64] and $\phi \in [[0.01],[0.01,0.02],[0.01,0.02,0.03],[0.01,0.02,0.03,0.04],[0.01,0.02,0.03,0.04,$

0.05]]. We firstly fine-tune $SRoberta_{base}$ with different $\phi$ on MNLI and SNLI with Eq. 2, by using the entailment pairs as positive instances and the contradiction pairs as hard negative instances. The purpose of this step is to train several entropy models who can generate different semantic similar embedding for every sentence. Since the pre-trained model has already trained on these datasets, the training process will converge quickly. We use the early stopping method to quickly stop the training process if the loss doesn’t decrease within 3 steps. Then we continue to fine-tune the pre-trained $SRoberta_{base}$, by taking the augmented embeddings and the positive instances and entailment instances of MNLI and SNLI into the contrastive objectives with the Eq. 3. We train this model for 1 epoch, evaluate it every 10% of samples on the development set of STS-B by Spearman-Correlation. When the batch size is 64 and $\phi =[0.01,0.0,0.03,0.04]$, our model achieves the best accuracy.

For FAQ Datasets. We implement RCL on the same pre-trained model as above. And apply the same grid-search as above for batch size and $\phi$. We firstly fine-tune an initial model based on the pre-trained $SRoberta_{base}$ for several epochs with Eq. 1 and save the model who has the highest MAP value on the development dataset. Based on this initial model, we continue to fine-tune the entropy model with different $\phi$ with Eq. 2. Finally, we fine-tune the final retrieval model based on the pre-trained $SRoberta_{base}$ with the augmented embeddings for several epochs, saving the model who has the highest MAP value on the development dataset. When batch size is 64 and $\phi =[0.01,0.02,0.03,0.04]$, the retrieval model achieves the best accuracy.