Abstract
Query-document topic match is one of the central signals for ranked information retrieval. Returning documents which are too broad or insufficient or off-topic compared to the query leads to a poor user experience. Thus, given a query and a document, it is critical to estimate degree of topical mismatch between the two. Previous work has either focused on very broad topics (like health, education, science, etc.) or predicted extremely-fine-level topic mismatch using query reformulations which suffer from sparsity. Predicting query-document topic mismatch is difficult because it needs semantic understanding of both the query and the document. In this paper, we model the problem as a five-class classification problem using a novel Transformer-based architecture. Our technique takes the query as input along with a detailed document representation including title, URL, snippet, key-phrases and topic distribution, and outputs one of these five grades of topic match: Very Unsatisfactory, Unsatisfactory, Neutral, Satisfactory, and Very Satisfactory. On a large dataset of \(\sim \)2.43M query-document pairs, we show that our proposed method can provide an AUC of 0.75.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A snippet is a short summary of webpage that appears in the Bing search results. Snippets are generated based on the search term and are presented as part of a search result list.
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. JMLR 3(Jan), 993–1022 (2003)
Brenes, D.J., Gayo-Avello, D., Pérez-González, K.: Survey and evaluation of query intent detection methods. In: Workshop on Web Search Click Data, pp. 1–7 (2009)
Cao, Z., Qin, T., Liu, T.Y., Tsai, M.F., Li, H.: Learning to rank: from pairwise approach to listwise approach. In: Proceedings of the 24th International Conference on Machine Learning, pp. 129–136 (2007)
Cer, D., et al.: Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018)
Chapelle, O., Chang, Y.: Yahoo! learning to rank challenge overview. In: Proceedings of the Learning to Rank Challenge, pp. 1–24 (2011)
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670–680 (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Gayo-Avello, D.: A survey on session detection methods in query logs and a proposal for future evaluation. Inf. Sci. 179(12), 1822–1843 (2009)
Gupta, M., Agrawal, P.: Compression of deep learning models for text: a survey. arXiv preprint arXiv:2008.05221 (2020)
Hashemi, H.B., Asiaee, A., Kraft, R.: Query intent detection using convolutional neural networks. In: International Conference on Web Search and Data Mining, Workshop on Query Understanding (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Jelodar, H., et al.: Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey. Multimedia Tools Appl. 78(11), 15169–15211 (2019)
Jones, R., Klinkner, K.L.: Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs. In: CIKM, pp. 699–708 (2008)
Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International Conference on Machine Learning, pp. 1188–1196 (2014)
Li, J., Luong, M.T., Jurafsky, D.: A hierarchical neural autoencoder for paragraphs and documents. arXiv preprint arXiv:1506.01057 (2015)
Li, X., de Rijke, M.: Do topic shift and query reformulation patterns correlate in academic search? In: Jose, J.M., et al. (eds.) ECIR 2017. LNCS, vol. 10193, pp. 146–159. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56608-5_12
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp. 3111–3119 (2013)
Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)
Xiong, L., Hu, C., Xiong, C., Campos, D., Overwijk, A.: Open domain web keyphrase extraction beyond language modeling. In: EMNLP-IJCNLP, pp. 5178–5187 (2019)
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chelaramani, S. et al. (2022). Query-Document Topic Mismatch Detection. In: Bhattacharya, A., et al. Database Systems for Advanced Applications. DASFAA 2022. Lecture Notes in Computer Science, vol 13247. Springer, Cham. https://doi.org/10.1007/978-3-031-00129-1_35
Download citation
DOI: https://doi.org/10.1007/978-3-031-00129-1_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-00128-4
Online ISBN: 978-3-031-00129-1
eBook Packages: Computer ScienceComputer Science (R0)