Bhaskar Mitra
Nick Craswell
ABSTRACT
In the last few years, neural representation learning approaches have achieved very good performance on many natural language processing tasks, such as language modelling and machine translation. This suggests that neural models will also achieve good performance on information retrieval (IR) tasks, such as relevance ranking, addressing the query-document vocabulary mismatch problem by using a semantic rather than lexical matching. Although initial iterations of neural models do not outperform traditional lexical-matching baselines, the level of interest and effort in this area is increasing, potentially leading to a breakthrough. The popularity of the recent SIGIR 2016 workshop on Neural Information Retrieval provides evidence to the growing interest in neural models for IR. While recent tutorials have covered some aspects of deep learning for retrieval tasks, there is a significant scope for organizing a tutorial that focuses on the fundamentals of representation learning for text retrieval. The goal of this tutorial will be to introduce state-of-the-art neural embedding models and bridge the gap between these neural models with early representation learning approaches in IR (e.g., LSA). We will discuss some of the key challenges and insights in making these models work in practice, and demonstrate one of the toolsets available to researchers interested in this area.
- A. Atreya and C. Elkan. Latent semantic indexing (lsi) fails for trec collections. ACM SIGKDD Explorations Newsletter, 12 (2): 5--10, 2011. Google Scholar
Digital Library
- D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. pharXiv preprint arXiv:1409.0473, 2014.Google Scholar
- D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3: 993--1022, 2003.Google Scholar
- N. Craswell, W. B. Croft, J. Guo, B. Mitra, and M. de Rijke. Report on the sigir 2016 workshop on neural information retrieval (neu-ir). 2016.Google Scholar
- S. C. Deerwester, S. T. Dumais, T. K. Landauer, G. W. Furnas, and R. A. Harshman. Indexing by latent semantic analysis. JASIS, 41 (6): 391--407, 1990. Google ScholarCross Ref
- F. Diaz, B. Mitra, and N. Craswell. Query expansion with locally-trained word embeddings. In Proc. ACL, 2016. Google ScholarCross Ref
- A. M. Elkahky, Y. Song, and X. He. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In Proc. WWW, pages 278--288, 2015. Google ScholarDigital Library
- H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al. From captions to visual concepts and back. arXiv preprint arXiv:1411.4952, 2014.Google Scholar
- D. Ganguly, D. Roy, M. Mitra, and G. J. Jones. Word embedding based generalized language model for information retrieval. In Proc. SIGIR, pages 795--798. ACM, 2015. Google ScholarDigital Library
- J. Gao, P. Pantel, M. Gamon, X. He, L. Deng, and Y. Shen. Modeling interestingness with deep neural networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2014. Google ScholarCross Ref
- M. Grbovic, N. Djuric, V. Radosavljevic, and N. Bhamidipati. Search retargeting using directed query embeddings. In Proc. WWW, pages 37--38. International World Wide Web Conferences Steering Committee, 2015. Google ScholarDigital Library
- P. Gupta, K. Bali, R. E. Banchs, M. Choudhury, and P. Rosso. Query expansion for mixed-script information retrieval. In Proc. SIGIR, pages 677--686. ACM, 2014. Google ScholarDigital Library
- X. He, R. Srivastava, J. Gao, and L. Deng. Joint learning of distributed representations for images and texts. arXiv preprint arXiv:1504.03083, 2015.Google Scholar
- F. Hill, K. Cho, S. Jean, C. Devin, and Y. Bengio. Not all neural embeddings are born equal. arXiv preprint arXiv:1410.0718, 2014.Google Scholar
- T. Hofmann. Probabilistic latent semantic indexing. In Proc. SIGIR, pages 50--57. ACM, 1999. Google ScholarDigital Library
- B. Hu, Z. Lu, H. Li, and Q. Chen. Convolutional neural network architectures for matching natural language sentences. In Proc. NIPS, pages 2042--2050, 2014.Google ScholarDigital Library
- P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck. Learning deep structured semantic models for web search using clickthrough data. In Proc. CIKM, pages 2333--2338. ACM, 2013. Google ScholarDigital Library
- R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.Google Scholar
- N. Kalchbrenner, E. Grefenstette, and P. Blunsom. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188, 2014.Google Scholar
- T. Kenter and M. de Rijke. Short text similarity with word embeddings. In Proc. CIKM, volume 15, page 115. Google ScholarDigital Library
- Q. V. Le and T. Mikolov. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053, 2014.Google Scholar
- O. Levy, Y. Goldberg, and I. Ramat-Gan. Linguistic regularities in sparse and explicit word representations. CoNLL-2014, page 171, 2014. Google ScholarCross Ref
- H. Li and Z. Lu. Deep learning for information retrieval.Google Scholar
- T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.Google Scholar
- B. Mitra. Exploring session context using distributed representations of queries and reformulations. In Proc. SIGIR, pages 3--12. ACM, 2015. Google ScholarDigital Library
- B. Mitra and N. Craswell. Query auto-completion for rare prefixes. In Proc. CIKM. ACM, To appear, 2015. Google ScholarDigital Library
- B. Mitra, F. Diaz, and N. Craswell. Learning to match using local and distributed representations of text for web search. arXiv preprint arXiv:1610.08136, 2016.Google Scholar
- Mitra, Nalisnick, Craswell, and Caruana]mitra2016desmB. Mitra, E. Nalisnick, N. Craswell, and R. Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.Google Scholar
- E. Nalisnick, B. Mitra, N. Craswell, and R. Caruana. Improving document ranking with dual word embeddings. In Proc. WWW, 2016. Google ScholarDigital Library
- J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. Proc. EMNLP, 12: 1532--1543, 2014. Google ScholarCross Ref
- S. Robertson. Understanding inverse document frequency: on theoretical arguments for idf. Journal of documentation, 60 (5): 503--520, 2004. Google ScholarCross Ref
- D. Roy, D. Paul, M. Mitra, and U. Garain. Using word embeddings for automatic query expansion. arXiv preprint arXiv:1606.07608, 2016.Google Scholar
- R. Salakhutdinov and G. Hinton. Semantic hashing. International Journal of Approximate Reasoning, 50 (7): 969--978, 2009. Google ScholarDigital Library
- G. Salton, A. Wong, and C.-S. Yang. A vector space model for automatic indexing. Communications of the ACM, 18 (11): 613--620, 1975. Google ScholarDigital Library
- A. Severyn and A. Moschitti. Learning to rank short text pairs with convolutional deep neural networks. In Proc. SIGIR, pages 373--382. ACM, 2015. Google ScholarDigital Library
- Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil. Learning semantic representations using convolutional neural networks for web search. In Proc. WWW, pages 373--374, 2014. Google ScholarDigital Library
- F. Sun, J. Guo, Y. Lan, J. Xu, and X. Cheng. Learning word representations by jointly modeling syntagmatic and paradigmatic relations. In Proc. ACL, 2015. Google ScholarCross Ref
- L. Vilnis and A. McCallum. Word representations via gaussian embedding. arXiv preprint arXiv:1412.6623, 2014.Google Scholar
- I. Vulić and M.-F. Moens. Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In Proc. SIGIR, pages 363--372. ACM, 2015. Google ScholarDigital Library
- X. Yan, J. Guo, S. Liu, X. Cheng, and Y. Wang. Learning topics in short texts by non-negative matrix factorization on term correlation matrix. In Proceedings of the SIAM International Conference on Data Mining, 2013. Google ScholarCross Ref
- D. Yu, A. Eversole, M. Seltzer, K. Yao, Z. Huang, B. Guenter, O. Kuchaiev, Y. Zhang, F. Seide, H. Wang, et al. An introduction to computational networks and the computational network toolkit. Technical report, Tech. Rep. MSR, Microsoft Research, 2014, http://codebox/cntk, 2014.Google Scholar
- G. Zheng and J. Callan. Learning to reweight terms with distributed representations. In Proc. SIGIR, pages 575--584. ACM, 2015. Google ScholarDigital Library
Index Terms
- Neural Text Embeddings for Information Retrieval
Recommendations
A Model for Adaptive Information Retrieval
The paper presents a network model that can be used to produce conceptual and logical schemas for Information Retrieval applications. The model has interesting adaptability characteristics and can be instantiated in various effective ways. The paper also ...
SIGIR 2017 Workshop on Neural Information Retrieval (Neu-IR'17)
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information RetrievalIn recent years, deep neural networks have yielded significant performance improvements in application areas such as speech recognition, computer vision, and machine translation. This has led to expectations in the information retrieval (IR) community ...
Text Retrieval based on Least Information Measurement
ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information RetrievalWe developed a new information retrieval framework based on the Least Information (LI) metric. We derived multiple term weighting schemes and combined them with a vector space representation for ad hoc retrieval. Given probability distributions in a ...
Comments