research-article

DeText: A Deep Text Ranking Framework with BERT

Authors:

Bee-Chung Chen,

Deepak AgarwalAuthors Info & Claims

CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

Pages 2509 - 2516

https://doi.org/10.1145/3340531.3412699

Published: 19 October 2020 Publication History

Abstract

Ranking is the most important component in a search system. Most search systems deal with large amounts of natural language data, hence an effective ranking system requires a deep understanding of text semantics. Recently, deep learning based natural language processing (deep NLP) models have generated promising results on ranking systems. BERT is one of the most successful models that learn contextual embedding, which has been applied to capture complex query-document relations for search ranking. However, this is generally done by exhaustively interacting each query word with each document word, which is inefficient for online serving in search product systems. In this paper, we investigate how to build an efficient BERT-based ranking model for industry use cases. The solution is further extended to a general ranking framework, DeText, that is open sourced and can be applied to various ranking productions. Offline and online experiments of DeText on three real-world search systems present significant improvement over state-of-the-art approaches.

Supplementary Material

MP4 File (3340531.3412699.mp4)

DeText: A Deep Text Ranking Framework with BERT

Download
54.04 MB

References

[1]

Selim Aksoy and Robert M Haralick. 2001. Feature normalization and likelihoodbased similarity measures for image retrieval. Pattern recognition letters (2001).

[2]

Ziv Bar-Yossef and Naama Kraus. 2011. Context-sensitive query auto-completion. In WWW.

[3]

Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11 (2010).

[4]

Christopher J Burges, Robert Ragno, and Quoc V Le. 2007. Learning to rank with nonsmooth cost functions. In NeurIPS.

[5]

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In KDD.

Digital Library

[6]

Zhuyun Dai and Jamie Callan. 2019. Deeper Text Understanding for IR with Contextual Neural Language Modeling. In SIGIR.

[7]

Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In WSDM.

[8]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.

[9]

Tom Fawcett. 2006. An introduction to ROC analysis. Pattern recognition letters (2006), 861--874.

Digital Library

[10]

Mihajlo Grbovic and Haibin Cheng. 2018. Real-time personalization using embeddings for search ranking at airbnb. In KDD.

[11]

Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In CIKM.

[12]

Sepp Hochreiter and JÃigrgen Schmidhuber. 1997. Long short-term memory. In Neural computation.

[13]

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In CIKM.

[14]

Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In EMNLP.

[15]

Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.

[16]

Yann LeCun and Yoshua Bengio. 1995. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks (1995).

Digital Library

[17]

Pan Li, Zhen Qin, Xuanhui Wang, and Donald Metzler. 2019. Combining Decision Trees and Neural Networks for Learning-to-Rank in Personal Search. In KDD.

[18]

Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085 (2019).

[19]

Sankar K Pal and Sushmita Mitra. 1992. Multilayer perceptron, fuzzy sets, and classification. IEEE Transactions on neural networks (1992).

Digital Library

[20]

Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. 2016. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. TASLP (2016).

[21]

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP.

[22]

Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019. Understanding the Behaviors of BERT in Ranking. arXiv preprint arXiv:1904.07531 (2019).

[23]

Rohan Ramanath, Hakan Inan, Gungor Polatkan, Bo Hu, Qi Guo, Cagri Ozcaglar, Xianren Wu, Krishnaram Kenthapadi, and Sahin Cem Geyik. 2018. Towards Deep and Representation Learning for Talent Search at LinkedIn. In CIKM.

[24]

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL.

[25]

Dan Shacham, Uri Merhav, Qi He, and Angela Jiang. 2017. Context-aware map from entities to canonical forms. US Patent App. 15/189,974.

[26]

Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM.

[27]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.

[28]

Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In SIGIR.

[29]

Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, Chikashi Nobata, et al. 2016. Ranking relevance in yahoo search. In KDD.

[30]

Hamed Zamani, Bhaskar Mitra, Xia Song, Nick Craswell, and Saurabh Tiwary. 2018. Neural ranking models with multiple document fields. In WSDM.

Cited By

Mai H(2024)Functional Analysis of English Carriers and Related Resources of Cultural Communication in Internet MediaEconomics10.1515/econ-2022-007518:1Online publication date: 10-Apr-2024
https://doi.org/10.1515/econ-2022-0075
Wang HYu SChen CTurhan BZhu X(2024)Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning ProjectsACM Transactions on Software Engineering and Methodology10.1145/363824533:4(1-22)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3638245
Hong ZWang HQian CChen WHe TZou YLiu QWang XChua TNgo CKumar RLauw HKa-Wei Lee R(2024)PRINT: Personalized Relevance Incentive Network for CTR Prediction in Sponsored SearchCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3648316(190-195)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3648316
Show More Cited By

Index Terms

DeText: A Deep Text Ranking Framework with BERT
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

Re-ranking search results using query logs
CIKM '06: Proceedings of the 15th ACM international conference on Information and knowledge management

This work addresses two common problems in search, frequently occurring with underspecified user queries: the top-ranked results for such queries may not contain documents relevant to the user's search intent, and fresh and relevant pages may not get ...
iRANK: A rank-learn-combine framework for unsupervised ensemble ranking

The authors address the problem of unsupervised ensemble ranking. Traditional approaches either combine multiple ranking criteria into a unified representation to obtain an overall ranking score or to utilize certain rank fusion or aggregation techniques ...
Effective rank aggregation for metasearching

Nowadays, mashup services and especially metasearch engines play an increasingly important role on the Web. Most of users use them directly or indirectly to access and aggregate information from more than one data sources. Similarly to the rest of the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

October 2020

3619 pages

ISBN:9781450368599

DOI:10.1145/3340531

General Chairs:
Mathieu d'Aquin
DSI, Insight, NUI Galway, Ireland
,
Stefan Dietze
GESIS, Cologne, Germany, Heinrich-Heine-University Düsseldorf, Germany, L3S Research Center, Germany
,
Program Chairs:
Claudia Hauff
TU Delft, The Netherlands
,
Edward Curry
DSI, Insight, NUI Galway, Ireland
,
Philippe Cudre Mauroux
eXascale, University of Fribourg, Switzerland

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '20

Sponsor:

CIKM '20: The 29th ACM International Conference on Information and Knowledge Management

October 19 - 23, 2020

Virtual Event, Ireland

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
771
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)5

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mai H(2024)Functional Analysis of English Carriers and Related Resources of Cultural Communication in Internet MediaEconomics10.1515/econ-2022-007518:1Online publication date: 10-Apr-2024
https://doi.org/10.1515/econ-2022-0075
Wang HYu SChen CTurhan BZhu X(2024)Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning ProjectsACM Transactions on Software Engineering and Methodology10.1145/363824533:4(1-22)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3638245
Hong ZWang HQian CChen WHe TZou YLiu QWang XChua TNgo CKumar RLauw HKa-Wei Lee R(2024)PRINT: Personalized Relevance Incentive Network for CTR Prediction in Sponsored SearchCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3648316(190-195)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3648316
Fan YTu XLi R(2024)An Inverse Retrieval Method via Query Generation for Xiaohongshu’s Search EngineAdvanced Intelligent Computing Technology and Applications10.1007/978-981-97-5675-9_31(362-373)Online publication date: 1-Aug-2024
https://doi.org/10.1007/978-981-97-5675-9_31
Herwanto GQuirchmayr GTjoa A(2024)Learning to Rank Privacy Design Patterns: A Semantic Approach to Meeting Privacy RequirementsRequirements Engineering: Foundation for Software Quality10.1007/978-3-031-57327-9_4(57-73)Online publication date: 30-Mar-2024
https://doi.org/10.1007/978-3-031-57327-9_4
Wang DSalamatian KXia YDeng WZhang QSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-textual Features for CTR PredictionProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599780(5039-5050)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599780
Pilliang MMunawar Hadi MFirmansyah GTjahjono B(2022)Predicting Risk Matrix in Software Development Projects using BERT and K-Means2022 9th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI)10.23919/EECSI56542.2022.9946637(137-142)Online publication date: 6-Oct-2022
https://doi.org/10.23919/EECSI56542.2022.9946637
Baltescu PChen HPancha NZhai ALeskovec JRosenberg CZhang ARangwala H(2022)ItemSage: Learning Product Embeddings for Shopping Recommendations at PinterestProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539170(2703-2711)Online publication date: 14-Aug-2022
https://dl.acm.org/doi/10.1145/3534678.3539170
Shi JJiang CGupta AZhou MOuyang YXiao QSong QWu YWei HGao HZhang ARangwala H(2022)Generalized Deep Mixed ModelsProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539103(3869-3877)Online publication date: 14-Aug-2022
https://dl.acm.org/doi/10.1145/3534678.3539103
Wang DYan SXia YSalamatian KDeng WZhang QZhang ARangwala H(2022)Learning Supplementary NLP Features for CTR Prediction in Sponsored SearchProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539064(4010-4020)Online publication date: 14-Aug-2022
https://dl.acm.org/doi/10.1145/3534678.3539064
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten