skip to main content
research-article

The neural hype, justified!: a recantation

Published: 23 March 2021 Publication History

Abstract

One year ago, in the SIGIR Forum issue of December 2018, I ranted about the "neural hype" [9]. One year later, I write again to publicly recant my heretical beliefs. What a difference a year makes! In accelerated "deep learning" time, a year seems like an eternity---so much exciting progress has been made in the previous months!

References

[1]
A. Arampatzis, T. Tsoris, C. H. A. Koster, and T. P. van der Weide. Phrase-based information retrieval. Information Processing and Management, 34(6):693--707, December 1998.
[2]
T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. Improvements that don't add up: Ad-hoc retrieval results since 1998. In Proceedings of the 18th International Conference on Information and Knowledge Management (CIKM 2009), pages 601--610, Hong Kong, China, 2009.
[3]
P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, and T. Wang. MS MARCO: A human generated MAchine Reading COmprehension dataset. arXiv:1611.09268v3, 2018.
[4]
M. F. Dacrema, P. Cremonesi, and D. Jannach. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys '19), pages 101--109, Copenhagen, Denmark, 2019.
[5]
Z. Dai and J. Callan. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), pages 985--988, Paris, France, 2019.
[6]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171--4186, Minneapolis, Minnesota, June 2019.
[7]
J. L. Fagan. Experiments in automatic phrase indexing for document retrieval: A comparison of syntactic and non-syntactic methods. Technical Report TR87-868, Cornell University, Department of Computer Science, September 1987.
[8]
S. Hofstätter and A. Hanbury. Let's measure run time! Extending the IR replicability infrastructure to include performance aspects. In Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019): CEUR Workshop Proceedings Vol-2409, pages 12--16, Paris, France, 2019.
[9]
J. Lin. The neural hype and comparisons against weak baselines. SIGIR Forum, 52(2):40--51, 2018.
[10]
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692, 2019.
[11]
S. MacAvaney, A. Yates, A. Cohan, and N. Goharian. CEDR: Contextualized embeddings for document ranking. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), pages 1101--1104, Paris, France, 2019.
[12]
R. Nogueira and K. Cho. Passage re-ranking with BERT. arXiv:1901.04085, 2019.
[13]
R. Nogueira, W. Yang, J. Lin, and K. Cho. Document expansion by query prediction. arXiv:1904.08375, 2019.
[14]
H. Padigela, H. Zamani, and W. B. Croft. Investigating the successes and failures of BERT for passage re-ranking. arXiv:1905.01758, 2019.
[15]
M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227--2237, New Orleans, Louisiana, June 2018.
[16]
Y. Qiao, C. Xiong, Z. Liu, and Z. Liu. Understanding the behaviors of BERT in ranking. arXiv:1904.07531, 2019.
[17]
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training, 2018.
[18]
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683, 2019.
[19]
M. Sanderson. Word-sense disambiguation and information retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1994), pages 142--151, Dublin, Ireland, 1994.
[20]
A. F. Smeaton, R. O'Donnell, and F. Kelledy. Indexing structures derived from syntax in TREC-3: System description. In Proceedings of the Third Text REtrieval Conference (TREC-3), Gaithersburg, Maryland, 1994.
[21]
E. M. Voorhees. Using WordNet to disambiguate word senses for text retrieval. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993), pages 171--180, Pittsburgh, Pennsylvania, 1993.
[22]
W. Yang, K. Lu, P. Yang, and J. Lin. Critically examining the "neural hype": weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), pages 1129--1132, Paris, France, 2019.
[23]
W. Yang, H. Zhang, and J. Lin. Simple applications of BERT for ad hoc document retrieval. In arXiv:1903.10972, 2019.
[24]
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv:1906.08237, 2019.
[25]
Z. A. Yilmaz, W. Yang, H. Zhang, and J. Lin. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3481--3487, Hong Kong, China, 2019.

Cited By

View all
  • (2023)Text Information Retrieval in TetunAdvances in Information Retrieval10.1007/978-3-031-28241-6_48(429-435)Online publication date: 2-Apr-2023
  • (2022)Sparse Pairwise Re-ranking with Pre-trained TransformersProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3539813.3545140(72-80)Online publication date: 23-Aug-2022
  • (2022)Another Look at Information Retrieval as Statistical TranslationProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531717(2749-2754)Online publication date: 6-Jul-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum
ACM SIGIR Forum  Volume 53, Issue 2
December 2019
125 pages
ISSN:0163-5840
DOI:10.1145/3458553
Issue’s Table of Contents
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 March 2021
Published in SIGIR Volume 53, Issue 2

Check for updates

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)16
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Text Information Retrieval in TetunAdvances in Information Retrieval10.1007/978-3-031-28241-6_48(429-435)Online publication date: 2-Apr-2023
  • (2022)Sparse Pairwise Re-ranking with Pre-trained TransformersProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3539813.3545140(72-80)Online publication date: 23-Aug-2022
  • (2022)Another Look at Information Retrieval as Statistical TranslationProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531717(2749-2754)Online publication date: 6-Jul-2022
  • (2022)An Evaluation Study of Generative Adversarial Networks for Collaborative FilteringAdvances in Information Retrieval10.1007/978-3-030-99736-6_45(671-685)Online publication date: 10-Apr-2022
  • (2021)The Simplest Thing That Can Possibly Work: (Pseudo-)Relevance Feedback via Text ClassificationProceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3471158.3472261(123-129)Online publication date: 11-Jul-2021
  • (2021)Query Embedding Pruning for Dense RetrievalProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482162(3453-3457)Online publication date: 26-Oct-2021
  • (2021)On Approximate Nearest Neighbour Selection for Multi-Stage Dense RetrievalProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482156(3318-3322)Online publication date: 26-Oct-2021
  • (2021)A Troubling Analysis of Reproducibility and Progress in Recommender Systems ResearchACM Transactions on Information Systems10.1145/343418539:2(1-49)Online publication date: 6-Jan-2021
  • (2021)Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking LeaderboardProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463034(2283-2287)Online publication date: 11-Jul-2021
  • (2021)MS MARCO: Benchmarking Ranking Models in the Large-Data RegimeProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3462804(1566-1576)Online publication date: 11-Jul-2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media