ABSTRACT
Traditional TREC-style pooling methodology relies on using predicted relevance by systems to select documents for judgment. This coincides with typical search behaviour (e.g., web search). In the case of temporally ordered streams of documents, the order that users encounter documents is in this temporal order and not some predetermined rank order. We investigate a user oriented pooling methodology focusing on the documents that simulated users would likely read in such temporally ordered streams. Under this user model, many of the relevant documents found in the TREC 2013 Temporal Summarization Track's pooling effort would never be read. Not only does our pooling strategy focus on pooling documents that will be read by (simulated) users, the resultant pools are different from the standard TREC pools.
- J. Allan, R. Gupta, and V. Khandelwal. Temporal Summaries of New Topics. In SIGIR, pp. 10--18, 2001. Google ScholarDigital Library
- J. Aslam, F. Diaz, M. Ekstrand-Abueg, V. Pavlu, and T. Sakai. TREC 2013 Temporal Summarization. In TREC, 2013.Google Scholar
- G. Baruah, M. D. Smucker, and C. L. A. Clarke. Evaluating Streams of Evolving News Events. In Proc. SIGIR, 2015. Google ScholarDigital Library
- C. Buckley, D. Dimmick, I. Soboroff, and E. Voorhees. Bias and the Limits of Pooling. In Proc. SIGIR, pp. 619--620, 2006. Google ScholarDigital Library
- B. Carterette, J. Allan, and R. Sitaraman. Minimal Test Collections for Retrieval Evaluation. In Proc. SIGIR, pp. 268--275, 2006. Google ScholarDigital Library
- C. L. A. Clarke, L. Freund, M. D. Smucker, and E. Yilmaz. Report on the SIGIR 2013 Workshop on Modeling User Behavior for Information Retrieval Evaluation (MUBE 2013). SIGIR Forum, 47(2):84--95, Jan. 2013. Google ScholarDigital Library
- C. L. A. Clarke and M. D. Smucker. Time Well Spent. In IIiX'14, pp. 205--214, 2014. Google ScholarDigital Library
- G. V. Cormack and T. R. Lynam. Power and Bias of Subset Pooling Strategies. In Proc. SIGIR, pp. 837--838, 2007. Google ScholarDigital Library
- M. Johanson. Facebook changes: Users overwhelmingly 'dislike' new news feed and ticker, Sept. 2011. International Business Times. http://www.ibtimes.com/facebook-changes-users-overwhelmingly-dislike-new-news-feed-ticker-317124.Google Scholar
- D. D. Lewis. The TREC-4 Filtering Track. In Proc. TREC-4, 1995.Google Scholar
- R. McCreadie, C. Macdonald, and I. Ounis. Incremental Update Summarization: Adaptive Sentence Selection Based on Prevalence and Novelty. In CIKM, pp. 301--310, 2014. Google ScholarDigital Library
- I. Soboroff, I. Ounis, J. Lin, and I. Soboroff. Overview of the TREC-2012 Microblog Track. In TREC, 2012.Google Scholar
- K. Sparck-Jones and C. Van Rijsbergen. Report on the need for and provision of an "ideal" Information Retrieval Test Collection. 1975.Google Scholar
- E. M. Voorhees and D. K. Harman. TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, 2005. Google ScholarDigital Library
- J. Zobel. How Reliable are the Results of Large-scale Information Retrieval experiments? In SIGIR, pp. 307--314, 1998. Google ScholarDigital Library
Index Terms
Pooling for User-Oriented Evaluation Measures
Recommendations
A unified model for metasearch, pooling, and system evaluation
CIKM '03: Proceedings of the twelfth international conference on Information and knowledge managementWe present a unified model which, given the ranked lists of documents returned by multiple retrieval systems in response to a given query, simultaneously solves the problems of (1) fusing the ranked lists of documents in order to obtain a high-quality ...
Rank-based pooling for deep convolutional neural networks
Pooling is a key mechanism in deep convolutional neural networks (CNNs) which helps to achieve translation invariance. Numerous studies, both empirically and theoretically, show that pooling consistently boosts the performance of the CNNs. The ...
Bias and the limits of pooling
SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrievalModern retrieval test collections are built through a process called pooling in which only a sample of the entire document set is judged for each topic. The idea behind pooling is to find enough relevant documents such that when unjudged documents are ...
Comments