ABSTRACT
While search technology is widely used for learning-oriented information needs, the results provided by popular services such as Web search engines are optimized primarily for generic relevance, not effective learning outcomes. As a result, the typical information trail that a user must follow while searching to achieve a learning goal may be an inefficient one involving unnecessarily easy or difficult content, or material that is irrelevant to actual learning progress relative to a user's existing knowledge. We address this problem by introducing a novel theoretical framework, algorithms, and empirical analysis of an information retrieval model that is optimized for learning outcomes instead of generic relevance. We do this by formulating an optimization problem that incorporates a cognitive learning model into a retrieval objective, and then give an algorithm for an efficient approximate solution to find the search results that represent the best 'training set' for a human learner. Our model can personalize results for an individual user's learning goals, as well as account for the effort required to achieve those goals for a given set of retrieval results. We investigate the effectiveness and efficiency of our retrieval framework relative to a commercial search engine baseline ('Google') through a crowdsourced user study involving a vocabulary learning task, and demonstrate the effectiveness of personalized results from our model on word learning outcomes.
- Peter Bailey, Liwei Chen, Scott Grosenick, Li Jiang, Yan Li, Paul Reinholdtsen, Charles Salada, Haidong Wang, and Sandy Wong. 2012. User task understanding: a web search engine perspective. In NII Shonan Meeting on Whole-Session Evaluation of Interactive Information Retrieval Systems, Kanagawa, Japan.Google Scholar
- Benjamin S. Bloom. 1956. Taxonomy of educational objectives: The classification of educational goals. New York, Longmans, Green.Google Scholar
- Benjamin S. Bloom. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher 13, 6 (1984), 4--16. Google ScholarCross Ref
- Andrei Broder. 2002. A Taxonomy of Web Search. SIGIR Forum 36, 2 (Sept. 2002), 3--10. Google ScholarDigital Library
- Kevyn Collins-Thompson, Paul N. Bennett, Ryen W. White, Sebastian de la Chica, and David Sontag. 2011. Personalizing Web Search Results by Reading Level. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM '11). ACM, New York, NY, USA, 403--412. Google ScholarDigital Library
- Kevyn Collins-Thompson, Soo Young Rieh, Carl C. Haynes, and Rohail Syed. 2016. Assessing Learning Outcomes in Web Search: A Comparison of Tasks and Query Strategies. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR '16). ACM, New York, NY, USA, 163--172. Google ScholarDigital Library
- Albert Corbett. 2001. Cognitive computer tutors: Solving the two-sigma problem. In International Conference on User Modeling. Springer, 137--147. Google ScholarCross Ref
- Antonella De Angeli, Lynne Coventry, Graham Johnson, and Karen Renaud. 2005. Is a picture really worth a thousand words? Exploring the feasibility of graphical authentication systems. International Journal of Human-Computer Studies 63, 1 (2005), 128--152. Google ScholarDigital Library
- Sergio Duarte Torres, Djoerd Hiemstra, and Pavel Serdyukov. 2010. An Analysis of Queries Intended to Search Information for Children. In Proceedings of the Third Symposium on Information Interaction in Context (IIiX '10). ACM, New York, NY, USA, 235--244. Google ScholarDigital Library
- Carsten Eickhoff, Jaime Teevan, Ryen White, and Susan Dumais. 2014. Lessons from the Journey: A Query Log Analysis of Within-session Learning. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (WSDM '14). ACM, New York, NY, USA, 223--232. Google ScholarDigital Library
- Brian W. Junker. 1999. Some statistical models and computational methods that may be useful for cognitively-relevant assessment. Prepared for the National Research Council Committee on the Foundations of Assessment. Retrieved April 2 (1999), 81.Google Scholar
- David R. Krathwohl. 2002. A revision of Bloom's taxonomy: An overview. Theory into Practice 41, 4 (2002), 212--218. Google ScholarCross Ref
- Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods 44, 4 (2012), 978--990. Google Scholar
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).Google Scholar
- Peter Pirolli and Sanjay Kairam. 2013. A knowledge-tracing model of learning from a social tagging system. User Modeling and User-Adapted Interaction 23, 2--3 (2013), 139--168.Google ScholarCross Ref
- Filip Radlinski and Susan Dumais. 2006. Improving Personalized Web Search Using Result Diversification. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '06). ACM, New York, NY, USA, 691--692. Google ScholarDigital Library
- Karthik Raman, Paul N. Bennett, and Kevyn Collins-Thompson. 2014. Understanding intrinsic diversity in web search: Improving whole-session relevance. ACM Transactions on Information Systems (TOIS) 32, 4 (2014), 20.Google ScholarDigital Library
- Daniel E. Rose and Danny Levinson. 2004. Understanding User Goals in Web Search. In Proceedings of the 13th International Conference on World Wide Web (WWW '04). ACM, New York, NY, USA, 13--19. Google ScholarDigital Library
- Mark D. Smucker and Charles L. A. Clarke. 2012. Time-based Calibration of Effectiveness Measures. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '12). ACM, New York, NY, USA, 95--104. Google ScholarDigital Library
- Rohail Syed and Kevyn Collins-Thompson. 2017. Optimizing search results for human learning goals. Information Retrieval Journal (2017), 1--18.Google Scholar
- Emine Yilmaz, Manisha Verma, Nick Craswell, Filip Radlinski, and Peter Bailey. 2014. Relevance and Effort: An Analysis of Document Utility. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM '14). ACM, New York, NY, USA, 91--100. Google ScholarDigital Library
- Xiaojin Zhu. 2013. Machine teaching for bayesian learners in the exponential family. In Advances in Neural Information Processing Systems. 1905--1913.Google Scholar
Index Terms
- Retrieval Algorithms Optimized for Human Learning
Recommendations
Optimizing search results for human learning goals
AbstractWhile past research has shown that learning outcomes can be influenced by the amount of effort students invest during the learning process, there has been little research into this question for scenarios where people use search engines to learn. ...
Ranking by Language Similarity for Resource Scarce Southern Bantu Languages
ICTIR '21: Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information RetrievalResource Scarce Languages (RSLs) lack sufficient resources to use Cross-Lingual Information Retrieval (CLIR) techniques and tools such as machine translation. Consequentially, searching using RSLs is frustrating and usually ends in unsuccessful ...
Comments