Skip to main content
SpringerLink
Log in

TaskGenie: Crowd-Powered Task Generation for Struggling Search

  • Conference paper
  • First Online:
Web Information Systems Engineering – WISE 2020 (WISE 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12343))

Included in the following conference series:

Abstract

Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of ‘repeated similar queries’ and ‘quick-back clicks’, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://waps.io/study/?uid=123.

  2. 2.

    Wikinews: https://en.wikinews.org/; wikivoyage: https://www.wikivoyage.org/.

  3. 3.

    Anonymized URL– https://github.com/sst20190816/WISE2020.

  4. 4.

    https://j11y.io/snippets/.

  5. 5.

    http://figure-eight.com/.

  6. 6.

    http://curlie.org/.

  7. 7.

    https://en.wikipedia.org/wiki/List_of_wikis.

References

  1. Ageev, M., Guo, Q., Lagun, D., Agichtein, E.: Find it if you can: a game for modeling different types of web search success using interaction data. In: SIGIR, pp. 345–354 (2011)

    Google Scholar 

  2. Aula, A., Khan, R.M., Guan, Z.: How does search behavior change as search becomes more difficult? In: SIGCHI, pp. 35–44 (2010)

    Google Scholar 

  3. Baykan, E., Henzinger, M., Marian, L., Weber, I.: Purely URL-based topic classification. In: WWW, pp. 1109–1110 (2009)

    Google Scholar 

  4. Bhagat, R., Hovy, E.: What is a paraphrase? Comput. Linguist. 39(3), 463–472 (2013)

    Article  Google Scholar 

  5. Braarud, P.Ø., Kirwan, B.: Task complexity: what challenges the crew and how do they cope. In: Skjerve, A., Bye, A. (eds.) Simulator-based Human Factors Studies Across 25 Years, pp. 233–251. Springer, London (2010). https://doi.org/10.1007/978-0-85729-003-8_15

    Chapter  Google Scholar 

  6. Capra, R., Arguello, J., O’Brien, H., Li, Y., Choi, B.: The effects of manipulating task determinability on search behaviors and outcomes. In: SIGIR, pp. 445–454 (2018)

    Google Scholar 

  7. De Beaugrande, R.A., Dressler, W.U.: Introduction to Text Linguistics, vol. 1. Longman, London (1981)

    Book  Google Scholar 

  8. Gadiraju, U., et al.: Crowdsourcing versus the laboratory: towards human-centered experiments using the crowd. In: Archambault, D., Purchase, H., Hoßfeld, T. (eds.) Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. LNCS, vol. 10264, pp. 6–26. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66435-4_2

    Chapter  Google Scholar 

  9. Gadiraju, U., Yu, R., Dietze, S., Holtz, P.: Analyzing knowledge gain of users in informational search sessions on the web. In: CHIIR 2018 (2018)

    Google Scholar 

  10. Hassan, A.: A semi-supervised approach to modeling web search satisfaction. In: SIGIR, pp. 275–284 (2012)

    Google Scholar 

  11. Hassan, A., Jones, R., Klinkner, K.L.: Beyond DCG: user behavior as a predictor of a successful search. In: WSDM, pp. 221–230 (2010)

    Google Scholar 

  12. Hassan, A., White, R.W., Dumais, S.T., Wang, Y.M.: Struggling or exploring?: disambiguating long search sessions. In: WSDM, pp. 53–62 (2014)

    Google Scholar 

  13. Hassan Awadallah, A., White, R.W., Pantel, P., Dumais, S.T., Wang, Y.M.: Supporting complex search tasks. In: CIKM, pp. 829–838 (2014)

    Google Scholar 

  14. Kellar, M., Watters, C., Shepherd, M.: A goal-based classification of web information tasks. ASIST 43(1), 1–22 (2006)

    Google Scholar 

  15. Kenter, T., De Rijke, M.: Short text similarity with word embeddings. In: CIKM, pp. 1411–1420 (2015)

    Google Scholar 

  16. Kim, Y., Hassan, A., White, R.W., Zitouni, I.: Modeling dwell time to predict click-level satisfaction. In: WSDM, pp. 193–202 (2014)

    Google Scholar 

  17. Kolln, M., Funk, R.: Understanding English Grammar. Longman, London (1982)

    Google Scholar 

  18. Liu, C., Liu, J., Cole, M., Belkin, N.J., Zhang, X.: Task difficulty and domain knowledge effects on information search behaviors. ASIS&T 49(1), 1–10 (2012)

    Google Scholar 

  19. Liu, J., Kim, C.S., Creel, C.: Why do users feel search task difficult? In: The 76th ASIS&T. American Society for Information Science (2013)

    Google Scholar 

  20. Mai, J.E.: Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior. Emerald Group Publishing, Bingley (2016)

    Google Scholar 

  21. Mitra, B.: Exploring session context using distributed representations of queries and reformulations. In: SIGIR, pp. 3–12 (2015)

    Google Scholar 

  22. Odijk, D., White, R.W., Hassan Awadallah, A., Dumais, S.T.: Struggling and success in web search. In: CIKM, pp. 1551–1560 (2015)

    Google Scholar 

  23. Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: IA, vol. 5, pp. 2–4 (2005)

    Google Scholar 

  24. Robinson, P.: Task complexity, task difficulty, and task production: exploring interactions in a componential framework. Appl. Linguist. 22(1), 27–57 (2001)

    Article  Google Scholar 

  25. Singer, G., Norbisrath, U., Lewandowski, D.: Ordinary search engine users carrying out complex search tasks. J. Inf. Sci. 39(3), 346–358 (2013)

    Article  Google Scholar 

  26. Singer, P., et al.: Why we read Wikipedia. In: WWW, pp. 1591–1600 (2017)

    Google Scholar 

  27. White, R.W.: Interactions with Search Systems. Cambridge University Press, Cambridge (2016)

    Book  Google Scholar 

  28. White, R.W., Roth, R.A.: Exploratory search: beyond the query-response paradigm. In: Synthesis Lectures on Information Concepts, Retrieval, and Services, vol. 1, no. 1, pp. 1–98 (2009)

    Google Scholar 

  29. Wilson, M.L., Kules, B., Shneiderman, B., et al.: From keyword search to exploration: designing future search interfaces for the web. Found. Trends® Web Sci. 2(1), 1–97 (2010)

    Google Scholar 

  30. Xu, L., Zhou, X.: Generating tasks for study of struggling search. In: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, pp. 267–270 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Renmin University of China, Beijing, 100872, China

    Luyan Xu

  2. East China Normal University, Shanghai, 200062, China

    Xuan Zhou

  3. Delft University of Technology, Delft, The Netherlands

    Ujwal Gadiraju

Authors
  1. Luyan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ujwal Gadiraju
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luyan Xu .

Editor information

Editors and Affiliations

  1. VU Amsterdam, Amsterdam, The Netherlands

    Zhisheng Huang

  2. VU Amsterdam, Amsterdam, The Netherlands

    Wouter Beek

  3. Victoria University, Melbourne, VIC, Australia

    Hua Wang

  4. Swinburne University of Technology, Hawthorn, VIC, Australia

    Rui Zhou

  5. Victoria University, Melbourne, VIC, Australia

    Yanchun Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, L., Zhou, X., Gadiraju, U. (2020). TaskGenie: Crowd-Powered Task Generation for Struggling Search. In: Huang, Z., Beek, W., Wang, H., Zhou, R., Zhang, Y. (eds) Web Information Systems Engineering – WISE 2020. WISE 2020. Lecture Notes in Computer Science(), vol 12343. Springer, Cham. https://doi.org/10.1007/978-3-030-62008-0_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62008-0_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62007-3

  • Online ISBN: 978-3-030-62008-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation