Skip to main content
SpringerLink
Log in

Translating Theory into Methodological Practice

  • Chapter
  • First Online:
Why Engagement Matters

Abstract

This chapter focuses on the User Engagement Scale (UES), a 31 item experiential questionnaire that measures six dimensions of user engagement: novelty, felt involvement, focused attention, aesthetic appeal, perceived usability, and endurability. The UES has been adopted and adapted to measure engagement in various domains, including online search, news, and video; educational, haptic, consumer, and social networking applications; and video games. Over 50 studies are synthesized to examine the dimensionality, reliability, validity and generalizability of the UES. In addition to illuminating the strengths and weaknesses of the UES as a measurement tool, this chapter uses the UES to explore the reciprocal relationship between theory and the development and application of measurement instruments in user experience research

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For description of some of these and other self-report questionnaires, please see “Chapter 2: Approaches Based on Self-Report Methods” in [27].

  2. 2.

    For an in-depth description of the UES development and evaluation process, please see [34] or [40].

References

  1. Agarwal, R., Karahanna, E.: Time flies when you’re having fun: cognitive absorption and beliefs about information technology usage. Manag. Inf. Syst. Q. 24 (4), 665–694 (2000)

    Article  Google Scholar 

  2. Arapakis, I., Lalmas, M., Cambazoglu, B., Marcos, M., Jose, J.M.: User engagement in online news: under the scope of sentiment, interest, affect, and gaze. J. Assoc. Inf. Sci. Technol. 65 (10), 1988–2005 (2014)

    Article  Google Scholar 

  3. Arapakis, I., Lalmas, M., Valkanas, G.: Understanding within-content engagement through pattern analysis of mouse gestures. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pp. 1439–1448. ACM, New York (2014)

    Google Scholar 

  4. Arapakis, I., Bai, X., Cambazoglu, B.B.: Impact of response latency on user behavior in web search. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 103–112. ACM, New York (2014)

    Google Scholar 

  5. Arguello, J., Wu, W.-C., Kelly, D., Edwards, A.: Task complexity, vertical display and user interaction in aggregated search. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 435–444. ACM, New York (2012)

    Google Scholar 

  6. Bangcuyo, R.G., Smith, K.J., Zumach, J.L., Pierce, A.M., Guttman, G.A., Simons, C.T.: The use of immersive technologies to improve consumer testing: the role of ecological validity, context and engagement in evaluating coffee. Food Qual. Prefer. 41, 84–95 (2015)

    Article  Google Scholar 

  7. Banhawi, F., Ali, N.M.: Measuring user engagement attributes in social networking application. In: Proceedings of the 2011 International Conference on Semantic Technology and Information Retrieval (STAIR), pp. 297–301. IEEE, New York (2011)

    Google Scholar 

  8. Bödker, S.: When second wave HCI meets third wave challenges. In: Proceedings of the 4th Nordic Conference on Human-Computer Interaction (NordCHI ’06), pp. 1–8. ACM, New York (2006). doi:10.1145/1182475.1182476

    Google Scholar 

  9. Bron, M., van Gorp, J., Nack, F., Baltussen, L.B., de Rijke, M.: Aggregated search interface preferences in multi-session search tasks. In: Proceedings of the 36th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 123–132. ACM, New York (2013)

    Google Scholar 

  10. Brooke, J.: SUS-A quick and dirty usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, A.L. (eds.) Usability Evaluation in Industry, pp. 4–7. Taylor and Francis, London (1996)

    Google Scholar 

  11. Busselle, R., Bilandzic, H.: Measuring narrative engagement. Media Psychol. 12 (4), 321–347 (2009)

    Article  Google Scholar 

  12. Bustillo, J., Garaizar, P.: Scratching the surface of digital literacy…but we need to go deeper. In: 2014 IEEE Frontiers in Education Conference Proceedings, pp. 1440–1443. IEEE, New York (2014)

    Google Scholar 

  13. Choi, H.S.: The impact of visuospatial characteristics of video games on improvements in cognitive abilities. Unpublished Doctoral Dissertation, North Carolina State University (2013)

    Google Scholar 

  14. Cian, L., Krishna, A., Elder, R.S.: This logo moves me: dynamic imagery from static image. J. Mark. Res. 51 (2), 184–197 (2013)

    Article  Google Scholar 

  15. Creswell, J.W.: Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage, Los Angeles (2003)

    Google Scholar 

  16. De Moor, K., Mazza, F., Hupont, I., Ríos Quintero, M., Mäki, T., Varela, M.: Chamber QoE: a multi-instrumental approach to explore affective aspects in relation to quality of experience. In: Proceedings of SPIE 9014, Human Vision and Electronic Imaging XIX, 90140U (2014). doi:10.1117/12.2042243

    Google Scholar 

  17. Deci, E.L., Ryan, R.M.: Handbook of Self-Determination Research. University Rochester Press, New York (2012)

    Google Scholar 

  18. Devellis, R.F.: Scale Development: Theory and Applications, 2nd edn. Sage, Thousand Oaks (2003)

    Google Scholar 

  19. Fuchsberger, V., Sellner, W., Moser, C., Tscheligi, M.: Benefits and hurdles for older adults in intergenerational online interactions. In: Miesenberger, K., Karshmer, A., Penaz, P., Zagler, W. (eds.) Computers Helping People with Special Needs. Lecture Notes in Computer Science, vol. 7382, pp. 697–704. Springer, Berlin/Heidelberg (2012)

    Chapter  Google Scholar 

  20. Grafsgaard, J.F.: Multimodal affect modeling in task-oriented tutorial dialogue. Unpublished Doctoral Dissertation, North Carolina State University (2014)

    Google Scholar 

  21. Halpern, D.: Towards a networked public sphere: how social media triggers civic engagement through news consumption and political discussion. Unpublished Doctoral Dissertation, Rutgers University (2013)

    Google Scholar 

  22. Hong, T.: Internet health search: when process complements goals. J. Am. Soc. Inf. Sci. Technol. 63 (11), 2283–2293 (2012)

    Article  Google Scholar 

  23. Jacques, R.D.: The nature of engagement and its role in hypermedia evaluation and design. Unpublished Doctoral Dissertation, South Bank University (1996)

    Google Scholar 

  24. Kajalainen, K.: Increasing the enjoyment of online video increasing the enjoyment of online video content with topical interactivity. Unpublished Doctoral Dissertation, Aalto University (2015)

    Google Scholar 

  25. Kelly, D.: Methods for evaluating interactive information retrieval systems with users. Found. Trends Inf. Retr. 3 (1–2), 1–224 (2009)

    Google Scholar 

  26. Kwak, N., Williams, A.E., Wang, X., Lee, H.: Talking politics and engaging politics: an examination of the interactive relationships between structural features of political talk and discussion engagement. Commun. Res. 32 (1), 87–111 (2005)

    Article  Google Scholar 

  27. Lalmas, M. O’Brien, H., Yom-Tov, E.: Measuring user engagement. Synth. Lect. Inf. Concepts Retr. Serv. 6 (4), 1–132 (2014)

    Google Scholar 

  28. Lawrence, B., Fournier, S., Brunel, F.: When companies don’t make the ad: a multimethod inquiry into the differential effectiveness of consumer-generated advertising. J. Advert. 42 (4), 292–307 (2013)

    Article  Google Scholar 

  29. Lee, Y.-C., Lin, W.-C., Cherng, F.-Y., Wang, H.-C., Sung, C.-Y., King, J.-T.: Using time-anchored peer comments to enhance social interaction in online educational videos. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 689–698. ACM, New York (2015)

    Google Scholar 

  30. Levesque, V., Oram, L., MacLean, K., Cockburn, A., Marchuk, N.D., Johnson, D., Colgate, J.E., Peshkin, M.A.: Enhancing physicality in touch interaction with programmable friction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2481–2490. ACM, New York (2011)

    Google Scholar 

  31. Lindgaard, G., Kirakowski, J.: Introduction to the special issue: the tricky landscape of developing rating scales in HCI. Interact. Comput. 25 (4), 271–277 (2013)

    Article  Google Scholar 

  32. McCay-Peet, L., Lalmas, M., Navalpakkam, V.: On salience, affect and focused attention. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 541–550. ACM, New York (2012). doi:10.1145/2207676.2207751

    Google Scholar 

  33. Moshfeghi, Y., Matthews, M., Blanco, R., Jose, J.M.: Influence of timeline and named-entity components on user engagement. In: Serdyukov, P., Braslavski, P., Kuznetsov, S., Kamps, J., Rüger, S., Agichtein, E., Segalovich, I., Yilmaz, E. (eds.) Advances in Information Retrieval. Lecture Notes in Computer Science, vol. 7814, pp. 305–317. Springer, Berlin/Heidelberg (2013)

    Chapter  Google Scholar 

  34. O’Brien, H.: Defining and measuring user experiences with technology. Unpublished Doctoral Dissertation, Dalhousie University (2008)

    Google Scholar 

  35. O’Brien, H.: The influence of hedonic and utilitarian motivations on user engagement: the case of online shopping experiences. Interact. Comput. 22 (5), 344–352 (2010)

    Article  Google Scholar 

  36. O’Brien, H.: The role of story and media in user engagement with online news. In: Proceedings of the Annual Conference of Canadian Association of Information Science (CAIS), CAIS, Victoria (2013)

    Google Scholar 

  37. O’Brien, H., Cairns, P.: An empirical evaluation of the user engagement scale (UES) in online news environments. Inf. Process. Manag. 51 (4), 413–427 (2015)

    Article  Google Scholar 

  38. O’Brien, H., Lebow, M.: A mixed methods approach to measuring user experience in online news interactions. J. Assoc. Inf. Sci. Technol. 64 (8), 1543–1556 (2013)

    Article  Google Scholar 

  39. O’Brien, H., Toms, E.G.: What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inf. Sci. Technol. 59 (6), 938–955 (2008)

    Google Scholar 

  40. O’Brien, H., Toms, E.G.: The development and evaluation of a survey to measure user engagement in e-commerce environments. J. Am. Soc. Inf. Sci. Technol. 61 (1), 50–69 (2010)

    Article  Google Scholar 

  41. O’Brien, H., Toms, E.G.: Is there a universal instrument for measuring interactive information retrieval? The case of the user engagement scale. In: Proceedings of Information Interaction in Context (IIiX), pp. 335–340. ACM, Rutgers (2010). doi:10.1145/1840784.1840835

    Google Scholar 

  42. O’Brien, H., Toms, E.G.: Examining the generalizability of the user engagement scale (UES) in exploratory search. Inf. Process. Manag. 49 (5), 1092–1107 (2013)

    Article  Google Scholar 

  43. Okoro, E.M.: A study of different representation conventions during investigatory sensemaking. Unpublished Masters Thesis, Middlesex University (2014)

    Google Scholar 

  44. Parra, D., Brusilovsky, P.: User-controllable personalization: a case study with SetFusion. Int. J. Hum. Comput. Stud. 77, 43–67 (2014)

    Google Scholar 

  45. Peterson, R.A.: Constructing Effective Questionnaires. Sage Publications, Thousand Oaks (2000)

    Google Scholar 

  46. Reitz, A.R.: Online consumer engagement: understanding the antecedents and outcomes. Unpublished Doctoral Dissertation, Colorado State University (2012)

    Google Scholar 

  47. Seedorf, S., Thum, C., Schulze, T., Pfrogner, L.: Social co-browsing in online shopping: the impact of real-time collaboration on user engagement. In: Proceedings of the Twenty Second European Conference on Information Systems, Tel Aviv 2014. AIS Electronic Library (2014)

    Google Scholar 

  48. Sharek, D.J.: Investigating real-time predictors of engagement: implications for adaptive video games and online training. Unpublished Doctoral Dissertation, North Carolina State University (2012)

    Google Scholar 

  49. Shirzad, N.: The use of physiological signals and motor performance metrics in task difficulty adaptation: improving engagement in robot-assisted movement therapy. Unpublished Doctoral Dissertation, The University of British Columbia (2013)

    Google Scholar 

  50. Tabachnick, B.G., Fidell, L.S.: Using Multivariate Statistics, 6th International edition (cover) edn. Sage Publications, Thousand Oaks (2013)

    Google Scholar 

  51. Vail, A.K., Boyer, K.E., Wiebe, E.N., Lester, J.C.: The Mars and Venus effect: the influence of user gender on the effectiveness of adaptive task support. In: Ricci, F., Bontcheva, K., Conlan, O., Lawless, S. (eds.) User Modeling, Adaptation and Personalization. Lecture Notes in Computer Science, vol. 9146, pp. 265–276. Springer, Berlin/Heidelberg (2015)

    Chapter  Google Scholar 

  52. Warnock, D., Lalmas, M.: An exploration of cursor tracking data. arXiv preprint (2015). arXiv:1502.00317

    Google Scholar 

  53. Webster, J., Ho, H.: Audience engagement in multimedia presentations. Data Base Adv. Inf. Syst. 28 (2), 63–77 (1997)

    Article  Google Scholar 

  54. Whitman, L.: The effectiveness of interactivity in multimedia software tutorials. Unpublished Doctoral Dissertation, North Carolina State University (2013)

    Google Scholar 

  55. Wiebe, E.N., Lamb, A., Hardy, M., Sharek, D.: Measuring engagement in video game-based environments: investigation of the user engagement scale. Comput. Educ. 32, 123–132 (2014)

    Google Scholar 

  56. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence questionnaire. Presence: Teleoper. Virtual Environ. 7 (3), 225–240 (1998)

    Article  Google Scholar 

  57. Zhu., Y., Heynderick, I., Redi, J.A.: Alone or together: measuring users’ viewing experience in different social contexts. In: Proceedings of SPIE 9014, Human Vision and Electronic Imaging XIX, 90140W (2014). doi:10.1117/12.2042867

    Google Scholar 

Download references

Acknowledgements

I wish to express appreciation to the Social Science and Humanities Research Council of Canada and the Networks of Centres of Excellence in Graphics, Animation and New Media Project for funding support.

Author information

Authors and Affiliations

  1. iSchool, The University of British Columbia, Vancouver, BC, Canada

    Heather O’Brien

Authors
  1. Heather O’Brien
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Heather O’Brien .

Editor information

Editors and Affiliations

  1. University of British Columbia, Vancouver, British Columbia, Canada

    Heather O'Brien

  2. University of York, York, United Kingdom

    Paul Cairns

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

O’Brien, H. (2016). Translating Theory into Methodological Practice. In: O'Brien, H., Cairns, P. (eds) Why Engagement Matters. Springer, Cham. https://doi.org/10.1007/978-3-319-27446-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27446-1_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27444-7

  • Online ISBN: 978-3-319-27446-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation