Skip to main content
SpringerLink
Log in

A Holistic Approach to Measuring User Engagement

  • Chapter
  • First Online:
Book cover New Directions in Third Wave Human-Computer Interaction: Volume 2 - Methodologies

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

This chapter provides an overview of methodological approaches and current work in the evaluation of user engagement (UE). Using a series of propositions about the nature of engagement, I review a selection of recent research that utilizes varied methodological approaches to study UE in various human-computer interaction settings. The propositions and the reviewed literature are used to propose a methodological framework to guide decision making and reflection regarding how UE will be evaluated in a given context. The chapter concludes with reflections on broader issues related to how researchers’ methodological stances influence the evaluation of UE. Overall, the chapter argues that UE should be measured using a thoughtful mix of qualitative and quantitative methods, considering the particulars of the use context, and balancing established and emerging subjective and objective metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For more in-depth discussion on different UE evaluation approaches, please see Lalmas et al. (2014) and Yardley et al. (2016).

  2. 2.

    For a more in-depth discussion of issues inherent in questionnaire development, selection, adaptation, analysis and reporting, please see O’Brien and McKay-Peet (2017).

References

  • Absar R, O’Brien H, Webster ET (2014) Exploring social context in mobile information behavior. In: Proceedings of the American Society for Information Science and Technology, Seattle, WA, 31 October – 5 November 2014

    Google Scholar 

  • Agarwal R, Karahanna E (2000) Time flies when you're having fun: cognitive absorption and beliefs about information technology usage. MIS Quart 24(4):665–694

    Article  Google Scholar 

  • Andujar M, Morreale P, Jimenez Y, Jimenez L, Gilbert JE (2017) Evaluation of user’s affective engagement while interacting with educational technologies: a pilot study. In: Advances in Neuroergonomics and Cognitive Engineering. Springer, Heidelberg, pp 97–105

    Chapter  Google Scholar 

  • Anzalone SM, Boucenna S, Ivaldi S, Chetouani M (2015) Evaluating the engagement with social robots. Int J Soc Robot 7(4):465–478

    Article  Google Scholar 

  • Aristeidou M (2016) Citizen inquiry: engaging citizens in online communities of scientific inquiries. Dissertation, Institute of Educational Technology

    Google Scholar 

  • Bagnara S, Pozzi S (2016) The third wave of human computer interaction: from interfaces to digital ecologies. DigitCult 1(1):59–65

    Google Scholar 

  • Bødker S (2015) Third-wave HCI, 10 years later—participation and sharing. Interactions 22(5):24–31

    Article  Google Scholar 

  • de Guinea AO, Titah R, Léger PM, Micheneau T (2012) Neurophysiological correlates of information systems commonly used self-reported measures: a multitrait multimethod study. In: Hawaii International Conference on System Science (HICSS), IEEE, Maui, HI, 4–7 January 2012, pp. 562–571

    Google Scholar 

  • Flores FDN (2016) Engaging emerging adults in the design process of technology driven health interventions. Dissertation, The University of Wisconsin-Madison

    Google Scholar 

  • Freund L, Kopak R, O’Brien H (2016) The effects of textual environment on reading comprehension: implications for searching as learning. J Inf Sci 42(1):79–93

    Article  Google Scholar 

  • Gosling SD, Rentfrow PJ, Swann WB (2003) A very brief measure of the big-five personality domains. J Res Pers 37(6):504–528

    Article  Google Scholar 

  • Harrison S, Tatar D, Sengers P (2007) The three paradigms of HCI. In: Alt. Chi. session at SIGCHI conference on human factors in computing systems, San Jose, CA, 28 April – 3 May, 2007

    Google Scholar 

  • Jacques RD (1996) The nature of engagement and its role in hypermedia evaluation and design. Dissertation, South Bank University, London

    Google Scholar 

  • Jacques R, Preece J, Carey T (1995) Engagement as a design concept for multimedia. Can J Educ Commun 24(1):49–59

    Google Scholar 

  • Jensen ML, Lee YH, Piercy CW, Dunbar NE, Elizondo J, Bessarabova E, Twyman NW, Burgoon JK, Valacich JS, Adame B, Miller C (2016) Exploring failure and engagement in a complex digital training game: a multi-method examination. AIS Trans Hum Comput Int 8(1):1–20

    Article  Google Scholar 

  • Lalmas M, O'Brien HL, Yom-Tov E (2014) Measuring user engagement. Syn Lect Inform Concept Retr Serv 6(4):1–132

    Google Scholar 

  • Laurel B (1993) Computers as theatre. Addison-Wesley, Boston

    Google Scholar 

  • Lebow M, O’Brien HL (2012) Is there a role for physiological methods in the evaluation of human-information interaction? https://doi.org/10.14288/1.0107446. Accessed 15 May 2017

  • Leiker AM, Miller M, Brewer L, Nelson M, Siow M, Lohse K (2016) The relationship between engagement and neurophysiological measures of attention in motion-controlled video games: a randomized controlled trial. JMIR Serious Games 4(1):e4. https://doi.org/10.2196/games.5460

    Article  Google Scholar 

  • Li C, Rusak Z, Horvath I, Kooijman A, Ji L (2016) Implementation and validation of engagement monitoring in an engagement enhancing rehabilitation system. IEEE Trans Neural Syst Rehabil Eng 99:726. https://doi.org/10.1109/TNSRE.2016.2591183

    Article  Google Scholar 

  • Lohse KR, Boyd LA, Hodges NJ (2016) Engaging environments enhance motor skill learning in a computer gaming task. J Motor Behav 48(2):172–182

    Article  Google Scholar 

  • Martey RM, Kenski K, Folkestad J, Feldman L, Gordis E, Shaw A, Stromer-Galley J, Clegg B, Zhang H, Kaufman N, Rabkin AN (2014) Measuring game engagement: multiple methods and construct complexity. Simulat Gaming 45(4–5):528–547

    Article  Google Scholar 

  • Mathur A, Lane ND, Kawsar F (2016) September. Engagement-aware computing: modelling user engagement from mobile contexts. In: Proceedings of the ACM international joint conference on pervasive and ubiquitous computing, Heidelberg, Germany, 12–16 September 2016, pp. 622–633

    Google Scholar 

  • McAuley E, Duncan T, Tammen VV (1989) Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: a confirmatory factor analysis. Res Q Exercise Sport 60(1):48–58

    Article  Google Scholar 

  • McCarthy J, Wright P (2004) Technology as experience. MIT Press, Cambridge, MA

    Book  Google Scholar 

  • Meza-Kubo V, Morán AL, Carrillo I, Galindo G, García-Canseco E (2016) Assessing the user experience of older adults using a neural network trained to recognize emotions from brain signals. J Biomed Inform 62:202–209

    Article  Google Scholar 

  • Nguyen C (2015) Understanding user engagement in the open collaboration model of crowdsourcing. Dissertation, University of Nebraska at Omaha

    Google Scholar 

  • O’Brien HL (2008) Defining and measuring engagement in user experiences with technology. Ph.D. dissertation, Dalhousie University, Canada. http://ezproxy.library.ubc.ca/​login?url=; https://search-proquestcomezproxy.library.ubc.ca/docview/304405933?accountid=14656. Accessed 3 Apr 2018

    Google Scholar 

  • O’Brien HL (2011) Exploring user engagement in online news interactions. In: Proceedings of the association for information science and technology, New Orleans, LA, 9–12 October 2011

    Google Scholar 

  • O’Brien HL (2016a) Theoretical perspectives on user engagement. In: O’Brien HL, Cairns P (eds) Why engagement matters: cross disciplinary perspectives on user engagement, springer international publishing. Springer, Cham, pp 1–26

    Google Scholar 

  • O’Brien HL (2016b) Translating theory into methodological practice. In: O’Brien HL, Cairns P (eds) Why engagement matters: cross disciplinary perspectives on user engagement, springer international publishing. Springer, Cham, pp 27–52

    Google Scholar 

  • O’Brien HL, Cairns P (2015) An empirical evaluation of the User Engagement Scale (UES) in online news environments. Inform Process Manag 51(4):413–427

    Article  Google Scholar 

  • O’Brien HL, Lebow M (2013) Mixed-methods approach to measuring user experience in online news interactions. J Assoc Inf Sci Tech 64(8):1543–1556

    Article  Google Scholar 

  • O’Brien HL, McKay-Peet L (2017) Asking good questions: questionnaire design and analysis in interactive information retrieval research. In: Proceedings of the conference on conference human information interaction and retrieval, Oslo, Norway, 7–12 March, pp. 27–36

    Google Scholar 

  • O’Brien HL, Toms EG (2008) What is user engagement? A conceptual framework for defining user engagement with technology. J Assoc Inf Sci Tech 59(6):938–955

    Article  Google Scholar 

  • O’Brien HL, Toms EG (2010) The development and evaluation of a survey to measure user engagement. J Assoc Inf Sci Tech 61(1):50–69

    Article  Google Scholar 

  • O’Brien HL, Freund L, Kopak R (2016) Investigating the role of user engagement in digital reading environments. In: Proceedings of the ACM on conference on human information interaction and retrieval, Carrboro, NC, 13–17 March 2017, pp. 71–80

    Google Scholar 

  • Ponciano L, Brasileiro F (2015) Finding volunteers’ engagement profiles in human computation for citizen science projects. arXiv preprint arXiv:1501.02134. Accessed 15 May 2017

    Google Scholar 

  • Stroop JR (1935) Studies of interference in serial verbal reactions. J Exp Psychol 18(6):643–662

    Article  Google Scholar 

  • Watson D, Clark LA, Tellegen A (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol 54(6):1063–1070

    Article  Google Scholar 

  • Webster J, Ho H (1997) Audience engagement in multimedia presentations. ACM SIGMIS Database 28(2):63–77

    Article  Google Scholar 

  • Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, Merchant GC, Naughton F, Blandford A (2016) Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med 51(5):833–842

    Article  Google Scholar 

  • Zhang J, Hamilton WL, Danescu-Niculescu-Mizil C, Jurafsky D, Leskovec J (2017) Community identity and user engagement in a multi-community landscape. In: Proceedings of the interactional conference on web and social media, Montreal, QC, 15–18 May 2017

    Google Scholar 

Download references

Acknowledgements

I wish to thank the Social Science and Humanities Council of Canada (SSHRC), the UBC Hampton Research Fund, and the Networks of Centres of Excellent Graphics, Animation and New Media Project (NCE GRAND) for generous funding support over the past several years. I would also like to acknowledge the colleagues – including and especially graduate students at the School of Library, Archival and Information Studies – who have worked with me over the years on various research projects for their insights, efforts, dedication and enthusiasm.

Author information

Authors and Affiliations

  1. School of Library, Archival and Information Studies, University of British Columbia, Vancouver, BC, Canada

    Heather L. O’Brien

Authors
  1. Heather L. O’Brien
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Heather L. O’Brien .

Editor information

Editors and Affiliations

  1. School of Interactive Arts & Technology, Simon Fraser University, Surrey, BC, Canada

    Michael Filimowicz

  2. School of Communications, Simon Fraser University, Burnaby, BC, Canada

    Veronika Tzankova

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

O’Brien, H.L. (2018). A Holistic Approach to Measuring User Engagement. In: Filimowicz, M., Tzankova, V. (eds) New Directions in Third Wave Human-Computer Interaction: Volume 2 - Methodologies . Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-73374-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-73374-6_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-73373-9

  • Online ISBN: 978-3-319-73374-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation