skip to main content
10.1145/3532106.3533537acmconferencesArticle/Chapter ViewAbstractPublication PagesdisConference Proceedingsconference-collections
research-article

Flexibility and Social Disconnectedness: Assessing University Students’ Well-Being Using an Experience Sampling Chatbot and Surveys Over Two Years of COVID-19

Published: 13 June 2022 Publication History

Abstract

COVID-19 caused an abrupt switch from face-to-face to online teaching. This led to unknown challenges and consequences for students and lecturers. In the first semester after its outbreak, we developed a messenger-based chatbot to perform an experience sampling study to evaluate students’ well-being and experiences (n = 31) with the radical changes in higher education. Finding a decrease in students’ perceived motivation but an increase in productivity, we conducted a follow-up survey to compare the development a year later (n = 41). Our results revealed two main student profiles, one feeling severely impacted by the persisting social distance in their study performance and the other appreciating the flexibility and expended free time due to the changes in the teaching formats. Based on our findings, we introduce implications for the overall design of higher education and show the benefits and challenges of combining chatbot-enabled experience sampling with traditional surveys.

Supplemental Material

ZIP File
Supplemental files

References

[1]
Fehmi Ben Abdesslem, Iain Parris, and Tristan Nicholas Hoang Henderson. 2010. Mobile experience sampling: Reaching the parts of Facebook other methods cannot reach. In Proceedings of the Privacy and Usability Methods Pow-wow. Dundee, United Kingdom.
[2]
Firoj al Mamun, Ismail Hosen, Jannatul Mawa Misti, Mark Mohan Kaggwa, and Mohammed A. Mamun. 2021. Mental Disorders of Bangladeshi Students During the COVID-19 Pandemic: A Systematic Review. Psychology Research and Behavior Management Volume 14 (May 2021), 645–654. https://doi.org/10.2147/PRBM.S315961
[3]
Joseph Amankwah-Amoah, Zaheer Khan, Geoffrey Wood, and Gary Knight. 2021. COVID-19 and digitalization: The great acceleration. Journal of Business Research 136 (2021), 602–611. https://doi.org/10.1016/j.jbusres.2021.08.011
[4]
Florian Bemmann, Ramona Schoedel, Niels Van Berkel, and Daniel Buschek. 2021. Chatbots for Experience Sampling-Initial Opportunities and Challenges. In Adjunct Proceedings of the ACM Intelligent User Interface Conference, Vol. 2903. Association for Computing Machinery, United States.
[5]
Niels Van Berkel, Denzil Ferreira, and Vassilis Kostakos. 2017. The Experience Sampling Method on Mobile Devices. Comput. Surveys 50, 6 (Dec. 2017), 1–40. https://doi.org/10.1145/3123988
[6]
Virginia Braun and Victoria Clarke. 2017. Applied Qualitative Research in Psychology. Applied Qualitative Research in Psychology 0887, 2006 (2017). https://doi.org/10.1057/978-1-137-35913-1
[7]
Daniel Buschek, Mariam Hassib, and Florian Alt. 2018. Personal Mobile Messaging in Context: Chat Augmentations for Expressiveness and Awareness. ACM Transactions on Computer-Human Interaction 25, 4 (Sept. 2018), 1–33. https://doi.org/10.1145/3201404
[8]
Daniel Buschek, Mariam Hassib, and Florian Alt. 2018. Personal Mobile Messaging in Context: Chat Augmentations for Expressiveness and Awareness. ACM Transactions on Computer-Human Interaction 25, 4, Article 23 (aug 2018), 33 pages. https://doi.org/10.1145/3201404
[9]
Yu-Lin Chang, Yung-Ju Chang, and Chih-Ya Shen. 2019. She is in a Bad Mood Now: Leveraging Peers to Increase Data Quantity via a Chatbot-Based ESM. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, Taipei Taiwan, 1–6. https://doi.org/10.1145/3338286.3344406
[10]
Kunal Chaturvedi, Dinesh Kumar Vishwakarma, and Nidhi Singh. 2021. COVID-19 and its impact on education, social life and mental health of students: A survey. Children and Youth Services Review 121 (2021), 105866. https://doi.org/10.1016/j.childyouth.2020.105866
[11]
Chola Chhetri. 2020. ”I Lost Track of Things”: Student Experiences of Remote Learning in the Covid-19 Pandemic. In Proceedings of the 21st Annual Conference on Information Technology Education (Virtual Event, USA) (SIGITE ’20). Association for Computing Machinery, New York, NY, USA, 314–319. https://doi.org/10.1145/3368308.3415413
[12]
Karen Church, Mauro Cherubini, and Nuria Oliver. 2014. A large-scale study of daily information needs captured in situ. ACM Transactions on Computer-Human Interaction 21, 2 (Feb. 2014), 1–46. https://doi.org/10.1145/2552193
[13]
Karen Church and Rodrigo de Oliveira. 2013. What’s up with whatsapp?: comparing mobile instant messaging behaviors with traditional SMS. In Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services - MobileHCI ’13. ACM Press, Munich, Germany, 352. https://doi.org/10.1145/2493190.2493225
[14]
Rebecca A Croxton. 2014. The role of interactivity in student satisfaction and persistence in online learning. Journal of Online Learning and Teaching 10, 2 (2014), 314. Publisher: Multimedia Educational Resource for Learning and Online Teaching (MERLOT).
[15]
Mihaly Csikszentmihalyi and Reed Larson. 2014. Validity and Reliability of the Experience-Sampling Method. In Flow and the Foundations of Positive Psychology. Springer Netherlands, Dordrecht, 35–54. https://doi.org/10.1007/978-94-017-9088-8_3
[16]
Sir John Daniel. 2020. Education and the COVID-19 pandemic. PROSPECTS 49, 1-2 (Oct. 2020), 91–96. https://doi.org/10.1007/s11125-020-09464-3
[17]
Dan Davis, Guanliang Chen, Claudia Hauff, and Geert-Jan Houben. 2018. Activating learning at scale: A review of innovations in online learning strategies. Computers & Education 125 (2018), 327–344. https://doi.org/j.compedu.2018.05.019
[18]
Sara Isabella de Freitas, John Morgan, and David Gibson. 2015. Will MOOCs transform learning and teaching in higher education? Engagement and course retention in online learning provision: Engagement and course retention in online learning provision. British Journal of Educational Technology 46, 3 (May 2015), 455–471. https://doi.org/10.1111/bjet.12268
[19]
John David N Dionisio, William G Burns III, and Richard Gilbert. 2013. 3D virtual worlds and the metaverse: Current status and future possibilities. ACM Computing Surveys (CSUR) 45, 3 (2013), 1–38. https://doi.org/10.1145/2480741.2480751
[20]
Fiona Draxler, Christina Schneegass, Jonas Safranek, and Heinrich Hussmann. 2021. Why Did you Stop? - Investigating Origins and Effects of Interruptions during Mobile Language Learning. In Mensch und Computer 2021. ACM, Ingolstadt Germany, 21–33. https://doi.org/10.1145/3473856.3473881
[21]
Aziz Essadek and Thomas Rabeyron. 2020. Mental health of French students during the Covid-19 pandemic. Journal of Affective Disorders 277 (Dec. 2020), 392–393. https://doi.org/10.1016/j.jad.2020.08.042
[22]
Jon Froehlich, Mike Y. Chen, Sunny Consolvo, Beverly Harrison, and James A. Landay. 2007. MyExperience: a system for in situtracing and capturing of user feedback on mobile phones. In Proceedings of the 5th international conference on Mobile systems, applications and services - MobiSys ’07. ACM Press, San Juan, Puerto Rico, 57. https://doi.org/10.1145/1247660.1247670
[23]
Hong Gao, Efe Bozkir, Lisa Hasenbein, Jens-Uwe Hahn, Richard Göllner, and Enkelejda Kasneci. 2021. Digital transformations of classrooms in virtual reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–10. https://doi.org/10.1145/3411764.3445596
[24]
Darren Gergle and Eszter Hargittai. 2018. A methodological pilot for gathering data through text-messaging to study question-asking in everyday life. Mobile Media & Communication 6, 2 (May 2018), 197–214. https://doi.org/10.1177/2050157917741333
[25]
Asma Ghandeharioun, Daniel McDuff, Mary Czerwinski, and Kael Rowan. 2019. Towards Understanding Emotional Intelligence for Behavior Change Chatbots. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, Cambridge, United Kingdom, 8–14. https://doi.org/10.1109/ACII.2019.8925433
[26]
Erwin Handoko, Susie L. Gronseth, Sara G. McNeil, Curtis J. Bonk, and Bernard R. Robin. 2019. Goal Setting and MOOC Completion. The International Review of Research in Open and Distributed Learning 20, 3 (Feb. 2019). https://doi.org/10.19173/irrodl.v20i4.4270
[27]
Rick Harrington and Donald A. Loffredo. 2010. MBTI personality type and other factors that relate to preference for online versus face-to-face instruction. The Internet and Higher Education 13, 1-2 (Jan. 2010), 89–95. https://doi.org/10.1016/j.iheduc.2009.11.006
[28]
Jeremy F Huckins, Alex W daSilva, Weichen Wang, Elin Hedlund, Courtney Rogers, Subigya K Nepal, Jialing Wu, Mikio Obuchi, Eilis I Murphy, Meghan L Meyer, Dylan D Wagner, Paul E Holtzheimer, and Andrew T Campbell. 2020. Mental Health and Behavior of College Students During the Early Phases of the COVID-19 Pandemic: Longitudinal Smartphone and Ecological Momentary Assessment Study. Journal of Medical Internet Research 22, 6 (June 2020), e20185. https://doi.org/10.2196/20185
[29]
Nataliya V. Ivankova and Sheldon L. Stick. 2007. Students’ Persistence in a Distributed Doctoral Program in Educational Leadership in Higher Education: A Mixed Methods Study. Research in Higher Education 48, 1 (Feb. 2007), 93–135. https://doi.org/10.1007/s11162-006-9025-4
[30]
M. Jun, S. Lee, and T. Shim. 2021. First-Year College Student Life Experiences during COVID-19 in South Korea. International journal of environmental research and public health 18 (2021), 9895. Issue 18. https://doi.org/10.3390/ijerph18189895
[31]
Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300316
[32]
Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300316
[33]
René F. Kizilcec, Mar Pérez-Sanagustín, and Jorge J. Maldonado. 2017. Self-regulated learning strategies predict learner behavior and goal attainment in Massive Open Online Courses. Computers & Education 104 (Jan. 2017), 18–33. https://doi.org/10.1016/j.compedu.2016.10.001
[34]
Yuchen Li, Yue Wang, Jingwen Jiang, Unnur A. Valdimarsdóttir, Katja Fall, Fang Fang, Huan Song, Donghao Lu, and Wei Zhang. 2021. Psychological distress among health professional students during the COVID-19 outbreak. Psychological Medicine 51, 11 (Aug. 2021), 1952–1954. https://doi.org/10.1017/S0033291720001555
[35]
Abdelsalam M. Maatuk, Ebitisam K. Elberkawi, Shadi Aljawarneh, Hasan Rashaideh, and Hadeel Alharbi. 2021. The COVID-19 pandemic and E-learning: challenges and opportunities from the perspective of students and instructors.Journal of Computing Higher Education(2021).
[36]
Berta Rodrigues Maia and Paulo César Dias. 2020. Ansiedade, depressão e estresse em estudantes universitários: o impacto da COVID-19. Estudos de Psicologia (Campinas) 37 (2020), e200067. https://doi.org/10.1590/1982-0275202037e200067
[37]
Karola Marky, Florian Müller, Markus Funk, Alexander Geiß, Sebastian Günther, Martin Schmitz, Jan Riemann, and Max Mühlhäuser. 2019. Teachyverse: Collaborative E-Learning in Virtual Reality Lecture Halls. In Proceedings of Mensch und Computer 2019 on - MuC’19. ACM Press, Hamburg, Germany, 831–834. https://doi.org/10.1145/3340764.3344917
[38]
Abhinav Mehrotra, Jo Vermeulen, Veljko Pejovic, and Mirco Musolesi. 2015. Ask, but don’t interrupt: the case for interruptibility-aware mobile experience sampling. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers - UbiComp ’15. ACM Press, Osaka, Japan, 723–732. https://doi.org/10.1145/2800835.2804397
[39]
Colin Milligan, Allison Littlejohn, and Anoush Margaryan. 2013. Patterns of engagement in connectivist MOOCs. Journal of Online Learning and Teaching 9, 2 (2013), 149–159. Publisher: MERLOT.
[40]
J.E. Mitchell. 2020. How do we think about labs and practical skills in an online context? (35 ed.). Godalming: Engineering Professors’ Council. https://doi.org/10.1007/s12528-021-09274-2
[41]
Gaëlle Molinari, Maxence Trannois, Aurélien Tabard, and Elise Lavoué. 2016. EMORE-L: an emotion reporting tool for distance learning. In Actes de la 28ième conférence francophone sur l’Interaction Homme-Machine on - IHM ’16. ACM Press, Fribourg, Switzerland, 167–176. https://doi.org/10.1145/3004107.3004126
[42]
Joyce Neroni, Celeste Meijs, Hieronymus J.M. Gijselaers, Paul A. Kirschner, and Renate H.M. de Groot. 2019. Learning strategies and academic performance in distance education. Learning and Individual Differences 73 (July 2019), 1–7. https://doi.org/10.1016/j.lindif.2019.04.007
[43]
Lorenz S Neuwirth, Svetlana Jović, and B Runi Mukherji. 2021. Reimagining higher education during and post-COVID-19: Challenges and opportunities. Journal of Adult and Continuing Education 27, 2 (2021), 141–156. https://doi.org/10.1177/1477971420947738
[44]
Gabriella Oliveira, Jorge Grenha Teixeira, Ana Torres, and Carla Morais. 2021. An exploratory study on the emergency remote education experience of higher education students and teachers during the COVID‐19 pandemic. British Journal of Educational Technology 52, 4 (July 2021), 1357–1376. https://doi.org/10.1111/bjet.13112
[45]
Prajwal Paudyal, Ayan Banerjee, Yijian Hu, and Sandeep Gupta. 2019. Davee: A deaf accessible virtual environment for education. In Proceedings of the 2019 on Creativity and Cognition. 522–526. https://doi.org/10.1145/3325480.3326546
[46]
Reinhard Pekrun, Anne C. Frenzel, Thomas Goetz, and Raymond P. Perry. 2007. The Control-Value Theory of Achievement Emotions. In Emotion in Education. Elsevier, 13–36. https://doi.org/10.1016/B978-012372545-5/50003-4
[47]
Chrysi Rapanta, Luca Botturi, Peter Goodyear, Lourdes Guàrdia, and Marguerite Koole. 2020. Online University Teaching During and After the Covid-19 Crisis: Refocusing Teacher Presence and Learning Activity. Postdigital Science and Education 2, 3 (Oct. 2020), 923–945. https://doi.org/10.1007/s42438-020-00155-y
[48]
Shazia Rashid and Sunishtha Singh Yadav. 2020. Impact of Covid-19 Pandemic on Higher Education and Research. Indian Journal of Human Development 14, 2 (2020), 340–343. https://doi.org/10.1177/0973703020946700
[49]
Daniel J. Rough and Aaron Quigley. 2020. End-User Development of Experience Sampling Smartphone Apps -Recommendations and Requirements. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (June 2020), 1–19. https://doi.org/10.1145/3397307
[50]
Samara Ruiz, Sven Charleer, Maite Urretavizcaya, Joris Klerkx, Isabel Fernández-Castro, and Erik Duval. 2016. Supporting learning by considering emotions: tracking and visualization a case study. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ’16. ACM Press, Edinburgh, United Kingdom, 254–263. https://doi.org/10.1145/2883851.2883888
[51]
Erika K. Smith and Ece Kaya. 2021. Online University Teaching at the time of COVID-19 (2020): An Australian Perspective. IAFOR Journal of Education 9, 2 (April 2021), 183–200. https://doi.org/10.22492/ije.9.2.11
[52]
Changwon Son, Sudeep Hegde, Alec Smith, Xiaomei Wang, and Farzan Sasangohar. 2020. Effects of COVID-19 on College Students’ Mental Health in the United States: Interview Survey Study. Journal of Medical Internet Research 22, 9 (Sept. 2020), e21279. https://doi.org/10.2196/21279
[53]
C-H. Su and C-H. Cheng. 2015. A mobile gamification learning system for improving the learning motivation and achievements: A mobile gamification learning system. Journal of Computer Assisted Learning 31, 3 (June 2015), 268–286. https://doi.org/10.1111/jcal.12088
[54]
Bernardo Tabuenca, Marco Kalz, Hendrik Drachsler, and Marcus Specht. 2015. Time will tell: The role of mobile learning analytics in self-regulated learning. Computers & Education 89 (Nov. 2015), 53–74. https://doi.org/10.1016/j.compedu.2015.08.004 Publisher: Elsevier.
[55]
Ella Tallyn, Hector Fried, Rory Gianni, Amy Isard, and Chris Speed. 2018. The Ethnobot: Gathering Ethnographies in the Age of IoT. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–13. https://doi.org/10.1145/3173574.3174178
[56]
Faiza Tazi, Sunny Shrestha, Dan Norton, Kathryn Walsh, and Sanchari Das. 2021. Parents, Educators, Caregivers Cybersecurity & Privacy Concerns for Remote Learning During COVID-19. In CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter (Online (Athens, Greece), Greece) (CHI Greece 2021). Association for Computing Machinery, New York, NY, USA, Article 16, 5 pages. https://doi.org/10.1145/3489410.3489426
[57]
Virginia Thomas and Margarita Azmitia. 2016. Tapping Into the App: Updating the Experience Sampling Method for the 21st Century. Emerging Adulthood 4, 1 (Feb. 2016), 60–67. https://doi.org/10.1177/2167696815618489
[58]
Niels van Berkel, Jorge Goncalves, Peter Koval, Simo Hosio, Tilman Dingler, Denzil Ferreira, and Vassilis Kostakos. 2019. Context-Informed Scheduling and Analysis: Improving Accuracy of Mobile Self-Reports. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300281
[59]
Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T. Campbell. 2014. StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp ’14 Adjunct. ACM Press, Seattle, Washington, 3–14. https://doi.org/10.1145/2632048.2632054
[60]
Xiaomei Wang, Sudeep Hegde, Changwon Son, Bruce Keller, Alec Smith, and Farzan Sasangohar. 2020. Investigating Mental Health of US College Students During the COVID-19 Pandemic: Cross-Sectional Survey Study. Journal of Medical Internet Research 22, 9 (Sept. 2020), e22817. https://doi.org/10.2196/22817
[61]
Ziang Xiao, Michelle X. Zhou, Q. Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. 2020. Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-Ended Questions. ACM Transactions on Computer-Human Interaction 27, 3, Article 15 (jun 2020), 37 pages. https://doi.org/10.1145/3381804
[62]
Jane Y-K Yau, Mike Joy, and Stephan Dickert. 2010. A Mobile Context-aware Framework for Managing Learning Schedules–Data Analysis from a Diary Study. Journal of Educational Technology & Society 13, 3 (2010), 22–32.
[63]
Saijing Zheng, Pamela Wisniewski, Mary Beth Rosson, and John M Carroll. 2016. Ask the Instructors: Motivations and Challenges of Teaching Massive Open Online Courses. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing - CSCW ’16. ACM Press, San Francisco, California, USA, 205–220. https://doi.org/10.1145/2818048.2820082

Cited By

View all
  • (2024)Investigating the Effects of Real-time Student Monitoring Interface on Instructors’ Monitoring Practices in Online TeachingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642845(1-11)Online publication date: 11-May-2024
  • (2023)Digitale Methoden in Bildungsforschung und BildungspraxisDatafizierung (in) der Bildung10.14361/9783839465820-006(81-102)Online publication date: 4-Dec-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
DIS '22: Proceedings of the 2022 ACM Designing Interactive Systems Conference
June 2022
1947 pages
ISBN:9781450393584
DOI:10.1145/3532106
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. chatbot
  2. distance teaching
  3. experience sampling
  4. higher education

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Data Availability

Funding Sources

Conference

DIS '22
Sponsor:
DIS '22: Designing Interactive Systems Conference
June 13 - 17, 2022
Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Upcoming Conference

DIS '25
Designing Interactive Systems Conference
July 5 - 9, 2025
Funchal , Portugal

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)71
  • Downloads (Last 6 weeks)9
Reflects downloads up to 03 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Investigating the Effects of Real-time Student Monitoring Interface on Instructors’ Monitoring Practices in Online TeachingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642845(1-11)Online publication date: 11-May-2024
  • (2023)Digitale Methoden in Bildungsforschung und BildungspraxisDatafizierung (in) der Bildung10.14361/9783839465820-006(81-102)Online publication date: 4-Dec-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media