Abstract
The Peer Data Labelling System (PDLS) is a novel and extensible approach to generating labelled data suitable for training supervised machine learning (ML) algorithms for use in Child Computer Interaction (CCI) research and development. For a supervised ML model to make accurate predictions it requires accurate data on which to train. Poor quality input data to systems results in poor quality outputs often referred to as garbage in, garbage out (GIGO) systems.
PDLS is an alternative system to commonly employed approaches to facial and emotion recognition such as the Facial Action Coding System (FACS) or algorithmic approaches such as AFFDEX or FACET.
This paper presents the approaches taken to evaluate the effectiveness of PDLS. Algorithmic approaches did not produce consistent classifications and major amendments to the PDLS would be required if that validation route was pursued. The human review process found that the pupil observers and reviewers reached consensus in classifying most of the data as engaged. Recognising disengagement is more challenging, and further work is required to ensure that there is more consistency in what the participants recognise as engagement and disengagement.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bishay, M., Preston, K., Strafuss, M., Page, G., Turcot, J., Mavadati, M.: Affdex 2.0: a real-time facial expression analysis toolkit. arXiv preprint arXiv:2202.12059 (2022)
Bryant, D., Howard, A.: A comparative analysis of emotion-detecting AI systems with respect to algorithm performance and dataset diversity. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 377–382 (2019)
Christenson, S., Reschly, A.L., Wylie, C., et al.: Handbook of Research on Student Engagement, vol. 840. Springer, Heidelberg (2012). https://doi.org/10.1007/978-1-4614-2018-7
Cohn, J.F., Ambadar, Z., Ekman, P.: Observer-based measurement of facial expression with the facial action coding system. In: The Handbook of Emotion Elicitation and Assessment, vol. 1, no. 3, pp. 203–221 (2007)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Dupré, D., Krumhuber, E.G., Küster, D., McKeown, G.J.: A performance comparison of eight commercially available automatic classifiers for facial affect recognition. PLoS ONE 15(4), e0231968 (2020)
Ekman, P.: Facial action coding system, January 2020. https://www.paulekman.com/facial-action-coding-system/
Ekman, P., Friesen, W.V.: Facial action coding system. Environ. Psychol. Nonverbal Behav. (1978)
Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: potential of the concept, state of the evidence. Rev. Educ. Res. 74(1), 59–109 (2004)
Groccia, J.E.: What is student engagement? New Dir. Teach. Learn. 2018(154), 11–20 (2018)
iMotions: Unpack human behavior, November 2022. https://imotions.com/
Khan, R.A., Crenn, A., Meyer, A., Bouakaz, S.: A novel database of children’s spontaneous facial expressions (LIRIS-CSE). Image Vis. Comput. 83, 61–69 (2019)
Kulke, L., Feyerabend, D., Schacht, A.: A comparison of the Affectiva Imotions facial expression analysis software with EMG for identifying facial expressions of emotion. Front. Psychol. 11, 329 (2020)
Littlewort, G., et al.: The computer expression recognition toolbox (CERT). In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition (FG), pp. 298–305. IEEE (2011)
LoBue, V., Thrasher, C.: The child affective facial expression (CAFE) set: validity and reliability from untrained adults. Front. Psychol. 5, 1532 (2015)
McDuff, D., Mahmoud, A., Mavadati, M., Amr, M., Turcot, J., Kaliouby, R.E.: Affdex SDK: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 3723–3726 (2016)
Negrão, J.G., et al.: The child emotion facial expression set: a database for emotion recognition in children. Front. Psychol. 12, 666245 (2021)
Parsonage, G., Horton, M., Read, J.: The peer data labelling system (PDLS). A participatory approach to classifying engagement in the classroom. In: Abdelnour Nocera, J., Kristın Lárusdóttir, M., Petrie, H., Piccinno, A., Winckler, M. (eds.) INTERACT 2023. LNCS, vol. 14143, pp. 224–233. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-42283-6_13
Stöckli, S., Schulte-Mecklenbeck, M., Borer, S., Samson, A.C.: Facial expression analysis with affdex and facet: a validation study. Behav. Res. Methods 50(4), 1446–1460 (2018)
Testerman, M.: Databases (a-z) - face image databases - research guides at Princeton university, January 2022. https://libguides.princeton.edu/facedatabases. Accessed 22 Feb 2024
Webb, R., Ayers, S., Endress, A.: The city infant faces database: a validated set of infant facial expressions. Behav. Res. Methods 50(1), 151–159 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Parsonage, G., Horton, M., Read, J. (2024). Evaluating the Effectiveness of the Peer Data Labelling System (PDLS). In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2024. Lecture Notes in Computer Science(), vol 14734. Springer, Cham. https://doi.org/10.1007/978-3-031-60606-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-60606-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-60605-2
Online ISBN: 978-3-031-60606-9
eBook Packages: Computer ScienceComputer Science (R0)