research-article

PupilNet, Measuring Task Evoked Pupillary Response using Commodity RGB Tablet Cameras: Comparison to Mobile, Infrared Gaze Trackers for Inferring Cognitive Load

Authors:

Chatchai Wangwiwattana,

Eric C. LarsonAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 4

Article No.: 171, Pages 1 - 26

https://doi.org/10.1145/3161164

Published: 08 January 2018 Publication History

Abstract

Pupillary diameter monitoring has been proven successful at objectively measuring cognitive load that might otherwise be unobservable. This paper compares three different algorithms for measuring cognitive load using commodity cameras. We compare the performance of modified starburst algorithm (from previous work) and propose two new algorithms: 2 Level Snakuscules and a convolutional neural network which we call PupilNet. In a user study with eleven participants, our comparisons show PupilNet outperforms other algorithms in measuring pupil dilation, is robust to various lighting conditions, and robust to different eye colors. We show that the difference between PupilNet and a gold standard head-mounted gaze tracker varies only from -2.6% to 2.8%. Finally, we also show that PupilNet gives similar conclusions about cognitive load during a longer duration typing task.

References

[1]

2015. Tobii Pro Glasses 2 wearable eye tracker. (Jun 2015). https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/

[2]

Alexandra Branzan Albu, Ben Widsten, Tiange Wang, Julie Lan, and Jordana Mah. 2008. A computer vision-based system for real-time detection of sleep onset in fatigued drivers. In 2008 IEEE Intelligent Vehicles Symposium. IEEE, 25--30. https://doi.org/10.1109/IVS.2008.4621133

[3]

Gary Aston-Jones and Jonathan D Cohen. 2005. An Integrative Theory of Locus Coeruleus-Norepinephrine Function: Adaptive Gain and Optimal Performance. Annual review of neuroscience 28 (2005), 403--50. https://doi.org/10.1146/

[4]

Gary Aston-Jones, Janusz Rajkowski, Piotr Kubiak, and Tatiana Alexinsky. 1994. Locus coeruleus neurons in monkey are selectively activated by attended cues in vigilance tasks. Journal of Neuroscience 14 (1994), 4467--4480.

[5]

Tadas Baltrušaitis, Peter Robinson, and Louis Philippe Morency. 2013. Constrained local neural fields for robust facial landmark detection in the wild. Proceedings of the IEEE International Conference on Computer Vision (2013), 354--361. https://doi.org/10.1109/ICCVW.2013.54

Digital Library

[6]

Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. OpenFace: an open source facial behavior analysis toolkit. In IEEE Winter Conference on Applications of Computer Vision.

[7]

Jackson Beatty. 1982. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological bulletin 91, 2 (1982), 276--292. https://doi.org/10.1037/0033-2909.91.2.276

[8]

Jackson Beatty. 1982. Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources. (1982), 276--292 pages.

[9]

Jackson Beatty and Brennis Lucero-Wagoner. 2000. The pupillary system. 142--162 pages. http://prx.library.gatech.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=psyh&AN=2000-03927-005&site=ehost-live

[10]

Craig W Berridge and Barry D Waterhouse. 2003. The locus coeruleusâĂŞnoradrenergic system: modulation of behavioral state and state-dependent cognitive processes. Brain Research Reviews 42, 1 (2003), 33--84. https://doi.org/10.1016/S0165-0173(03)00143-7

[11]

Zhijian Chen and Nelson Cowan. 2005. Chunk limits and length limits in immediate recall: a reconciliation. Journal of experimental psychology. Learning, memory, and cognition 31, 6 (11 2005), 1235--49. https://doi.org/10.1037/0278-7393.31.6.1235

[12]

John Daugman. 2004. How Iris Recognition Works. In IEEE Transactions on Circuits and Systems for Video Technology. Vol. 14. https://doi.org/10.1109/TCSVT.2003.818350

Digital Library

[13]

Pan Du, Warren A Kibbe, and Simon M Lin. 2006. Improved peak detection in mass spectrum by incorporating continuous wavelet transform-based pattern matching. Bioinformatics 22, 17 (2006), 2059--2065.

Digital Library

[14]

Maria K. Eckstein, BelÃľn Guerra-Carrillo, Alison T. Miller Singley, and Silvia A. Bunge. 2017. Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? Developmental Cognitive Neuroscience 25 (2017), 69--91. https://doi.org/10.1016/j.dcn.2016.11.001

[15]

Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2015. Excuse: Robust pupil detection in real-world scenarios. In International Conference on Computer Analysis of Images and Patterns. Springer, 39--51.

[16]

Kunihiko Fukushima and Sei Miyake. 1982. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets. Springer, 267--285.

[17]

Sanyam Garg, Abhinav Tripathi, and Edward Cutrell. 2016. Accurate eye center localization using Snakuscule. 2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016 (2016). https://doi.org/10.1109/WACV.2016.7477673

[18]

Alaa Hilal, Bassam Daya, and Pierre Beauseroy. [n. d.]. Hough Transform and Active Contour for Enhanced Iris Segmentation. ([n. d.]). https://www.ijcsi.org/papers/IJCSI-9-6-2-1-10.pdf

[19]

Qiong Huang, Ashok Veeraraghavan, and Ashutosh Sabharwal. 2015. TabletGaze: unconstrained appearance-based gaze estimation in mobile tablets. arXiv preprint arXiv:1508.01244 (2015).

[20]

Shamsi T Iqbal, Xianjun Sam Zheng, and Brian P Bailey. 2004. Task-evoked pupillary response to mental workload in human-computer interaction. Extended abstracts of the 2004 conference on Human factors and computing systems CHI 04 (2004), 1477. https://doi.org/10.1145/985921.986094

Digital Library

[21]

Amir-Homayoun Javadi, Zahra Hakimi, Morteza Barati, Vincent Walsh, and Lili Tcheang. 2015. SET: a pupil detection method using sinusoidal approximation. Frontiers in neuroengineering 8 (2015).

[22]

John S Kafka. 2016. Psychoanalysis and the Temporal Trace. Time and Trace: Multidisciplinary Investigations of Temporality (2016), 197.

[23]

Daniel Kahneman and Jackson Beatty. 1966. Pupil Diameter and Load on Memory. Source: Science, New Series 154, 3756 (12 1966), 1583--1585. http://www.jstor.org/stable/1720478http://www.jstor.org.proxy.libraries.smu.edu/stable/pdfplus/10.2307/1720478.pdf?acceptTC=truehttp://about.jstor.org/terms

[24]

Koray Kara, Dursun Karaman, Uzeyir Erdem, Mehmet Ayhan Congologlu, Ibrahim Durukan, and Abdullah Ilhan. 2013. Investigation of Autonomic Nervous System Functions by Pupillometry in Children with Attention-Deficit/ Hyperactivity Disorder Investigation of autonomic nervous system functions by pupillometry in children with Attention-Deficit/Hyperactivity Disorder. Bulletin of Clinical Psychopharmacology 23, 1 (2013). https://doi.org/10.5455/bcp.20121130085850

[25]

Canan Karatekin, David J Marcus, J W Couperous, and Jane W Couperus. 2007. Regulations of cognitive resources during sustained attention and working memory in 10-year-olds and adults. Psychophysiology 44, 1 (1 2007), 128--144. https://doi.org/10.1111/j.1469-8986.2006.00477.x

[26]

Michael Kass, Andrew Witkin, and Demetri Terzopoulos. 1988. Snakes: Active contour models. International journal of computer vision 1, 4 (1988), 321--331.

[27]

Vahid Kazemi and Josephine Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.13140/2.1.1212.2243

Digital Library

[28]

Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam:. International Conference on Learning Representations (ICLR2015) (12 2015). https://doi.org/10.1145/1830483.1830503

Digital Library

[29]

Jeff Klingner. 2010. Measuring cognitive load during visual tasks by combining pupillometry and eye tracking. Perspective May (2010), 130.

[30]

Jeff Klingner, Rakshit Kumar, and Pat Hanrahan. 2008. Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the 2008 symposium on Eye tracking research 8 applications - ETRA ‘08 1, 212 (2008), 69. https://doi.org/10.1145/1344471.1344489

Digital Library

[31]

Jaehan Koh, Venu Govindaraju, and Vipin Chaudhary. [n. d.]. A Robust Iris Localization Method Using an Active Contour Model and Hough Transform. ([n. d.]). https://pdfs.semanticscholar.org/4709/a9e2920f4083264f04e94c71463b528af128.pdf

[32]

Bruno Laeng, Marte Ørbo, Terje Holmlund, and Michele Miozzo. 2011. Pupillary Stroop effects. Cognitive processing 12, 1 (2 2011), 13--21. https://doi.org/10.1007/s10339-010-0370-z

[33]

Daniel Lafond, René Proulx, Alexis Morris, William Ross, Alexandre Bergeron-Guyard, and Mihaela Ulieru. 2014. Hci dilemmas for context-aware support in intelligence analysis. In Adapt. 2014, Sixth Int. Conf. Adapt. Self-Adaptive Syst. Appl. 68--72.

[34]

Yann LeCun et al. 1989. Generalization and network design strategies. Connectionism in perspective (1989), 143--155.

[35]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.

[36]

Dongheng Li, David Winfield, and Derrick J Parkhurst. 2012. Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. (2012). https://pdfs.semanticscholar.org/db1d/7f94e91feea0a0e0b2f4563f2d05b0338732.pdf

[37]

Irene E. Loewenfeld. 1993. The pupil: Anatomy, physiology, and clinical applications. Wayne State University Press. Google Scholar, Detroit, MI.

[38]

George A. Miller. 1956. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review 63, 2 (1956), 81--97. https://doi.org/10.1037/h0043158

[39]

Shwetak Patel. 2008. Infrastructure Mediated Sensing. August (2008), 274. http://hdl.handle.net/1853/24829

[40]

Ken Pfeuffer, Jason Alexander, and Hans Gellersen. 2016. Partially-indirect Bimanual Input with Gaze, Pen, and Touch for Pan, Zoom, and Ink Interaction. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (2016), 2845--2856. https://doi.org/10.1145/2858036.2858201

Digital Library

[41]

Jan L Plass, Roxana Moreno, and Roland Brünken. 2010. Cognitive Load Theory. Vol. 55. 286 pages. https://doi.org/10.1016/B978-0-12-387691-1.00002-8 arXiv:arXiv:1011.1669v3

[42]

Sohail Rafiqi, Chatchai Wangwiwattana, Ephrem Fernandez, Suku Nair, and Eric C. Larson. 2015. Work-in-progress, PupilWare-M: Cognitive load estimation using unmodified smartphone cameras. In Proceedings - 2015 IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems, MASS 2015. https://doi.org/10.1109/MASS.2015.31

Digital Library

[43]

Sohail Rafiqi, Chatchai Wangwiwattana, Jasmine Kim, Ephrem Fernandez, Suku Nair, and Eric C. Larson. 2015. PupilWare: Towards pervasive cognitive load measurement using commodity devices. In 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2015 - Proceedings. https://doi.org/10.1145/2769493.2769506

Digital Library

[44]

Gerulf Rieger and Ritch C Savin-Williams. 2012. The eyes have it: sex and sexual orientation differences in pupil dilation patterns. PloS one 7, 8 (1 2012), e40256. https://doi.org/10.1371/journal.pone.0040256

[45]

Kaushik Roy, Prabir Bhattacharya, and Ching Y Suen. 2010. Unideal Iris Segmentation Using Region-Based Active Contour Model. LNCS 6112 (2010), 256--265. https://pdfs.semanticscholar.org/a5da/0a5fbfe89bd678d099c504a7d94bce955019.pdf

[46]

Wayne J. Ryan, Damon L. Woodard, Andrew T. Duchowski, and Stan T. Birchfield. 2008. Adapting Starburst for Elliptical Iris Segmentation. In 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems. IEEE, 1--7. https://doi.org/10.1109/BTAS.2008.4699340

[47]

Lech Świrski, Andreas Bulling, and Neil Dodgson. 2012. Robust real-time pupil tracking in highly off-axis images. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 173--176.

Digital Library

[48]

Philippe Thevenaz and Michael Unser. 2006. The Snakuscule. 2006 International Conference on Image Processing (2006), 1633--1636. https://doi.org/10.1109/ICIP.2006.312658

[49]

Warren Tryon W. 1975. Pupillometry: A Survey of Sources of Variation. Psychophysiology 12 (1975). https://doi.org/10.1111/j.1469-8986.1975.tb03068.x

[50]

Alex Waibel, Toshiyuki Hanazawa, Geoffrey Hinton, Kiyohiro Shikano, and Kevin J Lang. 1989. Phoneme recognition using time-delay neural networks. IEEE transactions on acoustics, speech, and signal processing 37, 3 (1989), 328--339.

[51]

Richard P Wildes, Jane C Asmuth, Gilbert L Green, Stephen C Hsu, Raymond J Kolczynski, James R Matey, Sterling E McBride, Richard P Wildes, Jane C Asmuth, Gilbert L Green, Stephen C Hsu, Raymond J Kolczynski, James R Matey, and Sterling E McBride. 1994. A system for automated iris recognition. In Applications of Computer Vision, 1994., Proceedings of the Second IEEE Workshop on. IEEE, IEEE Comput. Soc. Press, 121--128. https://doi.org/10.1109/ACV.1994.341298

[52]

Erroll Wood and Andreas Bulling. 2014. EyeTab: Model-based gaze estimation on unmodified tablet computers. In Proceedings of the Symposium on Eye Tracking Research and Applications. 207--210.

Digital Library

[53]

Jie Xu, Yang Wang, Fang Chen, and Eric Choi. 2011. Pupillary Response Based Cognitive Workload Measurement under Luminance Changes. 178--185. https://doi.org/10.1007/978-3-642-23771-3{_}14

[54]

Beste F Yuksel, Kurt B Oleson, Lane Harrison, Evan M Peck, Daniel Afergan, Remco Chang, and Robert J K Jacob. 2016. Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty based on Brain State. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (2016), 5372--5384. https://doi.org/10.1145/2858036.2858388

Digital Library

Cited By

Sarker PZaman NOng JWaisberg ELee ATavakkoli A(2025)XR-Pupillometry: A novel approach to pupillometry for clinical care and beyondThe Pan-American Journal of Ophthalmology10.4103/pajo.pajo_62_247:1Online publication date: Jan-2025
https://doi.org/10.4103/pajo.pajo_62_24
Hwang JChoi WKim A(2025)Effects of in-vehicle auditory interactions on takeover performance in SAE L2 semi-automated vehiclesInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103401196(103401)Online publication date: Feb-2025
https://doi.org/10.1016/j.ijhcs.2024.103401
Spence ABangay S(2024)Domain-Agnostic Representation of Side-ChannelsEntropy10.3390/e2608068426:8(684)Online publication date: 13-Aug-2024
https://doi.org/10.3390/e26080684
Show More Cited By

Index Terms

PupilNet, Measuring Task Evoked Pupillary Response using Commodity RGB Tablet Cameras: Comparison to Mobile, Infrared Gaze Trackers for Inferring Cognitive Load

Recommendations

Measuring the task-evoked pupillary response with a remote eye tracker
ETRA '08: Proceedings of the 2008 symposium on Eye tracking research & applications

The pupil-measuring capability of video eye trackers can detect the task-evoked pupillary response: subtle changes in pupil size which indicate cognitive load. We performed several experiments to measure cognitive load using a remote video eye tracker, ...
Pupillary response based cognitive workload index under luminance and emotional changes
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing Systems

Pupillary response has been widely accepted as a physiological index of cognitive workload. It can be reliably measured with video-based eye trackers in a non-intrusive way. However, in practice commonly used measures such as pupil size or dilation might ...
Indexing cognitive workload based on pupillary response under luminance and emotional changes
IUI '13: Proceedings of the 2013 international conference on Intelligent user interfaces

Pupillary response is a popular physiological index of cognitive workload that can be used for design and evaluation of adaptive interface in various areas of human-computer interaction (HCI) research. However, in practice various confounding factors ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 4

December 2017

1298 pages

EISSN:2474-9567

DOI:10.1145/3178157

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 January 2018

Accepted: 01 October 2017

Received: 01 August 2017

Published in IMWUT Volume 1, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
388
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sarker PZaman NOng JWaisberg ELee ATavakkoli A(2025)XR-Pupillometry: A novel approach to pupillometry for clinical care and beyondThe Pan-American Journal of Ophthalmology10.4103/pajo.pajo_62_247:1Online publication date: Jan-2025
https://doi.org/10.4103/pajo.pajo_62_24
Hwang JChoi WKim A(2025)Effects of in-vehicle auditory interactions on takeover performance in SAE L2 semi-automated vehiclesInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103401196(103401)Online publication date: Feb-2025
https://doi.org/10.1016/j.ijhcs.2024.103401
Spence ABangay S(2024)Domain-Agnostic Representation of Side-ChannelsEntropy10.3390/e2608068426:8(684)Online publication date: 13-Aug-2024
https://doi.org/10.3390/e26080684
Du LJia JZhang XLan G(2024)PrivateGaze: Preserving User Privacy in Black-box Mobile Gaze Tracking ServicesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785958:3(1-28)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678595
Filipa Ferreira DFerreira SMateus CBarbosa-Rocha NCoelho LRodrigues M(2024)Advancing the understanding of pupil size variation in occupational safety and health: A systematic review and evaluation of open-source methodologiesSafety Science10.1016/j.ssci.2024.106490175(106490)Online publication date: Jul-2024
https://doi.org/10.1016/j.ssci.2024.106490
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Shen XJiang HLiu DYang KDeng FLui JLiu JDustdar SLuo J(2022)PupilRec: Leveraging Pupil Morphology for Recommending on SmartphonesIEEE Internet of Things Journal10.1109/JIOT.2022.31816079:17(15538-15553)Online publication date: 1-Sep-2022
https://doi.org/10.1109/JIOT.2022.3181607
Spence ABangay S(2022)Security beyond cybersecurity: side-channel attacks against non-cyber systems and their countermeasuresInternational Journal of Information Security10.1007/s10207-021-00563-621:3(437-453)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1007/s10207-021-00563-6
Wilson JNair SScielzo SLarson E(2021)Objective Measures of Cognitive Load Using Deep Multi-Modal LearningProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34481115:1(1-35)Online publication date: 30-Mar-2021
https://dl.acm.org/doi/10.1145/3448111
Pasquali DGonzalez-Billandon JRea FSandini GSciutti ABethel CPaiva ABroadbent EFeil-Seifer DSzafir D(2021)Magic iCubProceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3434073.3444682(293-302)Online publication date: 8-Mar-2021
https://dl.acm.org/doi/10.1145/3434073.3444682
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents