short-paper

EMIP Toolkit: A Python Library for Customized Post-processing of the Eye Movements in Programming Dataset

Authors:

Jonathan MaleticAuthors Info & Claims

ETRA '21 Short Papers: ACM Symposium on Eye Tracking Research and Applications

Article No.: 34, Pages 1 - 6

https://doi.org/10.1145/3448018.3457425

Published: 25 May 2021 Publication History

Abstract

The use of eye tracking in the study of program comprehension in software engineering allows researchers to gain a better understanding of the strategies and processes applied by programmers. Despite the large number of eye tracking studies in software engineering, very few datasets are publicly available. The existence of the large Eye Movements in Programming Dataset (EMIP) opens the door for new studies and makes reproducibility of existing research easier. In this paper, a Python library (the EMIP Toolkit) for customized post-processing of the EMIP dataset is presented. The toolkit is specifically designed to make using the EMIP dataset easier and more accessible. It implements features for fixation detection and correction, trial visualization, source code lexical data enrichment, and mapping fixation data over areas of interest. In addition to the toolkit, a filtered token-level dataset with scored recording quality is presented for all Java trials (accounting for 95.8% of the data) in the EMIP dataset.

References

[1]

2011. iView X System Manual. https://psychologie.unibas.ch/fileadmin/user_upload/psychologie/Forschung/N-Lab/SMI_iView_X_Manual.pdf

[2]

Naser S Al Madi. 2020. Modeling Eye Movement for the Assessment of Programming Proficiency. Ph.D. Dissertation. Kent State University.

[3]

Nasir Ali, Zohreh Sharafl, Yann-Gaël Guéhéneuc, and Giuliano Antoniol. 2012. An empirical study on requirements traceability using eye-tracking. In 2012 28th IEEE International Conference on Software Maintenance (ICSM). IEEE, 191–200.

Digital Library

[4]

Christoph Aschwanden and Martha Crosby. 2006. Code scanning patterns in program comprehension. In Proceedings of the 39th hawaii international conference on system sciences.

[5]

Roman Bednarik, Teresa Busjahn, Agostino Gibaldi, Alireza Ahadi, Maria Bielikova, Martha Crosby, Kai Essig, Fabian Fagerholm, Ahmad Jbara, Raymond Lister, 2020. EMIP: The eye movements in programming dataset. Science of Computer Programming 198 (2020), 102520.

[6]

Roman Bednarik and Markku Tukiainen. 2006. An eye-tracking methodology for characterizing program comprehension processes. In Proceedings of the 2006 symposium on Eye tracking research & applications. ACM, 125–132.

Digital Library

[7]

Roman Bednarik and Markku Tukiainen. 2007. Analysing and Interpreting Quantitative Eye-Tracking Data in Studies of Programming: Phases of Debugging with Multiple Representations. In PPIG. Citeseer, 13.

[8]

Dave Binkley, Marcia Davis, Dawn Lawrie, Jonathan I Maletic, Christopher Morrell, and Bonita Sharif. 2013. The impact of identifier style on effort and comprehension. Empirical Software Engineering 18, 2 (2013), 219–276.

Digital Library

[9]

Teresa Busjahn, Carsten Schulte, and Andreas Busjahn. 2011. Analysis of code reading to gain more insight in program comprehension. In Proceedings of the 11th Koli Calling International Conference on Computing Education Research. 1–9.

Digital Library

[10]

Michael L Collard, Michael J Decker, and Jonathan I Maletic. 2011. Lightweight transformation and fact extraction with the srcML toolkit. In 2011 IEEE 11th international working conference on source code analysis and manipulation. IEEE, 173–184.

Digital Library

[11]

Martha E Crosby and Jan Stelovsky. 1990. How do we read algorithms? A case study. Computer 23, 1 (1990), 25–35.

Digital Library

[12]

Benoít De Smet, Lorent Lempereur, Zohreh Sharafi, Yann-Gaël Guéhéneuc, Giuliano Antoniol, and Naji Habra. 2014. Taupe: Visualizing and analyzing eye-tracking data. Science of Computer Programming 79 (2014), 260–278.

Digital Library

[13]

Hacı Ali Duru, Murat Perit Çakır, and Veysi İşler. 2013. How does software visualization contribute to software comprehension? A grounded theory approach. International Journal of Human-Computer Interaction 29, 11(2013), 743–763.

[14]

Yann-Gaël Guéhéneuc. 2006. TAUPE: towards understanding program comprehension. In Proceedings of the 2006 conference of the Center for Advanced Studies on Collaborative research. 1–es.

Digital Library

[15]

Prateek Hejmady and N Hari Narayanan. 2012. Visual attention patterns during program debugging with an IDE. In Proceedings of the Symposium on Eye Tracking Research and Applications. 197–200.

Digital Library

[16]

Sebastien Jeanmart, Yann-Gael Gueheneuc, Houari Sahraoui, and Naji Habra. 2009. Impact of the visitor pattern on program comprehension and maintenance. In 2009 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE, 69–78.

Digital Library

[17]

G Karthik, J Amudha, and C Jyotsna. 2019. A Custom Implementation of the Velocity Threshold Algorithm for Fixation Identification. In 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT). IEEE, 488–492.

[18]

Meng-Lung Lai, Meng-Jung Tsai, Fang-Ying Yang, Chung-Yuan Hsu, Tzu-Chien Liu, Silvia Wen-Yu Lee, Min-Hsien Lee, Guo-Li Chiou, Jyh-Chong Liang, and Chin-Chung Tsai. 2013. A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational research review 10 (2013), 90–115.

[19]

Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke. 2014. On the comprehension of program comprehension. ACM Transactions on Software Engineering and Methodology (TOSEM) 23, 4(2014), 1–37.

Digital Library

[20]

Naser Al Madi, Cole S Peterson, Bonita Sharif, and Jonathan Maletic. 2020. Can the ez reader model predict eye movements over code? towards a model of eye movements over source code. In ACM Symposium on Eye Tracking Research and Applications. 1–4.

Digital Library

[21]

Naser Al Madi, Cole S Peterson, Bonita Sharif, and Jonathan Maletic. 2021. From Novice to Expert: Analysis of Token Level Effects in a Longitudinal Eye Tracking Study. In 29th IEEE/ACM International Conference on Program Comprehension.

[22]

Marcus Nyström and Kenneth Holmqvist. 2010. An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data. Behavior research methods 42, 1 (2010), 188–204.

[23]

Unaizah Obaidellah, Mohammed Al Haek, and Peter C-H Cheng. 2018. A survey on the usage of eye-tracking in computer programming. ACM Computing Surveys (CSUR) 51, 1 (2018), 5.

Digital Library

[24]

Gerardo Cepeda Porras and Yann-Gaël Guéhéneuc. 2010. An empirical study on the efficiency of different design pattern representations in UML class diagrams. Empirical Software Engineering 15, 5 (2010), 493–522.

Digital Library

[25]

Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research.Psychological bulletin 124, 3 (1998), 372.

[26]

Pablo Romero, Richard Cox, Benedict du Boulay, and Rudi Lutz. 2002. Visual attention and representation switching during java program debugging: A study using the restricted focus viewer. In International Conference on Theory and Application of Diagrams. Springer, 221–235.

Digital Library

[27]

Zohreh Sharafi, Bonita Sharif, Yann-Gaël Guéhéneuc, Andrew Begel, Roman Bednarik, and Martha Crosby. 2020. A practical guide on conducting eye tracking studies in software engineering. Empirical Software Engineering 25, 5 (2020), 3128–3174.

Digital Library

[28]

Zohreh Sharafi, Zéphyrin Soh, and Yann-Gaël Guéhéneuc. 2015. A systematic literature review on the usage of eye-tracking in software engineering. Information and Software Technology 67 (2015), 79–107.

Digital Library

[29]

Bonita Sharif and Jonathan I Maletic. 2010. An eye tracking study on camelcase and under_score identifier styles. In 2010 IEEE 18th International Conference on Program Comprehension. IEEE, 196–205.

Digital Library

[30]

Bonita Sharif, John Meinken, Timothy Shaffer, and Huzefa Kagdi. 2017. Eye movements in software traceability link recovery. Empirical Software Engineering 22, 3 (2017), 1063–1102.

Digital Library

[31]

Kshitij Sharma, Daniela Caballero, Himanshu Verma, Patrick Jermann, and Pierre Dillenbourg. 2015. Looking AT versus looking THROUGH: A dual eye-tracking study in MOOC context. International Society of the Learning Sciences, Inc.[ISLS].

[32]

Zéphyrin Soh, Zohreh Sharafi, Bertrand Van den Plas, Gerardo Cepeda Porras, Yann-Gaël Guéhéneuc, and Giuliano Antoniol. 2012. Professional status and expertise for UML class diagram comprehension: An empirical study. In 2012 20th IEEE International Conference on Program Comprehension (ICPC). IEEE, 163–172.

[33]

Randy Stein and Susan E Brennan. 2004. Another person’s eye gaze as a cue in solving programming problems. In Proceedings of the 6th international conference on Multimodal interfaces. 9–15.

Digital Library

[34]

Rachel Turner, Michael Falcone, Bonita Sharif, and Alina Lazar. 2014. An eye-tracking study assessing the comprehension of C++ and Python source code. In Proceedings of the Symposium on Eye Tracking Research and Applications. 231–234.

Digital Library

[35]

Braden Walters, Timothy Shaffer, Bonita Sharif, and Huzefa Kagdi. 2014. Capturing software traceability links from developers’ eye gazes. In Proceedings of the 22nd International Conference on Program Comprehension. 201–204.

Digital Library

[36]

Shehnaaz Yusuf, Huzefa Kagdi, and Jonathan I Maletic. 2007. Assessing the comprehension of UML class diagrams via eye tracking. In 15th IEEE International Conference on Program Comprehension (ICPC’07). IEEE, 113–122.

Digital Library

Cited By

Ueda RDoi HFurui AShimatani KHayashi HTsuji T(2025)Discriminative Analysis of Autistic Tendencies at 18 Months of Age Using Eye Gaze Characteristics in 4-, 10-, and 18-month-old Infants2025 IEEE/SICE International Symposium on System Integration (SII)10.1109/SII59315.2025.10870970(265-270)Online publication date: 21-Jan-2025
https://doi.org/10.1109/SII59315.2025.10870970
Li ZHolly D(2024)Assessing Programming Proficiency Through Eye Gaze Analysis Using Fine-Tuned Large Language Model2024 10th IEEE International Conference on High Performance and Smart Computing (HPSC)10.1109/HPSC62738.2024.00009(7-12)Online publication date: 10-May-2024
https://doi.org/10.1109/HPSC62738.2024.00009
Lee SKim HYi THyun K(2024)BIGazeAdvanced Engineering Informatics10.1016/j.aei.2023.10215958:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.aei.2023.102159

Index Terms

EMIP Toolkit: A Python Library for Customized Post-processing of the Eye Movements in Programming Dataset

Index terms have been assigned to the content through auto-classification.

Recommendations

Can the E-Z Reader Model Predict Eye Movements Over Code? Towards a Model of Eye Movements Over Source Code
ETRA '20 Short Papers: ACM Symposium on Eye Tracking Research and Applications

Studies of eye movements during source code reading have supported the idea that reading source code differs fundamentally from reading natural text. The paper analyzed an existing data set of natural language and source code eye movement data using the ...
An eye tracking dataset for point of gaze detection
ETRA '12: Proceedings of the Symposium on Eye Tracking Research and Applications

This paper presents a new, publicly available eye tracking dataset, aimed to be used as a benchmark for Point of Gaze (PoG) detection algorithms. The dataset consists of a set of videos recording the eye motion of human test subjects as they were ...
Analysing EOG signal features for the discrimination of eye movements with wearable devices
PETMEI '11: Proceedings of the 1st international workshop on pervasive eye tracking & mobile eye-based interaction

Eye tracking research in human-computer interaction and experimental psychology traditionally focuses on stationary devices and a small number of common eye movements. The advent of pervasive eye tracking promises new applications, such as eye-based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '21 Short Papers: ACM Symposium on Eye Tracking Research and Applications

May 2021

232 pages

ISBN:9781450383455

DOI:10.1145/3448018

Editors:
Andreas Bulling
University of Stuttgart, Germany
,
Anke Huckauf
Ulm University, Germany
,
Hans Gellersen
Aarhus University, Denmark
,
Daniel Weiskopf
University of Stuttgart, Germany
,
Mihai Bace
ETH Zürich, Switzerland
,
Teresa Hirzle
Ulm University, Germany
,
Florian Alt
Bundeswehr University Munich, Germany
,
Thies Pfeiffer
Hochschule Emden/Leer, Germany
,
Roman Bednarik
University of Eastern Finland, Finland
,
Krzysztof Krejtz
SWPS University of Social Sciences and Humanities, Poland
,
Tanja Blascheck
University of Stuttgart, Germany
,
Michael Burch
University of Applied Sciences, Chur, Switzerland
,
Peter Kiefer
ETH Zurich, Switzerland
,
Michael Dodd
University of Nebraska-Lincoln, USA
,
Bonita Sharif
University of Nebraska-Lincoln, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

ETRA '21

Sponsor:

SIGGRAPH

ETRA '21: 2021 Symposium on Eye Tracking Research and Applications

May 25 - 27, 2021

Virtual Event, Germany

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

ETRA '25

Sponsor:
sigchi
sigchi

2025 Symposium on Eye Tracking Research and Applications

May 26 - 29, 2025

Tokyo , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
189
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ueda RDoi HFurui AShimatani KHayashi HTsuji T(2025)Discriminative Analysis of Autistic Tendencies at 18 Months of Age Using Eye Gaze Characteristics in 4-, 10-, and 18-month-old Infants2025 IEEE/SICE International Symposium on System Integration (SII)10.1109/SII59315.2025.10870970(265-270)Online publication date: 21-Jan-2025
https://doi.org/10.1109/SII59315.2025.10870970
Li ZHolly D(2024)Assessing Programming Proficiency Through Eye Gaze Analysis Using Fine-Tuned Large Language Model2024 10th IEEE International Conference on High Performance and Smart Computing (HPSC)10.1109/HPSC62738.2024.00009(7-12)Online publication date: 10-May-2024
https://doi.org/10.1109/HPSC62738.2024.00009
Lee SKim HYi THyun K(2024)BIGazeAdvanced Engineering Informatics10.1016/j.aei.2023.10215958:COnline publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.aei.2023.102159

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten