short-paper

Determining Differences in Reading Behavior Between Experts and Novices by Investigating Eye Movement on Source Code Constructs During a Bug Fixing Task

Authors:

Salwa Aljehane,

Jonathan MaleticAuthors Info & Claims

ETRA '21 Short Papers: ACM Symposium on Eye Tracking Research and Applications

Article No.: 30, Pages 1 - 6

https://doi.org/10.1145/3448018.3457424

Published: 25 May 2021 Publication History

Abstract

This research compares the eye movement of expert and novice programmers working on a bug fixing task. This comparison aims at investigating which source code elements programmers focus on when they review Java source code. Programmer code reading behaviors at the line and term levels are used to characterize the differences between experts and novices. The study analyzes programmers’ eye movements over identified source code areas using an existing eye tracking dataset of 12 experts and 10 novices. The results show that the difference between experts and novices is significant in source code element coverage. Specifically, novices read more method signatures, variable declarations, identifiers, and keywords compared to experts. However, experts are better at finishing the task using fewer source code elements when compared to novices. Moreover, programmers tend to focus on the method signatures the most while reading the code.

References

[1]

Nahla J. Abid, Jonathan I. Maletic, and Bonita Sharif. 2019a. Using Developer Eye Movements to Externalize the Mental Model Used in Code Summarization Tasks. Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, ACM, 13:1-13:9.

Digital Library

[2]

Nahla J. Abid, Bonita Sharif, Natalia Dragan, Hend Alrasheed, and Jonathan I. Maletic. 2019b. Developer Reading Behavior While Summarizing Java Methods: Size and Context Matters. 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 384–395.

[3]

Roman Bednarik. 2012. Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human-Computer Studies 70, 2, 143–155.

Digital Library

[4]

Andrew Begel and Hana Vrzakova. 2018. Eye movements in code review. Proceedings of the Workshop on Eye Movements in Programming - EMIP ’18, ACM Press, 1–5.

Digital Library

[5]

Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye Movements in Code Reading: Relaxing the Linear Order. 2015 IEEE 23rd International Conference on Program Comprehension, 255–265.

[6]

Teresa Busjahn, Carsten Schulte, and Andreas Busjahn. 2011. Analysis of code reading to gain more insight in program comprehension. Proceedings of the 11th Koli Calling International Conference on Computing Education Research - Koli Calling ’11, ACM Press, 1.

Digital Library

[7]

Michael L. Collard, Michael J. Decker, and Jonathan I. Maletic. 2011. Lightweight Transformation and Fact Extraction with the srcML Toolkit. 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation, IEEE, 173–184.

[8]

Martha E. Crosby, Jean Scholtz, and Susan Wiedenbeck. 2002. The Roles Beacons Play in Comprehension for Novice and Expert Programmers. 10.

[9]

Martha E. Crosby and Jan Stelovsky. 1990. How do we read algorithms? A case study. Computer 23, 1, 25–35.

Digital Library

[10]

Janet Feigenspan, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2012. Measuring programming experience. 2012 20th IEEE International Conference on Program Comprehension (ICPC), 73–82.

[11]

Thomas Fritz, Andrew Begel, Sebastian C. Müller, Serap Yigit-Elliott, and Manuela Züger. 2014. Using Psycho-physiological Measures to Assess Task Difficulty in Software Development. Proceedings of the 36th International Conference on Software Engineering, ACM, 402–413.

Digital Library

[12]

Myles Hollander, Douglas A. Wolfe, and Eric Chicken. 2013. Nonparametric Statistical Methods. John Wiley & Sons.

[13]

Katja Kevic, Braden M. Walters, Timothy R. Shaffer, Bonita Sharif, David C. Shepherd, and Thomas Fritz. 2015. Tracing Software Developers’ Eyes and Interactions for Change Tasks. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, 202–213.

Digital Library

[14]

Katja Kevic, Braden M. Walters, Timothy R. Shaffer, Bonita Sharif, David C. Shepherd, and Thomas Fritz. 2017. Eye gaze and interaction contexts for change tasks – Observations and potential. Journal of Systems and Software 128, 252–266.

Digital Library

[15]

Cole S. Peterson, Nahla J. Abid, Corey A. Bryant, Jonathan I. Maletic, and Bonita Sharif. 2019. Factors influencing dwell time during source code reading: a large-scale replication experiment. Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, Association for Computing Machinery, 1–4.

Digital Library

[16]

Paige Rodeghero and Collin McMillan. 2015. An Empirical Study on the Patterns of Eye Movement during Summarization Tasks. 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 1–10.

[17]

Paige Rodeghero, Collin McMillan, Paul W. McBurney, Nigel Bosch, and Sidney D'Mello. 2014. Improving automated source code summarization via an eye-tracking study of programmers. Proceedings of the 36th International Conference on Software Engineering - ICSE 2014, ACM Press, 390–401.

Digital Library

[18]

Timothy R. Shaffer, Jenna L. Wise, Braden M. Walters, Sebastian C. Müller, Michael Falcone, and Bonita Sharif. 2015. iTrace: Enabling Eye Tracking on Software Artifacts Within the IDE to Support Software Engineering Tasks. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, 954–957.

Digital Library

[19]

James Shanteau. 1992. Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes 53, 2, 252–266.

[20]

Zohreh Sharafi, Bonita Sharif, Yann-Gaël Guéhéneuc, Andrew Begel, Roman Bednarik, and Martha Crosby. 2020. A practical guide on conducting eye tracking studies in software engineering. Empirical Software Engineering 25, 5, 3128–3174.

Digital Library

[21]

Bonita Sharif. 2011. Empirical assessment of UML class diagram layouts based on architectural importance. 2011 27th IEEE International Conference on Software Maintenance (ICSM), 544–549.

Digital Library

[22]

Bonita Sharif, Michael Falcone, and Jonathan I. Maletic. 2012. An Eye-tracking Study on the Role of Scan Time in Finding Source Code Defects. Proceedings of the Symposium on Eye Tracking Research and Applications, ACM, 381–384.

[23]

Bonita Sharif and Jonathan I. Maletic. 2010a. The Effects of Layout on Detecting the Role of Design Patterns. 2010 23rd IEEE Conference on Software Engineering Education and Training, 41–48.

[24]

Bonita Sharif and Jonathan I. Maletic. 2010b. An Eye Tracking Study on camelCase and under_score Identifier Styles. 2010 IEEE 18th International Conference on Program Comprehension, 196–205.

[25]

Janet Siegmund, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2014. Measuring and modeling programming experience. Empirical Software Engineering 19, 5, 1299–1334.

Digital Library

[26]

Rachel Turner, Michael Falcone, Bonita Sharif, and Alina Lazar. 2014. An Eye-tracking Study Assessing the Comprehension of C++ and Python Source Code. Proceedings of the Symposium on Eye Tracking Research and Applications, ACM, 231–234.

Digital Library

[27]

Hidetake Uwano, Masahide Nakamura, Akito Monden, and Ken-ichi Matsumoto. 2006. Analyzing individual performance of source code review using reviewers’ eye movement. Proceedings of the 2006 symposium on Eye tracking research & applications - ETRA ’06, ACM Press, 133.

Digital Library

Cited By

Zhang YRutherford T(2024)Scaffolding Expertise: Evaluating Scaffolds for Block-Based Coding Among Experts and NovicesProceedings of the 2024 International Symposium on Artificial Intelligence for Education10.1145/3700297.3700345(277-282)Online publication date: 6-Sep-2024
https://dl.acm.org/doi/10.1145/3700297.3700345
Dörzapf TPeitek NWyrich MApel S(2024)Data Analysis Tools Affect Outcomes of Eye-Tracking StudiesProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686672(96-106)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686672
Alakmeh TReich DJäger LFritz T(2024)Predicting Code Comprehension: A Novel Approach to Align Human Gaze with Code using Deep Neural NetworksProceedings of the ACM on Software Engineering10.1145/36607951:FSE(1982-2004)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660795
Show More Cited By

Index Terms

Determining Differences in Reading Behavior Between Experts and Novices by Investigating Eye Movement on Source Code Constructs During a Bug Fixing Task

Index terms have been assigned to the content through auto-classification.

Recommendations

Studying Developer Eye Movements to Measure Cognitive Workload and Visual Effort for Expertise Assessment
ETRA

Eye movement data provides valuable insights that help test hypotheses about a software developer's comprehension process. The pupillary response is successfully used to assess mental processing effort and attentional focus. Relatively little is known ...
Using developer eye movements to externalize the mental model used in code summarization tasks
ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

Eye movements of developers are used to speculate the mental cognition model (i.e., bottom-up or top-down) applied during program comprehension tasks. The cognition models examine how programmers understand source code by describing the temporary ...
Detect Related Bugs from Source Code Using Bug Information
COMPSAC '10: Proceedings of the 2010 IEEE 34th Annual Computer Software and Applications Conference

Open source projects often maintain open bug repositories during development and maintenance, and the reporters often point out straightly or implicitly the reasons why bugs occur when they submit them. The comments about a bug are very valuable for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '21 Short Papers: ACM Symposium on Eye Tracking Research and Applications

May 2021

232 pages

ISBN:9781450383455

DOI:10.1145/3448018

Editors:
Andreas Bulling
University of Stuttgart, Germany
,
Anke Huckauf
Ulm University, Germany
,
Hans Gellersen
Aarhus University, Denmark
,
Daniel Weiskopf
University of Stuttgart, Germany
,
Mihai Bace
ETH Zürich, Switzerland
,
Teresa Hirzle
Ulm University, Germany
,
Florian Alt
Bundeswehr University Munich, Germany
,
Thies Pfeiffer
Hochschule Emden/Leer, Germany
,
Roman Bednarik
University of Eastern Finland, Finland
,
Krzysztof Krejtz
SWPS University of Social Sciences and Humanities, Poland
,
Tanja Blascheck
University of Stuttgart, Germany
,
Michael Burch
University of Applied Sciences, Chur, Switzerland
,
Peter Kiefer
ETH Zurich, Switzerland
,
Michael Dodd
University of Nebraska-Lincoln, USA
,
Bonita Sharif
University of Nebraska-Lincoln, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

ETRA '21

Sponsor:

SIGGRAPH

ETRA '21: 2021 Symposium on Eye Tracking Research and Applications

May 25 - 27, 2021

Virtual Event, Germany

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

ETRA '25

Sponsor:
sigchi
sigchi

2025 Symposium on Eye Tracking Research and Applications

May 26 - 29, 2025

Tokyo , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
364
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YRutherford T(2024)Scaffolding Expertise: Evaluating Scaffolds for Block-Based Coding Among Experts and NovicesProceedings of the 2024 International Symposium on Artificial Intelligence for Education10.1145/3700297.3700345(277-282)Online publication date: 6-Sep-2024
https://dl.acm.org/doi/10.1145/3700297.3700345
Dörzapf TPeitek NWyrich MApel S(2024)Data Analysis Tools Affect Outcomes of Eye-Tracking StudiesProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686672(96-106)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686672
Alakmeh TReich DJäger LFritz T(2024)Predicting Code Comprehension: A Novel Approach to Align Human Gaze with Code using Deep Neural NetworksProceedings of the ACM on Software Engineering10.1145/36607951:FSE(1982-2004)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660795
Grabinger LHauser FWolff CMottok J(2024)On Eye Tracking in Software EngineeringSN Computer Science10.1007/s42979-024-03045-35:6Online publication date: 26-Jul-2024
https://dl.acm.org/doi/10.1007/s42979-024-03045-3
Aljehane SSharif BMaletic J(2023)Studying Developer Eye Movements to Measure Cognitive Workload and Visual Effort for Expertise AssessmentProceedings of the ACM on Human-Computer Interaction10.1145/35911357:ETRA(1-18)Online publication date: 18-May-2023
https://dl.acm.org/doi/10.1145/3591135
Park KWeill-Tessier PBrown NSharif BJensen NKölling M(2023)An eye tracking study assessing the impact of background styling in code editors on novice programmers’ code understandingProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600133(444-463)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600133
Nashid NSintaha MMesbah A(2023)Embedding Context as Code Dependencies for Neural Program Repair2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00018(95-106)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00018
Kuang PSöderberg ENiehorster DHöst MBavota GHao D(2023)Toward Gaze-Assisted Developer ToolsProceedings of the 45th International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER58687.2023.00015(49-54)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-NIER58687.2023.00015
Peitek NBergum ARekrut MMucke JNadig MParnin CSiegmund JApel SRoychoudhury ACadar CKim M(2022)Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking studyProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549084(120-131)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549084
Sharif BMansoor N(2022)Humans in Empirical Software Engineering Studies: An Experience Report2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00154(1286-1292)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00154
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten