skip to main content
10.1145/3314111.3319834acmconferencesArticle/Chapter ViewAbstractPublication PagesetraConference Proceedingsconference-collections
research-article

Using developer eye movements to externalize the mental model used in code summarization tasks

Published: 25 June 2019 Publication History

Abstract

Eye movements of developers are used to speculate the mental cognition model (i.e., bottom-up or top-down) applied during program comprehension tasks. The cognition models examine how programmers understand source code by describing the temporary information structures in the programmer's short term memory. The two types of models that we are interested in are top-down and bottom-up. The top-down model is normally applied as-needed (i.e., the domain of the system is familiar). The bottom-up model is typically applied when a developer is not familiar with the domain or the source code. An eye-tracking study of 18 developers reading and summarizing Java methods is used as our dataset for analyzing the mental cognition model. The developers provide a written summary for methods assigned to them. In total, 63 methods are used from five different systems. The results indicate that on average, experts and novices read the methods more closely (using the bottom-up mental model) than bouncing around (using top-down). However, on average novices spend longer gaze time performing bottom-up (66s.) compared to experts (43s.)

References

[1]
Nahla J. Abid, Bonita Sharif, Natalia Dragan, Hend Alrasheed, and Jonathan I. Maletic. 2019. Developer Reading Behavior while Summarizing Java Methods : Size and Context Matters. In Proceedings of the 41th International Conference on Software Engineering (ICSE 2019).
[2]
Roman Bednarik and Markku Tukiainen. 2006. An Eye-tracking Methodology for Characterizing Program Comprehension Processes. In Proceedings of the 2006 Symposium on Eye Tracking Research & Applications (ETRA '06). 125--132.
[3]
Roman Bednarik and Markku Tukiainen. 2008. Temporal Eye-tracking Data: Evolution of Debugging Strategies with Multiple Representations. In Proceedings of the 2008 Symposium on Eye Tracking Research & Applications (ETRA '08). 99--102.
[4]
Dave Binkley, Marcia Davis, Dawn Lawrie, Jonathan I. Maletic, Christopher Morrell, and Bonita Sharif. 2013. The impact of identifier style on effort and comprehension. Empirical Software Engineering 18, 2 (01 Apr 2013), 219--276.
[5]
Ruven Brooks. 1983. Towards a theory of the comprehension of computer programs. International journal of man-machine studies 18, 6 (1983), 543--554.
[6]
Jean-Marie Burkhardt, Françoise Détienne, and Susan Wiedenbeck. 1997. Mental representations constructed by experts and novices in object-oriented program comprehension. In Human-Computer Interaction INTERACTâĂŹ97. Springer, 339--346.
[7]
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye Movements in Code Reading: Relaxing the Linear Order. In 2015 IEEE 23rd International Conference on Program Comprehension. 255--265.
[8]
Preetha Chatterjee, Manziba Akanda Nishi, Kostadin Damevski, Vinay Augustine, Lori Pollock, and Nicholas A Kraft. 2017. What information about code snippets is available in different software-related documents? an exploratory study. In Software Analysis, Evolution and Reengineering (SANER), 2017 IEEE 24th International Conference on. IEEE, 382--386.
[9]
Martha E. Crosby and Jan Stelovsky. 1990. How do we read algorithms? A case study. Computer 23, 1 (1990), 25--35.
[10]
Drew T. Guarnera, Corey A. Bryant, Ashwin Mishra, Jonathan I. Maletic, and Bonita Sharif. 2018. iTrace: Eye Tracking Infrastructure for Development Environments. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA '18). Article 105, 3 pages.
[11]
Myles Hollander and Douglas A. Wolfe. 1973. Nonparametric Statistical Methods. John Wiley and Sons, New York.
[12]
Katja Kevic, Braden M. Walters, Timothy R. Shaffer, Bonita Sharif, David C. Shepherd, and Thomas Fritz. 2015. Tracing Software Developers' Eyes and Interactions for Change Tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). 202--213.
[13]
Stanley Letovsky. 1987. Cognitive processes in program comprehension. Journal of Systems and software 7, 4 (1987), 325--339.
[14]
Pontus Olsson. 2007. Real-time and Offline Filters for Eye Tracking. (2007), 42 pages.
[15]
Nancy Pennington. 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive psychology 19, 3 (1987), 295--341.
[16]
Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. 124, 3 (00 1998), 372--422.
[17]
Paige Rodeghero and Collin McMillan. 2015. An Empirical Study on the Patterns of Eye Movement during Summarization Tasks. In 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Vol. 00. 1--10.
[18]
Paige Rodeghero, Collin McMillan, Paul W. McBurney, Nigel Bosch, and Sidney D'Mello. 2014. Improving Automated Source Code Summarization via an Eye-tracking Study of Programmers. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). 390--401.
[19]
Timothy R. Shaffer, Jenna L. Wise, Braden M. Walters, Sebastian C. Müller, Michael Falcone, and Bonita Sharif. 2015. iTrace: Enabling Eye Tracking on Software Artifacts Within the IDE to Support Software Engineering Tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). 954--957.
[20]
Bonita Sharif. 2011. Empirical assessment of UML class diagram layouts based on architectural importance. In 2011 27th IEEE International Conference on Software Maintenance (ICSM), Vol. 00. 544--549.
[21]
Bonita Sharif, Michael Falcone, and Jonathan I. Maletic. 2012. An Eye-tracking Study on the Role of Scan Time in Finding Source Code Defects. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12). 381--384.
[22]
Bonita Sharif and Jonathan I. Maletic. 2010a. The Effects of Layout on Detecting the Role of Design Patterns. In 2010 23rd IEEE Conference on Software Engineering Education and Training. 41--48. Bonita Sharif and Jonathan I. Maletic. 2010b. An Eye Tracking Study on camelCase and under score Identifier Styles. In 2010 18th IEEE International Conference on Program Comprehension. 196--205.
[23]
Ben Shneiderman and Richard Mayer. 1979. Syntactic/semantic interactions in programmer behavior: A model and experimental results. International Journal of Computer & Information Sciences 8, 3 (1979), 219--238.
[24]
Elliot Soloway and Kate Ehrlich. 1984. Empirical studies of programming knowledge. IEEE Transactions on software engineering 5 (1984), 595--609.
[25]
Margaret-Anne Storey. 2005. Theories, methods and tools in program comprehension: Past, present and future. In Program Comprehension, 2005. IWPC 2005. Proceedings. 13th International Workshop on. IEEE, 181--191.
[26]
Hidetake Uwano, Masahide Nakamura, Akito Monden, and Ken ichi Matsumoto. 2006. Analyzing Individual Performance of Source Code Review Using Reviewers' Eye Movement. In Proceedings of the 2006 Symposium on Eye Tracking Research & Applications (ETRA '06). 133--140.
[27]
Armeliese von Mayrhauser and A. Marie Vans. 1995. Program Comprehension During Software Maintenance and Evolution. Computer 28 (08 1995), 44--55.
[28]
Anneliese von Mayrhauser and A. Marie Vans. 1997. Program understanding behavior during debugging of large scale software. In Papers presented at the seventh workshop on Empirical studies of programmers. ACM, 157--179.
[29]
Xiaoran Wang, Lori L. Pollock, and K. Vijay-Shanker. 2014. Automatic Segmentation of Method Code into Meaningful Blocks: Design and Evaluation. Journal of Software: Evolution and Process 26, 1 (2014), 27--49.

Cited By

View all
  • (2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
  • (2024)Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code SummarizationProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644434(47-51)Online publication date: 15-Apr-2024
  • (2024)EyeTrans: Merging Human and Machine Attention for Neural Code SummarizationProceedings of the ACM on Software Engineering10.1145/36437321:FSE(115-136)Online publication date: 12-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications
June 2019
623 pages
ISBN:9781450367097
DOI:10.1145/3314111
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. code summarization
  2. eye tracking study
  3. program comprehension

Qualifiers

  • Research-article

Conference

ETRA '19

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)59
  • Downloads (Last 6 weeks)4
Reflects downloads up to 12 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
  • (2024)Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code SummarizationProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644434(47-51)Online publication date: 15-Apr-2024
  • (2024)EyeTrans: Merging Human and Machine Attention for Neural Code SummarizationProceedings of the ACM on Software Engineering10.1145/36437321:FSE(115-136)Online publication date: 12-Jul-2024
  • (2024)An Extractive-and-Abstractive Framework for Source Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/363274233:3(1-39)Online publication date: 14-Mar-2024
  • (2024)Esale: Enhancing Code-Summary Alignment Learning for Source Code SummarizationIEEE Transactions on Software Engineering10.1109/TSE.2024.342227450:8(2077-2095)Online publication date: Aug-2024
  • (2024)On Eye Tracking in Software EngineeringSN Computer Science10.1007/s42979-024-03045-35:6Online publication date: 26-Jul-2024
  • (2023)Assessing the Effect of Programming Language and Task Type on Eye Movements of Computer Science StudentsACM Transactions on Computing Education10.1145/363253024:1(1-38)Online publication date: 14-Nov-2023
  • (2023)Towards Modeling Human Attention from Eye Movements for Neural Source Code SummarizationProceedings of the ACM on Human-Computer Interaction10.1145/35911367:ETRA(1-19)Online publication date: 18-May-2023
  • (2023)Studying Developer Eye Movements to Measure Cognitive Workload and Visual Effort for Expertise AssessmentProceedings of the ACM on Human-Computer Interaction10.1145/35911357:ETRA(1-18)Online publication date: 18-May-2023
  • (2023)G-DAIC: A Gaze Initialized Framework for Description and Aesthetic-Based Image CroppingProceedings of the ACM on Human-Computer Interaction10.1145/35911327:ETRA(1-19)Online publication date: 18-May-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media