Abstract
The paper introduces a fundamental technological problem with collecting high-speed eye tracking data while studying software engineering tasks in an integrated development environment. The use of eye trackers is quickly becoming an important means to study software developers and how they comprehend source code and locate bugs. High quality eye trackers can record upwards of 120 to 300 gaze points per second. However, it is not always possible to map each of these points to a line and column position in a source code file (in the presence of scrolling and file switching) in real time at data rates over 60 gaze points per second without data loss. Unfortunately, higher data rates are more desirable as they allow for finer granularity and more accurate study analyses. To alleviate this technological problem, a novel method for eye tracking data collection is presented. Instead of performing gaze analysis in real time, all telemetry (keystrokes, mouse movements, and eye tracker output) data during a study is recorded as it happens. Sessions are then replayed at a much slower speed allowing for ample time to map gaze point positions to the appropriate file, line, and column to perform additional analysis. A description of the method and corresponding tool, Deja Vu, is presented. An evaluation of the method and tool is conducted using three different eye trackers running at four different speeds (60 Hz, 120 Hz, 150 Hz, and 300 Hz). This timing evaluation is performed in Visual Studio, Eclipse, and Atom IDEs. Results show that Deja Vu can playback 100% of the data recordings, correctly mapping the gaze to corresponding elements, making it a well-founded and suitable post processing step for future eye tracking studies in software engineering. Finally, a proof of concept replication analysis of four tasks from two previous studies is performed. Due to using the Deja Vu approach, this replication resulted in richer collected data and improved on the number of distinct syntactic categories that gaze was mapped on in the code.












Similar content being viewed by others
References
Abid NJ, Sharif B, Dragan N, Alrasheed H, Maletic JI (2019) Developer reading behavior while summarizing java methods: size and context matters. In: Atlee JM, Bultan T, Whittle J (eds) Proceedings of the 41st international conference on software engineering, ICSE 2019, Montreal, QC, Canada, May 25–31, 2019. https://doi.org/10.1109/ICSE.2019.00052. IEEE/ACM, pp 384–395
Andersson R, Nyström M, Holmqvist K (2010) Sampling frequency and eye-tracking measures: how speed affects durations, latencies, and more. J Eye Mov Res 3(3). https://doi.org/10.16910/jemr.3.3.6
Andersson R, Larsson L, Holmqvist K, Stridh M, Nyström M (2017) One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms. Behav Res Methods 49:616–637
Bao L, Ye D, Xing Z, Xia X, Wang X (2015) Activityspace: a remembrance framework to support interapplication information needs. In: Cohen MB, Grunske L, Whalen M (eds) 30th IEEE/ACM international conference on automated software engineering, ASE 2015, Lincoln, NE, USA, November 9–13, 2015. https://doi.org/10.1109/ASE.2015.90. IEEE Computer Society, pp 864–869
Bao L, Xing Z, Xia X, Lo D, Hassan AE (2018) Inference of development activities from interaction with uninstrumented applications. Empir Softw Eng 23(3):1313–1351. https://doi.org/10.1007/s10664-017-9547-8
Bernal-Cárdenas C, Cooper N, Moran K, Chaparro O, Marcus A, Poshyvanyk D (2020) Translating video recordings of mobile app usages into replayable scenarios. In: Rothermel G, Bae D (eds) ICSE ’20: 42nd international conference on software engineering, Seoul, South Korea, 27 June–19 July, 2020. https://doi.org/10.1145/3377811.3380328. ACM, pp 309–321
Brooks R (1983) Towards a theory of the comprehension of computer programs. Int J Man-Mach Stud 18 (6):543–554. https://doi.org/10.1016/S0020-7373(83)80031-5. https://www.sciencedirect.com/science/article/pii/S0020737383800315
Brown NCC, AlTadmri A, Sentance S, Kölling M (2018) Blackbox, five years on: an evaluation of a large-scale programming data collection project. In: Malmi L, Korhonen A, McCartney R, Petersen A (eds) Proceedings of the 2018 ACM conference on international computing education research, ICER 2018, Espoo, Finland, August 13–15, 2018. https://doi.org/10.1145/3230977.3230991. ACM, pp 196–204
Burg B, Bailey R, Ko AJ, Ernst MD (2013) Interactive record/replay for web application debugging. In: Izadi S, Quigley AJ, Poupyrev I, Igarashi T (eds) The 26th annual ACM symposium on user interface software and technology, UIST’13, St. Andrews, United Kingdom, October 8–11, 2013. https://doi.org/10.1145/2501988.2502050. ACM, pp 473–484
Collard ML, Decker MJ, Maletic J (2013) srcML: an infrastructure for the exploration, analysis, and manipulation of source code: a tool demonstration. In: 2013 IEEE International conference on software maintenance. https://doi.org/10.1109/ICSM.2013.85, pp 516–519
Dodd MD, der Stigchel SV, Hollingworth A (2009) Novelty is not always the best policy: inhibition of return and facilitation of return as a function of visual task. Psychol Sci 20(3):333–339. https://doi.org/10.1111/j.1467-9280.2009.02294.x. pMID: 19222812
Duchowski A (2007) Eye tracking methodology: theory and practice. https://doi.org/10.1007/978-1-84628-609-4
Duchowski A, Krejtz K, Gehrer N, Bafna T, Baekgaard P (2020a) The low/high index of pupillary activity. In: 2020 CHI conference on human factors in computing systems. https://doi.org/10.1145/3313831.3376394, pp 1–12
Duchowski AT, Krejtz K, Żurawska J, House DH (2020b) Using microsaccades to estimate task difficulty during visual search of layered surfaces. IEEE Trans Visual Comput Graph 26(9):2904–2918. https://doi.org/10.1109/TVCG.2019.2901881
Engbert R, Kliegl R (2003) Microsaccades uncover the orientation of covert attention. Vis Res 43 (9):1035–1045. https://doi.org/10.1016/S0042-6989(03)00084-1. https://www.sciencedirect.com/science/article/pii/S0042698903000841
Eriksen C (1995) The flankers task and response competition: a useful tool for investigating a variety of cognitive problems. Vis Cogn 2:101–118
Fakhoury S, Ma Y, Arnaoudova V, Adesope O (2018) The effect of poor source code lexicon and readability on developers’ cognitive load. In: Proceedings of the 26th conference on program comprehension, ICPC ’18. https://doi.org/10.1145/3196321.3196347. http://doi.acm.org/10.1145/3196321.3196347. ACM, New York, pp 286–296
Fakhoury S, Roy D, Pines H, Cleveland T, Peterson CS, Arnaoudova V, Sharif B, Maletic J (2021) gazel: supporting source code edits in eye-tracking studies. In: 2021 IEEE/ACM 43rd international conference on software engineering: companion proceedings (ICSE-Companion). https://doi.org/10.1109/ICSE-Companion52605.2021.00038, pp 69–72
Floyd B, Santander T, Weimer W (2017) Decoding the representation of code in the brain: an fmri study of code review and expertise. In: 2017 IEEE/ACM 39th international conference on software engineering (ICSE). https://doi.org/10.1109/ICSE.2017.24, pp 175–186
Goldberg JH, Stimson MJ, Lewenstein M, Scott N, Wichansky AM (2002) Eye tracking in web search tasks: design implications. In: Proceedings of the 2002 symposium on eye tracking research & applications, ETRA ’02. https://doi.org/10.1145/507072.507082. http://doi.acm.org/10.1145/507072.507082. ACM, New York, pp 51–58
Guarnera DT, Bryant CA, Mishra A, Maletic JI, Sharif B (2018) itrace: eye tracking infrastructure for development environments. In: Proceedings of the 2018 ACM symposium on eye tracking research & applications. ACM, p 105
Guo J, Li S, Lou J, Yang Z, Liu T (2019) Sara: self-replay augmented record and replay for android in industrial cases. In: Zhang D, Møller A (eds) Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2019, Beijing, China, July 15–19, 2019. https://doi.org/10.1145/3293882.3330557. ACM, pp 90–100
Hafed ZM, Clark JJ (2002) Microsaccades as an overt measure of covert attention shifts. Vis Res 42(22):2533–2545. https://doi.org/10.1016/S0042-6989(02)00263-8. https://www.sciencedirect.com/science/article/pii/S0042698902002638
Holmqvist K, Andersson R (2017) Eye-tracking: a comprehensive guide to methods, paradigms and measures. Oxford University Press, Oxford
Just M, Carpenter P (1980) A theory of reading: from eye fixations to comprehension. Psychol Rev 87(4):329–54
Kelleher C, Hnin W (2019) Predicting cognitive load in future code puzzles. Association for Computing Machinery, New York, pp 1–12. https://doi.org/10.1145/3290605.3300487
Kersten M, Murphy GC (2006) Using task context to improve programmer productivity. In: Young M, Devanbu PT (eds) Proceedings of the 14th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2006, Portland, Oregon, USA, November 5–11, 2006. https://doi.org/10.1145/1181775.1181777. ACM, pp 1–11
Kevic K, Walters BM, Shaffer TR, Sharif B, Fritz T, Shepherd DC (2015) Tracing software developers eyes and interactions for change tasks. In: Proceedings of the 10th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering
Kevic K, Walters B, Shaffer T, Sharif B, Shepherd DC, Fritz T (2017) Eye gaze and interaction contexts for change tasks—observations and potential. J Syst Softw 128:252–266. https://doi.org/10.1016/j.jss.2016.03.030
Klein RM, MacInnes WJ (1999) Inhibition of return is a foraging facilitator in visual search. Psychol Sci 10(4):346–352. https://doi.org/10.1111/1467-9280.00166
Letovsky S (1987) Cognitive processes in program comprehension. J Syst Softw 7(4):325–339. https://doi.org/10.1016/0164-1212(87)90032-X. https://www.sciencedirect.com/science/article/pii/016412128790032X
Lowet E, Gomes B, Srinivasan K, Zhou H, Desimone R (2018) Enhanced neural processing by covert attention only during microsaccades directed toward the attended stimulus. Neuron 99:207–214.e3
Lupiáñez J (2010) Inhibition of return. Scholarpedia 3:17–34
Microsoft (2018) mouse_event function (winuser.h). https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mouse_event
Minelli R, Mocci A, Lanza M, Kobayashi T (2014) Quantifying program comprehension with interaction data. In: 2014 14th International conference on quality software. https://doi.org/10.1109/QSIC.2014.11, pp 276–285
Minelli R, Mocci A, Lanza M (2015) I know what you did last summer—an investigation of how developers spend their time. In: 2015 IEEE 23rd International conference on program comprehension. https://doi.org/10.1109/ICPC.2015.12, pp 25–35
Minelli R, Mocci A, Lanza M (2016) Measuring navigation efficiency in the ide. p to be published. https://doi.org/10.1109/IWESEP.2016.11
Nelson GL, Ko AJ (2018) On use of theory in computing education research. In: Malmi L, Korhonen A, McCartney R, Petersen A (eds) Proceedings of the 2018 ACM conference on international computing education research, ICER 2018, Espoo, Finland, August 13–15, 2018. https://doi.org/10.1145/3230977.3230992. ACM, pp 31–39
Niño IJ, de la Ossa B, Gil JA, Sahuquillo J, Pont A (2005) CARENA: a tool to capture and replay web navigation sessions. In: Al-Shaer E, Pras A, Owezarski P (eds) Third IEEE/IFIP workshop on end-to-end monitoring techniques and services, e2EMON 2005, 15th May 2005. https://doi.org/10.1109/E2EMON.2005.1564474. IEEE Computer Society, Nice, pp 127–141
Obaidellah U, Al Haek M, Cheng PCH (2018) A survey on the usage of eye-tracking in computer programming. ACM Comput Surv 51(1):5:1–5:58. https://doi.org/10.1145/3145904. http://doi.acm.org/10.1145/3145904
Olsson P (2007) Real-time and offline filters for eye tracking. KTH Electrical Engineering, Stockholm
Park K, Sharif B (2021) Assessing perceived sentiment in pull requests with emoji: evidence from tools and developer eye movements. In: 6th IEEE/ACM international workshop on emotion awareness in software engineering, SEmotion@ICSE 2021, Madrid, Spain, May 31, 2021. https://doi.org/10.1109/SEmotion52567.2021.00009. IEEE, pp 1–6
Pennington N (1987) Stimulus structures and mental representations in expert comprehension of computer programs. Cogn Psychol 19(3):295–341. https://doi.org/10.1016/0010-0285(87)90007-7. https://www.sciencedirect.com/science/article/pii/0010028587900077
Peterson CS, Abid NJ, Bryant CA, Maletic JI, Sharif B (2019a) Factors influencing dwell time during source code reading: a large-scale replication experiment. In: Krejtz K, Sharif B (eds) Proceedings of the 11th ACM symposium on eye tracking research & applications, ETRA 2019, Denver, CO, USA, June 25–28, 2019. https://doi.org/10.1145/3314111.3319833. ACM, pp 38:1–38:4
Peterson CS, Saddler JA, Halavick NM, Sharif B (2019b) A gaze-based exploratory study on the information seeking behavior of developers on stack overflow. In: Mandryk RL, Brewster SA, Hancock M, Fitzpatrick G, Cox AL, Kostakos V, Perry M (eds) Extended abstracts of the 2019 CHI conference on human factors in computing systems, CHI 2019, Glasgow, Scotland, UK, May 04–09, 2019. https://doi.org/10.1145/3290607.3312801. ACM
Ramler R, Gattringer M, Pichler J (2020) Live replay of screen videos: Automatically executing real applications as shown in recordings. In: Kontogiannis K, Khomh F, Chatzigeorgiou A, Fokaefs M, Zhou M (eds) 27th IEEE international conference on software analysis, evolution and reengineering, SANER 2020, London, ON, Canada, February 18–21, 2020. https://doi.org/10.1109/SANER48275.2020.9054833. IEEE, pp 664–665
Rayner K (1978) Eye movements in reading and information processing. Psychol Bull 85(3):618–660
Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3):372–422
Rist RS (1986) Plans in programming: definition, demonstration, and development. In: Papers presented at the first workshop on empirical studies of programmers on empirical studies of programmers. Ablex Publishing Corp., USA, pp 28–47
Saddler J A, Peterson C S, Sama S, Nagaraj S, Baysal O, Guerrouj L, Sharif B (2020) Studying developer reading behavior on stack overflow during api summarization tasks. In: 2020 IEEE 27th international conference on software analysis, evolution and reengineering (SANER). IEEE, pp 195–205
Salvucci DD, Goldberg JH (2000) Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 symposium on eye tracking research & applications, ETRA ’00. https://doi.org/10.1145/355017.355028. http://doi.acm.org/10.1145/355017.355028. ACM, New York, pp 71–78
Sharafi Z, Shaffer T, Sharif B, Guéhéneuc Y (2015a) Eye-tracking metrics in software engineering. In: Sun J, Reddy YR, Bahulkar A, Pasala A (eds) 2015 Asia-Pacific software engineering conference, APSEC 2015, New Delhi, India, December 1–4, 2015. https://doi.org/10.1109/APSEC.2015.53. IEEE Computer Society, pp 96–103
Sharafi Z, Soh Z, Guéhéneuc YG (2015b) A systematic literature review on the usage of eye-tracking in software engineering. Inf Softw Technol (IST)
Sharafi Z, Sharif B, Guéhéneuc Y, Begel A, Bednarik R, Crosby M E (2020) A practical guide on conducting eye tracking studies in software engineering. Empir Softw Eng 25(5):3128–3174. https://doi.org/10.1007/s10664-020-09829-4
Sharif B, Maletic J (2016a) itrace: overcoming the limitations of short code examples in eye tracking experiments. In: 2016 IEEE International conference on software maintenance and evolution, ICSME 2016, Raleigh, NC, USA, October 2–7, 2016. https://doi.org/10.1109/ICSME.2016.61. IEEE Computer Society, p 647
Sharif B, Maletic J (2016b) itrace: overcoming the limitations of short code examples in eye tracking experiments. In: 2016 IEEE International conference on software maintenance and evolution (ICSME). https://doi.org/10.1109/ICSME.2016.61, pp 647–647
Sharif B, Meinken J, Shaffer T, Kagdi H (2016a) Eye movements in software traceability link recovery. Empir Softw Eng 1–40. https://doi.org/10.1007/s10664-016-9486-9
Sharif B, Shaffer T, Wise JL, Maletic JI (2016b) Tracking developers’ eyes in the IDE. IEEE Softw 33(3):105–108. https://doi.org/10.1109/MS.2016.84
Sharif B, Peterson C, Guarnera D, Bryant C, Buchanan Z, Zyrianov V, Maletic J (2019) Practical eye tracking with itrace. In: 2019 IEEE/ACM 6th international workshop on eye movements in programming (EMIP). https://doi.org/10.1109/EMIP.2019.00015, pp 41–42
Soloway E, Ehrlich K (1984) Empirical studies of programming knowledge. Software Engineering. IEEE Trans SE 10:595–609. https://doi.org/10.1109/TSE.1984.5010283
Stigchel S, Theeuwes J (2006) Our eyes deviate away from a location where a distractor is expected to appear. Exp Brain Res. Experimentelle Hirnforschung Expérimentation Cérébrale 169:338–49. https://doi.org/10.1007/s00221-005-0147-2
Stigchel S, Mills M, Dodd M (2010) Shift and deviate: Saccades reveal that shifts of covert attention evoked by trained spatial stimuli are obligatory. Atten Percept Psychophys 72:1244–50. https://doi.org/10.3758/APP.72.5.1244
Storey MD (2006) Theories, tools and research methods in program comprehension: past, present and future. Softw Qual J 14(3):187–208. https://doi.org/10.1007/s11219-006-9216-4
Sun Y, Chen D, Jiao W, Huang G (2014) An online education approach using web operation record and replay techniques. In: IEEE 38th Annual computer software and applications conference, COMPSAC 2014, Vasteras, Sweden, July 21–25, 2014. https://doi.org/10.1109/COMPSAC.2014.68. IEEE Computer Society, pp 456–465
Sun Y, Chen D, Xin C, Jiao W (2015) Automating repetitive tasks on web-based ides via an editable and reusable capture-replay technique. In: Ahamed SI, Chang CK, Chu WC, Crnkovic I, Hsiung P, Huang G, Yang J (eds) 39th IEEE annual computer software and applications conference, COMPSAC 2015, Taichung, Taiwan, July 1–5, 2015, vol 2. https://doi.org/10.1109/COMPSAC.2015.12. IEEE Computer Society, pp 666–675
Sun J, Zhang S, Huang S, Hui Z (2018) Design and application of a sikuli based capture-replay tool. In: 2018 IEEE International conference on software quality, reliability and security companion, QRS companion 2018, Lisbon, Portugal, July 16–20, 2018. https://doi.org/10.1109/QRS-C.2018.00021. IEEE, pp 42–44
Van der Stigchel S, Theeuwes J (2005) The influence of attending to multiple locations on eye movements. Vis Res 45(15):1921–1927. https://doi.org/10.1016/j.visres.2005.02.002. https://www.sciencedirect.com/science/article/pii/S0042698905000945
Von Mayrhauser A, Vans A (1995) Program comprehension during software maintenance and evolution. Computer 28(8):44–55. https://doi.org/10.1109/2.402076
Yan F, Qi Z, Xia M, Liu X (2018) Efficient and deterministic replay for web-enabled android apps. In: Chaudron M, Crnkovic I, Chechik M, Harman M (eds) Proceedings of the 40th international conference on software engineering: companion proceedings, ICSE 2018, Gothenburg, Sweden, May 27–June 03, 2018. https://doi.org/10.1145/3183440.3194994. ACM, pp 329–330
Zyrianov V, Guarnera DT, Peterson CS, Sharif B, Maletic JI (2020) Automated recording and semantics-aware replaying of high-speed eye tracking and interaction data to support cognitive studies of software engineering tasks. In: IEEE International conference on software maintenance and evolution, ICSME 2020, Adelaide, Australia, September 28–October 2, 2020. https://doi.org/10.1109/ICSME46990.2020.00051. IEEE, pp 464–475
Acknowledgements
The authors would like to thank the anonymous reviewers for their insightful comments and suggestions. This work has been partly funded by the US NSF under Grant Numbers CNS 17-30181, CNS 18-55753, and CCF 18-55756
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing Interests
The authors have no competing interests with the editorial board members or editors. The authors have no other competing interests, financial or otherwise.
The work is supported in part by a grant from the US National Science Foundation CNS 17- 30181/30307.
Additional information
Communicated by: Zhenchang Xing, Kelly Blincoe
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article belongs to the Topical Collection: Software Maintenance and Evolution (ICSME)
About this article
Cite this article
Zyrianov, V., Peterson, C.S., Guarnera, D.T. et al. Deja Vu: semantics-aware recording and replay of high-speed eye tracking and interaction data to support cognitive studies of software engineering tasks—methodology and analyses. Empir Software Eng 27, 168 (2022). https://doi.org/10.1007/s10664-022-10209-3
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-022-10209-3