Skip to main content

Advertisement

Log in

Deja Vu: semantics-aware recording and replay of high-speed eye tracking and interaction data to support cognitive studies of software engineering tasks—methodology and analyses

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

The paper introduces a fundamental technological problem with collecting high-speed eye tracking data while studying software engineering tasks in an integrated development environment. The use of eye trackers is quickly becoming an important means to study software developers and how they comprehend source code and locate bugs. High quality eye trackers can record upwards of 120 to 300 gaze points per second. However, it is not always possible to map each of these points to a line and column position in a source code file (in the presence of scrolling and file switching) in real time at data rates over 60 gaze points per second without data loss. Unfortunately, higher data rates are more desirable as they allow for finer granularity and more accurate study analyses. To alleviate this technological problem, a novel method for eye tracking data collection is presented. Instead of performing gaze analysis in real time, all telemetry (keystrokes, mouse movements, and eye tracker output) data during a study is recorded as it happens. Sessions are then replayed at a much slower speed allowing for ample time to map gaze point positions to the appropriate file, line, and column to perform additional analysis. A description of the method and corresponding tool, Deja Vu, is presented. An evaluation of the method and tool is conducted using three different eye trackers running at four different speeds (60 Hz, 120 Hz, 150 Hz, and 300 Hz). This timing evaluation is performed in Visual Studio, Eclipse, and Atom IDEs. Results show that Deja Vu can playback 100% of the data recordings, correctly mapping the gaze to corresponding elements, making it a well-founded and suitable post processing step for future eye tracking studies in software engineering. Finally, a proof of concept replication analysis of four tasks from two previous studies is performed. Due to using the Deja Vu approach, this replication resulted in richer collected data and improved on the number of distinct syntactic categories that gaze was mapped on in the code.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Abid NJ, Sharif B, Dragan N, Alrasheed H, Maletic JI (2019) Developer reading behavior while summarizing java methods: size and context matters. In: Atlee JM, Bultan T, Whittle J (eds) Proceedings of the 41st international conference on software engineering, ICSE 2019, Montreal, QC, Canada, May 25–31, 2019. https://doi.org/10.1109/ICSE.2019.00052. IEEE/ACM, pp 384–395

  • Andersson R, Nyström M, Holmqvist K (2010) Sampling frequency and eye-tracking measures: how speed affects durations, latencies, and more. J Eye Mov Res 3(3). https://doi.org/10.16910/jemr.3.3.6

  • Andersson R, Larsson L, Holmqvist K, Stridh M, Nyström M (2017) One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms. Behav Res Methods 49:616–637

    Article  Google Scholar 

  • Bao L, Ye D, Xing Z, Xia X, Wang X (2015) Activityspace: a remembrance framework to support interapplication information needs. In: Cohen MB, Grunske L, Whalen M (eds) 30th IEEE/ACM international conference on automated software engineering, ASE 2015, Lincoln, NE, USA, November 9–13, 2015. https://doi.org/10.1109/ASE.2015.90. IEEE Computer Society, pp 864–869

  • Bao L, Xing Z, Xia X, Lo D, Hassan AE (2018) Inference of development activities from interaction with uninstrumented applications. Empir Softw Eng 23(3):1313–1351. https://doi.org/10.1007/s10664-017-9547-8

    Article  Google Scholar 

  • Bernal-Cárdenas C, Cooper N, Moran K, Chaparro O, Marcus A, Poshyvanyk D (2020) Translating video recordings of mobile app usages into replayable scenarios. In: Rothermel G, Bae D (eds) ICSE ’20: 42nd international conference on software engineering, Seoul, South Korea, 27 June–19 July, 2020. https://doi.org/10.1145/3377811.3380328. ACM, pp 309–321

  • Brooks R (1983) Towards a theory of the comprehension of computer programs. Int J Man-Mach Stud 18 (6):543–554. https://doi.org/10.1016/S0020-7373(83)80031-5. https://www.sciencedirect.com/science/article/pii/S0020737383800315

    Article  Google Scholar 

  • Brown NCC, AlTadmri A, Sentance S, Kölling M (2018) Blackbox, five years on: an evaluation of a large-scale programming data collection project. In: Malmi L, Korhonen A, McCartney R, Petersen A (eds) Proceedings of the 2018 ACM conference on international computing education research, ICER 2018, Espoo, Finland, August 13–15, 2018. https://doi.org/10.1145/3230977.3230991. ACM, pp 196–204

  • Burg B, Bailey R, Ko AJ, Ernst MD (2013) Interactive record/replay for web application debugging. In: Izadi S, Quigley AJ, Poupyrev I, Igarashi T (eds) The 26th annual ACM symposium on user interface software and technology, UIST’13, St. Andrews, United Kingdom, October 8–11, 2013. https://doi.org/10.1145/2501988.2502050. ACM, pp 473–484

  • Collard ML, Decker MJ, Maletic J (2013) srcML: an infrastructure for the exploration, analysis, and manipulation of source code: a tool demonstration. In: 2013 IEEE International conference on software maintenance. https://doi.org/10.1109/ICSM.2013.85, pp 516–519

  • Dodd MD, der Stigchel SV, Hollingworth A (2009) Novelty is not always the best policy: inhibition of return and facilitation of return as a function of visual task. Psychol Sci 20(3):333–339. https://doi.org/10.1111/j.1467-9280.2009.02294.x. pMID: 19222812

    Article  Google Scholar 

  • Duchowski A (2007) Eye tracking methodology: theory and practice. https://doi.org/10.1007/978-1-84628-609-4

  • Duchowski A, Krejtz K, Gehrer N, Bafna T, Baekgaard P (2020a) The low/high index of pupillary activity. In: 2020 CHI conference on human factors in computing systems. https://doi.org/10.1145/3313831.3376394, pp 1–12

  • Duchowski AT, Krejtz K, Żurawska J, House DH (2020b) Using microsaccades to estimate task difficulty during visual search of layered surfaces. IEEE Trans Visual Comput Graph 26(9):2904–2918. https://doi.org/10.1109/TVCG.2019.2901881

    Article  Google Scholar 

  • Engbert R, Kliegl R (2003) Microsaccades uncover the orientation of covert attention. Vis Res 43 (9):1035–1045. https://doi.org/10.1016/S0042-6989(03)00084-1. https://www.sciencedirect.com/science/article/pii/S0042698903000841

    Article  Google Scholar 

  • Eriksen C (1995) The flankers task and response competition: a useful tool for investigating a variety of cognitive problems. Vis Cogn 2:101–118

    Article  Google Scholar 

  • Fakhoury S, Ma Y, Arnaoudova V, Adesope O (2018) The effect of poor source code lexicon and readability on developers’ cognitive load. In: Proceedings of the 26th conference on program comprehension, ICPC ’18. https://doi.org/10.1145/3196321.3196347. http://doi.acm.org/10.1145/3196321.3196347. ACM, New York, pp 286–296

  • Fakhoury S, Roy D, Pines H, Cleveland T, Peterson CS, Arnaoudova V, Sharif B, Maletic J (2021) gazel: supporting source code edits in eye-tracking studies. In: 2021 IEEE/ACM 43rd international conference on software engineering: companion proceedings (ICSE-Companion). https://doi.org/10.1109/ICSE-Companion52605.2021.00038, pp 69–72

  • Floyd B, Santander T, Weimer W (2017) Decoding the representation of code in the brain: an fmri study of code review and expertise. In: 2017 IEEE/ACM 39th international conference on software engineering (ICSE). https://doi.org/10.1109/ICSE.2017.24, pp 175–186

  • Goldberg JH, Stimson MJ, Lewenstein M, Scott N, Wichansky AM (2002) Eye tracking in web search tasks: design implications. In: Proceedings of the 2002 symposium on eye tracking research & applications, ETRA ’02. https://doi.org/10.1145/507072.507082. http://doi.acm.org/10.1145/507072.507082. ACM, New York, pp 51–58

  • Guarnera DT, Bryant CA, Mishra A, Maletic JI, Sharif B (2018) itrace: eye tracking infrastructure for development environments. In: Proceedings of the 2018 ACM symposium on eye tracking research & applications. ACM, p 105

  • Guo J, Li S, Lou J, Yang Z, Liu T (2019) Sara: self-replay augmented record and replay for android in industrial cases. In: Zhang D, Møller A (eds) Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2019, Beijing, China, July 15–19, 2019. https://doi.org/10.1145/3293882.3330557. ACM, pp 90–100

  • Hafed ZM, Clark JJ (2002) Microsaccades as an overt measure of covert attention shifts. Vis Res 42(22):2533–2545. https://doi.org/10.1016/S0042-6989(02)00263-8. https://www.sciencedirect.com/science/article/pii/S0042698902002638

    Article  Google Scholar 

  • Holmqvist K, Andersson R (2017) Eye-tracking: a comprehensive guide to methods, paradigms and measures. Oxford University Press, Oxford

    Google Scholar 

  • Just M, Carpenter P (1980) A theory of reading: from eye fixations to comprehension. Psychol Rev 87(4):329–54

    Article  Google Scholar 

  • Kelleher C, Hnin W (2019) Predicting cognitive load in future code puzzles. Association for Computing Machinery, New York, pp 1–12. https://doi.org/10.1145/3290605.3300487

    Google Scholar 

  • Kersten M, Murphy GC (2006) Using task context to improve programmer productivity. In: Young M, Devanbu PT (eds) Proceedings of the 14th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2006, Portland, Oregon, USA, November 5–11, 2006. https://doi.org/10.1145/1181775.1181777. ACM, pp 1–11

  • Kevic K, Walters BM, Shaffer TR, Sharif B, Fritz T, Shepherd DC (2015) Tracing software developers eyes and interactions for change tasks. In: Proceedings of the 10th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering

  • Kevic K, Walters B, Shaffer T, Sharif B, Shepherd DC, Fritz T (2017) Eye gaze and interaction contexts for change tasks—observations and potential. J Syst Softw 128:252–266. https://doi.org/10.1016/j.jss.2016.03.030

    Article  Google Scholar 

  • Klein RM, MacInnes WJ (1999) Inhibition of return is a foraging facilitator in visual search. Psychol Sci 10(4):346–352. https://doi.org/10.1111/1467-9280.00166

    Article  Google Scholar 

  • Letovsky S (1987) Cognitive processes in program comprehension. J Syst Softw 7(4):325–339. https://doi.org/10.1016/0164-1212(87)90032-X. https://www.sciencedirect.com/science/article/pii/016412128790032X

    Article  Google Scholar 

  • Lowet E, Gomes B, Srinivasan K, Zhou H, Desimone R (2018) Enhanced neural processing by covert attention only during microsaccades directed toward the attended stimulus. Neuron 99:207–214.e3

    Article  Google Scholar 

  • Lupiáñez J (2010) Inhibition of return. Scholarpedia 3:17–34

    Google Scholar 

  • Microsoft (2018) mouse_event function (winuser.h). https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mouse_event

  • Minelli R, Mocci A, Lanza M, Kobayashi T (2014) Quantifying program comprehension with interaction data. In: 2014 14th International conference on quality software. https://doi.org/10.1109/QSIC.2014.11, pp 276–285

  • Minelli R, Mocci A, Lanza M (2015) I know what you did last summer—an investigation of how developers spend their time. In: 2015 IEEE 23rd International conference on program comprehension. https://doi.org/10.1109/ICPC.2015.12, pp 25–35

  • Minelli R, Mocci A, Lanza M (2016) Measuring navigation efficiency in the ide. p to be published. https://doi.org/10.1109/IWESEP.2016.11

  • Nelson GL, Ko AJ (2018) On use of theory in computing education research. In: Malmi L, Korhonen A, McCartney R, Petersen A (eds) Proceedings of the 2018 ACM conference on international computing education research, ICER 2018, Espoo, Finland, August 13–15, 2018. https://doi.org/10.1145/3230977.3230992. ACM, pp 31–39

  • Niño IJ, de la Ossa B, Gil JA, Sahuquillo J, Pont A (2005) CARENA: a tool to capture and replay web navigation sessions. In: Al-Shaer E, Pras A, Owezarski P (eds) Third IEEE/IFIP workshop on end-to-end monitoring techniques and services, e2EMON 2005, 15th May 2005. https://doi.org/10.1109/E2EMON.2005.1564474. IEEE Computer Society, Nice, pp 127–141

  • Obaidellah U, Al Haek M, Cheng PCH (2018) A survey on the usage of eye-tracking in computer programming. ACM Comput Surv 51(1):5:1–5:58. https://doi.org/10.1145/3145904. http://doi.acm.org/10.1145/3145904

    Google Scholar 

  • Olsson P (2007) Real-time and offline filters for eye tracking. KTH Electrical Engineering, Stockholm

    Google Scholar 

  • Park K, Sharif B (2021) Assessing perceived sentiment in pull requests with emoji: evidence from tools and developer eye movements. In: 6th IEEE/ACM international workshop on emotion awareness in software engineering, SEmotion@ICSE 2021, Madrid, Spain, May 31, 2021. https://doi.org/10.1109/SEmotion52567.2021.00009. IEEE, pp 1–6

  • Pennington N (1987) Stimulus structures and mental representations in expert comprehension of computer programs. Cogn Psychol 19(3):295–341. https://doi.org/10.1016/0010-0285(87)90007-7. https://www.sciencedirect.com/science/article/pii/0010028587900077

    Article  Google Scholar 

  • Peterson CS, Abid NJ, Bryant CA, Maletic JI, Sharif B (2019a) Factors influencing dwell time during source code reading: a large-scale replication experiment. In: Krejtz K, Sharif B (eds) Proceedings of the 11th ACM symposium on eye tracking research & applications, ETRA 2019, Denver, CO, USA, June 25–28, 2019. https://doi.org/10.1145/3314111.3319833. ACM, pp 38:1–38:4

  • Peterson CS, Saddler JA, Halavick NM, Sharif B (2019b) A gaze-based exploratory study on the information seeking behavior of developers on stack overflow. In: Mandryk RL, Brewster SA, Hancock M, Fitzpatrick G, Cox AL, Kostakos V, Perry M (eds) Extended abstracts of the 2019 CHI conference on human factors in computing systems, CHI 2019, Glasgow, Scotland, UK, May 04–09, 2019. https://doi.org/10.1145/3290607.3312801. ACM

  • Ramler R, Gattringer M, Pichler J (2020) Live replay of screen videos: Automatically executing real applications as shown in recordings. In: Kontogiannis K, Khomh F, Chatzigeorgiou A, Fokaefs M, Zhou M (eds) 27th IEEE international conference on software analysis, evolution and reengineering, SANER 2020, London, ON, Canada, February 18–21, 2020. https://doi.org/10.1109/SANER48275.2020.9054833. IEEE, pp 664–665

  • Rayner K (1978) Eye movements in reading and information processing. Psychol Bull 85(3):618–660

    Article  Google Scholar 

  • Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3):372–422

    Article  Google Scholar 

  • Rist RS (1986) Plans in programming: definition, demonstration, and development. In: Papers presented at the first workshop on empirical studies of programmers on empirical studies of programmers. Ablex Publishing Corp., USA, pp 28–47

  • Saddler J A, Peterson C S, Sama S, Nagaraj S, Baysal O, Guerrouj L, Sharif B (2020) Studying developer reading behavior on stack overflow during api summarization tasks. In: 2020 IEEE 27th international conference on software analysis, evolution and reengineering (SANER). IEEE, pp 195–205

  • Salvucci DD, Goldberg JH (2000) Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 symposium on eye tracking research & applications, ETRA ’00. https://doi.org/10.1145/355017.355028. http://doi.acm.org/10.1145/355017.355028. ACM, New York, pp 71–78

  • Sharafi Z, Shaffer T, Sharif B, Guéhéneuc Y (2015a) Eye-tracking metrics in software engineering. In: Sun J, Reddy YR, Bahulkar A, Pasala A (eds) 2015 Asia-Pacific software engineering conference, APSEC 2015, New Delhi, India, December 1–4, 2015. https://doi.org/10.1109/APSEC.2015.53. IEEE Computer Society, pp 96–103

  • Sharafi Z, Soh Z, Guéhéneuc YG (2015b) A systematic literature review on the usage of eye-tracking in software engineering. Inf Softw Technol (IST)

  • Sharafi Z, Sharif B, Guéhéneuc Y, Begel A, Bednarik R, Crosby M E (2020) A practical guide on conducting eye tracking studies in software engineering. Empir Softw Eng 25(5):3128–3174. https://doi.org/10.1007/s10664-020-09829-4

    Article  Google Scholar 

  • Sharif B, Maletic J (2016a) itrace: overcoming the limitations of short code examples in eye tracking experiments. In: 2016 IEEE International conference on software maintenance and evolution, ICSME 2016, Raleigh, NC, USA, October 2–7, 2016. https://doi.org/10.1109/ICSME.2016.61. IEEE Computer Society, p 647

  • Sharif B, Maletic J (2016b) itrace: overcoming the limitations of short code examples in eye tracking experiments. In: 2016 IEEE International conference on software maintenance and evolution (ICSME). https://doi.org/10.1109/ICSME.2016.61, pp 647–647

  • Sharif B, Meinken J, Shaffer T, Kagdi H (2016a) Eye movements in software traceability link recovery. Empir Softw Eng 1–40. https://doi.org/10.1007/s10664-016-9486-9

  • Sharif B, Shaffer T, Wise JL, Maletic JI (2016b) Tracking developers’ eyes in the IDE. IEEE Softw 33(3):105–108. https://doi.org/10.1109/MS.2016.84

    Article  Google Scholar 

  • Sharif B, Peterson C, Guarnera D, Bryant C, Buchanan Z, Zyrianov V, Maletic J (2019) Practical eye tracking with itrace. In: 2019 IEEE/ACM 6th international workshop on eye movements in programming (EMIP). https://doi.org/10.1109/EMIP.2019.00015, pp 41–42

  • Soloway E, Ehrlich K (1984) Empirical studies of programming knowledge. Software Engineering. IEEE Trans SE 10:595–609. https://doi.org/10.1109/TSE.1984.5010283

    Article  Google Scholar 

  • Stigchel S, Theeuwes J (2006) Our eyes deviate away from a location where a distractor is expected to appear. Exp Brain Res. Experimentelle Hirnforschung Expérimentation Cérébrale 169:338–49. https://doi.org/10.1007/s00221-005-0147-2

    Article  Google Scholar 

  • Stigchel S, Mills M, Dodd M (2010) Shift and deviate: Saccades reveal that shifts of covert attention evoked by trained spatial stimuli are obligatory. Atten Percept Psychophys 72:1244–50. https://doi.org/10.3758/APP.72.5.1244

    Article  Google Scholar 

  • Storey MD (2006) Theories, tools and research methods in program comprehension: past, present and future. Softw Qual J 14(3):187–208. https://doi.org/10.1007/s11219-006-9216-4

    Article  Google Scholar 

  • Sun Y, Chen D, Jiao W, Huang G (2014) An online education approach using web operation record and replay techniques. In: IEEE 38th Annual computer software and applications conference, COMPSAC 2014, Vasteras, Sweden, July 21–25, 2014. https://doi.org/10.1109/COMPSAC.2014.68. IEEE Computer Society, pp 456–465

  • Sun Y, Chen D, Xin C, Jiao W (2015) Automating repetitive tasks on web-based ides via an editable and reusable capture-replay technique. In: Ahamed SI, Chang CK, Chu WC, Crnkovic I, Hsiung P, Huang G, Yang J (eds) 39th IEEE annual computer software and applications conference, COMPSAC 2015, Taichung, Taiwan, July 1–5, 2015, vol 2. https://doi.org/10.1109/COMPSAC.2015.12. IEEE Computer Society, pp 666–675

  • Sun J, Zhang S, Huang S, Hui Z (2018) Design and application of a sikuli based capture-replay tool. In: 2018 IEEE International conference on software quality, reliability and security companion, QRS companion 2018, Lisbon, Portugal, July 16–20, 2018. https://doi.org/10.1109/QRS-C.2018.00021. IEEE, pp 42–44

  • Van der Stigchel S, Theeuwes J (2005) The influence of attending to multiple locations on eye movements. Vis Res 45(15):1921–1927. https://doi.org/10.1016/j.visres.2005.02.002. https://www.sciencedirect.com/science/article/pii/S0042698905000945

    Article  Google Scholar 

  • Von Mayrhauser A, Vans A (1995) Program comprehension during software maintenance and evolution. Computer 28(8):44–55. https://doi.org/10.1109/2.402076

    Article  Google Scholar 

  • Yan F, Qi Z, Xia M, Liu X (2018) Efficient and deterministic replay for web-enabled android apps. In: Chaudron M, Crnkovic I, Chechik M, Harman M (eds) Proceedings of the 40th international conference on software engineering: companion proceedings, ICSE 2018, Gothenburg, Sweden, May 27–June 03, 2018. https://doi.org/10.1145/3183440.3194994. ACM, pp 329–330

  • Zyrianov V, Guarnera DT, Peterson CS, Sharif B, Maletic JI (2020) Automated recording and semantics-aware replaying of high-speed eye tracking and interaction data to support cognitive studies of software engineering tasks. In: IEEE International conference on software maintenance and evolution, ICSME 2020, Adelaide, Australia, September 28–October 2, 2020. https://doi.org/10.1109/ICSME46990.2020.00051. IEEE, pp 464–475

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their insightful comments and suggestions. This work has been partly funded by the US NSF under Grant Numbers CNS 17-30181, CNS 18-55753, and CCF 18-55756

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jonathan I. Maletic.

Ethics declarations

Competing Interests

The authors have no competing interests with the editorial board members or editors. The authors have no other competing interests, financial or otherwise.

The work is supported in part by a grant from the US National Science Foundation CNS 17- 30181/30307.

Additional information

Communicated by: Zhenchang Xing, Kelly Blincoe

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Software Maintenance and Evolution (ICSME)

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zyrianov, V., Peterson, C.S., Guarnera, D.T. et al. Deja Vu: semantics-aware recording and replay of high-speed eye tracking and interaction data to support cognitive studies of software engineering tasks—methodology and analyses. Empir Software Eng 27, 168 (2022). https://doi.org/10.1007/s10664-022-10209-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-022-10209-3

Keywords