Skip to main content
SpringerLink
Log in

Extracting and analyzing time-series HCI data from screen-captured task videos

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Recent years have witnessed the increasing emphasis on human aspects in software engineering research and practices. Our survey of existing studies on human aspects in software engineering shows that screen-captured videos have been widely used to record developers’ behavior and study software engineering practices. The screen-captured videos provide direct information about which software tools the developers interact with and which content they access or generate during the task. Such Human-Computer Interaction (HCI) data can help researchers and practitioners understand and improve software engineering practices from human perspective. However, extracting time-series HCI data from screen-captured task videos requires manual transcribing and coding of videos, which is tedious and error-prone. In this paper we report a formative study to understand the challenges in manually transcribing screen-captured videos into time-series HCI data. We then present a computer-vision based video scraping technique to automatically extract time-series HCI data from screen-captured videos. We also present a case study of our scvRipper tool that implements the video scraping technique using 29-hours of task videos of 20 developers in two development tasks. The case study not only evaluates the runtime performance and robustness of the tool, but also performs a detailed quantitative analysis of the tool’s ability to extract time-series HCI data from screen-captured task videos. We also study the developer’s micro-level behavior patterns in software development from the quantitative analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://www.techsmith.com/snagit.html

  2. http://docs.opencv.org/modules/gpu/doc/introduction.html

References

  • Ammar N, Abi-Antoun M (2012) Empirical evaluation of diagrams of the run-time structure for coding tasks. In: Proceedings of the WCRE, pp 367–376

  • Bandura A (1986) Social foundations of thought and action: a social cognitive theory, vol 1, p 617

  • Banovic N, Grossman T, Matejka J, Fitzmaurice G (2012) Waken: reverse engineering usage information and interface structure from software videos. In: Proceedings of the UIST, pp 83–92

  • Bao L, Ye D, Xing Z, Xia X (2015a) ActivitySpace: a remembrance framework to support interapplication information needs. In: Proceedings 30th IEEE/ACM international conference on automated software engineering

  • Bao L, Li J, Xing Z, Wang X, Zhou B (2015b) Reverse engineering time-series interaction data from screen-captured videos. In: Proceedings of the SANER, pp 399–408

  • Bao L, Li J, Xing Z, Wang X, Zhou B (2015c) scvRipper: video scraping tool for modeling developers behavior using interaction data. In: Proceedings of the ICSE, pp 673–676

  • Bao L, Xing Z, Wang X, Zhou B (2015d) Tracking and analyzing cross-cutting activities in developers’ daily work. In: Proceedings of the 30th IEEE/ACM international conference on automated software engineering

  • Barabasi AL (2005) The origin of bursts and heavy tails in human dynamics. Nature 435(7039):207–211

    Article  Google Scholar 

  • Bateman S, Teevan J, White RW (2012) The search dashboard: how reflection and comparison impact search behavior. In: Proceedings of the CHI, 1785

  • Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Comp Vision Image Underst 110(3):346–359

    Article  Google Scholar 

  • Bordes A, Gabrilovich E (2014) Constructing and mining web-scale knowledge graphs: KDD 2014 tutorial. In: Proceedings of the KDD, p 1967

  • Brade K, Guzdial M, Steckel M, Soloway E (1992) Whorf: A visualization tool for software maintenance. In: Proceedings 1992 IEEE workshop on visual languages, pp 148–154

  • Brandt J, Guo PJ, Lewenstein J, Dontcheva M, Klemmer SR, Francisco S (2009) Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In: Proceedings of the CHI, pp 1589–1598

  • Brandt J, Dontcheva M, Weskamp M, Klemmer SR, Francisco S (2010) Example-centric programming: integrating web search into the development environment. In: Proceedings of the CHI, pp 513–522

  • Brown JS, Collins A, Duguid P (1989) Situated cognition and the culture of learning

  • Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell:679–698

  • Chang T-H, Yeh T, Miller R (2011) Associating the visual representation of user interfaces with their internal structures and metadata. In: Proceedings of the UIST, pp 245–256

  • Chang T-H, Yeh T, Miller RC (2010) GUI testing using computer vision. In: Proceeding of the CHI, pp 1535–1544

  • Colin Cameron A, Windmeijer FA (1997) An r-squared measure of goodness of fit for some common nonlinear regression models. J Econ 77(2):329–342

    Article  MathSciNet  MATH  Google Scholar 

  • Corritore CL, Wiedenbeck S (2000) Direction and scope of comprehension-related activities by procedural and object-oriented programmers: An empirical study. In: Proceedings of the IWPC. IEEE, pp 139– 148

  • Corritore CL, Wiedenbeck S (2001) An exploratory study of program comprehension strategies of procedural and object-oriented programmers. Int J Hum-Comput St 54(1):1–23

    Article  MATH  Google Scholar 

  • Dekel U, Herbsleb JD (2009) Reading the documentation of invoked API functions in program comprehension, pp 168–177

  • Dewan P, Agarwal P, Shroff G, Hegde R (2009) Distributed side-by-side programming. In: Proceedings of the 2009 ICSE workshop on cooperative and human aspects on software engineering, pp 48–55

  • Dixon M, Fogarty J (2010) Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure. In: Proceedings of the CHI, pp 1525–1534

  • Duala-Ekoko E, Robillard MP (2012) Asking and answering questions about unfamiliar APIs: an exploratory study. In: Proceedings of the ICSE, pp 266–276

  • Ester M, Kriegel H-P, Sander J, Xu X (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the KDD, vol 96, pp 226–231

  • Forsyth DA, Ponce J (2002) Computer vision: a modern approach. Prentice Hall Professional Technical Reference

  • Fritz T, Shepherd DC, Kevic K, Snipes W, Bräunlich C (2014) Developers’ code context models for change tasks. In: Proceedings of the FSE, pp 7–18

  • Gonzalez RC, Woods RE (2002) Digital image processing. Prentice hall Upper Saddle River, NJ

    Google Scholar 

  • Greiler M, van Deursen A, Storey M (2012) Test confessions: a study of testing practices for plug-in systems, pp 244–254

  • Guha R, Guha R, McCool R, McCool R, Miller E, Miller E (2003) Semantic search. In: Proceedings of the WWW, pp 700–709

  • Hartmann B, Macdougall D, Brandt J, Klemmer SR (2010) What would other programmers do? Suggesting solutions to error messages. In: Proceedings of the CHI, pp 1019–1028

  • Hilbert DM, Redmiles DF (2000) Extracting usability information from user interface events. ACM Comput Surv 32(4):384–421

    Article  Google Scholar 

  • Hundhausen CD, Brown JL, Farley S, Skarpas D (2006) A methodology for analyzing the temporal evolution of novice programs based on semantic components. In: Proceedings of the ACM international computing education research workshop, pp 59–71

  • Hurst A, Hudson SE, Mankoff J (2010) Automatically identifying targets users interact with during real world tasks. In: Proceedings of the IUI. ACM, pp 11–20

  • Kim JH, Gunn DV, Schuh E, Phillips B, Pagulayan RJ, Wixon D (2008) Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In: Proceedings of the CHI, pp 443–452

  • Kimble C, Hildreth PM, Bourdon I (2008) Communities of practice: creating learning environments for educators, vol 1. Information Age Publisher

  • Ko AJ, Myers BA (2004) Designing the whyline: a debugging interface for asking questions about program behavior. In: Proceedings of the CHI, pp 151–158

  • Ko AJ, Myers BA (2005) A framework and methodology for studying the causes of software errors in programming systems. J Visual Lang Comput 16(1):41–84

    Article  Google Scholar 

  • Ko AJ, Aung HH, Myers BA (2005a) Design requirements for more flexible structured editors from a study of programmers’ text editing. In: CHI’05 extended abstracts on human factors in computing systems. ACM, pp 1557–1560

  • Ko AJ, Aung HH, Myers BA (2005b) Eliciting design requirements for maintenance-oriented IDEs: a detailed study of corrective and perfective maintenance tasks. In: Proceedings of the ICSE, pp 126–135

  • Ko AJ, Myers BA, Coblenz MJ, Aung HH (2006) An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans Softw Eng 32(12):971–987

    Article  Google Scholar 

  • Koru AG, Ozok A, Norcio AF (2005) The effect of human memory organization on code reviews under different single and pair code reviewing scenarios. ACM SIGSOFT Software Engineering Notes 30:1–3

    Google Scholar 

  • Kushman N, Katabi D (2010) Enabling configuration-independent automation by non-expert users. In: Proceedings of the ninth USENIX symposium on operating systems design and implementation, pp 223–236

  • Lawrance J, Bellamy R, Burnett M, Rector K (2008) Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks. In: Proceedings of the CHI. ACM, pp 1323–1332

  • Lawrance J, Bogart C, Burnett M, Bellamy R, Rector K, Fleming SD (2013) How programmers debug, revisited: an information foraging theory perspective. IEEE Trans Softw Eng 39(2):197–215

    Article  Google Scholar 

  • Lawrence J, Clarke S, Burnett M, Rothermel G (2005) How well do professional developers test with code coverage visualizations? An empirical study. In: Proceedings of the VL/HCC, pp 53–60

  • Leary MR (1991) Introduction to behavioral research methods. Wadsworth Publishing Company

  • Li H, Xing Z, Peng X, Zhao W (2013) What help do developers seek, when and how?. In: Proceedings of the WCRE, pp 142–151

  • Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the ICCV, vol 2, pp 1150–1157

  • Matejka J, Li W (2009) CommunityCommands: command recommendations for software applications. In: Proceedings of the UIST, pp 193–202

  • Matejka J, Grossman T, Fitzmaurice G (2011) Ambient help. In: Proceeding of the CHI, pp 2751–2760

  • Mavrikis M, Grawemeyer B, Hansen A, Gutierrez-Santos S (2014) Exploring the potential of speech recognition to support problem solving and reflection. In: Open learning and teaching in educational communities. Springer, Berlin, pp 263–276

    Google Scholar 

  • Muja M, Lowe DG (2009) Fast approximate nearest neighbors with automatic algorithm configuration. In: VISAPP (1), pp 331–340

  • Murphy-Hill ER, Zimmermann T, Nagappan N (2014) Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development?. In: Proceeding of the ICSE, pp 1–11

  • Nie Z, Zhang Y (2005) Object-level ranking: bringing order to web objects. In: Proceeding of the WWW, pp 567–574

  • Nie Z, Ma Y, Shi S, Wen J-r, Ma W-y (2007) Web Object Retrieval. In: Proceeding of the WWW, pp 81–90

  • Parzen E (1962) On estimation of a probability density function and mode. Ann Math Stat 33(3):1065–1076

    Article  MathSciNet  MATH  Google Scholar 

  • Piorkowski D, Fleming SD, Scaffidi C, John L, Bogart C, John BE, Burnett M, Bellamy R (2011) Modeling programmer navigation: a head-to-head empirical evaluation of predictive models. In: Proceeding of the VL/HCC, pp 109–116

  • Ponzanelli L, Bacchelli A, Lanza M (2013) Seahawk: Stack overflow in the IDE. In: Proceeding of the ICSE, pp 1295–1298

  • Rabiner L, Juang BH (1986) An introduction to hidden Markov models. IEEE ASSP Mag 3(1):4–16

    Article  Google Scholar 

  • Rhodes B (1996) Remembrance agent: a continuously running automated information retrieval system. In: The proceedings of the first international conference on the practical application of intelligent agents and multi agent technology, pp 122–125

  • Robillard MP, Coelho W, Murphy GC (2004) How effective developers investigate source code: an exploratory study. IEEE Trans Softw Eng 30(12):889–903

    Article  Google Scholar 

  • Rosten E, Drummond T (2006) Machine learning for high-speed corner detection. In: Computer Vision–ECCV 2006. Springer, Berlin, pp 430–443

    Chapter  Google Scholar 

  • Sarma A, Maccherone L, Wagstrom P, Herbsleb J (2009) Tesseract: interactive visual exploration of socio-technical relationships in software development. In: Proceeding of the ICSE, pp 23–33

  • Sawadsky N, Murphy GC (2011) Fishtail: from task context to source code examples. In: Proceeding of the 1st workshop on Developing tools as plug-ins - TOPI, p 48

  • Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888–905

    Article  Google Scholar 

  • Sillito J, De Voider K, Fisher B, Murphy G (2005) Managing software change tasks: an exploratory study. In: International Symposium on Empirical Software Engineering, IEEE, p 10

  • Silverman BW (1986) Density estimation for statistics and data analysis, vol 26. CRC press

  • Sinha SN, Frahm J-M, Pollefeys M, Genc Y (2006) GPU-based video feature tracking and matching. In: EDGE, workshop on edge computing using new commodity architectures, vol 278, p 4321

  • Vakilian M, Chen N, Negara S, Rajkumar BA, Bailey BP, Johnson RE (2012) Use, disuse, and misuse of automated refactorings. In: Proceeding of the ICSE, pp 233–243

  • von Mayrhauser A, Vans AM (1997) Program understanding behavior during debugging of large scale software. In: Empirical Studies of Programmers, 7th Workshop, ACM. ACM, pp 157–179

  • Wang J, Peng X, Xing Z, Zhao W (2011) An exploratory study of feature location process: Distinct phases, recurring patterns, and elementary actions. In: Proceeding of the ICSM, pp 213–222

  • Wang J, Peng X, Xing Z, Zhao W (2013) Improving feature location practice with multi-faceted interactive exploration. In: Proceeding of the ICSE, pp 762–771

  • Weisstein EW (2011) Least squares fitting–exponential. MathWorld-A Wolfram Web Resource. http://mathworld.wolfram.com/LeastSquaresFittingExponential.html

  • Whittaker JA, Poore JH (1993) Markov analysis of software specifications

  • Wu D-C, Tsai W-H (2000) Spatial-domain image hiding using image differencing. Proc ICCVISP 147(1):29–37

    Google Scholar 

  • Yang J, McAuley J, Leskovec J, LePendu P, Shah N (2014) Finding progression stages in time-evolving event sequences. In: Proceeding of the WWW, pp 783–794

  • Yeh T, Chang T-H, Miller RC (2009) Sikuli: using GUI screenshots for search and automation. In: Proceeding of the UIST, pp 183–192

  • Zhang Q, Chen Y, Zhang Y, Xu Y (2008) SIFT implementation and optimization for multi-core systems. In: Proceeding of the IPDPS, pp 1–8

  • Zhu J, Nie Z, Liu X, Zhang B, Wen J-R (2009) StatSnowball: a statistical approach to extracting entity relationships. In: Proceeding of the WWW, p 101

Download references

Acknowledgments

This work was partially supported by the Major State Basic Research Development Program of China (973 ProgramNo.2015CB352201) and National Key Technology R&D Program of the Ministry of Science and Technology of China (No. 2013BAH01B01). This work is supported by NTU SUG M4081029.020 and MOE AcRF Tier1 M4011165.020.

Author information

Authors and Affiliations

  1. College of Computer Science, Zhejiang University, Hangzhou, China

    Lingfeng Bao, Xinyu Wang, Xin Xia & Bo Zhou

  2. School of Computer Engineering, Nanyang Technological University, Singapore, Singapore

    Jing Li & Zhenchang Xing

Authors
  1. Lingfeng Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenchang Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xin Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bo Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinyu Wang.

Additional information

Communicated by: Emerson Murphy-Hill

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bao, L., Li, J., Xing, Z. et al. Extracting and analyzing time-series HCI data from screen-captured task videos. Empir Software Eng 22, 134–174 (2017). https://doi.org/10.1007/s10664-015-9417-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-015-9417-1

Keywords

Search

Navigation