skip to main content
10.1145/2884781.2884834acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Revisit of automatic debugging via human focus-tracking analysis

Published: 14 May 2016 Publication History

Abstract

In many fields of software engineering, studies on human behavior have attracted a lot of attention; however, few such studies exist in automated debugging. Parnin and Orso conducted a pioneering study comparing the performance of programmers in debugging with and without a ranking-based fault localization technique, namely Spectrum-Based Fault Localization (SBFL). In this paper, we revisit the actual helpfulness of SBFL, by addressing some major problems that were not resolved in Parnin and Orso's study. Our investigation involved 207 participants and 17 debugging tasks. A user-friendly SBFL tool was adopted. It was found that SBFL tended not to be helpful in improving the efficiency of debugging. By tracking and analyzing programmers' focus of attention, we characterized their source code navigation patterns and provided in-depth explanations to the observations. Results indicated that (1) a short "first scan" on the source code tended to result in inefficient debugging; and (2) inspections on the pinpointed statements during the "follow-up browsing" were normally just quick skimming. Moreover, we found that the SBFL assistance may even slightly weaken programmers' abilities in fault detection. Our observations imply interference between the mechanism of automated fault localization and the actual assistance needed by programmers in debugging. To resolve this interference, we provide several insights and suggestions.

References

[1]
R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. An evaluation of similarity coefficients for software fault localization. In Proceedings of the 12th Pacific Rim International Symposium on Dependable Computing, pages 39--46, Riverside, USA, 2006.
[2]
R. Bednarik. Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human-Computer Studies, 70(2):143--155, 2012.
[3]
D. Binkley, M. Davis, D. Lawrie, J. I. Maletic, C. Morrell, and B. Sharif. The impact of identifier style on effort and comprehension. Empirical Software Engineering, 18(2):219--276, 2013.
[4]
C. Chen, H.-G. Gross, and A. Zaidman. Improving service diagnosis through increased monitoring granularity. In Proceedings of the 7th IEEE International Conference on Software Security and Reliability, pages 129--138, Gaithersburg, Maryland, USA, 2013.
[5]
M. Chen and V. Lim. Eye gaze and mouse cursor relationship in a debugging task. In Proceedings of HCI International Posters' Extended Abstracts, pages 468--472. Springer, 2013.
[6]
M. E. Crosby and J. Stelovsky. How do we read algorithms? a case study. Computer, 23(1):25--35, 1990.
[7]
M. A. Francel and S. Rugaber. The value of slicing while debugging. Science of Computer Programming, 40(2):151--169, 2001.
[8]
J. Freeman, R. Dale, and T. Farmer. Hand in motion reveals mind in motion. Frontiers in Psychology, 2:59, 2011.
[9]
J. D. Gannon. Human factors in software engineering. Computer, 12(12):6--7, 1979.
[10]
C. Gouveia, J. Campos, and R. Abreu. Using html5 visualizations in software fault localization. In Proceedings of the First IEEE Working Conference on Software Visualization, pages 1--10, Eindhoven, Netherlands, 2013.
[11]
J. Huang, R. W. White, and S. Dumais. No clicks, no problem: Using cursor movements to understand and improve search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1225--1234, Vancouver, BC, Canada, 2011.
[12]
J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering, pages 467--477, Florida, USA, 2002.
[13]
A. Ko, B. Myers, M. Coblenz, and H. Aung. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Transactions on Software Engineering, 32(12):971--987, 2006.
[14]
A. J. Ko and B. A. Myers. Designing the whyline: a debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 151--158, Vienna, Austria, 2004.
[15]
B. Korel and J. Laski. Dynamic program slicing. Information Processing Letters, 29(3):155--163, 1988.
[16]
J. Lawrance, C. Bogart, M. Burnett, R. Bellamy, K. Rector, and S. Fleming. How programmers debug, revisited: An information foraging theory perspective. IEEE Transactions on Software Engineering, 39(2):197--215, 2013.
[17]
T.-D. B. Le, D. LO, and F. Thung. Should i follow this fault localization tool's output? automated prediction of fault localization effectiveness. Empirical Software Engineering, pages 1--38, 2015.
[18]
C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead. Does bug prediction support human developers? findings from a google case study. In Proceedings of 35th IEEE/ACM International Conference on Software Engineering, pages 372--381, San Francisco, USA, 2013.
[19]
Mooctest. http://mooctest.net/.
[20]
L. Naish, H. J. Lee, and K. Ramamohanarao. A model for spectra-based software diagnosis. ACM Transactions on Software Engineering and Methodology, 20(3):11:1--11:32, 2011.
[21]
C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In Proceedings of the International Symposium on Software Testing and Analysis, pages 199--209, Toronto, Canada, 2011.
[22]
D. Piorkowski, S. Fleming, C. Scaffidi, L. John, C. Bogart, B. John, M. Burnett, and R. Bellamy. Modeling programmer navigation: A head-to-head empirical evaluation of predictive models. In Proceedings of 2011 IEEE Symposium on Visual Languages and Human-Centric Computing, pages 109--116, Pittsburgh, PA, USA, 2011.
[23]
P. Rodeghero, C. McMillan, P. W. McBurney, N. Bosch, and S. D'Mello. Improving automated source code summarization via an eye-tracking study of programmers. In Proceedings of the 36th International Conference on Software Engineering, pages 390--401, Hyderabad, India, 2014.
[24]
R. Saha, M. Lease, S. Khurshid, and D. Perry. Improving bug localization using structured information retrieval. In Proceedings of the 28th International Conference on Automated Software Engineering, pages 345--355, Palo Alto, California, USA, 2013.
[25]
I. Salman, A. T. Misirli, and N. Juristo. Are students representatives of professionals in software engineering experiments? In Proceedings of 37th IEEE/ACM International Conference on Software Engineering, pages 666--676, Florence, Italy, 2015.
[26]
B. Sharif, M. Falcone, and J. I. Maletic. An eye-tracking study on the role of scan time in finding source code defects. In Proceedings of the Symposium on Eye Tracking Research and Applications, pages 381--384, Santa Barbara, California, 2012.
[27]
B. Sharif, J. Maletic, et al. The effects of layout on detecting the role of design patterns. In Proceedings of the IEEE Conference on Software Engineering Education and Training, pages 41--48, Pittsburgh, USA, 2010.
[28]
J. Silva and O. Chitil. Combining algorithmic debugging and program slicing. In Proceedings of the 8th ACM SIGPLAN international conference on Principles and practice of declarative programming, pages 157--166, Venice, Italy, 2006.
[29]
I. Sommerville. Software Engineering. Pearson, 9 edition, 2009.
[30]
R. Stein and S. E. Brennan. Another person's eye gaze as a cue in solving programming problems. In Proceedings of the 6th international conference on Multimodal interfaces, pages 9--15, State College, USA, 2004.
[31]
Y. Tao, J. Kim, S. Kim, and C. Xu. Automatically generated patches as debugging aids: a human study. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 64--74, Hong Kong, China, 2014.
[32]
F. Thung, D. Lo, L. Jiang, et al. Are faults localizable? In Proceedings of the IEEE 9th Working Conference on Mining Software Repositories, pages 74--77, Zurich, Switzerland, 2012.
[33]
H. Uwano, M. Nakamura, A. Monden, and K.-i. Matsumoto. Analyzing individual performance of source code review using reviewers' eye movement. In Proceedings of the 2006 symposium on Eye tracking research & applications, pages 133--140, San Diego, USA, 2006.
[34]
Q. Wang, C. Parnin, and A. Orso. Evaluating the usefulness of ir-based fault localization techniques. In Proceedings of the 2015 International Symposium on Software Testing and Analysis, pages 1--11, Baltimore, Maryland, USA, 2015.
[35]
M. Weiser and J. Lyle. Experiments on slicing-based debugging aids. In Proceedings of the Workshop on Empirical Studies of Programmers, pages 187--197, USA, 1986.
[36]
W. E. Wong, V. Debroy, and B. Choi. A family of code coverage-based heuristics for effective fault localization. Journal of Systems and Software, 83(2):188--208, 2010.
[37]
X. Xie, T. Y. Chen, F.-C. Kuo, and B. Xu. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on Software Engineering and Methodology, 22(4):31:1--31:40, 2013.
[38]
J. Xuan and M. Monperrus. Learning to combine multiple ranking metrics for fault localization. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution, pages 191--200, Victoria, Canada, 2014.
[39]
J. Xuan and M. Monperrus. Test case purification for improving fault localization. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 52--63, Hong Kong, China, 2014.
[40]
X. Zhang, R. Gupta, and Y. Zhang. Precise dynamic slicing algorithms. In Proceedings of the 25th International Conference on Software Engineering, pages 319--329, Portland, Oregon, 2003.

Cited By

View all
  • (2024)SURE: A Visualized Failure Indexing Approach Using Program Memory SpectrumACM Transactions on Software Engineering and Methodology10.1145/367695833:8(1-43)Online publication date: 8-Jul-2024
  • (2024)Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code RepresentationProceedings of the ACM on Software Engineering10.1145/36607931:FSE(1937-1959)Online publication date: 12-Jul-2024
  • (2024)DAppFL: Just-in-Time Fault Localization for Decentralized Applications in Web3Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652116(137-148)Online publication date: 11-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '16: Proceedings of the 38th International Conference on Software Engineering
May 2016
1235 pages
ISBN:9781450339001
DOI:10.1145/2884781
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. attention tracking
  2. automated debugging
  3. fault comprehension
  4. navigation pattern
  5. spectrum-based fault localization
  6. user studies

Qualifiers

  • Research-article

Funding Sources

Conference

ICSE '16
Sponsor:

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)21
  • Downloads (Last 6 weeks)1
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)SURE: A Visualized Failure Indexing Approach Using Program Memory SpectrumACM Transactions on Software Engineering and Methodology10.1145/367695833:8(1-43)Online publication date: 8-Jul-2024
  • (2024)Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code RepresentationProceedings of the ACM on Software Engineering10.1145/36607931:FSE(1937-1959)Online publication date: 12-Jul-2024
  • (2024)DAppFL: Just-in-Time Fault Localization for Decentralized Applications in Web3Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652116(137-148)Online publication date: 11-Sep-2024
  • (2024)Automated Program Repair, What Is It Good For? Not Absolutely Nothing!Proceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639095(1-13)Online publication date: 20-May-2024
  • (2024)Deep learning-based software engineering: progress, challenges, and opportunitiesScience China Information Sciences10.1007/s11432-023-4127-568:1Online publication date: 24-Dec-2024
  • (2024)A Systematic Exploration of Mutation‐Based Fault Localization FormulaeSoftware Testing, Verification and Reliability10.1002/stvr.1905Online publication date: 11-Nov-2024
  • (2024)Richen: Automated enrichment of Git documentation with usage examples and scenariosJournal of Software: Evolution and Process10.1002/smr.2662Online publication date: 13-Mar-2024
  • (2024)Improving fault localization via weighted execution graph and graph attention networkJournal of Software: Evolution and Process10.1002/smr.261936:6Online publication date: 5-Jun-2024
  • (2023)Improving Spectrum-Based Localization of Multiple Faults by Iterative Test Suite ReductionProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598148(1445-1457)Online publication date: 12-Jul-2023
  • (2023)Responsibility in Context: On Applicability of Slicing in Semantic Regression Analysis2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00057(563-575)Online publication date: May-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media