skip to main content
research-article

End-user debugging strategies: A sensemaking perspective

Published: 04 May 2012 Publication History

Abstract

Despite decades of research into how professional programmers debug, only recently has work emerged about how end-user programmers attempt to debug programs. Without this knowledge, we cannot build tools to adequately support their needs. This article reports the results of a detailed qualitative empirical study of end-user programmers' sensemaking about a spreadsheet's correctness. Using our study's data, we derived a sensemaking model for end-user debugging and categorized participants' activities and verbalizations according to this model, allowing us to investigate how participants went about debugging. Among the results are identification of the prevalence of information foraging during end-user debugging, two successful strategies for traversing the sensemaking model, potential ties to gender differences in the literature, sensemaking sequences leading to debugging progress, and sequences tied with troublesome points in the debugging process. The results also reveal new implications for the design of spreadsheet tools to support end-user programmers' sensemaking during debugging.

References

[1]
Abraham, R. and Erwig, M. 2007. A type system based on end-user vocabulary. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 215--222.
[2]
Ayalew Y. and Mittermeir R. 2003. Spreadsheet debugging. In Proceedings of the European Spreadsheet Risks Interest Group, Dublin, Ireland, July 24--25.
[3]
Bandura, A. 1986. Social Foundations of Thought and Action. Prentice Hall.
[4]
Bates, M. 1990. Where should the person stop and the information search interface start? Inf. Process. Manag. 26, 5, 575--591.
[5]
Beckwith, L., Burnett, M., Wiedenbeck, S., Cook, C., Sorte, S., and Hastings, M. 2005. Effectiveness of end-user debugging software features: Are there gender issues? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 869--878.
[6]
Beckwith, L., Kissinger, C., Burnett, M., Wiedenbeck, S., Lawrance, J., Blackwell, A., and Cook, C. 2006. Tinkering and gender in end-user programmers' debugging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 231--240.
[7]
Beckwith, L., Inman, D., Rector, K., and Burnett, M. 2007. On to the real world: Gender and self-efficacy in Excel. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 119--126.
[8]
Boehm, B. and Basili, V. 2001. Software defect reduction top 10 list. Comput. 34, 1, 135--137.
[9]
Burnett, M., Cook, C., Pendse, O., Rothermel, G., Summet, J., and Wallace, C. 2003. End-User software engineering with assertions in the spreadsheet paradigm. In Proceedings of the International Conference on Software Engineering. IEEE, 93--103.
[10]
Burnett M., Cook C., and Rothermel G. 2004. End-User software engineering. Comm. ACM 47, 9, 53--58.
[11]
Busch, T. 1995. Gender differences in self-efficacy and attitudes toward computers. J. Educ. Comput. Res. 12, 2, 147--158.
[12]
Butler, R. 2000. Is this spreadsheet a tax evader? How HM customs and excise test spreadsheet applications. In Proceedings of the 33rd Annual Hawaii International Conference on System Sciences.
[13]
Byrnes, J. P., Miller, D. C., and Schafer, W. D. 1999. Gender differences in risk taking: A meta-analysis. Psychol. Bull. 125, 367--383.
[14]
Dervin, B. 1984. A theoretic perspective and research approach for generating research helpful to communication practice. Public Relat. Res. Educ. 1, 1, 30--45.
[15]
Dervin, B., Foreman-Wernet, L., and Launterbach, E. Eds. 2003. Sense-Making Methodology Reader: Selected Writings of Brenda Dervin. Hampton Press, Cresskill, NJ, 215--231.
[16]
EUSPRIG 2009. Spreadsheet mistakes news stories, european spreadsheet risks interests group site. http://www.eusprig.org/stories.htm.
[17]
Fern, X., Komireddy, C., Grigoreanu, V., and Burnett, M. 2009. Mining problem-solving strategies from HCI data. ACM Trans. Comput.-Hum. Interact. To appear.
[18]
Finucane, M., Slovic, P., Merz, C-K., Flynn, J., and Satterfield, T. 2000. Gender, race and perceived risk: The white male effect. Health, Risk Soc. 2, 2, 159--172.
[19]
Fisher II, M. and Rothermel, G. 2005. The EUSES spreadsheet corpus: A shared resource for supporting experimentation with spreadsheet dependability mechanism. In Proceedings of the 1st Workshop on End-User Software Engineering. 47--51.
[20]
Furnas, G. and Russell, D. 2005. Making sense of sensemaking. Extended Abstracts on Human Factors in Computing Systems (ACM CHI'05). ACM, 2115--2116.
[21]
Gallagher, A., De Lisi, R., Holst, P., McGillicuddy-De Lisi, A., Morely, M., and Cahalan, C. 2000. Gender differences in advanced mathematical problem solving. J. Experim. Child Psychol. 75, 3, 165--190.
[22]
Grigoreanu, V., Beckwith, L., Fern, X., Yang, S., Komireddy, C., Narayanan, V., Cook, C., and Burnett, M. 2006. Gender differences in end-user debugging, revisited: What the miners found. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 19--26.
[23]
Grigoreanu, V., Cao, J., Kulesza, T., Bogart, C., Rector, K., Burnett, M., and Wiedenbeck, S. 2008. Can feature design reduce the gender gap in end-user software development environments? In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 149--156.
[24]
Grigoreanu, V., Brundage, J., Bahna, E., Burnett, M., ElRif, P., and Snover, J. 2009. Males' and females' script debugging strategies. In Proceedings of the 2nd International Symposium on End-User Development. Springer, 205--224.
[25]
Hartzel, K. 2003. How self-efficacy and gender issues affect software adoption and use. Comm. ACM 46, 9, 167--171.
[26]
Ioannidou, A., Repenning, A., and Webb, D. 2008. Using scalable game design to promote 3D fluency: Assessing the AgentCubes incremental 3D end-user development framework. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 47--54.
[27]
Jeffries, R. A. 1982. Comparison of debugging behavior of novice and expert programmers. AERA Annual Meeting, Department of Psychology, Carnegie Mellon University, Pittsburgh, PA.
[28]
Jin, J. and Dabbish, L. 2009. Self-Interruption on the computer: A typology of discretionary task interleaving. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1799--1808.
[29]
Kelleher, C. and Pausch, R. 2005. Lowering the barriers to programming: A survey of programming environments and languages for novice programmers. ACM Comput. Surv. 37, 2.
[30]
Kelleher, C., Pausch, R., and Kiesler, S. 2007. Storytelling Alice motivates middle school girls to learn computer programming. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1455--1464.
[31]
Kissinger, C., Burnett, M., Stumpf, S., Subrahmaniyan, N., Beckwith, L., Yang, S., and Rosson, M. 2006. Supporting end-user debugging: What do users want to know? In Proceedings of the Working Conference on Advanced Visual Interfaces, ACM, 135--142.
[32]
Klein, G., Moon, B., and Hoffman, R. R. 2006. Making sense of sensemaking 1: Alternative perspectives. IEEE Intell. Syst. 21, 4, 70--73.
[33]
Ko, A. and Myers, B. 2004. Designing the whyline: A debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 151--158.
[34]
Ko, A., Myers, B., and Aung, Htet Htet 2004. Six learning barriers in end-user programming systems. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 199--206.
[35]
Lawrance, J., Bellamy, R., Burnett, M., and Rector, K. 2008. Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1323--1332.
[36]
Leedom, D. K. 2001. Final report: Sensemaking symposium. http://www.dodccrp.org/events/2001_sensemaking_symposium/docs/FinalReport/Sensemaking_Final_Report.htm.
[37]
Littman, D. C., Pinto, J., Letovsky, S., and Soloway, E. 1986. Mental models and software maintenance. In Proceedings of the 1st Workshop on Empirical Studies of Programmers on Empirical Studies of Programmers. Ablex Publishing Corporation, 80--98.
[38]
Meyers-Levy, J. 1989. Gender differences in information processing: A selectivity interpretation. In Cognitive and Affective Responses to Advertising. R. Cafferata and A. Tybout, Eds., Lexington Books, MA.
[39]
Myers, B., Ko, A., and Burnett, M. 2006. Invited research overview: End-User programming. In Extended Abtracts of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 75--80.
[40]
Nanja, N. and Cook, C. 1987. An analysis of the on-line debugging process. In Proceedings of the 2nd Workshop on Empirical Studies of Programmers. Ablex Publishing Corporation, 172--184.
[41]
Nardi, B. A. 1993. A Small Matter of Programming: Perspectives on End-User Computing. MIT Press, MA.
[42]
O'Donnell, E. and Johnson, E. 2001. The effects of auditor gender and task complexity on information processing efficiency. Int. J. Audit. 5, 91--105.
[43]
Pane, J. and Myers, B. 1996, Usability issues in the design of novice programming systems, Tech. rep. CMU-CS-96-132, School of Computer Science, Carnegie Mellon University.
[44]
Panko, R. 1998. What we know about spreadsheet errors. J. End User Comput. 10, 2, 15--21.
[45]
Panko, R. and Orday, N. 2005. Sarbanes-Oxley: What about all the spreadsheets? In Proceedings of the European Spreadsheet Research Information Group.
[46]
Phalgune, A., Kissinger, C., Burnett, M., Cook, C., Beckwith, L., and Ruthruff, J. 2005. Garbage in, garbage out? An empirical look at oracle mistakes by end-user programmers. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 45--52.
[47]
Pirolli, P. and Card, S. 1999. Information foraging. Psychol. Rev. 106, 4, 643--675.
[48]
Pirolli, P. and Card, S. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of the International Conference on Intelligence Analysis.
[49]
Powell, M. and Ansic, D. 1997. Gender differences in risk behaviour in financial decision-making: An experimental analysis. J. Econom. Psychol. 18, 6, 605--628.
[50]
Prabhakararao, S., Cook, D., Ruthruff, J., Creswick, E., Main, M., Durham, M., and Burnett, M. 2003. Strategies and behaviors of end-user programmers with interactive fault localization. In Proceedings of Symposium on Human-Centric Computing Languages and Environments. IEEE, 15--22.
[51]
Rode, J. and Rosson, M. 2003. Programming at runtime: Requirements and paradigms for nonprogrammer web application development, In Proceedings of Symposium on Human-Centric Computing Languages and Environments. IEEE. 23--30.
[52]
Romero, P., du Boulay, B., Cox, R., Lutz, R., and Bryant, S. Debugging strategies and tactics in a multi-representation software environment. Int. J. Hum.- Comput. Studies 61, 992--1009.
[53]
Rosson, M., Sinha, H., Bhattacharya, M., and Zhao, D. 2007. Design planning in end-user web development. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 189--196.
[54]
Russell, D. M., Stefik, M. J., Pirolli, P., and Card, S. K. 1993. The cost structure of sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 269--276.
[55]
Scaffidi, C., Shaw, M., and Myers, B. 2005. Estimating the numbers of end users and end user programmers, In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 207--214.
[56]
Shrager, J. and Klahr, D. 1986. Instructionless learning about a complex device: The paradigm and observations. Int. J. Man-Mach. Studies 25, 153--189.
[57]
Stefik, M., Baldonado, M., Bobrow, D., Card, S., Everett, J., Lavendel, G., Marimont, D., Newman, P., Russell, D., and Smoliar, S. 2002. The knowledge sharing challenge: The sensemaking. White paper. http://www2.parc.com/istl/groups/hdi/papers/sensemaking-whitepaper.pdf.
[58]
Subrahmaniyan, N., Beckwith, L., Grigoreanu, V., Burnett, M., Wiedenbeck, S., Narayanan, V., Bucht, K., Drummond, R., and Fern, X. 2008. Testing vs. code inspection vs. what else? Male and female end users' debugging strategies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 617--626.
[59]
Torkzadeh, G. and Koufteros, X. 1994. Factorial validity of a computer self-efficacy scale and the impact of computer training. Educ. Psychol. Measur. 54, 3, 813--821.
[60]
Wagner, E. and Lieberman, H. 2004. Supporting user hypotheses in problem diagnosis on the web and elsewhere. In Proceedings of the International Conference on Intelligent User Interfaces, ACM, 30--37.
[61]
Wilcox, E., Atwood, J., Burnett, M., Cadiz, J., and Cook, C. 1997. Does continuous visual feedback aid debugging in direct-manipulation programming systems? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 258--265.

Cited By

View all
  • (2024)Reclaiming the Unexplored in Hybrid Visual ProgrammingProceedings of the 2024 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/3689492.3690045(13-25)Online publication date: 17-Oct-2024
  • (2024)"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting StudyProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663389(1-21)Online publication date: 25-Jun-2024
  • (2024)In-Line Compositional Visual ProgrammingCompanion Proceedings of the 8th International Conference on the Art, Science, and Engineering of Programming10.1145/3660829.3660841(73-79)Online publication date: 11-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction
ACM Transactions on Computer-Human Interaction  Volume 19, Issue 1
March 2012
205 pages
ISSN:1073-0516
EISSN:1557-7325
DOI:10.1145/2147783
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 May 2012
Accepted: 01 July 2011
Revised: 01 April 2011
Received: 01 July 2009
Published in TOCHI Volume 19, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. End-user programming
  2. debugging
  3. debugging strategies
  4. end-user software engineering
  5. gender HCI
  6. gender differences
  7. sensemaking
  8. spreadsheets

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)78
  • Downloads (Last 6 weeks)12
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Reclaiming the Unexplored in Hybrid Visual ProgrammingProceedings of the 2024 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/3689492.3690045(13-25)Online publication date: 17-Oct-2024
  • (2024)"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting StudyProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663389(1-21)Online publication date: 25-Jun-2024
  • (2024)In-Line Compositional Visual ProgrammingCompanion Proceedings of the 8th International Conference on the Art, Science, and Engineering of Programming10.1145/3660829.3660841(73-79)Online publication date: 11-Mar-2024
  • (2024)Understanding and Evaluating Developer Behaviour in Programming TasksProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments10.1145/3643796.3648450(35-39)Online publication date: 20-Apr-2024
  • (2024)Spreadsheet quality assurance: a literature reviewFrontiers of Computer Science10.1007/s11704-023-2384-618:2Online publication date: 22-Jan-2024
  • (2023)Multiple-Representation Visual Compositional Dataflow ProgrammingCompanion Proceedings of the 7th International Conference on the Art, Science, and Engineering of Programming10.1145/3594671.3594681(39-47)Online publication date: 13-Mar-2023
  • (2023)What Did My AI Learn? How Data Scientists Make Sense of Model BehaviorACM Transactions on Computer-Human Interaction10.1145/354292130:1(1-27)Online publication date: 7-Mar-2023
  • (2023)Octave: An End-User Programming Environment for Analysis of Spatiotemporal Data for Construction Students2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL-HCC57772.2023.00014(51-59)Online publication date: 3-Oct-2023
  • (2023)FxD: a functional debugger for dysfunctional spreadsheets2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL-HCC57772.2023.00012(31-40)Online publication date: 3-Oct-2023
  • (2023)A sensemaking analysis of API learning using ReactJournal of Computer Languages10.1016/j.cola.2022.10118974(101189)Online publication date: Jan-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media