Skip to main content
SpringerLink
Log in

Empirical studies on programming language stimuli

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Comprehending and debugging computer programs are inherently difficult tasks. The current approach to building program execution and debugging environments is to use exclusively visual stimuli on programming languages whose syntax and semantics has often been designed without empirical guidance. We present an alternative: Sodbeans, an open-source integrated development environment designed to output carefully chosen spoken auditory cues to supplement empirically evaluated visual stimuli. Originally designed for the blind, earlier work suggested that Sodbeans may benefit sighted programmers as well. We evaluate Sodbeans in two experiments. First, we report on a formal debugging experiment comparing (1) a visual debugger, (2) an auditory debugger, and (3) a multimedia debugger, which includes both visual and auditory stimuli. The results from this study indicate that while auditory debuggers on their own are significantly less effective for sighted users when compared with visual and multimedia debuggers, multimedia debuggers might benefit sighted programmers under certain circumstances. Specifically, we found that while multimedia debuggers do not provide instant usability, once programmers have some practice, their performance in answering comprehension questions improves. Second, we created and evaluated a pilot survey analyzing individual elements in a custom programming language (called HOP) to garner empirical metrics on their comprehensibility. Results showed that some of the most widely used syntax and semantics choices in commercial programming languages are extraordinarily unintuitive for novices. For example, at an aggregate level, the word for , as in a for loop, was rated reliably worse than repeat by more than 673% by novices. After completing our studies, we implemented the HOP programming language and integrated it into Sodbeans.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Begel, A., & Graham, S. (2004). Spoken language support for software development. In IEEE symposium on visual languages and human-centric computing (VL/HCC) (pp. 271–272). Los Alamitos, CA: IEEE Computer Society Press.

  • Binkley, D., Davis, M., Lawrie, D., & Morrell, C. (2009). To camel case or under_score. In Program Comprehension, 2009. ICPC ’09. IEEE 17th International Conference on (pp 158–167). doi:10.1109/ICPC.2009.5090039.

  • Boardman, D. B., Greene, G., Khandelwal, V., & Mathur, A. P. (1995). Listen: A tool to investigate the use of sound for the analysis of program behavior. In Computer software and applications conference, 1995. COMPSAC 95. Proceedings., nineteenth annual international, Dallas, TX (pp. 184–189). doi:10.1109/CMPSAC.1995.524778.

  • Bonar, J., & Soloway, E. (1983). Uncovering principles of novice programming. In POPL ’83: Proceedings of the 10th ACM SIGACT-SIGPLAN symposium on principles of programming languages (pp. 10–13). NY, USA: ACM Press. doi:10.1145/567067.567069.

  • Brown, A., Pettifer, S., & Stevens, R. (2004). Evaluation of a non-visual molecule browser. In Assets ’04: Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility (pp. 40–47). New York, NY: ACM Press. doi:10.1145/1028630.1028639.

  • Brown, M. H., & Hershberger, J. (1991). Colour and sound in algorithm animation. In Proceedings of the IEEE workshop on visual languages (pp. 10–17). Los Alamitos, CA: IEEE Computer Society Press. doi:10.1109/WVL.1991.238856.

  • Cooper, S., Dann, W., & Pausch, R. (2000). Alice: A 3-d tool for introductory programming concepts. In CCSC ’00: Proceedings of the fifth annual CCSC northeastern conference on The Journal of Computing in Small Colleges (pp. 107–116). USA: Consortium for Computing Sciences in Colleges.

  • Dawkins, R. (2006). The selfish gene 30th anniversary edition. New York:Oxford University Press Inc.

    Google Scholar 

  • Deißenböck, F., & Pizka, M. (2005). Concise and consistent naming. In IWPC ’05: Proceedings of the 13th international workshop on program comprehension (pp. 97–106). Washington, DC, USA: IEEE Computer Society. doi:10.1109/WPC.2005.14.

  • Dougherty, J. P. (2007). Concept visualization in cs0 using alice. The Journal of Computing in Small Colleges, 22(3), 145–152.

    Google Scholar 

  • Ellis, B., Stylos, J., & Myers, B. (2007). The factory pattern in api design: A usability evaluation. In International Conference on Software Engineering. ICSE ’07 (pp. 302–312). doi:10.1109/ICSE.2007.85.

  • Francioni, J. M., Jackson, J. A., & Albright, L. (1991). The sounds of parallel programs. In: The sixth distributed memory computing conference (pp. 570–577).

  • Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Boston, MA: Addison-Wesley.

    Google Scholar 

  • Gaver, W. W. (1986). Auditory icons: Using sound in computer interfaces. Human-Computer Interaction 2, 167–177.

    Article  Google Scholar 

  • Green, T. R. G., & Petre, M. (1996). Usability analysis of visual programming environments: A ‘cognitive dimensions’ framework. Journal of Visual Languages and Computing 7(2), 131–174. URL: citeseer.ist.psu.edu/article/green96usability.html.

    Google Scholar 

  • Gross, P., & Powers, K. (2005). Evaluating assessments of novice programming environments. In: ICER ’05: Proceedings of the 2005 international workshop on computing education research (pp. 99–110). NY, USA: ACM Press. doi:10.1145/1089786.1089796.

  • Guzdial, M., & Soloway, E. (2002). Teaching the nintendo generation to program. Communications of the ACM 45(4), 17–21. doi:10.1145/505248.505261.

  • Høst, E. W., Østvold, B. M. (2007). The programmer’s lexicon, volume I: The verbs. In SCAM ’07: Proceedings of the seventh IEEE international working conference on source code analysis and manipulation (pp. 193–202). IEEE Computer Society, Washington, DC, USA, doi:10.1109/SCAM.2007.31.

  • Hundhausen, C. D., & Brown, J. L. (2005). Personalizing and discussing algorithms within cs1 studio experiences: an observational study. In ICER ’05: Proceedings of the first international workshop on computing education research (pp. 45–56). NY, USA: ACM. doi:10.1145/1089786.1089791.

  • Hundhausen, C. D., Brown, J. L., & Farley, S. (2006). Adding procedures and pointers to the alvis algorithm visualization software: A preliminary design. In SoftVis ’06: Proceedings of the 2006 ACM symposium on Software visualization (pp. 155–156). New York, NY, USA: ACM Press. doi:10.1145/1148493.1148521.

  • Hundhausen, C. D., Farley, S. F., & Brown, J. L. (2009). Can direct manipulation lower the barriers to computer programming and promote transfer of training? An experimental study. ACM Transactions on Computer-Human Interaction 16(3), 1–40, doi:10.1145/1592440.1592442.

    Article  Google Scholar 

  • Kelleher, C., & Pausch, R. (2005). Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Computing Surveys 37(2), 83–137, doi:10.1145/1089733.1089734.

    Article  Google Scholar 

  • Lewis, B., & Ducassé, M. (2003). Using events to debug java programs backwards in time. In OOPSLA ’03: Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications (pp. 96–97). New York, NY: ACM Press. doi:10.1145/949344.949367.

  • Mayrhauser, A. v., & Vans, A. M. (1997). Program understanding behavior during debugging of large scale software. In ESP ’97: Papers presented at the seventh workshop on empirical studies of programmers (pp. 157–179). New York, NY: ACM Press. doi:10.1145/266399.266414.

  • McIver, L. (1996). Seven deadly sins of introductory programming language design. In Proceedings from the international conference on software engineering: education and practice (pp. 309–316). Los Alamitos, CA: IEEE Computer Society Press.

  • McIver, L. (2000). The effect of programming language on error rates of novice programmers. In The 12th annual workshop of psychology of programmers interest group (PPIG), Cosenza, Italy.

  • McIver, L. K. (2001). Syntactic and semantic issues in introductory programming education. PhD thesis, Monash University.

  • McTear, M. F. (2002). Spoken dialogue technology: Enabling the conversational user interface. ACM Computing Surveys 34(1), 90–169. doi:10.1145/505282.505285.

    Article  Google Scholar 

  • Mullins, P., Whitfield, D., & Conlon, M. (2009). Using alice 2.0 as a first language. The Journal of Computing in Small Colleges, 24(3), 136–143.

    Google Scholar 

  • Myers, B., & Pane, J. (1996). Usability issues in the design of novice programming systems. School of computer science technical report CMU-CS-96-132, Carnegie Mellon University.

  • Myers, B. A., Pane, J. F., & Ko, A. (2004). Natural programming languages and environments. Communications of the ACM, 47(9), 47–52.

    Article  Google Scholar 

  • Myers, B. A., Ko, A. J., Park, S. Y., Stylos, J., LaToza, T. D., & Beaton, J. (2008). More natural end-user software engineering. In WEUSE ’08: Proceedings of the 4th international workshop on end-user software engineering, ACM, New York, NY, USA (pp. 30–34). doi:10.1145/1370847.1370854.

  • Palladino, D. K., & Walker, B. N. (2007). Learning rates for auditory menus enhanced with spearcons versus earcons. In Proceedings of the 13th international conference on auditory display (pp. 274–279). Montréal, Canada.

  • Parente, P. (2004). Audio enriched links: Web page previews for blind users. SIGACCESS Accessibility and Computing, 77–78, 2–8. doi:10.1145/1029014.1028633.

    Article  Google Scholar 

  • Pausch, R. (2008). Alice: A dying man’s passion. In SIGCSE ’08: Proceedings of the 39th SIGCSE technical symposium on computer science education (p. 1). ACM, New York, NY, USA, doi:10.1145/1352135.1352137.

  • Pennington, N. (1987). Comprehension strategies in programming. In G. M. Olson, S. Sheppard, E. Soloway & B. Shneiderman (Eds.), Empirical Studies of Programmers: Second Workshop (pp. 100–113). Westport, CT: Greenwood Publishing Group Inc.

  • Pothier, G., Tanter, E., & Piquer, J. (2007). Scalable omniscient debugging. In OOPSLA ’07: Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications (pp. 535–552). New York, NY, USA: ACM. doi:10.1145/1297027.1297067.

  • Ritchie, D. M. (1996). The development of the c programming language (pp. 671–698). doi:10.1145/234286.1057834.

  • Sammet, J. E. (1981). The early history of cobol (pp. 199–243). doi:10.1145/800025.1198367.

  • Smith, A. C., Cook, J. S., Francioni, J. M., Hossain, A., Anwar, M., & Rahman, M. F. (2004). Nonvisual tool for navigating hierarchical structures. In The tenth international ACM SIGACCESS conference on computers and accessibility (pp. 133–139). New York, NY: ACM Press. doi:10.1145/1029014.1028654.

  • Soloway, E., Bonar, J., & Ehrlich, K. (1983). Cognitive strategies and looping constructs: An empirical study. Communications of the ACM, 26(11), 853–860. doi:10.1145/182.358436.

    Article  Google Scholar 

  • Steele, G. L., & Gabriel, R. P. (1996). The evolution of lisp (pp. 233–330). doi:10.1145/234286.1057818.

  • Stefik, A. (2008). On the design of program execution environments for non-sighted computer programmers. PhD thesis, Washington State University.

  • Stefik, A., Fitz, K., & Alexander, R. (2006). Increasing fault detection effectiveness using layered program auralization. In Software engineering research and practice (pp. 959–965). Las Vegas, NV: CSREA Press.

  • Stefik, A., Alexander, R., Patterson, R., & Brown, J. (2007). WAD: A feasibility study using the wicked audio debugger. In ICPC ’07: Proceedings of the 15th IEEE international conference on program comprehension (ICPC’07). Los Alamitos, CA: IEEE Computer Society Press.

  • Stevens, R. (1996). Principles for the design of auditory interfaces to present complex information to blind people. PhD thesis, The University of York.

  • Stroustrup, B. (1996). A history of c++: 1979–1991 (pp. 699–769). doi:10.1145/234286.1057836.

  • Tabbers, H. K., Martens, R. L., & Van Merriënboer, J. J. G. (2001). The modality effect in multimedia instructions. In: Annual conference of the Cognitive Science Society.

  • Vickers, P., & Alty, J. L. (2002). When bugs sing. Interacting with Computers, 14(6), 793–819.

    Article  Google Scholar 

  • Vickers, P., & Alty, J. L. (2005). Musical program auralization: Empirical studies. ACM Transactions on Applied Perception, 2(4), 477–489, doi:10.1145/1101530.1101546.

    Article  Google Scholar 

  • Walker, B. N., & Kramer, G. (2005). Mappings and metaphors in auditory displays: An experimental assessment. ACM Transactions on Applied Perception, 2(4), 407–412. doi:10.1145/1101530.1101534.

    Article  Google Scholar 

  • Walker, B. N., Nance, A., & Lindsay, J. (2006). Spearcons: Speech-based earcons improve navigation performance in auditory menus. In Proceedings of the 12th international conference on auditory display, London, UK.

  • Whitaker, W. A. (1996). Ada—the project: The dod high order language working group (pp. 173–232). doi:10.1145/234286.1057816.

Download references

Acknowledgments

We would like to thank Catherine Daus for her assistance with parts of the statistical analysis in Empirical Study 2. We would also like to thank Richard Most, one of the blind computer programmers we collaborate with, for his help in getting better screen reader compatibility on Mac OS X into Sodbeans. And, we thank Neelima Samsani, Andrew Hauck, and Aaron Willows for their help in implementing Sodbeans, both the HOP virtual machine (Neelima and Aaron), the NetBeans Platform code (Neelima), and the auditory libraries (Andrew). We would also like to thank Tim Boudreau (Oracle) and Tom Wheeler (A NetBeans Platform expert), who, through their extraordinary technical expertise with the NetBeans Platform, helped make the Sodbeans 1.0 release possible. Lastly, we wish to express gratitude to the National Science Foundation under awards (CNS-0940521) and (DUE-0536770) for their support of this work.

Author information

Authors and Affiliations

  1. Southern Illinois University Edwardsville, Edwardsville, IL, USA

    Andreas Stefik

  2. Central Washington University, Ellensburg, WA, USA

    Ed Gellenbeck

Authors
  1. Andreas Stefik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ed Gellenbeck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Stefik.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stefik, A., Gellenbeck, E. Empirical studies on programming language stimuli. Software Qual J 19, 65–99 (2011). https://doi.org/10.1007/s11219-010-9106-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-010-9106-7

Keywords

Search

Navigation