skip to main content
10.1145/3632620.3671113acmconferencesArticle/Chapter ViewAbstractPublication PagesicerConference Proceedingsconference-collections
research-article

Overcoming Barriers in Scaling Computing Education Research Programming Tools: A Developer's Perspective

Published: 12 August 2024 Publication History

Abstract

Background and Context. Research software in the Computing Education Research (CER) domain frequently encounters issues with scalability and sustained adoption, which limits its educational impact. Despite the development of numerous CER programming (CER-P) tools designed to enhance learning and instruction, many fail to see widespread use or remain relevant over time. Previous research has primarily examined the challenges educators face in adopting and reusing CER tools, with few focusing on understanding the barriers to scaling and adoption practices from the tool developers’ perspective.
Objectives. To address this, we conducted semi-structured interviews with 16 tool developers within the computing education community, focusing on the challenges they encounter and the practices they employ in scaling their CER-P tools.
Method. Our study employs thematic analysis of the semi-structured interviews conducted with developers of CER-P tools.
Findings. Our analysis revealed several barriers to scaling highlighted by participants, including funding issues, maintenance burdens, and the challenge of ensuring tool interoperability for a broader user base. Despite these challenges, developers shared various practices and strategies that facilitated some degree of success in scaling their tools. These strategies include the development of teaching materials and units of curriculum, active marketing within the academic community, and the adoption of flexible design principles to facilitate easier adaptation and use by educators and students.
Implications. Our findings lay the foundation for further discussion on potential community action initiatives, such as the repository of CS tools and the community of tool developers, to allow educators to discover and integrate tools more easily in their classrooms and support tool developers by exchanging design practices to build high-quality education tools. Furthermore, our study suggests the potential benefits of exploring alternative funding models.

References

[1]
Lecia Barker, Christopher Hovey, and Jane Gruning. 2015. What Influences CS Faculty to Adopt Teaching Practices?Proceedings of the 46th ACM Technical Symposium on Computer Science Education (2015), 604–609. https://doi.org/10.1145/2676723.2677282
[2]
Michael Berry and Michael Kölling. 2014. The State of Play: A Notional Machine for Learning Programming. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education (Uppsala, Sweden) (ITiCSE ’14). Association for Computing Machinery, New York, NY, USA, 21–26. https://doi.org/10.1145/2591708.2591721
[3]
Jeremiah Blanchard, John R. Hott, Vincent Berry, Rebecca Carroll, Bob Edmison, Richard Glassey, Oscar Karnalim, Brian Plancher, and Seán Russell. 2022. Stop Reinventing the Wheel! Promoting Community Software in Computing Education. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 22 (dec 2022), 261–292. https://doi.org/10.1145/3571785.3574129
[4]
Neil CC Brown, Amjad Altadmri, Sue Sentance, and Michael Kölling. 2018. Blackbox, five years on: An evaluation of a large-scale programming data collection project. In Proceedings of the 2018 ACM Conference on International Computing Education Research. 196–204.
[5]
Peter Brusilovsky, Stephen Edwards, Amruth Kumar, Lauri Malmi, Luciana Benotti, Duane Buck, Petri Ihantola, Rikki Prince, Teemu Sirkiä, Sergey Sosnovsky, Jaime Urquiza, Arto Vihavainen, and Michael Wollowski. 2014. Increasing Adoption of Smart Learning Content for Computer Science Education. In Proceedings of the Working Group Reports of the 2014 on Innovation & Technology in Computer Science Education Conference (Uppsala, Sweden) (ITiCSE-WGR ’14). Association for Computing Machinery, New York, NY, USA, 31–57. https://doi.org/10.1145/2713609.2713611
[6]
Steve Carr and Jean Mayo. 2020. SecureCvisual: Visualization and Analysis for C Code Security. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education(SIGCSE ’20). ACM. https://doi.org/10.1145/3328778.3372540
[7]
Wanda Dann, Dennis Cosgrove, Don Slater, Dave Culyba, and Steve Cooper. 2012. Mediated Transfer: Alice 3 to Java. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (Raleigh, North Carolina, USA) (SIGCSE ’12). Association for Computing Machinery, New York, NY, USA, 141–146. https://doi.org/10.1145/2157136.2157180
[8]
Matthew Heinsen Egan and Chris McDonald. 2021. An evaluation of SeeC: a tool designed to assist novice C programmers with program understanding and debugging. Computer Science Education 31, 3 (2021), 340–373. https://doi.org/10.1080/08993408.2020.1777034
[9]
Eric Fouh, Ville Karavirta, Daniel A Breakiron, Sally Hamouda, Simin Hall, Thomas L Naps, and Clifford A Shaffer. 2014. Design and architecture of an interactive eTextbook–The OpenDSA system. Science of computer programming 88 (2014), 22–40.
[10]
Marcos J. Gomez, Marco Moresi, and Luciana Benotti. 2019. Text-based Programming in Elementary School: A Comparative Study of Programming Abilities in Children with and without Block-based Experience. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’19). ACM. https://doi.org/10.1145/3304221.3319734
[11]
Philip Guo. 2021. Ten Million Users and Ten Years Later: Python Tutor’s Design Guidelines for Building Scalable and Sustainable Research Software in Academia. UIST 2021 - Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (oct 2021), 1235–1251. https://doi.org/10.1145/3472749.3474819
[12]
Lassi Haaranen, Lukas Ahrenberg, and Arto Hellas. 2023. Decades of Striving for Pedagogical and Technological Alignment. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research (, Koli, Finland, ) (Koli Calling ’23). Association for Computing Machinery, New York, NY, USA, Article 23, 8 pages. https://doi.org/10.1145/3631802.3631809
[13]
David Hammer and Leema K Berland. 2014. Confusing claims for data: A critique of common practices for presenting qualitative research on learning. Journal of the Learning Sciences 23, 1 (2014), 37–46.
[14]
Christopher Lynnly Hovey, David P. Bunde, Zack Butler, and Cynthia Taylor. 2023. How Do i Get People to Use My Ideas? Lessons from Successful Innovators in CS Education. SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education 1 (mar 2023), 841–847. https://doi.org/10.1145/3545945.3569779
[15]
Ryosuke Ishizue, Kazunori Sakamoto, Hironori Washizaki, and Yoshiaki Fukazawa. 2018. PVC: Visualizing C Programs on Web Browsers for Novices. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (Baltimore, Maryland, USA) (SIGCSE ’18). Association for Computing Machinery, New York, NY, USA, 245–250. https://doi.org/10.1145/3159450.3159566
[16]
Masita Jalil, Shahrul Azman Noah, and Sufian Idris. 2010. Evaluating the Effectiveness of a Pattern Application Support Tool for Novices. In Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education (Bilkent, Ankara, Turkey) (ITiCSE ’10). Association for Computing Machinery, New York, NY, USA, 239–243. https://doi.org/10.1145/1822090.1822158
[17]
Mike Joy, Nathan Griffiths, and Russell Boyatt. 2005. The boss online submission and assessment system. Journal on Educational Resources in Computing (JERIC) 5, 3 (2005), 2–es.
[18]
Erkki Kaila, Teemu Rajala, Mikko-Jussi Laakso, and Tapio Salakoski. 2010. Effects of Course-Long Use of a Program Visualization Tool. In Proceedings of the Twelfth Australasian Conference on Computing Education - Volume 103 (Brisbane, Australia) (ACE ’10). Australian Computer Society, Inc., AUS, 97–106.
[19]
Oscar Karnalim and Simon. 2021. Explanation in Code Similarity Investigation. IEEE Access 9 (2021), 59935–59948. https://doi.org/10.1109/ACCESS.2021.3073703
[20]
Tobias Kohn and Bill Manaris. 2020. Tell Me What’s Wrong: A Python IDE with Error Messages. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE ’20). Association for Computing Machinery, New York, NY, USA, 1054–1060. https://doi.org/10.1145/3328778.3366920
[21]
Tobias Kohn and Bill Manaris. 2020. Tell Me What’s Wrong: A Python IDE with Error Messages. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 1054–1060.
[22]
Michael Kölling, Bruce Quig, Andrew Patterson, and John Rosenberg. 2003. The BlueJ system and its pedagogy. Computer Science Education 13, 4 (2003), 249–268.
[23]
Zachary Kurmas. 2023. DLUnit: A Unit Testing Framework for Simulated Digital Logic Circuits. In 2023 IEEE Frontiers in Education Conference (FIE). IEEE, 1–6.
[24]
Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny. 2023. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research(Koli Calling ’23). ACM. https://doi.org/10.1145/3631802.3631830
[25]
David Liu, Jonathan Calver, and Michelle Craig. 2024. A Static Analysis Tool in CS1: Student Usage and Perceptions of PythonTA. In Proceedings of the 26th Australasian Computing Education Conference. 172–181.
[26]
Nuno Macedo, Alcino Cunha, José Pereira, Renato Carvalho, Ricardo Silva, Ana C R Paiva, Miguel Sozinho Ramalho, and Daniel Silva. 2021. Experiences on teaching alloy with an automated assessment platform. Science of Computer Programming 211 (2021), 102690. https://doi.org/10.1016/j.scico.2021.102690
[27]
Nuno Macedo, Alcino Cunha, José Pereira, Renato Carvalho, Ricardo Silva, Ana C R Paiva, Miguel Sozinho Ramalho, and Daniel Silva. 2021. Experiences on teaching alloy with an automated assessment platform. Science of Computer Programming 211 (2021), 102690. https://doi.org/10.1016/j.scico.2021.102690
[28]
Jane Mills, Ann Bonner, and Karen Francis. 2006. The development of constructivist grounded theory. International journal of qualitative methods 5, 1 (2006), 25–35.
[29]
Jesús Moreno-León, Gregorio Robles, and Marcos Román-González. 2015. Dr. Scratch: Automatic analysis of scratch projects to assess and foster computational thinking. RED. Revista de Educación a Distancia46 (2015), 1–23.
[30]
Lijun Ni. 2009. What makes CS teachers change? factors influencing CS teachers’ adoption of curriculum innovations. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education (Chattanooga, TN, USA) (SIGCSE ’09). Association for Computing Machinery, New York, NY, USA, 544–548. https://doi.org/10.1145/1508865.1509051
[31]
Rohan Padhye, Koushik Sen, and Paul N. Hilfinger. 2019. ChocoPy: A Programming Language for Compilers Courses. In Proceedings of the 2019 ACM SIGPLAN Symposium on SPLASH-E (Athens, Greece) (SPLASH-E 2019). Association for Computing Machinery, New York, NY, USA, 41–45. https://doi.org/10.1145/3358711.3361627
[32]
James H. Paterson, John Haddow, and Ka Fai Cheng. 2008. PatternCoder: A Programming Support Tool for Learning Binary Class Associations and Design Patterns. In Proceedings of the 8th International Conference on Computing Education Research (Koli, Finland) (Koli ’08). Association for Computing Machinery, New York, NY, USA, 96–100. https://doi.org/10.1145/1595356.1595375
[33]
Lutz Prechelt and Guido Malpohl. 2003. Finding Plagiarisms among a Set of Programs with JPlag. Journal of Universal Computer Science 8 (2003).
[34]
Thomas Price, Rui Zhi, and Tiffany Barnes. 2017. Evaluation of a Data-Driven Feedback Algorithm for Open-Ended Programming.International Educational Data Mining Society (2017), 192–197.
[35]
Thomas W. Price, Neil C.C. Brown, Dragan Lipovac, Tiffany Barnes, and Michael Kölling. 2016. Evaluation of a Frame-based Programming Editor. In Proceedings of the 2016 ACM Conference on International Computing Education Research(ICER ’16). ACM, 33–42. https://doi.org/10.1145/2960310.2960319
[36]
Yizhou Qian and James D Lehman. 2019. Using Targeted Feedback to Address Common Student Misconceptions in Introductory Programming: A Data-Driven Approach. SAGE Open 9, 4 (2019), 2158244019885136. https://doi.org/10.1177/2158244019885136
[37]
Kelly Rivers. 2017. Automated data-driven hint generation for learning programming. Ph. D. Dissertation. Carnegie Mellon University.
[38]
Susan H. Rodger, Julian Genkins, Ian McMahon, and Peggy Li. 2013. Increasing the Experimentation of Theoretical Computer Science with New Features in JFLAP. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education (Canterbury, England, UK) (ITiCSE ’13). Association for Computing Machinery, New York, NY, USA, 351. https://doi.org/10.1145/2462476.2466521
[39]
Guido Rößling, Markus Schüler, and Bernd Freisleben. 2000. The ANIMAL Algorithm Animation Tool. SIGCSE Bull. 32, 3 (jul 2000), 37–40. https://doi.org/10.1145/353519.343069
[40]
Pálma Rozalia Osztián, Zoltán Kátai, and Erika Osztián. 2020. Algorithm Visualization Environments: Degree of interactivity as an influence on student-learning. In 2020 IEEE Frontiers in Education Conference (FIE). 1–8. https://doi.org/10.1109/FIE44824.2020.9273892
[41]
Seán Russell. 2022. Automated code tracing exercises for cs1. In Proceedings of the 6th Conference on Computing Education Practice. 13–16.
[42]
Clifford A Shaffer, Ville Karavirta, Ari Korhonen, and Thomas L Naps. 2011. OpenDSA: beginning a community active-ebook project. In Proceedings of the 11th Koli Calling International Conference on computing education research. 112–117.
[43]
SIGCSESpecialPrograms 2023. SIGCSE Special Programs. Retrieved August 17, 2023 from https://sigcse.org/programs/special/
[44]
John Stamper, Michael Eagle, Tiffany Barnes, and Marvin Croy. 2013. Experimental evaluation of automatic hint generation for a logic tutor. International Journal of Artificial Intelligence in Education 22, 1-2 (2013), 3–17.
[45]
Cynthia Taylor, Jaime Spacco, David P. Bunde, Zack Butler, Heather Bort, Christopher Lynnly Hovey, Francesco Maiorana, and Thomas Zeume. 2018. Propagating the adoption of CS educational innovations. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018 Companion). Association for Computing Machinery, New York, NY, USA, 217–235. https://doi.org/10.1145/3293881.3295785
[46]
Teacherspayteachers 2023. Teacherspayteachers. Retrieved August 17, 2023 from https://www.teacherspayteachers.com/
[47]
Artturi Tilanterä, Giacomo Mariani, Ari Korhonen, and Otto Seppälä. 2021. Towards a json-based algorithm animation language. In 2021 working conference on software visualization (vissoft). IEEE, 135–139.
[48]
Dwayne Towell and Brent Reeves. 2010. From Walls to Steps: Using online automatic homework checking tools to improve learning in introductory programming courses. (2010).
[49]
Kelsey Van Haaster and Dianne Hagan. 2004. Teaching and Learning with BlueJ: an Evaluation of a Pedagogical Tool.Issues in Informing Science & Information Technology 1 (2004).
[50]
Paul J. Wagner. 2020. The SQL File Evaluation (SQLFE) Tool: A Flexible and Extendible System for Evaluation of SQL Queries. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE ’20). Association for Computing Machinery, New York, NY, USA, 1334. https://doi.org/10.1145/3328778.3372599
[51]
Paul J Wagner. 2020. The sql file evaluation (sqlfe) tool: A flexible and extendible system for evaluation of sql queries. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 1334–1334.

Index Terms

  1. Overcoming Barriers in Scaling Computing Education Research Programming Tools: A Developer's Perspective

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICER '24: Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1
    August 2024
    539 pages
    ISBN:9798400704758
    DOI:10.1145/3632620
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 August 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    • Honorable Mention

    Author Tags

    1. computing education tools
    2. scaling research tools
    3. tool developer

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    Conference

    ICER 2024
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 189 of 803 submissions, 24%

    Upcoming Conference

    ICER 2025
    ACM Conference on International Computing Education Research
    August 3 - 6, 2025
    Charlottesville , VA , USA

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 161
      Total Downloads
    • Downloads (Last 12 months)161
    • Downloads (Last 6 weeks)15
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media