Skip to main content
SpringerLink
Log in

Automatic Assessment via Intelligent Analysis of Students’ Program Output Patterns

  • Conference paper
  • First Online:
Blended Learning. Enhancing Learning Success (ICBL 2018)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10949))

Included in the following conference series:

Abstract

Automatic assessment of computer programming exercises offers a number of benefits to both learners and educators, including timely and customised feedback, as well as saving of human effort in grading. However, due to the high variety of programs submitted by students, exact matching between the expected output and different output variants is undesirable and how to do the matching properly is a challenging and practical problem. Existing approaches to address this problem adopt various inexact matching strategies, but typically they are unscalable, incapable of expressing a diversity of program outputs, or require high level of expertise. In this paper, we propose Hierarchical Program Output Structure (HiPOS), which provides higher expressiveness and flexibility, to model the program output. Based on HiPOS, we design different levels of matching rules in the matching rule hierarchy to determine the admissible program output variants in a flexible and scalable manner. We conducted experiments and compare our approach of automatic assessment to human judgement. The results show that our proposed method achieved an accuracy of 0.8467 in determining the admissible program output variants.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Academia Stack Exchange, Use of automated assessment of programming assignments. http://academia.stackexchange.com/questions/20578/use-of-automated-assessment-of-programming-assignments. Accessed 20 Dec 2015

  2. Ala-Mutka, K.: A survey of automated assessment approaches for programming assignments. Comput. Sci. Educ. 15(2), 83–102 (2005)

    Article  Google Scholar 

  3. Chen, D., Manning, C.D.: A fast and accurate dependency parser using neural networks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 740–750 (2014)

    Google Scholar 

  4. Curator: an Electronic Submission Management Environment, Computer Science @ Virginia Tech.. http://courses.cs.vt.edu/curator/. Accessed 20 Dec 2015

  5. Douce, C., Livingstone, D., Orwell, J.: Automatic test-based assessment of programming: a review. ACM J. Educ. Resour. Comput. 5(3) (2005). Article 4

    Article  Google Scholar 

  6. Fonte, D., da Cruz, D., Gançarski, A.L., Henriques, P.R.: A flexible dynamic system for automatic grading of programming exercises. In: Proceedings of the Symposium on Languages, Applications and Technologies (SLATE 2013), pp. 129–144 (2013)

    Google Scholar 

  7. Gerdes, A., Heeren, B., Jeuring, J., van Binsbergen, L.T.: Ask-Elle: an adaptable programming tutor for Haskell giving automated feedback. Int. J. Artif. Intell. Educ. 27(1), 65–100 (2017)

    Article  Google Scholar 

  8. Gómez-Albarrán, M.: The teaching and learning of programming: a survey of supporting software tools. Comput. J. 48(2), 130–144 (2005)

    Article  Google Scholar 

  9. Gulwani, S., Radiček, I., Zuleger, F.: Feedback generation for performance problems in introductory programming assignments. In: Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014), pp. 41–51 (2014)

    Google Scholar 

  10. Han, L., Kashyap, A., Finin, T., Mayfield, J., Weese, J.: UMBC EBIQUITY-CORE: semantic textual similarity systems. In: Proceedings of the Joint Conference on Lexical and Computational Semantics, pp. 44–52 (2013)

    Google Scholar 

  11. Higgins, C., Gray, G., Symeonidis, P., Tsintsifas, A.: Automated assessment and experiences of teaching programming. ACM J. Educ. Resour. Comput. 5(3) (2005). Article 5

    Article  Google Scholar 

  12. Ihantola, P., Ahoniemi, T., Karavirta, V., Seppälä, O.: Review of recent systems for automatic assessment of programming assignments. In: Proceedings of the Koli Calling International Conference on Computing Education Research (Koli Calling 2010), pp. 86–93 (2010)

    Google Scholar 

  13. Jackson, D.: Using software tools to automate the assessment of student programs. Comput. Educ. 17(2), 133–143 (1991)

    Article  Google Scholar 

  14. Jacobs, C., Wilson, M., Barker, C.: Using the latest developments in mobile technology to improve students’ engagement with assessment feedback. In: Proceedings of the 2015 IEEE Conference on Interactive Mobile Communication Technologies and Learning (IMCL), pp. 124–126 (2015)

    Google Scholar 

  15. Joy, M., Griffiths, N., Royatt, R.: The BOSS online submission and assessment system. ACM J. Educ. Resour. Comput. 5(3) (2005). Article 2

    Article  Google Scholar 

  16. Law, K.M.Y., Lee, V.C.S., Yu, Y.T.: Learning motivation in e-learning facilitated computer programming courses. Comput. Educ. 55(1), 218–228 (2010)

    Article  Google Scholar 

  17. Lewis, D.D., Jones, K.S.: Natural language processing for information retrieval. Commun. ACM 39(1), 92–101 (1996)

    Article  Google Scholar 

  18. Morris, D.S.: Automatic grading of student’s programming assignments: an interactive process and suite of programs. In: Proceedings of the ASEE/IEEE Frontiers in Education Conference (FIE 2003), pp. S3F-1–6 (2003)

    Google Scholar 

  19. The Penn Treebank Project. https://www.cis.upenn.edu/~treebank/. Accessed 20 Dec 2015

  20. Pieterse, V.: Automated assessment of programming assignments. In: Proceedings of the Computer Science Education Research Conference (CSERC 2013), pp. 45–56 (2013)

    Google Scholar 

  21. Poon, C.K., Wong, T.L., Yu, Y.T., Lee, V.C.S., Tang, C.M.: Toward more robust automatic analysis of student program outputs for assessment and learning. In: Proceedings of the 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC), pp. 780–785 (2016)

    Google Scholar 

  22. Tang, C.M., Yu, Y.T.: An exploratory study on instructors’ agreement on the correctness of computer program outputs. In: Cheung, Simon K.S., Fong, J., Fong, W., Wang, F.L., Kwok, L.F. (eds.) ICHL 2013. LNCS, vol. 8038, pp. 69–80. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39750-9_7

    Chapter  Google Scholar 

  23. Tang, C.M., Yu, Y.T.: An exploratory study on instructors’ agreement on the correctness of computer program outputs. In: Cheung, Simon K.S., Fong, J., Fong, W., Wang, F.L., Kwok, L.F. (eds.) ICHL 2013. LNCS, vol. 8038, pp. 69–80. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39750-9_7

    Chapter  Google Scholar 

  24. Tang, C.M., Yu, Y.T., Poon, C.K.: An approach towards automatic testing of student programs using token patterns. In: Proceedings of the International Conference on Computers in Education (ICCE 2009), pp. 188–190 (2009)

    Google Scholar 

  25. Tang, C.M., Yu, Y.T., Poon, C.K.: An experimental prototype for automatically testing student programs using token patterns. In: Proceedings of the International Conference on Computer Supported Education (CSEDU 2010), pp. 144–149 (2010)

    Google Scholar 

  26. UMBC Phrase Similarity Service. http://swoogle.umbc.edu/SimService/phrase_similarity.html. Accessed 20 Dec 2015

  27. Wong, T.L., Lam, W., Chen, B.: Mining employment market via text block detection and adaptive cross-domain information extraction. In: Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), pp. 283–290 (2009)

    Google Scholar 

  28. Yu, Y.T., Poon, C.K., Choy, M.: Experiences with PASS: developing and using a programming assignment assessment system. In: Proceedings of the International Conference on Quality Software (QSIC 2006), pp. 360–365 (2006)

    Google Scholar 

Download references

Acknowledgement

The work described in this paper is fully supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. UGC/FDS11/E02/15).

Author information

Authors and Affiliations

  1. Caritas Institute of Higher Education, Tseung Kwan O New Town, Hong Kong SAR

    Chung Keung Poon & Jacky Kin Lun Li

  2. Douglas College, New Westminster, BC, Canada

    Tak-Lam Wong

  3. City University of Hong Kong, Kowloon Tong, Hong Kong SAR

    Chung Man Tang, Yuen Tak Yu & Victor Chung Sing Lee

Authors
  1. Chung Keung Poon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tak-Lam Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chung Man Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jacky Kin Lun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuen Tak Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Victor Chung Sing Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tak-Lam Wong .

Editor information

Editors and Affiliations

  1. The Open University of Hong Kong, Hong Kong, China

    Simon K.S. Cheung

  2. City University of Hong Kong, Hong Kong, China

    Lam-for Kwok

  3. Kansai University, Osaka, Japan

    Kenichi Kubota

  4. The Open University of Hong Kong, Hong Kong, Hong Kong

    Lap-Kei Lee

  5. Kwansei Gakuin University, Osaka, Japan

    Jumpei Tokito

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Poon, C.K., Wong, TL., Tang, C.M., Li, J.K.L., Yu, Y.T., Lee, V.C.S. (2018). Automatic Assessment via Intelligent Analysis of Students’ Program Output Patterns. In: Cheung, S., Kwok, Lf., Kubota, K., Lee, LK., Tokito, J. (eds) Blended Learning. Enhancing Learning Success. ICBL 2018. Lecture Notes in Computer Science(), vol 10949. Springer, Cham. https://doi.org/10.1007/978-3-319-94505-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-94505-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-94504-0

  • Online ISBN: 978-3-319-94505-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation