Skip to main content
SpringerLink
Log in

An Experimental Evaluation of Tools for Grading Concurrent Programming Exercises

  • Conference paper
  • First Online:
Formal Techniques for Distributed Objects, Components, and Systems (FORTE 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13910))

  • 151 Accesses

Abstract

Automatic grading based on unit tests is a key feature of massive open online courses (MOOC) on programming, as it allows instant feedback to students and enables courses to scale up. This technique works well for sequential programs, by checking outputs against a sample of inputs, but unfortunately it is not adequate for detecting races and deadlocks, which precludes its use for concurrent programming, a key subject in parallel and distributed computing courses. In this paper we provide a hands-on evaluation of verification and testing tools for concurrent programs, collecting a precise set of requirements, and describing to what extent they can or can not be used for this purpose. Our conclusion is that automatic grading of concurrent programming exercises remains an open challenge.

This work is financed by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia, within project LA/P/0063/2020.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/mj-ramos/FORTE2023.

References

  1. Codeboard. https://codeboard.io/. Accessed 30 Apr 2023

  2. Deadlock not being detected. https://groups.google.com/g/java-pathfinder/c/rzkaeuNDZCY. Accessed 04 Jan 2023

  3. Dl-Check. https://github.com/devexperts/dlcheck. Accessed 02 Jan 2023

  4. Google groups thread: java.lang.error: java.lang.nosuchfieldexception: tid. https://groups.google.com/g/java-pathfinder/c/t1n73xdyrFI. Accessed 29 April 2023

  5. JaDA. http://jada.cs.unibo.it/demo.html. Accessed 30 Apr 2023

  6. Java Pathfinder. https://github.com/javapathfinder/jpf-core

  7. Java Pathfinder (master branch). https://github.com/javapathfinder/jpf-core/tree/45a4450cd0bd1193df5419f7c9d9b89807d00db6. Accessed 04 Jan 2023

  8. JCarder. http://www.jcarder.org/download.html. Accessed 30 Apr 2023

  9. JUnit. https://junit.org/junit5/. Accessed 30 Apr 2023

  10. RV-Predict. https://runtimeverification.com/predict/. Accessed 30 Apr 2023

  11. ThreadSafe. http://www.contemplateltd.com/. Accessed 27 Dec 2022

  12. Visual Threads. http://www.unix.digital.com/visualthreads/index.html. Accessed 30 Apr 2023

  13. White box testing techniques, tools and advantages – a quick guide (2022). https://www.xenonstack.com/insights/what-is-white-box-testing. Accessed 03 Jan 2023

  14. Blackshear, S., Gorogiannis, N., O’Hearn, P., Sergey, I.: RacerD: compositional static race detection. In: Proceedings of the ACM Conference on Programming Languages 2(OOPSLA), pp. 1–28 (2018). https://doi.org/10.1145/3276514

  15. Cai, Y., Wu, S., Chan, W.: ConLock: a constraint-based approach to dynamic checking on deadlocks in multithreaded programs. In: Proceedings of the 36th International Conference on Software Engineering, pp. 491–502. ACM (2014). https://doi.org/10.1145/2568225.2568312

  16. Calcagno, C., et al.: Moving fast with software verification. In: Havelund, K., Holzmann, G., Joshi, R. (eds.) NFM 2015. LNCS, vol. 9058, pp. 3–11. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-17524-9_1

    Chapter  Google Scholar 

  17. Elmas, T., Qadeer, S., Tasiran, S.: Goldilocks: a race-aware Java runtime. Commun. ACM 53, 85–92 (2010). https://doi.org/10.1145/1839676.1839698

  18. Engler, D., Ashcraft, K.: RacerX: effective, static detection of race conditions and deadlocks. In: Proceedings of the19th ACM Symposium on Operating Systems Principles, pp. 237–252. ACM (2003). https://doi.org/10.1145/945445.945468

  19. Erickson, J., Musuvathi, M., Burckhardt, S., Olynyk, K.: Effective data-race detection for the kernel, pp. 151–162 (2010)

    Google Scholar 

  20. Eslamimehr, M., Palsberg, J.: Sherlock: scalable deadlock detection for concurrent programs. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 353–365. FSE 2014, Computing Machinery (2014). https://doi.org/10.1145/2635868.2635918

  21. Flanagan, C., Leino, K., Lillibridge, M., Nelson, G., Saxe, J., Stata, R.: Extended static checking for Java. In: Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation, pp. 234–245. ACM (2002). https://doi.org/10.1145/512529.512558

  22. Flanagan, C., Freund, S.N.: FastTrack: efficient and precise dynamic race detection. SIGPLAN Not. 44(6), 121–133 (2009). https://doi.org/10.1145/1543135.1542490

  23. Gao, J., Yang, X., Jiang, Y., Liu, H., Ying, W., Zhang, X.: Jbench: a dataset of data races for concurrency testing. In: Proceedinsg of the 15th IEEE/ACM 15th International Conference on Mining Software Repositories, pp. 6–9. ACM (2018). https://doi.org/10.1145/3196398.3196451

  24. Holt, R.: Some deadlock properties of computer systems. ACM Comput. Surv. 4(3), 179–196 (1972). https://doi.org/10.1145/356603.356607

    Article  MathSciNet  Google Scholar 

  25. Holzmann, G.: The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Professional, Boston (2011)

    Google Scholar 

  26. Huang, J., Meredith, P., Roşu, G.: Maximal sound predictive race detection with control flow abstraction. In: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 337–348. ACM (2014). https://doi.org/10.1145/2666356.2594315

  27. Huang, J., Zhang, C.: Persuasive prediction of concurrency access anomalies. In: Proceedings of the 2011 International Symposium on Software Testing and Analysis, pp. 144–154. ISSTA 2011. Association for Computing Machinery (2011). https://doi.org/10.1145/2001420.2001438

  28. Joshi, P., Naik, M., Park, C.-S., Sen, K.: CalFuzzer: an extensible active testing framework for concurrent programs. In: Bouajjani, A., Maler, O. (eds.) CAV 2009. LNCS, vol. 5643, pp. 675–681. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02658-4_54

    Chapter  Google Scholar 

  29. Lamport, L.: The PlusCal algorithm language. In: Leucker, M., Morgan, C. (eds.) ICTAC 2009. LNCS, vol. 5684, pp. 36–60. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03466-4_2

    Chapter  Google Scholar 

  30. Luo, Q., Zhang, S., Zhao, J., Hu, M.: A lightweight and portable approach to making concurrent failures reproducible. In: Rosenblum, D.S., Taentzer, G. (eds.) FASE 2010. LNCS, vol. 6013, pp. 323–337. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12029-9_23

    Chapter  Google Scholar 

  31. Marino, D., Musuvathi, M., Narayanasamy, S.: LiteRace: effective sampling for lightweight data-race detection. In: Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 134–143. PLDI 2009. Association for Computing Machinery (2009). https://doi.org/10.1145/1542476.1542491

  32. Melo, S., Souza, S., Silva, R., Souza, P.: Concurrent software testing in practice: a catalog of tools. In: Proceedings of the 6th International Workshop on Automating Test Case Design, Selection and Evaluation, pp. 31–40. ACM (2015). https://doi.org/10.1145/2804322.2804328

  33. Naik, M., Aiken, A., Whaley, J.: Effective static race detection for Java. In: Proceedings of the 27th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 308–319. ACM (2006). https://doi.org/10.1145/1133981.1134018

  34. Netzer, R., Miller, B.: What are race conditions? Some issues and formalizations. ACM Lett. Program. Lang. Syst. 1(1), 74–88 (1992). https://doi.org/10.1145/130616.130623

    Article  Google Scholar 

  35. Nir-Buchbinder, Y., Ur, S.: ConTest listeners: a concurrency-oriented infrastructure for Java test and heal tools. In: Pezzè, M. (ed.) Proceedings of the 4th International Workshop on Software Quality Assurance, SOQUA 2007, in conjunction with the 6th ESEC/FSE joint meeting, pp. 9–16. ACM (2007). https://doi.org/10.1145/1295074.1295077

  36. Pugh, W., Ayewah, N.: Unit testing concurrent software. In: Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering, pp. 513–516. ASE 2007. Association for Computing Machinery (2007). https://doi.org/10.1145/1321631.1321722

  37. Said, M., Wang, C., Yang, Z., Sakallah, K.: Generating data race witnesses by an SMT-based analysis, vol. 6617 (2011). https://doi.org/10.1007/978-3-642-20398-5_23

  38. Samak, M., Ramanathan, M.K.: Trace driven dynamic deadlock detection and reproduction. SIGPLAN Not. 49(8), 29–42 (2014). https://doi.org/10.1145/2692916.2555262

  39. Savage, S., Burrows, M., Nelson, G., Sobalvarro, P., Anderson, T.: Eraser: a dynamic data race detector for multithreaded programs. ACM Trans. Comput. Syst. 15(4), 391–411 (1997). https://doi.org/10.1145/265924.265927

    Article  Google Scholar 

  40. Sen, K., Agha, G.: A race-detection and flipping algorithm for automated testing of multi-threaded programs. In: Bin, E., Ziv, A., Ur, S. (eds.) HVC 2006. LNCS, vol. 4383, pp. 166–182. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-70889-6_13

    Chapter  Google Scholar 

  41. Şerbănuţă, T., Chen, F., Roşu, G.: Maximal causal models for sequentially consistent systems. In: Qadeer, S., Tasiran, S. (eds.) Proceedings of the 3rd International Conference on Runtime Verification. LNCS, vol. 7687, pp. 136–150. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35632-2_16

  42. Zhai, K., Xu, B., Chan, W., Tse, T.: CARISMA: a context-sensitive approach to race-condition sample-instance selection for multithreaded applications. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, pp. 221–231. ACM (2012). https://doi.org/10.1145/2338965.2336780

Download references

Author information

Authors and Affiliations

  1. INESC TEC & U. Minho, Braga, Portugal

    Manuel Barros, Maria Ramos, Alexandre Gomes, Alcino Cunha, José Pereira & Paulo Sérgio Almeida

Authors
  1. Manuel Barros
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Ramos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexandre Gomes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alcino Cunha
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. José Pereira
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Paulo Sérgio Almeida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manuel Barros .

Editor information

Editors and Affiliations

  1. University of Twente, Enschede, The Netherlands

    Marieke Huisman

  2. NOVA School of Science and Technology and NOVA LINCS, Caparica, Portugal

    António Ravara

Rights and permissions

Reprints and permissions

Copyright information

© 2023 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Barros, M., Ramos, M., Gomes, A., Cunha, A., Pereira, J., Almeida, P.S. (2023). An Experimental Evaluation of Tools for Grading Concurrent Programming Exercises. In: Huisman, M., Ravara, A. (eds) Formal Techniques for Distributed Objects, Components, and Systems. FORTE 2023. Lecture Notes in Computer Science, vol 13910. Springer, Cham. https://doi.org/10.1007/978-3-031-35355-0_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35355-0_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35354-3

  • Online ISBN: 978-3-031-35355-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation