Skip to main content
SpringerLink
Log in

A recommendation approach for programming online judges supported by data preprocessing techniques

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The use of programming online judges (POJ) to support students acquiring programming skills is common nowadays because this type of software contains a large collection of programming exercises to be solved by students. A POJ not only provides exercises but also automates the code compilation and its evaluation process. A common problem that students face when using POJ is information overload, as choosing the right problem to solve can be quite frustrating due to the large number of problems offered. The integration of current POJs into e-learning systems such as Intelligent Tutoring Systems (ITSs) is hard because of the lack of necessary information in ITSs. Hence, the aim of this paper is to support students with the information overload problem by using a collaborative filtering recommendation approach that filters out programming problems suitable for students’ programming skills. It uses an enriched user-problem matrix that implies a better student role representation, facilitating the computation of closer neighborhoods and hence a more accurate recommendation. Additionally a novel data preprocessing step that manages anomalous users’ behaviors that could affect the recommendation generation is also integrated in the recommendation process. A case study is carried out on a POJ real dataset showing that the proposal outperforms other previous approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Adomavicius G, Tuzhilin AT (2005) Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749

    Article  Google Scholar 

  2. Ala-Mutka KM (2005) A survey of automated assessment approaches for programming assignments. Comput Sci Educ 15(2):83–102

    Article  Google Scholar 

  3. Amatriain X, Pujol JM (2015) Data mining methods for recommender systems. In: Recommender Systems Handbook. Springer, pp 227–262

  4. Amatriain X, Pujol JM, Oliver N (2009a) I like it... i like it not: Evaluating user ratings noise in recommender systems. In: User modeling, adaptation, and personalization. Springer, pp 247–258

  5. Amatriain X, Pujol JM, Tintarev N (2009b) Rate it again: increasing recommendation accuracy by user re-rating. In: Proceedings of the third ACM conference on Recommender systems. ACM, pp 173–180

  6. Arefin AS (2006) Art of Programming Contest. Gyankosh Prokashonia

  7. Caiza J, Del Amo J (2013) Programming assignments automatic grading: Review of tools and implementations. In: Proceedings of INTED, vol 2013, pp 5691–5700

  8. Campos PG, Díez F, Cantador I (2014) Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Model User-Adap Inter 24(1-2):67–119

    Article  Google Scholar 

  9. Castro J, Yera Toledo R, Martínez L (2016) An empirical study of natural noise management in group recommendation systems. Decision Support Systems. doi:10.1016/j.dss.2016.09.020

  10. Chen LC, Kuo PJ, Liao IE (2015) Ontology-based library recommender system using mapreduce. Clust Comput 18(1):113–121

    Article  Google Scholar 

  11. Christensen I, Schiaffino S (2014) Social influence in group recommender systems. Online Inf Rev 38 (4):524–542

    Article  Google Scholar 

  12. Dascalu MI, Bodea CN, Mihailescu MN, Tanase EA, Ordoñez de Pablos P (2016) Educational recommender systems and their application in lifelong learning. Behav Inform Technol 35(4):290–297

    Article  Google Scholar 

  13. De Maio C, Fenza G, Gaeta M, Loia V, Orciuoli F, Senatore S (2012) Rss-based e-learning recommendations exploiting fuzzy fca for knowledge modeling. Appl Soft Comput 12(1):113–124

    Article  Google Scholar 

  14. De Oliveira MG, Ciarelli PM, Oliveira E (2013) Recommendation of programming activities by multi-label classification for a formative assessment of students. Expert Syst Appl 40(16):6641–6651

    Article  Google Scholar 

  15. Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1(1):3–18

    Article  Google Scholar 

  16. Drachsler H, Verbert K, Santos OC, Manouselis N (2015) Panorama of recommender systems to support learning. In: Recommender systems handbook. Springer, pp 421–451

  17. Fenza G, Orciuoli F (2016) Building pedagogical models by formal concept analysis. In: International Conference on Intelligent Tutoring Systems. Springer, pp 144–153

  18. Gaeta M, Orciuoli F, Paolozzi S, Salerno S (2011) Ontology extraction for knowledge reuse: The e-learning perspective. IEEE Trans Syst Man Cybern Part A Syst Humans 41(4):798–809

    Article  Google Scholar 

  19. Gruber TR (1995) Toward principles for the design of ontologies used for knowledge sharing? Int J Hum Comput Stud 43(5):907–928

    Article  Google Scholar 

  20. Guarino N, Giaretta P (1995) Ontologies and knowledge bases towards a terminological clarification. In: Towards very large knowledge bases: knowledge building & knowledge sharing. IOS Press, pp 25–32

  21. Gunawardana A, Shani G (2009) A Survey of Accuracy Evaluation Metrics of Recommendation Tasks. J Mach Learn Res 10:2935–2962

    MathSciNet  MATH  Google Scholar 

  22. Hsiao IH, Sosnovsky S, Brusilovsky P (2010) Guiding students to the right questions: adaptive navigation support in an e-learning system for java programming. J Comput Assist Learn 26(4):270–283

    Article  Google Scholar 

  23. Huang Z, Zeng D, Chen H (2007) A comparison of collaborative-filtering recommendation algorithms for e-commerce. IEEE Intell Syst 5:68–78

    Article  Google Scholar 

  24. Klaṡnja-Milićević A, Vesin B, Ivanović M, Budimac Z (2011) E-learning personalization based on hybrid recommendation strategy and learning style identification. Comput Educ 56(3):885–899

    Article  Google Scholar 

  25. Klaṡnja-Milićević A, Ivanović M, Nanopoulos A (2015) Recommender systems in e-learning environments: a survey of the state-of-the-art and possible extensions. Artif Intell Rev 44(4):571–604

    Article  Google Scholar 

  26. Krishnan S, Patel J, Franklin MJ, Goldberg K (2014) A methodology for learning, analyzing, and mitigating social influence bias in recommender systems. In: Proceedings of the 8th ACM Conference on Recommender systems. ACM, pp 137–144

  27. Kurnia A, Lim A, Cheang B (2001) Online judge. Comput Educ 36(4):299–315

    Article  Google Scholar 

  28. Leal JP, Silva F (2003) Mooshak: a web-based multi-site programming contest system. Software: Practice and Experience 33(6):567–581

    Google Scholar 

  29. Li B, Chen L, Zhu X, Zhang C (2013) Noisy but non-Malicious user detection in social recommender systems. World Wide Web 16(5-6):677–699

    Article  Google Scholar 

  30. Llana L, Martin-Martin E, Pareja-Flores C, Velázquez-Iturbide JA ́ (2014) Flop: a user-friendly system for automated program assessment. Journal of Universal Computer Science 20(9):1304–1326

    Google Scholar 

  31. Lops P, Gemmis M, Semeraro G (2011) Content-based recommender systems: State of the art and trends. In: Recommender systems handbook springer US, chap, vol 3, pp 73–105

  32. Lu J, Wu D, Mao M, Wang W, Zhang G (2015) Recommender system application developments: a survey. Decis Support Syst 74:12–32

    Article  Google Scholar 

  33. Martínez L, Pérez L G, Barranco M (2007) A multigranular linguistic content-based recommendation model. Int J Intell Syst 22(5):419–434

    Article  Google Scholar 

  34. Martínez L, Barranco MJ, Pérez LG, Espinilla M (2008) A knowledge based recommender system with multigranular linguistic information. International Journal of Computational Intelligence Systems 1(3):225–236

    Article  Google Scholar 

  35. Miranda S, Orciuoli F, Sampson DG (2016) A skos-based framework for subject ontologies to improve learning experiences. Comput Hum Behav 61:609–621

    Article  Google Scholar 

  36. Murray T (1999) Authoring intelligent tutoring systems: an analysis of the state of the art. Int J Artif Intell Educ 10:98–129

    Google Scholar 

  37. Nadolski RJ, Van den Berg B, Berlanga AJ, Drachsler H, Hummel HG, Koper R, Sloep PB (2009) Simulating light-weight personalised recommender systems in learning networks: a case for pedagogy-oriented and rating-based hybrid recommendation strategies. Journal of Artificial Societies and Social Simulation 12(1):4

  38. Ning X, Desrosiers C, Karypis G (2015) A comprehensive survey of neighborhood-based recommendation methods. In: Recommender Systems Handbook. Springer, pp 37–76

  39. Noguera J, Barranco M, Segura R, Martínez L (2012) A mobile 3d-gis hybrid recommender system for tourism. Inf Sci 215:37– 52

    Article  Google Scholar 

  40. O’Mahony MP, Hurley NJ, Silvestre G (2006) Detecting noise in recommender system databases. In: Proceedings of the 11th international conference on Intelligent user interfaces. ACM, pp 109–115

  41. Petit J, Giménez O, Roura S (2012) Jutge. org: an educational programming judge. In: Proceedings of the 43rd ACM technical symposium on Computer Science Education. ACM, pp 445– 450

  42. Pham HX, Jung JJ (2013) Preference-based user rating correction process for interactive recommendation systems. Multimedia tools and applications 65(1):119–132

    Article  Google Scholar 

  43. Piramuthu S, Kapoor G, Zhou W, Mauw S (2012) Input online review data and related bias in recommender systems. Decis Support Syst 53(3):418–424

    Article  Google Scholar 

  44. Polson MC, Richardson JJ (2013) Foundations of intelligent tutoring systems. Psychology Press

  45. Regueras LM, Verdú E, Muňoz MF, Pérez MA, De Castro JP, Verdú MJ (2009) Effects of competitive e-learning tools on higher education students: a case study. IEEE Trans Educ 52(2):279–285

    Article  Google Scholar 

  46. Ricci F (2015) Recommender systems handbook. Springer, Shapira B

    Book  MATH  Google Scholar 

  47. Ruiz-Iniesta A, Jimenez-Diaz G, Gomez-Albarran M (2014) A semantically enriched context-aware oer recommendation strategy and its application to a computer science oer repository. IEEE Trans Educ 57(4):255–260

    Article  Google Scholar 

  48. Said A, Jain BJ, Narr S, Plumbaum T (2012) Users and noise: The magic barrier of recommender systems. In: User modeling, Adaptation, and Personalization. Springer, pp 237–248

  49. Skiena SS (2006) Revilla MA. The programming contest training manual. Springer Science & Business Media, Programming challenges

    Google Scholar 

  50. Sleeman D, Brown JS (1982) Intelligent tutoring systems. Academic Press, London

    Google Scholar 

  51. Thai-Nghe N, Drumond L, Horváth T, Nanopoulos A, Schmidt-Thieme L (2011) Matrix and tensor factorization for predicting student performance. In: Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU), pp 69–78

  52. Verdú E, Regueras LM, Verdú MJ, Leal JP, de Castro JP, Queirós R (2012) A distributed system for learning programming on-line. Comput Educ 58(1):1–10

  53. Vesin B, Klaṡnja-Milićević A, Ivanović M, Budimac Z (2013) Applying recommender systems and adaptive hypermedia for e-learning personalization. Computing and Informatics 32(3):629–659

    Google Scholar 

  54. Wang GP, Chen SY, Yang X, Feng R (2016) Ojpot: online judge & practice oriented teaching idea in programming courses. Eur J Eng Educ 41(3):304–319

    Article  Google Scholar 

  55. Wang PY, Yang HC (2012) Using collaborative filtering to support college students’ use of online forum for english learning. Comput Educ 59(2):628–637

    Article  Google Scholar 

  56. Winoto P, Tang TY, McCalla GI (2012) Contexts in a paper recommendation system with collaborative filtering. The International Review of Research in Open and Distributed Learning 13(5):56–75

    Article  Google Scholar 

  57. Yera Toledo R, Caballero Mota Y (2014) An e-learning collaborative filtering approach to suggest problems to solve in programming online judges. International Journal of Distance Education Technologies 12(2):51–65

    Article  Google Scholar 

  58. Yera Toledo R, Caballero Mota Y, Garcia Borroto M (2013) A regularity-based preprocessing method for collaborative recommender systems. J Inf Process Syst 9(3):435– 460

    Article  Google Scholar 

  59. Yera Toledo R, Caballero Mota Y, Martínez L (2015) Correcting noisy ratings in collaborative recommender systems. Knowl-Based Syst 76:96–108

    Article  Google Scholar 

  60. Yera Toledo R, Castro J, Martínez L (2016) A fuzzy model for managing natural noise in recommender systems. Appl Soft Comput 40:187–198

    Article  Google Scholar 

  61. Yu R, Cai Z, Du X, He M, Wang Z, Yang B, Chang P (2015) The research of the recommendation algorithm in online learning. International Journal of Multimedia and Ubiquitous Engineering 10(4):71–80

    Article  Google Scholar 

Download references

Acknowledgments

This research work was partially supported by the Research Project TIN2015-66524-P.

Author information

Authors and Affiliations

  1. University of Ciego de Ávila, Ciego de Ávila, Cuba

    Raciel Yera

  2. Computer Science Department, University of Jaén, Jaén, Spain

    Luis Martínez

Authors
  1. Raciel Yera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luis Martínez.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yera, R., Martínez, L. A recommendation approach for programming online judges supported by data preprocessing techniques. Appl Intell 47, 277–290 (2017). https://doi.org/10.1007/s10489-016-0892-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-016-0892-x

Keywords

Search

Navigation