skip to main content
10.1145/3286606.3286834acmotherconferencesArticle/Chapter ViewAbstractPublication PagessmartcityappConference Proceedingsconference-collections
research-article

Enhancing MOOCs peer reviews validity and reliability by a fuzzy coherence measure

Published: 10 October 2018 Publication History

Abstract

Massive Open Online Course (MOOC) are gaining more and more popularity permitting a large access for education. In doing so, they are confronted to new challenges, such as assessing a huge number of participants. A common idea lies on peer reviewing. However, students are not formed to assess others contributions. Hence, their evaluation are unreliable and may be biased. Since peer reviewing imposes itself in the actual MOOCs context, several approaches were considered to improve some aspects of its outcome. Here we proposed a fuzzy based approach that aims enhancing validity and reliability.

References

[1]
García-Peñalvo, F. J., Fidalgo-Blanco, Á., and Sein-Echaluce, M. L., 2018, "An Adaptive Hybrid MOOC Model: Disrupting the MOOC Concept in Higher Education," Telemat. Inform., 35(4), 1018--1030.
[2]
Haber, J., 2014, MOOCs, The MIT Press, Cambridge, Massachusetts.
[3]
Ashton, S., and Davies, R. S., 2015, "Using Scaffolded Rubrics to Improve Peer Assessment in a MOOC Writing Course," Distance Educ., 36(3), 312--334.
[4]
Jemni, M., Kinshuk, and Khribi, M. K., eds., 2017, Open Education: From OERs to MOOCs, Springer-Verlag, Berlin Heidelberg.
[5]
Pomerol, J.-C., Epelboin, Y., and Thoury, C., 2015, MOOCs: Design, Use and Business Models, ISTE Ltd., London, UK?; Hoboken, NJ.
[6]
Association, I. R. M., 2014, Open Source Technology: Concepts, Methodologies, Tools, and Applications, IGI Global, Hershey, PA.
[7]
Wellen, R., 2013, "Open Access, Megajournals, and MOOCs: On the Political Economy of Academic Unbundling," SAGE Open, 3(4), 1--16.
[8]
Despujol, I. M., Turró, C., Castañeda, L., and Busquets, J., 2017, "Effect of Free Certificate Discontinuation in Completion Rates of MOOC," Digital Education: Out to the World and Back to the Campus, Springer, Cham, 182--187.
[9]
Alario-Hoyos, C., Pérez-Sanagustín, M., Delgado-Kloos, C., G, H. A. P., Muñoz-Organero, M., and Rodríguez-de-las-Heras, A., 2013, "Analysing the Impact of Built-In and External Social Tools in a MOOC on Educational Technologies," Scaling up Learning for Sustained Impact, Springer, Berlin, Heidelberg, 5--18.
[10]
Formanek, M., Wenger, M. C., Buxner, S. R., Impey, C. D., and Sonam, T., 2017, "Insights about Large-Scale Online Peer Assessment from an Analysis of an Astronomy MOOC," Comput. Educ., 113, 243--262.
[11]
Wilson, M. J., Diao, M. M., and Huang, L., 2015, "'I'm Not Here to Learn How to Mark Someone Else's Stuff': An Investigation of an Online Peer-to-Peer Review Workshop Tool," Assess. Eval. High. Educ., 40(1), 15--32.
[12]
Usher, M., and Barak, M., 2018, "Peer Assessment in a Project-Based Engineering Course: Comparing between on-Campus and Online Learning Environments," Assess. Eval. High. Educ., 43(5), 745--759.
[13]
Bordel, B., Alcarria, R., Martín, D., and Sánchez-de-Rivera, D., 2017, "Improving MOOC Student Learning Through Enhanced Peer-to-Peer Tasks," Digital Education: Out to the World and Back to the Campus, Springer, Cham, 140--149.
[14]
Mulder, R., Baik, C., Naylor, R., and Pearce, J., 2014, "How Does Student Peer Review Influence Perceptions, Engagement and Academic Outcomes? A Case Study," Assess. Eval. High. Educ., 39(6), 657--677.
[15]
Meek, S. E. M., Blakemore, L., and Marks, L., 2017, "Is Peer Review an Appropriate Form of Assessment in a MOOC? Student Participation and Performance in Formative Peer Review," Assess. Eval. High. Educ., 42(6), 1000--1013.
[16]
Suen, H. K., 2014, "Peer Assessment for Massive Open Online Courses (MOOCs)," Int. Rev. Res. Open Distrib. Learn., 15(3), 312--327.
[17]
James, S., Pan, L., Wilkin, T., and Yin, L., 2017, "Online Peer Marking with Aggregation Functions," 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), 1--6.
[18]
Rhoads, R. A., 2015, MOOCs, High Technology, and Higher Learning, Johns Hopkins University Press.
[19]
Zheng, Q., Chen, L., and Burgos, D., 2018, "The International Comparison and Trend Analysis of the Development of MOOCs in Higher Education," The Development of MOOCs in China, Springer, Singapore, 1--9.
[20]
Ulrich, C., and Nedelcu, A., 2015, "MOOCs in Our University: Hopes and Worries," Procedia - Soc. Behav. Sci., 180, 1541--1547.
[21]
Pfeiffer, L., 2015, MOOC, COOC - La formation professionnelle à l'ère du digital, Dunod, Paris.
[22]
Al-Atabi, M., and DeBoer, J., 2014, "Teaching Entrepreneurship Using Massive Open Online Course (MOOC)," Technovation, 34(4), 261--264.
[23]
Jemni, M., and Khribi, M. K., 2017, "Toward Empowering Open and Online Education in the Arab World Through OER and MOOCs," Open Education: From OERs to MOOCs, Springer, Berlin, Heidelberg, 73--100.
[24]
Ortega-Arranz, A., Sanz-Martínez, L., Álvarez-Álvarez, S., Muñoz-Cristóbal, J. A., Bote-Lorenzo, M. L., Martínez-Monés, A., and Dimitriadis, Y., 2017, "From Low-Scale to Collaborative, Gamified and Massive-Scale Courses: Redesigning a MOOC," Digital Education: Out to the World and Back to the Campus, Springer, Cham, 77--87.
[25]
Reischer, M., Khalil, M., and Ebner, M., 2017, "Does Gamification in MOOC Discussion Forums Work?," Digital Education: Out to the World and Back to the Campus, Springer, Cham, 95--101.
[26]
Spector, J. M., 2017, "A Critical Look at MOOCs," Open Education: From OERs to MOOCs, Springer, Berlin, Heidelberg, 135--147.
[27]
Shapiro, H. B., Lee, C. H., Wyman Roth, N. E., Li, K., Çetinkaya-Rundel, M., and Canelas, D. A., 2017, "Understanding the Massive Open Online Course (MOOC) Student Experience: An Examination of Attitudes, Motivations, and Barriers," Comput. Educ., 110, 35--50.
[28]
Jung, Y., and Lee, J., 2018, "Learning Engagement and Persistence in Massive Open Online Courses (MOOCS)," Comput. Educ., 122, 9--22.
[29]
Dhorne, L., Deflandre, J.-P., Bernaert, O., Bianchi, S., and Thirouard, M., 2017, "Mentoring Learners in MOOCs: A New Way to Improve Completion Rates?," Digital Education: Out to the World and Back to the Campus, Springer, Cham, 29--37.
[30]
Deng, R., Benckendorff, P., and Gannaway, D., 2017, "Understanding Learning and Teaching in MOOCs from the Perspectives of Students and Instructors: A Review of Literature from 2014 to 2016," Digital Education: Out to the World and Back to the Campus, Springer, Cham, 176--181.
[31]
Zhao, Z., Wu, Q., Chen, H., and Wan, C., 2016, "Learning Quality Evaluation of MOOC Based on Big Data Analysis," Smart Computing and Communication, Springer, Cham, 277--286.
[32]
Dyomin, V., Mozhaeva, G., Babanskaya, O., and Zakharova, U., 2017, "MOOC Quality Evaluation System: Tomsk State University Experience," Digital Education: Out to the World and Back to the Campus, Springer, Cham, 197--202.
[33]
Betanzos, M., Costa-jussà, M. R., and Belanche, L., 2017, "Tradares: A Tool for the Automatic Evaluation of Human Translation Quality within a MOOC Environment," Appl. Artif. Intell., 31(3), 288--297.
[34]
Marshall, S. J., 2018, "Open Education: A Parable of Change in Higher Education," Shaping the University of the Future, Springer, Singapore, 213--251.
[35]
Boyatt, R., Joy, M., Rocks, C., and Sinclair, J., 2014, "What (Use) Is a MOOC?," The 2nd International Workshop on Learning Technology for Education in Cloud, Springer, Dordrecht, 133--145.
[36]
Pollack Ichou, R., 2018, "Can MOOCs Reduce Global Inequality in Education?," Australas. Mark. J. AMJ.
[37]
Steels, L., 2015, Music Learning with Massive Open Online Courses, IOS Press, Amsterdam?; Washington, DC.
[38]
Feldmann, B., 2014, "Two Decades of E-Learning in Distance Teaching - From Web 1.0 to Web 2.0 at the University of Hagen," Learning Technology for Education in Cloud. MOOC and Big Data, Springer, Cham, 163--172.
[39]
Trudy, H. B., Ronald, B. H., James, C. P., 2014, Budget and Finance in the American Community College: New Directions for Community Colleges, Number 168, Jossey-Bass.
[40]
Gaston, P. L., and Ochoa, E. M., 2013, Higher Education Accreditation: How It's Changing, Why It Must, Stylus Publishing, Sterling, Virginia.
[41]
Kearney, E. M., 2013, "Assessing Learning," On Becoming a Teacher, SensePublishers, Rotterdam, 85--89.
[42]
Sale, D., 2014, "Assessing Learning," The Challenge of Reframing Engineering Education, Springer, Singapore, 59--80.
[43]
Ettarres, Y., 2017, "Evaluation of Online Assignments and Quizzes Using Bayesian Networks," Innovations in Smart Learning, Springer, Singapore, 39--44.
[44]
Govindarajan, K., Boulanger, D., Seanosky, J., Bell, J., Pinnell, C., Kumar, V. S., and Kinshuk, 2017, "Assessing Learners' Progress in a Smart Learning Environment Using Bio-Inspired Clustering Mechanism," Innovations in Smart Learning, Springer, Singapore, 49--58.
[45]
Zhu, M., Sari, A., and Lee, M. M., 2018, "A Systematic Review of Research Methods and Topics of the Empirical MOOC Literature (2014-2016)," Internet High. Educ., 37, 31--39.
[46]
Staubitz, T., Petrick, D., Bauer, M., Renz, J., and Meinel, C., 2016, "Improving the Peer Assessment Experience on MOOC Platforms," Proceedings of the Third (2016) ACM Conference on Learning @ Scale, ACM, New York, NY, USA, 389--398.
[47]
Rust, C., 2001, "A Briefing on Assessment of Large Groups," LTSN Generic Centre: Assessment Series.
[48]
Purcell, M. E., and Hawtin, M., "Piloting External Peer Review as a Model for Performance Improvement in Third-Sector Organizations," Nonprofit Manag. Leadersh., 20(3), 357--374.
[49]
Wu, J., 2018, "Empirical Analysis of Evaluation of English Teachers' Educational Ability under MOOC Environment," 2018 International Conference on Intelligent Transportation, Big Data Smart City (ICITBS), 303--306.
[50]
Yin, Z., 2018, "Educational Ability Evaluation of Japanese Language Teacher under MOOC Environment," 2018 International Conference on Intelligent Transportation, Big Data Smart City (ICITBS), 299--302.
[51]
Koç, E. S., 2015, "An Evaluation of the Effectiveness of Committees of Teachers According to the Teachers' Views, Ankara Province Sample," Procedia - Soc. Behav. Sci., 174, 3--9.
[52]
Song, Y., Hu, Z., and Gehringer, E. F., 2017, "Collusion in Educational Peer Assessment: How Much Do We Need to Worry about It?," 2017 IEEE Frontiers in Education Conference (FIE), 1--8.
[53]
Luo, H., Robinson, A. C., and Park, J.-Y., 2014, "Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects," J. Asynchronous Learn. Netw., 18(2), 1--14.
[54]
Derrick, G., 2018, The Evaluators' Eye: Impact Assessment and Academic Peer Review, Palgrave Macmillan.
[55]
Zheng, Q., Chen, L., and Burgos, D., 2018, "Emergence and Development of MOOCs," The Development of MOOCs in China, Springer, Singapore, 11--24.
[56]
Waks, L. J., 2016, The Evolution and Evaluation of Massive Open Online Courses: MOOCs in Motion, Palgrave Macmillan US.
[57]
Jackson, M., and Marks, L., 2016, "Improving the Effectiveness of Feedback by Use of Assessed Reflections and Withholding of Grades," Assess. Eval. High. Educ., 41(4), 532--547.
[58]
Gamage, D., Whiting, M., Rajapakshe, T., Thilakarathne, H., Perera, I., and Fernando, S., 2017, "Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives," ArXiv170306169 Cs, 315--318.
[59]
Adachi, C., Tai, J. H.-M., and Dawson, P., 2018, "Academics' Perceptions of the Benefits and Challenges of Self and Peer Assessment in Higher Education," Assess. Eval. High. Educ., 43(2), 294--306.
[60]
Alias, M., Masek, A., and Salleh, H. H. M., 2015, "Self, Peer and Teacher Assessments in Problem Based Learning: Are They in Agreements?," Procedia - Soc. Behav. Sci., 204, 309--317.
[61]
Cho, K., Schunn, C. D., and Wilson, R. W., 2006, "Validity and Reliability of Scaffolded Peer Assessment of Writing from Instructor and Student Perspectives," J. Educ. Psychol., 98(4), 891--901.
[62]
Wahid, U., Chatti, M. A., and Schroeder, U., 2016, "A Systematic Analysis of Peer Assessment in the MOOC Era and Future Perspectives," 64--69.
[63]
Zadeh, L. A., 1965, "Fuzzy Sets," Inf. Control, 8(3), 338--353.
[64]
Bellman, R. E., and Zadeh, L. A., 1970, "Decision-Making in a Fuzzy Environment," Manag. Sci., 17(4), B141-B164.
[65]
Capuano, N., Loia, V., and Orciuoli, F., 2017, "A Fuzzy Group Decision Making Model for Ordinal Peer Assessment," IEEE Trans. Learn. Technol., 10(2), 247--259.
[66]
Lubis, F. F., Rosmansyah, Y., and Supangkat, S. H., 2016, "Experience in Learners Review to Determine Attribute Relation for Course Completion," 2016 International Conference on ICT For Smart Society (ICISS), 32--36.
[67]
Ospina-Delgado, J., and Zorio-Grima, A., 2016, "Innovation at Universities: A Fuzzy-Set Approach for MOOC-Intensiveness," J. Bus. Res., 69(4), 1325--1328.
[68]
El Alaoui, M., 2018, "SMART Grid Evaluation Using Fuzzy Numbers and TOPSIS," IOP Conf. Ser. Mater. Sci. Eng., 353(1), 12--19.
[69]
El Alaoui, M., Ben-Azza, H., and Zahi, A., 2016, "New Multi-Criteria Decision-Making Based on Fuzzy Similarity, Distance and Ranking," Proceedings of the Third International Afro-European Conference for Industrial Advancement - AECIA 2016, Springer, Cham, 138--148.
[70]
Lee, H.-S., 2002, "Optimal Consensus of Fuzzy Opinions under Group Decision Making Environment," Fuzzy Sets Syst., 132(3), 303--315.
[71]
Chen, C.-T., Lin, C.-T., and Huang, S.-F., 2006, "A Fuzzy Approach for Supplier Evaluation and Selection in Supply Chain Management," Int. J. Prod. Econ., 102(2), 289--301.
[72]
Skalna, I., Rębiasz, B., Gaweł, B., Basiura, B., Duda, J., Opiła, J., and Pełech-Pilichowsk, T., 2015, Advances in Fuzzy Decision Making - Theory and Practice, Springer International Publishing.
[73]
Chai, K. C., Tay, K. M., and Lim, C. P., 2015, "A New Fuzzy Peer Assessment Methodology for Cooperative Learning of Students," Appl. Soft Comput., 32, 468--480.

Cited By

View all
  • (2023)Improving Peer Assessment Validity and Reliability Through a Fuzzy Coherence MeasureIEEE Transactions on Learning Technologies10.1109/TLT.2023.332166016:6(892-899)Online publication date: 1-Dec-2023
  • (2022)Sustainable Consensus in an Uncertain EnvironmentEarth Systems Protection and Sustainability10.1007/978-3-030-85829-2_11(273-290)Online publication date: 31-Jan-2022
  • (2020)Fuzzy Similarity Relations in Decision MakingHandbook of Research on Emerging Applications of Fuzzy Algebraic Structures10.4018/978-1-7998-0190-0.ch020(369-385)Online publication date: 2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SCA '18: Proceedings of the 3rd International Conference on Smart City Applications
October 2018
580 pages
ISBN:9781450365628
DOI:10.1145/3286606
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Fuzzy logic
  2. Group Decision Making
  3. Massive Open Online Course
  4. Peer Reviewing
  5. Reliability
  6. Validity
  7. Weighting Opinions

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SCA '18

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 14 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Improving Peer Assessment Validity and Reliability Through a Fuzzy Coherence MeasureIEEE Transactions on Learning Technologies10.1109/TLT.2023.332166016:6(892-899)Online publication date: 1-Dec-2023
  • (2022)Sustainable Consensus in an Uncertain EnvironmentEarth Systems Protection and Sustainability10.1007/978-3-030-85829-2_11(273-290)Online publication date: 31-Jan-2022
  • (2020)Fuzzy Similarity Relations in Decision MakingHandbook of Research on Emerging Applications of Fuzzy Algebraic Structures10.4018/978-1-7998-0190-0.ch020(369-385)Online publication date: 2020
  • (2019)Factors considered for the success of the Massive Open Online Course in the era of Smart Education : Systematic Literature Review2019 International Conference on ICT for Smart Society (ICISS)10.1109/ICISS48059.2019.8969844(1-5)Online publication date: Nov-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media