Rating monitoring as a means to mitigate rater effects and controversial evaluations

Assessment is a key aspect of any instructional process, as it is the main mean of determining the competences of the students. Nowadays, the assessment and scoring carried out by groups of reviewers, namely evaluation of final year project or thesis works or even peer evaluation, are becoming more and more frequent. However, the assessment and scoring of a work in such scenarios can be affected by each rater's thinking processes, knowledge level and personal preferences among other issues. These idiosyncrasies are known as rater effects and can dramatically affect the evaluation process. Although many works point out that the use of certain evaluation instruments, e.g., evaluation rubrics, can increase the fairness and impartiality of the evaluation, rater effects may be still present and remarkably affect the scoring. Furthermore, some works might present controversy on their assessment, i.e., the evaluators of a certain work might strongly disagree on its quality. Therefore, the identification of the rater effects and controversial evaluations is crucial to be able to take remediation actions and to guarantee a fair evaluation. However, this identification process is often hard for scoring leaders. Consequently, tools that help leaders in this process are necessary. This paper presents the visualizations used by RaMon (a system for monitoring raters and controversial evaluations) to help the monitoring process, along with the support it provides to take remediation actions.