The implementation and evaluation of a mobile self- and peer-assessment system

Abstract

Recently, more and more researchers have been exploring uses of mobile technology that support new instructional strategies. Based on research findings related to peer and self assessment, this study developed a Mobile Assessment Participation System (MAPS) using Personal Digital Assistants (PDAs) as the platform. In addition, the study proposes an implementation model of the MAPS that should facilitate the effectiveness of self- and peer-assessment in classrooms. The researcher argues that teachers and students can benefit from MAPS in various regards including more flexible assessment arrangement, more efficient use of time, and more opportunities for student reflection on learning and assessment. Thirty-seven students taking teacher-education courses with the researcher participated in this study, and these students employed the MAPS to conduct two-round assessment activities that would help these students assess both their own and one another’s final projects. Both the students’ valid responses in a survey herein and scores obtained from the assessment activities confirmed the benefits of the MAPS and its implementation model. Yet, the students voiced such concerns as the objectivity of peer-assessment and the difficulty of providing constructive feedback, and the correlation analysis indicated a lack of consistency between teacher-grading and student-grading.

Introduction

High-quality assessment can facilitate learning, but poor-quality assessment can discourage high-quality learning or even be harmful to students (Novak, Mintzes, & Wandersee, 1999). Except for adherents to traditional norm-referenced assessment, most theorists in the field currently emphasize the importance of finding weaknesses and strengths in student knowledge. Assessment should be ongoing, so that teachers can keep providing useful feedback to students and so that students can learn how to monitor their own performance (Bransford et al., 2000, Perkins et al., 1999; Shepard, 2001). Researchers have been encouraging teachers to employ self- and peer-assessment that helps students reflect on their own learning and enter into classroom-based collaborative learning. For example, Orsmond, Merry, and Reiling (2000) asked first-year undergraduate biology students to conduct self- and peer-assessment on a poster assignment, and argued that self- and peer-assessment strengthened the association between feedback and learning improvement. Before conducting self- and peer-assessment, teachers and students should engage in meaningful discussion and communication to select explicit criteria about how student performance and student work will be assessed. Thus, students will pay more attention to the processes of undertaking learning activities rather than focus solely on final products or grades. And thus, self- and peer-assessment stresses the importance of understanding student progress at different phases of learning.

The use of peer-assessment can be very flexible. Peer-assessment can help in assessing the performance of individual students as well as of groups, and can help in formative evaluation as well as summative evaluation. Students can assess their group members or other groups’ members (McLuckie & Topping, 2004). Although many students have less expertise and experience in assessment than teachers have, student assessment need not always be less effective than teacher assessment. Magin (2001) employed statistics techniques to compare peer-assessment results with single-teacher assessment results gathered from two courses, Introductory Clinical & Behavioural Studies and Science & Technology Studies at a University in UK. Magin found that the single-teacher assessment and the average of the two- or three student assessment reached similar levels of reliability. Hence, appropriately implementing peer-assessment can attain sufficient reliability, and students can thereby receive more information and feedback to improve their performance than otherwise would be the case. Yet, Gibbs (2006) argued that over-emphasizing the reliability of student marking misses the point and dilutes the benefit of self- and peer-assessment wherein students engage in learning by internalizing academic standards and by making judgments about their own and peer performance in relation to these standards.

Regarding how to effectively involve students in assessment processes, Bloxham and West (2004) in their study on sports-studies students’ assessment of peer work suggested that teachers use strategies such as discussing assessment criteria with students, providing scoring rubrics, encouraging feedback-giving, and evaluating the appropriateness of student assessment. Through this kind of systematic participation, students can better understand how their performance and work will be assessed and make revisions based on the feedback. There are various studies on self- and peer-assessment in higher education, and they involve participants from different disciplines including education, health, business, computer science, and the humanities (e.g., Ballantyne et al., 2002, Falchikov and Goldfinch, 2000, Magin, 2001, Price and O’Donovan, 2006, Prins et al., 2005, Purchase, 2000). Their findings indicate that explicitly explaining and defining assessment criteria should be a top priority for teachers to conduct meaningful self or peer-assessment; therefore, teachers can deliberately involve students in the formation of assessment criteria and scoring rubrics. Co-deciding assessment criteria through meaningful dialogues and feedback-giving can enhance assessment effectiveness and facilitate collaborative learning (Price and O’Donovan, 2006, Prins et al., 2005).

Peer-assessment is beneficial for student learning in various ways. It improves students’ ability to relate instructional objectives to assessment activities, to understand assessment criteria and processes, to identify the strengths and weaknesses of the students’ own performance, to improve their understanding of and their confidence in the subject matter at hand, and to improve their future performance (Ballantyne et al., 2002). Nevertheless, there are cautions that teachers should take while conducting peer-assessment. For instance, Ashworth, Bannister, and Thorne (1997) found that assessment methods, when conducted in contexts more informal than the contexts of controlled and supervised examinations, created situations in which University students had greater opportunities to plagiarize and to commit other acts of cheating. Ballantyne et al. also reviewed studies addressing such problems of peer-assessment as the limited experience, expertise, and confidence of teachers and students relative to this assessment method. Many students doubt whether they themselves have the ability to provide constructive feedback and appropriate marks. Most students consider assessment to be the primary responsibility of teachers, and most teachers and students tend to value outcomes of expert assessment over the outcomes of peer-assessment. Many students feel uncomfortable with criticizing others’ performance, and students usually feel peer-assessment to be very demanding and time-consuming (Ballantyne et al., 2002, Davies, 2000, Lin et al., 2001, Miller, 2003, Topping et al., 2000, Tsai et al., 2002).

Appropriate technology use in assessment processes could alleviate some of the aforementioned shortcomings. For example, Davies (2000) developed a computerized peer-assessment system and implemented this system in an undergraduate module within a school of computing. The students made marks and comments on peers’ reports; some comments even concerned the student assessors’ detection of plagiarism by means of web searches for the references listed in the reports. Nicol and Milligan (2006) stressed that applying technology to the assessment process should focus on how the application can support the assessment process to promote dialogic feedback and to enhance students’ abilities of conceptualizing performance standards and being self-regulated learners. These emphases are the goals the present study strives to achieve. The next section will further discuss how technology may enhance the effectiveness and efficiency of self- and peer-assessment.

Although peer-assessment can save teachers a good deal of time that they would otherwise spend on grading student work, teachers have to devote considerable time and effort to organizing and managing assessment processes. Integrating technology into assessment processes may resolve potential obstacles to implementing self- and peer-assessment (Topping, 1998). Lin et al. (2001) developed an Internet-based peer-assessment system, and they described several advantages of this system after using it in a computer-science course in Taiwan. For example, to maintain anonymity during assessment processes, teachers may spend much time on encoding. Well-designed computer programs enable students to assess each other anonymously and demand much less time and effort from teachers, and thus, students are more willing to criticize others’ performance. Computer networks enable students to conduct assessment activities anytime anywhere, and teachers can flexibly log in online to check assessment progress. Also, online assessment can eliminate the time and the cost that teachers and students would otherwise invest in printing out student work. Sung, Chang, Chiou, and Hou (2005) highlighted a web-based self- and peer-assessment system that can do away with conditions restricting various assessment activities to the classroom. The researchers proposed a progressively focused assessment procedure supported by the web-based system, and Taiwanese ninth-grader participants improved their assessment objectivity and the quality of their webpage-design assignments.

Asking each student assessor to evaluate a student’s performance translates into a situation where students need less time to complete the assessment process than would be the case if each student assessor had to evaluate the performance of a group of peers. However, every student consequently receives less feedback from peers. Scenarios in which teachers ask their students to evaluate more and more work of peers translates into a situation where each student receives more and more feedback than he or she would receive if only the teachers were evaluating the students’ work. However, asking students to evaluate significant quantities of peer-produced work would take up a significant amount of time and might negatively influence the efficiency and the effectiveness of peer-assessment (Sung et al., 2005). Hence, deciding the number of “assessed students” for which each “assessing student” should be responsible is a crucial task. Sung et al., 2003, Sung et al., 2005 found that if the students conducted peer-assessment several times, the difference between students’ assessment outcomes and the teacher’s assessment outcomes would gradually decrease. This finding indicates that asking students to assess peer performance iteratively could give them a better chance to differentiate superior work from inferior work and to grasp assessment criteria, so that the students would become more competent assessors. Moreover, through giving and receiving feedback in self- and peer-assessment, the students in the aforementioned study enhanced their ability of reflection and improved their work. The web-based assessment system enabled teachers and students to assess and revise student work multiple times with much less time spent than would characterize situations without the technology.

Tsai et al. (2002) developed a web-based assessment system in which Taiwanese pre-service students conduct three cycles of peer-assessment and revision of a science-activity design project. The researchers found that those students who gave detailed and constructive comments to their peers benefited more from the assessment processes and had better ideas about how to improve their own work than was the case for other students. The researchers stated that positive effects of peer-assessment might not emerge right after assessment activities and that the results of peer-assessment might not be an appropriate indicator of student performance. Furthermore, some students reported that they felt such an assessment scheme to be time-consuming and that they would be unwilling to spend extra time assessing others’ work after class. Most students were uncomfortable with criticizing others’ performance in spite of the secure anonymity, and many students provided vague and irrelevant feedback, which was useless for recipients of feedback seeking to improve their performance. Some students even purposely gave peers unjustifiably low scores so that the scorers, themselves, could possess advantageous positions. Tsai et al. suggest that teachers try to ensure that (1) students engage in peer-assessment, (2) the teacher and students have an open and explicit discussion about assessment criteria in advance, and (3) the teacher continually monitor assessment processes to prevent students’ inappropriate grading of other students.

Pownell and Bailey (2002) differentiated four main phases of educational computing, and they envisioned the latest trend in educational computing: anyone can access information and communicate with others anytime and anywhere via small-size computers and wireless networks. Because of the characteristics of mobile technology such as ubiquity, smaller size, comparative affordability, and the prevalence of wireless networks, more and more researchers have been exploring supportive application of such handheld computers as mobile phones, laptop computers, PDAs, and graphic calculators to new educational strategies. Arising from these endeavors to conduct mobile-learning studies is the critical issue of how to use handheld computers in innovative ways that enhance assessment (Penuel et al., 2007, Penuel and Yarnall, 2005, Shin et al., 2007). For instance, Students can flexibly conduct project-based learning and self assessment inside and outside classrooms via PDAs or mobile phones; teachers and students can record teaching and learning processes for portfolio assessment; teachers can receive student feedback and can, thus, monitor student understanding through an Interactive Response System (IRS); and students can key in their answers anonymously via handheld remotes and can compare their thoughts with peers’ (Davis, 2003, Draper and Brown, 2004, Penuel et al., 2007, Smodal and Gregory, 2003, Vahey and Crawford, 2002, Wang et al., 2004).

However, not much research has reported findings about how to use mobile technology for self- and peer-assessment, and this issue merits further investigation. Hence, the current study presents a novel self- and peer-assessment system using PDAs as the system platform and proposes an implementation model of this system. In addition, 37 pre-service teachers participated in this study to assess their lesson-plan projects with both the developed system and the proposed implementation model. The researcher who conducted and authored this study collected and analyzed both the students’ scores corresponding to the lesson-plan projects and the students’ responses to the assessment activities.