Opening The Door: An evaluation of the efficacy of a problem-based learning game

Abstract

As higher education institutions seek to improve undergraduate education, initiatives are underway to target instructional methods, re-examine curricula, and apply innovative technologies to better engage students with content. This article discusses the findings of an exploratory study focused on a course redesign that game elements, PBL methods, and 3-D communication tools in an introductory computing course. Some of these findings included an appreciation for how the technology skills gained in the course applied to the world of work, an understanding of the significant role that interpersonal communications play in learning and in career success, a sense of empowerment fostered by access to resources, and an increased willingness to play, explore, and experiment with tools, content, and design processes.

Introduction

In an effort to enhance the quality of the undergraduate experience in large-enrollment introductory courses, one state in the southwestern United States provided universities with small grants that target initiating and supporting research focused on the redesign of these courses. Goals of the course redesign project included using innovative instructional design methods, small group instruction, and communications technologies to improve student satisfaction, critical thinking and problem-solving skills in large group instruction (LGI). The specific course targeted by this redesign was an introductory course in basic computer applications used in educational settings. Not only was this a large-enrollment course, the pre-existing format of the course was problematic for students because it targeted overly fine-grained learning objectives that have shown little transfer from the narrow context of the course tasks themselves to the course assessments, let alone support skill transference into future school and work contexts.

In addition to the university’s goals, a major goal of this course redesign was to better prepare undergraduates for their future work beyond the classroom as professionals in the workplace, using technology tools to communicate effectively with peers and future employers. This article details the analysis leading to the resulting curricular redesign, provides an overview of the key elements of the design, and an evaluation of the effectiveness of those design elements in the pilot implementation of the course. Evaluation of the course redesign involved mixed methods, both quantitative and qualitative. Quantitative measures included drop, failure, and withdrawal rates, analysis of posttest achievement scores, and summative course evaluation scores. The qualitative methodology included constant-comparative analysis of student semi-structured interviews. These methods were used to better contextualize the quantitative findings and develop a richer understanding of student experiences in the redesigned course. The results of our evaluation of this redesign along with the theoretical foundations and description of the redesign are reported in this article.

Prior to redesigning the course, the instructional design team conducted a needs analysis of the existing course to identify structural problems to be addressed by the redesign (Warren & Dondlinger, 2008). This involved both document analysis of the course materials conducted according to methods suggested by Robson (2002) followed by a series of interviews with current and past instructors as well as twelve of the nineteen students in one section of the course. Questions in these interviews were open-ended to allow instructors and students to provide whatever information they felt was relevant.

The original course used an Adobe Flash™-based computer-based instruction (Kulik & Kulik, 1991) program called SAM 2003 (Thomson Course Technology, 2007). Following completion of software modules, students take an examination to show that they have acquired the skills and knowledge provided through the software training tasks. No help is provided during the exam, and the instructor can set a limit to the number of times a student may take it. A typical week in the existing face-to-face course design had students attend class for 3 h each week and individually work on the tutorials, practice, and exams in the SAM 2003 software. The instructor was expected to help students as needed and troubleshoot technical problems with the software; however, no direct instruction through lecture or other means was provided as the software was expected to do this.

From the instructor and student semi-structured interviews, the design team specifically identified the following challenges with the software and the course:

• Most learning objectives focused on a narrow set of skills such as how to open, save, or close a file, and the functions in one program were not linked to others. The creation of a chart in Excel has applications in Word, Publisher, and PowerPoint. Understanding how the programs can be used in a complementary fashion is an important objective that wasn’t addressed in the existing curriculum. Moreover, the ways to perform functions, such as opening or saving a file, are the same for each program. Yet the tutorials took students through each of the steps for these actions with each program in the suite. Students found this boring and repetitive, particularly since most of them were already familiar with these basic actions from previous computer applications courses at the secondary level.

• Computer-based assessments and instruction were too rigid. There are commonly several ways to perform an action in Microsoft Office, but the program recognized only one. In some instances, the practice exercise asked a student to complete a task in one manner, while the assessment compelled another. Students expressed frustration with this problem throughout the course.

• Applications of knowledge were decontextualized and lack authenticity. Students felt that the applications of learning expected in both the training practice and exams did not fit well with their life experiences. Students saw little relevance between course content and their future work. They also found the repetitive drill and practice unengaging.

• Computer-based instruction provided little interaction with peers and the instructor. Although the software program provided some student-to-system interaction, it did not provide ways for students to interact with each other. Moreover, students felt that the software program essentially relegated the instructor to the role of grade collector and troubleshooter of the software rather than experts on effectively using the Office Suite. Instructors also felt disengaged and some reported that they sought to integrate elements of the Scholarship of Teaching and Learning (Bass, 1999) or Chickering and Gamson’s (1987) Seven Principles for Good Practice in Undergraduate Education to improve student learning and their own teaching, but were restricted by the system.

Further, while the existing course explored the history of the Internet and the basic construction of a web page, it did not introduce students to online productivity and collaboration tools that could be invaluable to them in both their academic and professional careers. Many of these computer literacy skills were also repetitive for students who were expected to have mastered them by 8th grade according to state standards (Texas Education Agency, 2010). Students and instructors also found that the course did not leverage online tools, such as wikis, weblogs, and social networking sites, that can be used to improve the viability and productivity of Office™ programs.

University requirements dictated that course redesigns include technology and teaching methods that improved student satisfaction and better engaged students in meaningful interactions with peers and instructors. Following the preliminary analysis of issues with the existing course, the design team examined the literature on technology and student satisfaction. While the previous course design also included a technology component, institutional data indicated low student satisfaction as well as high drop, failure, and withdrawal rates. Thus, we felt it important to understand the situational factors in technology integration that lead to increases or decreases in student satisfaction. We also examined problem-based learning (PBL), known for enhancing critical thinking and engagement. Since members of the design team have expertise in the area of games for learning, they also explored how they might combine elements of game design with PBL in an emerging game genre, the alternate reality game (AltRG), to further engage students in compelling and meaningful interaction with course content, peers, the instructor, and their future careers.

Using technology to engage students and increase their overall satisfaction with learning is not new. Many studies have confirmed that satisfaction in technology-based learning environments is equal to that of traditional face-to-face classrooms (Schoech, 2000, Spooner et al., 1999, Wernet et al., 2000. Indeed, a recent study by the International Center for Media and the Public Agenda led by Moeller (2010) at the University of Maryland asked students to go 24 h without technology; most reported their experience in terms of psychological withdrawal symptoms as it was such a central part of their lives. As far back as 2001, a study by Kubey, Lavin, and Barrows correlated high Internet use with lower academic performance while a more recent study in 2009 by Karpinski and Duberstein similarly correlated high use of the social network Facebook™ among students with lower academic performance. According to a report by Borreson Caruso and Salaway (2007) for EDUCAUSE, undergraduates spend an average of 18 h a week using technology just for course work, with more than 80% preferring moderate or high use of information technology in their courses.

Nevertheless, all studies did not yield the same results. Rivera and Rice (2002) compared a traditional course to a hybrid and full-web course and found satisfaction to be substantially lower in the full-web course and highest in the traditional course. Irons, Jung, and Keel (2002) compared an interactive television course that used the web to enhance the course. Satisfaction was lower in the course that required students to use the web. However, it was not the web-enhancements to the course that dissatisfied students; it was the lack of access to the Internet and posted materials. Hara and Kling (1999) found that student frustrations were high in a text-based Multi Object Oriented environment (MOO). Nevertheless, it wasn’t the environment that aggravated students; it was confusion over what to do and how to do it once inside it. Thus, ensuring access to materials and providing clear instructions for learning activities are key to satisfying learner needs.

Other studies found interaction to be the key to student satisfaction (Ahern and Repman, 1994, Frederickson et al., 2002). Further, Hassenplug and Harnish (1998) found that the mere perception of interaction, “rather than the actual amount, may relate to student satisfaction with the distance learning experience” (p. 599). However, too much interaction can cause learners to feel overwhelmed, resulting in cognitive overload (Hara and Kling, 1999, Mason and Weller, 2001). Nevertheless, an absence of interaction leaves students feeling isolated. Thus educators using technology for teaching should encourage interaction, yet carefully guide student interaction to maximize learning and curb frustration. These findings correlate with students’ frustrations in the course selected for this redesign as students and instructors both found the lack of interaction to be a major problem in the previous technology-based format.

The burst of development in Internet technologies and their application to learning environments spawned the now famous debate between Clark, 1994a, Clark, 1994b and Kozma, 1991, Kozma, 1994 regarding instructional media versus methods. Clark maintained that media can never influence learning, and it is instead the instructional method responsible. In contrast, Kozma asserted that media may influence certain aspects of cognition and further that media and method are dependent upon one another to effectively deliver learning experiences. This particular redesign and evaluation effort does not attempt to evaluate the various media embedded in the instructional design. Instead, it focuses on the instructional methods that shaped the design. While a variety of media were deployed in the redesign, it played a supporting role in providing a situated learning context for student activities and instruction (Anderson et al., 1996, Brown et al., 1989, Lave and Wenger, 1991, Vanderbilt, 1993).

Course designers chose a problem-based learning approach because PBL has been correlated with improving post-secondary learning experiences by providing authentic contexts for learning (Bonk, Kirkley, Hara, & Denned, 2001) and compelling students to engage in story-driven, problem-centered tasks (Jonassen, 1999, Jonassen and Hernandez-Serrano, 2002, Warren, 2006a, Warren, 2006b). Tiwari and Lai (2002) also found that PBL encourages learners to hone a variety of thinking skills: “analyze and synthesize data; develop hypotheses; apply deductive reasoning to a problem situation; draw conclusions after analysis, synthesis and evaluation of new information; synthesize strategies/solutions; and monitor and evaluate their own thinking process” (p. 2).

Other authors have met with success when they combined a problem-based learning approach with proven critical thinking strategies to enhance the existing strengths of PBL methods and provide additional scaffolding for learners (DiPasquale et al., 2003, Elder and Paul, 2002, Keller, 2002, Willis, 2002). One technique has been to require student self assessment of their own thinking and encourage metacognitive processing (Bransford et al., 2003, Meyerson and Adams, 2003). Another study found that increased metacognition was correlated with the use of PBL and may also stimulate increased transfer of knowledge to new contexts and settings (Lin, Hmelo, & Kinzer, 1999). Furthermore, the reported satisfaction rates of students engaged in some PBL-based courses has been found to be higher than those rates reported by students in traditional learning environments (Ge & Land, 2003). It is therefore reasonable to expect that using such methods in a post-secondary course would lead to comparable results.

Savery and Duffy (1995) have noted that “knowledge evolves through social negotiation and the viability of individual understandings” (p. 2) where the interaction between student and peers as well student and instructor becomes central to learning. These principles formed the framework for redesign of the course, particularly because interaction was an element that the students and instructors indicated they missed in the previous course format. These principles also provided a cogent set of criteria to include in the instructional design and its evaluation:

• Anchor all learning activities to a larger task or problem.

• Support the learner in developing ownership for the overall problem or task.

• Design an authentic task.

• Design the task and the learning environment to reflect the complexity of the environment they should be able to function in at the end of learning.

• Give the learner ownership of the process used to develop a solution.

• Design the learning environment to support and challenge the learner’s thinking.

• Encourage testing ideas against alternative views and alternative contexts.

• Provide opportunity for and support reflection on both the content learned and the learning process. (p. 3–6)

Elder and Paul (2002) and other authors have successfully used each of these to design instruction in undergraduate and graduate courses. However, they sought to further engage students by leveraging today’s technologies, such as digital games, as a means of embedding scaffolds for student learning and providing a coherent narrative within which to place the ill-structured problems that would drive knowledge construction.

As part of the process of developing the course, the instructional designers analyzed games to determine where they intersect with problem-based learning principles as means to support learning and engagement. Researchers such as Dede, Ketelhut, and Ruess (2006) and Squire (2006) point to and experiment with the use of digital games as a means of supporting inquiry-based learning in areas ranging from social studies to science. A sense of play present in these learning environments that is thought to better prepare learners for a world in constant change and evolution because it allows them to test ideas and concepts more freely. By play here, we refer back to the Vygotsky-influenced (1978) definition that two of the authors of this study applied in their earlier work on game-based learning environments, “Play…(is an activity that) allows for the mind’s exploration of the rules and consequences of engaging with or breaking them” (Warren, Dondlinger, & Barab, 2009, p. 490). Although it was published long after the design and research reported here was conducted, the recent book, A New Culture of Learning: Cultivating the Imagination for a World of Constant Change, affirms these principles. In the book, Thomas and Brown (2011) treat learning as a means of inquiry that is integral to learning through games.

We analyzed the list of social constructivist design elements given by Savery and Duffy (1995) above against elements of game structures identified by Salen and Zimmerman (2004) and Crawford (2003), and noted several overlapping components. Specifically, both games and constructivist elements include:

• strategically constructed conflict or problem

• context for engagement with the conflict/problem

• goals or objectives for play or learning

• normative rules or conditions governing play or learning

• a quantifiable outcome with a means for assessing success

• scaffolds to support learners when challenges are too great

• cognitive conflict emerging from interaction with a designed problem

Fig. 1 illustrates the parallel components of PBL and game environments. Where these two environments overlap is in situating learning tasks (rather than other types of game goals) in the narrative context as the problem to be resolved and in providing instruction through in-narrative characters known as pedagogical agents.

Both the university’s timeline and budget prohibited designing and developing a learning game in a stand-alone, immersive world. However, a new genre of game, the Alternate Reality Game (AltRG) provided a welcome alternative. The AltRG distributes game challenges, tasks, and rewards across a variety of media both digital and real. As described by the International Game Developers Association (Martin & Chatfield, 2006), “Alternate Reality Games take the substance of everyday life and weave it into narratives that layer additional meaning, depth, and interaction upon the real world” (p. 6). CNET staff writer, John Borland (2005), depicts them as “an obsession-inspiring genre that blends real-life treasure hunting, interactive storytelling, video games and online community” (para. 4). Some AltRGs such as ilovebees.com served as marketing for the release of Microsoft’s Halo 2. Others have an educational focus such as Hexagon Challenge which sought to “address decision-making skills, after-action report generation, and adaptation to performance” (Bogost, 2007). While not explicitly educational, others have dealt with social, economic, and environmental justice challenges. For example, Jane McGonigal has masterminded AltRGs as a means to construct what she describes as “collective intelligence.” The purpose of her 2007 ALTRG World Without Oil (Ernst, 2007) was to “play our way to a set of ideas about how to manage that crisis [a dramatic decrease in oil availability]” (cited in Strickland, 2007, p. 1). McGonigal noted that learners developed strategies for coping with a peak oil crisis and further claimed changes to their real world activities (Strickland, 2007). We thus conjecture that the simulated problem in the AltRG prompted players to create real world applications of knowledge constructed with their peers in a simulated play space.