Elsevier

Computers in Human Behavior

Volume 61, August 2016, Pages 378-385
Computers in Human Behavior

Full length article
Modelling experts' behavior with e-valUAM to measure computer science skills

https://doi.org/10.1016/j.chb.2016.03.044Get rights and content

Highlights

  • Evaluation of Computer Science skills.

  • Modelling experts' behavior.

  • Automatic students' evaluation.

Abstract

In this work we present an evaluation method that focuses on experts' behavior instead of the traditional scores based just on the number of correct answers. The method presented here is especially suitable to measure the skills in Computer Science since this is a wide discipline very difficult to evaluate due to the many facts publicly available on the Internet. By using traditional evaluation tools, it is very difficult to measure the real knowledge of the users since they can correctly answer even without having acquired formal academic knowledge. To use this method, we have developed a test that can detect significant differences between standard users and experts in Computer Science. The test is applied by the e-valUAM application, which has been modified to store several parameters from the users' answers. By optimizing the parameters by a linear model, we have developed an equation that can be used to quantitatively compare the results of a single user with the results from the reference group of experts. This optimization is only possible because this group shows good stability and gives statistically different results compared to the other groups. The scores achieved with our method can be used to predict the formal knowledge of the users and modify their training when needed.

Introduction

The concept of Technology-Enhanced Learning (TEL) has an increasing presence in educational institutions, leading Blended Learning, which makes use of Computer-Aided Learning (CAL), a keynote practice where teachers and students are computer-assisted in their teaching/learning processes (Molins-Ruano et al., 2013). Completely on-line learning environments, such as Massive Open Online Courses (MOOCs) (McAuley, Stewart, Cormier, & Siemens, 2010), must be assisted by CAL environments. In this scenario, the success of using new technologies in the learning process clearly depends on the actual knowledge and familiarity of the users with technology.

When applying new technologies in any learning process, we must start from the certainty of not considering the students as digital natives (Kirschner & van Merriënboer, 2013). In other words, we cannot be sure that students in digital environments are the best manager of their own learning since we cannot be sure if they know how to use technology and digital media properly (Atrio, 2010, Area Moreira et al., 2015). Moreover, in the case of using and applying new technologies, the fact that students can be confident about their knowledge without having adequate skills is another source of error that must be taken into account (Finn & Tauber, 2015). Previous experiences with MOOCs have demonstrated that there are difficulties in the learning process that can be related to prior level of schooling and expected hours devoted to the Course (Greene, Oswald and Pomerantz, 2015). These facts can be related also to the real level and skills acquired in the use of the technology. The students' abilities and custom in the use of new technologies are also defined as critical in another experimental study that reveals that the format how information is shown to the students is critical in their performance (Ruf & Ploetzner, 2014). In conclusion, since new technologies are already present in our contemporary learning environment and it is not clear if students and teachers have acquired enough skills to properly manage them, we must be very careful when introducing any technology or computer application as a tool in a learning process.

For this reason, it is essential to get good information about the users' ability before starting the development of any computer application. If we do not get this information, we could expend time and money in an application that can be useless for not having, for example, an appropriate interface or documentation. The problem is that Computer Science is extremely influenced by the very fast advance of the technology and a huge amount of information from very different sources is available, being not all of these sources adequate to get correct or formal knowledge. For example, it is common to find people using and talking about USB, Bluetooth, encrypted messaging, etc., without knowing anything about the technology underneath. Although wrong or inaccurate sources have been always around, Computer Science skills are much more exposed than the Physics, Mathematics or Biology ones. For example, it would be very easy to get correct answers if we ask “What is Windows?” For sure, many users will answer “it is an operating system.” However, how many of them really know what an operating system is?

In summary, the problem that we are facing here, and must be solved before going further in the use of new technologies for learning, is composed by two factors: 1) We cannot assume that users are ready to use new technologies. 2) It is very difficult to measure users' skills in new technologies. In this work we propose a new method to determine the real Computer Science skills of any kind of users that is based on two main concepts: 1) formatting Computer Adaptive Tests (CAT) and 2) modelling expert behavior.

On the one hand, CAT has already been used for very different purposes such as instantaneous scoring (Wainer et al., 2000), language proficiency improvement (Chapelle & Douglas, 2006), learning styles identification (Ortigosa, Paredes, & Rodriguez, 2010), measurement of chess playing proficiency (Van Der Maas, & Wagenmakers, 2005), Maths ability (Klinkenberg, Straatemeier, & Van der Maas, 2011), improvement of the efficiency of personality (Stark, Chernyshenko, Drasgow, & White, 2012), or heath status assessment (Revicki & Cella, 1997). Different versions of CAT have been used in learning environments in order to personalize the learning process (Conejo et al., 2004, Lilley et al., 2011, Sands et al., 1997, Weiss, 1982). In that framework, CAT allows seeing the students as individuals, taking their own characteristics into account. Typically, CAT systems are able to adapt the items presented to the student depending on their former answers, often including some kind of personalized feedback (Kumar, 2004). In this context, systems which provide any kind of feedback to students look to be more effective (Bravo, van Joolingen, & de Jong, 2009). We can even find assessment tools for some specific learning domains that are able to give advice (Castro-Schez et al., 2014, Economides, 2005, He et al., 2009) or feedback (Antal & Koncz, 2011) to the students.

On the other hand, modelling expert behavior is included in our model because one of the most important difficulties in CAT systems is the need of item pre-calibration. When evaluating Computer skills from a general point of view, we must also take into account that we are evaluating a very wide concept. This fact strongly increases the difficulty of selecting and pre-calibrating items for the test. The complexity of pre-calibration has led to different approximations and models, being one of them the development of experts' models which could help in the pre-calibration task (Antal and Koncz, 2011, Virvou and Troussas, 2011, Gálvez et al., 2009, Dorça et al., 2013, González-Brenes et al., 2014). Although these kinds of models are widely spread in procedural areas, they are not that common in Computer Science. However, there are some related works that supports the idea of using experts' modelling in this area. For instance, it has been demonstrated that changing knowledge structures increases the Computer Science skills learning effectiveness (Davis & Yi, 2004). It has been also demonstrated that novice and expert users show different behavior when facing information problems (Brand-Gruwel, Wopereis, & Vermetten, 2005). Many parameters related to the personality of the users have been proposed before in the literature. In that sense, it has been demonstrated that adolescents that perceive themselves as competent in handling computers show a keener computer interest and in turn higher performance rates in basic computer science skills (Christoph, Goldhammer, Zylka, & Hartig, 2015). The influence of previous experiences of learning with information and communication technology (ICT) has been also related to the behavioral intentions of teachers towards using ICT for teaching (Valtonen et al., 2015). In addition, it is also common to find out learning models that include didactic content that adapts to learners' needs (Marciniak, 2014), which make use of the concept of intelligent tutoring systems (ITS) (Wang et al, 2015).

Following those previous results, we have included two modifications in our evaluation method that are 1) to develop an alternative evaluation system able to give scores by comparison with experts instead of by measuring the correct answers to a set of items that are not correctly pretested and 2) we have modified our test to include additional information about the opinion of the users about their own answers. For example, we give them the opportunity of including answers and mark that they are not absolutely sure. This kind of additional information will give us the opportunity of scoring not only the correct answer but also the behavior of the users. We propose to predict the formal knowledge by comparing users' results with the experts' behavior by a model that has been calibrated with all the additional useful information from our test.

The structure of this paper is as follows: firstly, we describe the new test model with a full description of all the information that can be acquired. Secondly, we apply the test to different groups of users and stablish that there are significant statistical differences between groups. Following, we demonstrate that the most stable and accurate group of students comes from a high course of Computer Science. Being this group different from the others, and also uniform in their answers, it is used as the reference group of experts by the model. Then, we describe a numerical method that is able to give a score based on the behavior of any standard user when compared to the group of experts. This new score system is compared to the traditional method of measuring only correct answers. We show that there are users that get good scores in both methods and others that show big differences. The use of our method gives us the opportunity to discriminate both kind of users and correct possible deficiencies in the second group. This information would not be available without using both methods together.

Section snippets

e-valUAM description

To evaluate users by the modified CAT system proposed in this work, we will use the e-valUAM application. e-valUAM is a free software based on adaptive test that uses previously tested numerical models to make the students follow different paths depending on their answers. The numerical models have been previously optimized to maximize the effect of following different paths (Molins-Ruano, González-Sacristán, Díez, Rodríguez, & Sacha, 2015). Since e-valUAM has been fully described in previous

Applying the test

The test has been applied to three different groups of students. The first one is the target of the experiment. Thas group (named Standard Users) is composed by 36 students from the fourth course of the Degree in Early Childhood Education at the Universidad Autónoma de Madrid. Those students do not have, in general, any formal education about computers. However, they use computers frequently. The second group (named Inf 2) is formed by 23 Computer Science students that are in the middle of

Conclusion

In this work we have demonstrated that scoring by comparison with a group of experts gives additional relevant information when Computer Science skills are being evaluated. The additional information acquired by this model allows us to discriminate users that do not have correct formal knowledge even when they are able to get good scores by traditional evaluation systems. To use this method, we have developed an adaptive test that gives us additional information about the users' behavior and

Acknowledgments

This work has been partially funded by projects TIN 2013-44586-R and e-Madrid (S2013/ICE-2715). GMS acknowledges support from the Spanish “Ramón y Cajal Program”. The authors would also like to thank the students who participated in this experience.

References (38)

  • T. Valtonen et al.

    The impact of authentic learning experiences with ICT on pre-service teachers' intentions to use ICT for teaching and learning

    Computers & Education

    (2015)
  • D. Wang et al.

    A problem solving oriented intelligent tutoring system to improve students' acquisition of basic computer skills

    Computers & Education

    (2015)
  • M. Area Moreira et al.

    Educar a la generación de los Millennials como ciudadanos cultos del ciberespacio: apuntes para la alfabetización digital

    Revista de Estudios de Juventud

    (2015)
  • S. Atrio

    El renacimiento del siglo XXI para la universidad europea. Una declaración de intenciones

    Revista Iberoamericana de Educación

    (2010)
  • C.A. Chapelle et al.

    Assessing language through computer technology

    (2006)
  • R. Conejo et al.

    SIETTE: a web-based tool for adaptive testing

    International Journal of Artificial Intelligence in Education

    (2004)
  • F.D. Davis et al.

    Improving computer skill training: behavior modeling, symbolic mental rehearsal, and the role of knowledge structures

    Journal of Applied Psychology

    (2004)
  • F.A. Dorça et al.

    Automatic student modeling in adaptive educational systems through probabilistic learning style combinations: a qualitative comparison between two innovative stochastic approaches

    Journal of the Brazilian Computer Society

    (2013)
  • A.A. Economides

    Personalized feedback in CAT

    WSEAS Transactions on Advances in Engineering Education

    (2005)
  • Cited by (17)

    • A new health care system enabled by machine intelligence: Elderly people's trust or losing self control

      2021, Technological Forecasting and Social Change
      Citation Excerpt :

      Heerink et al. (2010) use both a technology acceptance model (TAM) (Davis, 1989) and a unified theory of acceptance and use of technology (UTAUT) model (Venkatesh et al., 2012), adding some new constructs—trust, social presence, perceived sociability, perceived enjoyment, perceived adaptability, and anxiety—to measure behavioral intentions of elderly people. Researchers of marketing (Pavlou, 2003; Shareef and Kumar, 2012; Shareef et al., 2013) and virtual technology (Carter and Bélanger, 2005; Dwivedi et al., 2016; Kim, 2016; Molins-Ruano et al., 2016; Shareef et al., 2011) unanimously assert that for consumers to accept anything virtual, they must trust that system. Looking at consumers’ adoption behavior model for mobile health (Dwivedi et al., 2016), which developed its theoretical concept based on a UTAUT2 model (Venkatesh et al., 2012), it is observed that effort expectancy is an important predictor for consumer acceptance of any complicated technology.

    • Group behavior in social media: Antecedents of initial trust formation

      2020, Computers in Human Behavior
      Citation Excerpt :

      We investigate this overarching issue via a thoroughly designed experiment and survey. Over the last 60 years, marketing researchers (Fishbein, 1963; Ho & Dempsey, 2010; Mitchell & Olson, 1981; Molins-Ruano, Rodriguez, Atrio, & Sacha, 2016) have analyzed consumer behavior to understand the factors influencing consumer opinions towards a product/service. The theory on consumer attitudinal change suggests that negative attitude towards a product can be altered.

    • Students' performance in French subjects in the COVID-19 confinement

      2020, ACM International Conference Proceeding Series
    View all citing articles on Scopus
    View full text