Elsevier

Computers & Security

Volume 125, February 2023, 103049
Computers & Security

Evaluating protection motivation based cybersecurity awareness training on Kirkpatrick's Model

https://doi.org/10.1016/j.cose.2022.103049Get rights and content

Highlights

  • Evaluation of effectiveness of PMT based cybersecurity training was conducted

  • The training increased knowledge and cybersecurity behavioral intention of the students

  • Self-efficacy was the dominant antecedent in predicting the cybersecurity behavior

  • The high satisfaction of the students showed efficacy of the PMT based training

Abstract

Context

Cybersecurity behavioral literature has a significant number of studies on training and awareness. However, there is lack of theoretical underpinnings in developing intervention to allow for positive behavioral change and evaluating them. The evaluation of theory based cybersecurity training warrants the use of program evaluation techniques.

Objective

The protection motivation theory (PMT) was employed to understand the behavioral change after the implementation of cybersecurity training. The evaluation was done on three levels of Kirkpatrick's evaluation model – reaction, learning and behavior.

Method

A pre-post quasi experimental design was adopted in this research. A total of 154 undergraduate students from computing and digital arts backgrounds took part in the research.

Results

The results of the study showed that the PMT based training was effective in increasing the threat knowledge of the students along with the increase in information of countermeasure strategies. From the two components of the PMT, self-efficacy was found to be the significant predictor of the cybersecurity behavioral intention in both pre-test and post-test PMT models. The cybersecurity training increased the self-efficacy of the students significantly and contributed towards cybersecurity behavioral intention change. The findings of this study imply that in designing the cybersecurity trainings, educators should dominantly take into account the self-efficacy component of the PMT.

Introduction

The importance of studying awareness and the role of human factors in cybersecurity has recently gained ground (Zimmermann and Renaud, 2019) and academic research in the area is still nascent (Parsons et al., 2017). The higher educational institutes (HEIs) are one of the least secure environments (Bongiovanni, Sep. 2019) and cybersecurity has been ranked as the number one area of concern over a period of time (Farooq et al., 2015) in these institutes. The cybersecurity concerns for HEIs were first identified two decades (Luker and Petersen, 2003) earlier but its only recently that empirical evidence has started to emerge (Bongiovanni, Sep. 2019). A number of researchers have highlighted the rise in cyberattacks in the university settings (Chapman, 2019) which is backed by rising number of cybercrimes in HEIs globally. One of the factors attributed towards these cybercrimes is the inherent infrastructure of these institutes which is open-by-design, decentralized and transient in nature (Borgman, 2018). Multiple stakeholders interact with the universities’ platforms for teaching, research and innovation purposes (Bongiovanni, Sep. 2019). Students, academicians and staff members continuously exchange data in a multi model environment which results in a huge amount of data being generated that can potentially be exploited by cyber criminals (Zhang and Li, 2015). Studies have also reported (Rezgui and Marks, 2008; Katz, 2005) the exploitation of universities’ computational infrastructure in launching denial of service attacks and mining of cryptocurrency by using students’ and staff's data. With low level of cybersecurity awareness in university going students, a number of research calls have been made to cultivate cybersecurity awareness (Chen et al., 2021) in HEIs.

Security education training and awareness (SETA) programs impart security knowledge and skills to the individuals to inculcate cybersecurity consciousness (Cram et al., 2019; Burns et al., 2018). SETA programs are differentiated as awareness, training or educational programs based on the rigor of the program. The security awareness programs are at the basic level of SETA (Kruger and Kearney, 2006) and are aimed to draw attention to the importance of cybersecurity via asynchronous communication (Kolb and Abdullah, 2009) such as posters and banners. On the other hand, cybersecurity training programs build skills as well as knowledge of the individuals which help them in protection against the threats in the cyberspace (Amankwa et al., 2014; Puhakainen and Siponen, 2010). Whereas, the security education programs are the highest level of SETA (Jenkins and Durcikova, 2013) and are formal educational programs which integrate security knowledge and skills in a body of security knowledge (Hu et al., 2021). With studies reporting humans as the weakest link possessing low cybersecurity knowledge, the awareness and training has been discussed in the scientific community for over a decade (Siponen, 2000). A number of studies have been conducted to understand the association between SETA programs and the cybersecurity behaviors under different theoretical frameworks (Tsai et al., 2016; Verkijika, 2018; Rajab and Eydgahi, 2019; Hina et al., 2019). Protection motivation theory (PMT) is one of the most frequently used theory in cybersecurity behavioral research (Hutchinson and Ophoff, 2020; Boehmer et al., 2015; Johnston and Warkentin, 2010) and several studies have supported the predictive power of PMT in explaining cybersecurity behaviors. However, the PMT constructs are operationalized in organizational settings which have mandated cybersecurity compliance. PMT works well in non-mandated settings as it was originally developed to understand security where the behavior is volitional (Sommestad et al., 2015). Another issue is the lack of PMT's applicability to study cybersecurity behavioral change (Haag et al., 2021) as evident from a recent systematic literature review. Therefore, SETA initiatives for university going students should not only be designed using PMT but also evaluated to see the behavioral change.

According to a recent survey, SETA initiatives are taken by a number of organizations (Assenza et al., Oct. 2020). A number of studies have been carried out that report designing of the security training. However, such initiatives are rarely evaluated in terms of their impact and effectiveness (Assenza et al., Oct. 2020; Muronga et al., 2019). The use of a structured methodology for such evaluations is lacking. Most of the SETA initiatives are gauged by a priori qualitative assumptions relying on the experiences of the managers. The experience-based approach is inadequate and is ill suited for preventive orientated cybersecurity awareness. Due to the educational element of the SETA programs, the evaluation should be systematic (Rossi et al., 2018) in which the output of the assessment should improve the state of a security program's contents as well as the objectives. Cybersecurity research community has made research calls for employing systematic program evaluation techniques (Assenza et al., Oct. 2020; Rahim et al., 2015) specifically Kirkpatrick's evaluation model. Therefore, evaluation of cybersecurity training programs is warranted. In the Section 1.1, we first report on literature review of cybersecurity awareness and trainings in the cybersecurity behavioral research. We then discuss PMT and the short comings in its applicability in cybersecurity behavioral research in Section 1.2 followed by introduction of Kirkpatrick's evaluation model in Section 1.3. We then present our research objectives in Section 1.4.

A number of studies are published on SETA programs. A recent systematic literature review (Kävrestad and Nohlberg, 2021) classified the evaluation of SETA programs into three main categories which we have used in carrying out the comparison analysis of the literature. The classification is as follows; 1) perception which is focused on perception of the trainees, 2) knowledge which evaluates the gain in knowledge and 3) security outcome which is the actual security behavior in laboratory or naturalist settings. The study (Huynh et al., 2017) employed a serious game to enhance the phishing awareness of the university students based on activity theory. It was found that the students’ perception about the enjoy-ability of the game was positive. Similarly, a gamified cybersecurity awareness training (Gjertsen et al., 2017) given to the employees resulted in positive perception while the memorability of the security question was enhanced by another game based training (Micallef and Arachchilage, 2017). The study (Micallef and Arachchilage, 2017) found that the features and functionalities of the game were rated positive by the respondents. A game based phishing training was carried out on a large number of participants (CJ et al., 2018). The training was found to be effective in building the knowledge of the trainees. A pre-post surveys were carried out in Zhou et al. (2018) to test the effectiveness of a mobile training app for enhancing smartphone security. The results showed that the feedback received from the participants was positive (Zhou et al., 2018). In another study (Jayakrishnan et al., 2020), training program was developed to teach the password heuristics employing pre-post experimental set up. The positive perception of the global participants and the learning gained showed the effectiveness of the training (Jayakrishnan et al., 2020). In another study (Van Rensburg et al., 2018) knowledge about the smartphone security was imparted to the university going students in a blended learning environment. The results of the pre-posttest experiment (Van Rensburg et al., 2018) showed increased learning and knowledge of the students. The limitation of the above mentioned studies is that they lacked employing any theoretical framework to study the cybersecurity awareness and did not take into account the cybersecurity outcome.

There were also few studies that evaluated the effectiveness of the trainings based on some theoretical frameworks and made use of the security outcome. One of the earliest study (Puhakainen and Siponen, 2010) used elaboration likelihood model and universal constructive instructional theory in an action research to enhance employees compliance with the email policy. The evaluation (Puhakainen and Siponen, 2010) was carried out using surveys, questionnaires and observations. In another action research study (Gundu, 2019), general deterrence theory (GDT) and theory of planned behavior (TPB) were used to heighten the knowledge and actual compliance with the cybersecurity policies in organizational settings. The pre-post experiment in Gundu, (2019) contained two iterations; with the reward and punishment concepts of GDT being incorporated in the second iteration and yielded positive outcomes. A number of studies have been carried out to lower the susceptibility of the users to phishing attacks (Lim et al., 2016; Cuchta et al., 2019; Yang et al., 2017; Tschakert and Ngamsuriyaroj, 2019) [(Silic and Lowry, 2020), p.] (Daengsi et al., 2021). These training were evaluated by measuring the security outcome and taking into consideration the logs of the participants. Using a training system, (Lim et al., 2016) reported a decrease in clicking of phishing links. In another study (Cuchta et al., 2019), a controlled experiment was conducted on a large number of university students, faculty members and staff personnel to heighten their resilience towards phishing attacks. The study (Cuchta et al., 2019) used multiple types of trainings (including text, visuals and interactive game) and found that the visual document based training was better in reducing the number of clicks in phishing emails (Cuchta et al., 2019). In another field experiment (Yang et al., 2017), phishing training based on warning signs was delivered to university students. The results revealed that number of identified phishing emails/webpages was increased; and the training was found to be effective in reducing clicks on phishing links (Yang et al., 2017). Similarly, in Tschakert and Ngamsuriyaroj, (2019) multiple delivery methods (textual, video, game based and instructor led) were employed to mitigate phishing susceptibility of the university going students using signal detection theory (SDT). The results of the SDT based training were promising in reducing susceptibility to phishing (Tschakert and Ngamsuriyaroj, 2019). In another study (Jansen and van Schaik, 2019), PMT was employed to design fear appeal messages for improving protection against phishing. The experimental study that employed 786 individuals showed increase in the positive attitudes towards phishing and behavioral intentions of the participants (Jansen and van Schaik, 2019). A recent study carried out in a French organization [47, p.] made use of kernel theory and introduced gamification principles in security training to enhance phishing awareness. The study found that the training was effective in increasing learning about phishing and consequently lowering employees’ susceptibility towards it. Another phishing training (Daengsi et al., 2021) using a blend of e-Learning and face-to-face seminar was conducted and evaluated on a pool of 20,000 employees in Thailand. The pre-posttest simulations in Daengsi et al. (2021) evaluated the susceptibility of the employees in clicking phishing links and found that females performed better in terms of their awareness as compared to males. The study (Dincelli and Chengalur-Smith, 2020) employed interactive story telling mechanism to train the participants in improving their self-disclosure habits by carrying out longitudinal randomized control trials. The results (Dincelli and Chengalur-Smith, 2020) found that the training was effective in increasing perception and knowledge about the disclosure of the information on the social media as well as lowering actual habits of disclosing information. The literature above shows that not all studies make use of perception, knowledge and security outcome altogether to evaluate the SETA programs and are mostly carried out in mandated settings. Moreover, majority of them mainly focus on a specific security concept such as phishing and self-disclosure.

Some security trainings have been designed and evaluated for technical cybersecurity issues targeted toward cybersecurity professionals and experts that we also discuss here. A recent study conducted in 2021 (Švábenský et al., 2022) implemented offensive security training employing a web based interactive system. The evaluation in Švábenský et al. (2022) was based on actual data generated by the students and the use of data mining and clustering algorithms. The results revealed that the training was effective and the outcomes were useful in further use of data mining in assessing students’ actual behaviors (Švábenský et al., 2022). Another cybersecurity training was carried out in a blended learning environment under the Hyflex teaching pedagogical principles (Nweke et al., 2022). Zoom, WhatsApp, Slack, Discussion forums and email technologies were employed in Nweke et al. (2022) along with in-person attendance of the class. The perception and the knowledge gained by the trainees showed positive results (Nweke et al., 2022). A technical cybersecurity training (Beuran et al., 2019) for capture the flag competition was developed and integrated into learning management system using sharable content object model (SCORM) packages. The undergraduate students evaluated the training (Beuran et al., 2019) based on the online features, hands-on training support, delivery and management with positive results. The same authors conducted another cybersecurity training (Tan et al., 2020) based on the principles of adaptive learning and executed it on Moodle learning management system. The authors used natural language processing techniques for generating knowledge based questions to see the effectiveness of the quizzes. The training was evaluated using 8 participants and yielded positive results (Tan et al., 2020). Recently in 2022, a full cybersecurity course (Tsai et al., 2022) was taught to the university students based on the attention, relevance, confidence and satisfaction (ARCS) model which is a motivational learning design framework. The protection motivation theory was employed to see the change in the cybersecurity intention and behavior of the students after a 16 week period in a controlled experiment (Tsai et al., 2022). The results of the experiment reported that the ARCS model based course was well received by the students and was successful in increasing their protection motivation.

The summary of the SETA initiatives is given in Table 1. It shows that the literature falls short in employing theoretical frameworks to evaluate SETA programs and fails to use program evaluation techniques in non-mandated settings. Moreover, the security trainings lack in employing university students who are one of the most vulnerable groups.

Protection motivation theory (PMT) was first presented by Rogers in 1975 to understand the change in attitude and behaviors of individuals (Rogers, 1975) in relationship to fear appeals. Fear appeals are the persuasive messages about a threat that can manifest itself if a recommended action is not taken (Rogers, 1975). Rogers added more components to PMT in 1983 (Rogers, 1983) that garnered majority of the scholarly community's interest; which is taken into consideration in this study (Fig. 1).

The core of the PMT theory is that individuals enact certain behaviors to avert the threats. The decision to carry out these actions is governed by threat appraisal and coping appraisal – which are the two core components of PMT (Rogers, 1983) and they play a key role in behavior change. Threat appraisal consists of threat vulnerability (PV) – the probability of the threat and threat severity (PS) – the severity of the consequences if the threat is manifested. The coping appraisal comprises of self-efficacy (SE) – the perceived ability of an individual to enact a protective response, response efficacy (RE) – the perception about the effectiveness of that response in averting a threat and response cost (RC) – the cost associated with performing the protective response (Rogers, 1983). The individuals enact threat and coping appraisals upon recognition of a threat which instills emotional state of fear (Rogers and Prentice-Dunn, 1997). When an individual feels that there is a probability of a threat and the consequences of that threat are severe, the threat appraisal come into action. At the same time the individual evaluates his self-efficacy in enacting a protective action keeping in view the efficacy of the action and associated cost (Rogers, 1983). The first component, threat appraisal allows the user to evaluate how serious a threat is. The user sees one's vulnerability towards a particular threat and evaluates how severe the consequences of threat can be. At the same time the second component allows the user to evaluate the strategies that can be adopted to avert that particular threat. The outcome of these two appraisals is the motivation to protect oneself with PS, PV, SE and RE having a positive while RC a negative influence on it (Rogers, 1983) as shown in Fig. 1.

The applicability of PMT is in situations where any kind of threat in involved (Rogers and Prentice-Dunn, 1997). The threat can be of medical, economical, personal and social nature. But the influence of PMT's component in different situations is varied (Floyd et al., 2000). One of the main assumptions of PMT is that the perception of the threat is a pre-requisite in evaluation of the coping appraisal (Floyd et al., 2000). The user must believe that one is vulnerable to the threat and it is severe, possesses the ability to perform protective behavior which is effective in averting the threat and the cost associated with it are compensated by the benefits that are garnered (Rogers, 1983). Another assumption of PMT is that fear arousal is not necessarily a pre-requisite in its application and thus may not be an essential element (Rogers and Prentice-Dunn, 1997). Reason being alternative such as previous experience of similar threats contributes towards invoking coping appraisal. PMT also does not take into consideration a complete rational decision making in engagement of protective behaviors. This is due to the fact that human biases such as cognitive motivational biases affect the threat and coping appraisals (Maddux and Rogers, 1983; Rogers, 1983). For example inability to enact protective behaviors can cause feeling of helplessness and resorting to the maladaptive responses.

PMT has been used in cybersecurity behavioral research (Boehmer et al., 2015; Johnston and Warkentin, 2010). An individual's threat appraisal comes into action when one perceives to be vulnerable to viruses on the Internet and considers the consequences of falling victim to viruses in terms of personal data loss. The individual then appraises the response that can be enacted in order to protect oneself by considering the presence of knowledge to install anti-virus (self-efficacy), the usefulness of the anti-virus in protection against viruses (response efficacy) and the time and effort required by the individual to make use of anti-virus (response cost). By carefully considering the cost benefit analysis of comparing the risks of non-protective behavior with the cost of eradicating the risk, the individual enacts/not enacts the cybersecurity behavior.

A number of studies have been conducted to study the effect of threat and coping appraisals on the cybersecurity intentional behavior mostly in organizational settings (Vance et al., 2012; Herath and Rao, 2009; Hina and Dominic, 2018; Siponen et al., 2014; Johnston et al., 2015; Li et al., 2019), for general users (Tsai et al., 2016; Verkijika, 2018; van Bavel et al., 2019) and to a lesser extent for students (Rajab and Eydgahi, 2019; Hina et al., 2019). The studies by Vance et al. (2012), (Herath and Rao, 2009) reported that different components of PMT differently predicted the security policy compliance in employees. These results have also been mirrored by the studies done on students and for general users (Tsai et al., 2016; Verkijika, 2018; Rajab and Eydgahi, 2019; Hina et al., 2019). Previous work in PMT shows positive results; and the theory has been touted as an appropriate theory to study cybersecurity behavior. Moreover, the components of PMT map well to the security concepts and the theory is favored by security researchers (Sommestad et al., 2015). Although the previous mentioned studies use PMT to investigate its components’ predictive power in explaining cybersecurity behaviors, they fail to investigate PMT in inducing behavior change (Haag et al., 2021). Another concern in these studies is the operationalizing of the PMT constructs due to their adoption from studies done in organizational contexts (Vrhovec and Mihelič, 2021). The cybersecurity behavioral intentions in organizational settings are mandatory in nature where as PMT explains voluntary cybersecurity behavior better and has been originally designed for volitional behavior (Sommestad et al., 2015). Another difference is that threat appraisal in non-mandatory settings affects individuals not in the same way as in mandatory settings. Reason being the vulnerability and severity both are personal at individual level pertaining to the person's device as compared to the organizations assets (Moody et al., 2018). For example “I would be subjected to an information security threat if I were to do what Mattila did” relates to the individual and “My organization would be subjected to an information security threat if I were to do what Mattila did” is related to the organizations (Moody et al., 2018). Due to these differences, the previous PMT based studies conducted with university going students are error prone. Moreover they have been conducted to explain behavior rather than the behavior change. This shows that while PMT is effective in improving cybersecurity behavior, gaps still exist in its applications to investigate behavioral change in non-mandated settings.

PMT has also been used to design awareness messages about the cybersecurity. A study by van Bavel et al. (2019) made use of PMT based fear appeals, threat and coping messages to enhance the secure navigation of an e-commerce site. The experimental evaluation of the study found that coping messages were effective in bringing positive behavioral change. In another study (Menard et al., 2017), messages based on PMT and self-determination theory were used to investigate the security motivations to use password managers. It was found that determinants of self-determination theory were effective in predicting protection motivation. With the empirically noted effectiveness of PMT based messages in enhancing the security behaviors, the design of the cybersecurity trainings also warrants inclusion of PMT based design principles.

One of the models used for evaluating training programs is Kirkpatrick's evaluation model (Kirkpatrick and Kirkpatrick, 2006). It is a classical model and was first proposed by Kirkpatrick in 1959 (Kirkpatrick, 1979). Since then it is frequently used in terms of citations and is popular among the educationists (Salas and Cannon-Bowers, 2001; Bates, 2004). Alliger et al. (Alliger et al., 1997) augmented it to an extended model by refining the terminologies but the original one is still mostly used [(Shelton and Alliger, 1993), p. 4]. One of the reasons for its popularity is that it is a system for demonstration of the training results and can make use of different data for evaluations carried out at different levels [76, p. 4]. There are four levels in Kirkpatrick's evaluation model which are; 1) Reaction, 2) Learning, 3) Behavior and 4) Result. Kirkpatrick's evaluation model has been used in the higher education sector (Chrysafiadi and Virvou, 2013; Arthur Jr et al., 2003) and studies have employed it as an evaluation model for meta-analysis of educational interventions (Howard and Gutworth, 2020). However, its applicability in evaluating the trainings in cybersecurity domain is limited. There has been numerous research calls for the use of Kirkpatrick's model in cybersecurity domain by many researchers (Abawajy et al., 2008; Mitrovic et al., 2019; Karjalainen and Siponen, 2011).

The discussion above has brought to light that previous SETA programs lack evaluation for their effectiveness using program evaluation techniques such as Kirkpatrick's model (Abawajy et al., 2008; Mitrovic et al., 2019; Karjalainen and Siponen, 2011). The reaction, learning and behavior change has not been gauged by the previous literature as shown in Table 1. Moreover, the use of PMT as a theoretical framework in SETA literature is in the mandated settings and does not cater for the university going students. Most studies employ PMT to study the behavior and fail to understand the predictive power of PMT's components in terms of behavior change. Therefore, this study evaluates a PMT based cybersecurity training intervention to induce behavioral change by employing Kirkpatrick's evaluation model.

This study aims to develop and evaluate cybersecurity training program based on protection motivation theory. Based on PMT, we aim to assess the effectiveness of the training program on the first three levels of Kirkpatrick's evaluation model. Our research objectives are;

  • 1.

    Assess the significance of PMT-based cybersecurity training for university going students in non-mandated settings.

  • 2.

    Assess the significance of protection motivation theory in changing cybersecurity behaviors statistically.

  • 3.

    Assess the effectiveness of PMT-based training in increasing cybersecurity knowledge and behavior taking into account program evaluation model (Kirkpatrick's model).

The study seeks to answer the following research question;

RQ. Does the PMT based cybersecurity training improve the three levels of Kirkpatrick's model - perception, knowledge and behavior - of the university going students?

Section snippets

Theoretical framework and hypothesis development

This study employs PMT as a theoretical lens to design cybersecurity awareness and evaluates the behavior change. PMT consists of two main components; Threat appraisal and coping appraisal. The threat appraisal and coping appraisal are coupled with cognitive processes and lead towards the motivation to protect oneself against cyber threats (Rogers, 1975; Rogers, 1983). The design of the training is based on design principles incorporated from (Heinrich et al., 2018). The design principles are

Methodology

We adopted quantitative methodology to answer research question of this study. A quasi experiment was chosen as a research method to test the hypotheses (Fig. 2). The details of the experimental design, instruments used and procedure are described in Sections 3.1-3.4.

Results

To test the hypotheses, we made use of structural equation modeling (SEM) technique and Wilcoxson signed rank test. Since we were interested in finding the relationships between PMT's constructs in explaining the cybersecurity behavior change, SEM is an appropriate technique. SEM makes use of measurement model and structural model which provide the information necessary to conduct hypothesis testing (Byrne). This study employs covariance based SEM (CB-SEM) which is an appropriate technique to

Discussion

This study made use of PMT based training to increase the cybersecurity knowledge and behavior of the undergraduate students. The evaluation of the cybersecurity training was based on Kirkpatrick's three levels in pre-post quasi experimental settings. Consistent with H2 and H3, the results revealed that students’ knowledge about the threats and countermeasures against them increased significantly which are in line with studies (Tschakert and Ngamsuriyaroj, 2019; Dincelli and

Conclusion and future work

This study reported on a quasi-experiment to evaluate the effectiveness of a PMT based cybersecurity training program. The evaluation was done on Kirkpatrick's first three levels of perception, learning and behavior. The training was very well received by the students from a tertiary institute. The pre-test and post-test results showed that training program was effective in enhancing the knowledge of the students regarding threats and their countermeasures. The PMT based models explained 48% of

CRediT authorship contribution statement

Naurin Farooq Khan: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing. Naveed Ikram: Formal analysis. Hajra Murtaza: Formal analysis, Investigation, Visualization, Writing – original draft, Writing – review & editing. Mehwish Javed: Data curation.

Declaration of Competing Interest

The authors of this manuscript claims that there's no financial/personal interest or belief that could affect the objectivity of the research

Naurin Farooq Khan: has done her MS in Computing and is currently working as a Senior Lecturer at Riphah International University. She has more than 10 years of teaching and research experience. Her research has been published in computing top journals such as Information and Software Technology and Artificial Intelligence Review. Her research areas are software engineering, cybersecurity, human behavior, cyberbullying and artificial intelligence. She is leading Cybersecurity Behavioral

References (127)

  • M.C. Howard et al.

    A meta-analysis of virtual reality training programs for social skill development

    Comput. Educat.

    (2020)
  • S. Hu et al.

    Security Education, Training, and Awareness Programs: Literature Review

    J.Comput. Informat. Systems

    (2021)
  • P. Ifinedo

    Understanding information systems security policy compliance: An integration of the theory of planned behavior and the protection motivation theory

    Comput. Secur.

    (2012)
  • J. Jansen et al.

    The design and evaluation of a theory-based intervention to promote security behaviour against phishing

    Int. J. Hum. Comput. Stud.

    (2019)
  • N.F. Khan et al.

    The Cybersecurity Behavioral Research: A Tertiary Study

    Comput.Secur.

    (2022)
  • H.A. Kruger et al.

    A prototype for assessing information security awareness

    Comput. Secur.

    (2006)
  • H. Li et al.

    Understanding compliance with internet use policy from the perspective of rational choice theory

    Decision Support Systems

    (2010)
  • L. Li et al.

    Investigating the impact of cybersecurity policy awareness on employees’ cybersecurity behavior

    Int. J. Inf. Manage.

    (2019)
  • J.E. Maddux et al.

    Protection motivation and self-efficacy: A revised theory of fear appeals and attitude change

    J. Exp. Soc. Psychol.

    (1983)
  • K. Parsons et al.

    Determining employee awareness using the human aspects of information security questionnaire (HAIS-Q)

    Comput. Secur.

    (2014)
  • K. Parsons et al.

    The human aspects of information security questionnaire (HAIS-Q): two further validation studies

    Comput. Secur.

    (2017)
  • M. Rajab et al.

    Evaluating the explanatory power of theoretical frameworks on intention to comply with information security policies in higher education

    Comput. Secur.

    (2019)
  • S. Saleem et al.

    Prevalence of cyberbullying victimization among Pakistani Youth

    Technol. Soc.

    (2021)
  • J. Abawajy et al.

    Investigation of stakeholders commitment to information security awareness programs

  • G.M. Alliger et al.

    A meta-analysis of the relations among training criteria

    Pers. Psychol.

    (1997)
  • E. Amankwa et al.

    A conceptual analysis of information security education, information security training and information security awareness definitions

  • I. Arpaci

    What drives students’ online self-disclosure behaviour on social media? A hybrid SEM and artificial intelligence approach

    Int. J. Mobile Commun.

    (2020)
  • W. Arthur Jr et al.

    Teaching effectiveness: The relationship between reaction and learning evaluation criteria

    Educ. Psychol.

    (2003)
  • W. Arthur Jr et al.

    Effectiveness of training in organizations: A meta-analysis of design and evaluation features

    J. Appl. Psychol.

    (2003)
  • G. Assenza et al.

    A Review of Methods for Evaluating Security Awareness Initiatives

    Eur J Secur Res

    (Oct. 2020)
  • R. Beuran et al.

    Supporting cybersecurity education and training via LMS integration: CyLMS

    Educat. Informat. Technolog.

    (2019)
  • C. Blackwood-Brown et al.

    Cybersecurity awareness and skills of senior citizens: a motivation perspective

    J. Comput. Informat. Systems

    (2021)
  • J. Boehmer et al.

    Determinants of online safety behaviour: Towards an intervention strategy for college students

    Behav. Informat. Techno.

    (2015)
  • C.L. Borgman

    Open data, grey data, and stewardship: Universities at the privacy frontier

    Berkeley Tech. LJ

    (2018)
  • B. Bulgurcu et al.

    Information security policy compliance: an empirical study of rationality-based beliefs and information security awareness

    MIS quarterly

    (2010)
  • A.J. Burns et al.

    Intentions to comply versus intentions to protect: A VIE theory approach to understanding the influence of insiders’ awareness of organizational SETA efforts

    Decision Sciences

    (2018)
  • Byrne, B.M., 2013. Structural Equation Modeling With AMOS: Basic Concepts, Applications, and Programming, 2nd Edition....
  • J. Chapman

    How Safe is Your Data?

    Cyber-security in Higher Education

    (2019)
  • G. CJ et al.

    Phishy-a serious game to train enterprise users on phishing awareness

  • J. Cohen

    Statistical Power Analysis For the Behavioral Sciences

    (2013)
  • W.A. Cram et al.

    Seeing the forest and the trees: a meta-analysis of the antecedents to information security policy compliance

    MIS Quarterly

    (2019)
  • T. Cuchta

    Human Risk Factors in Cybersecurity

  • J. D'Arcy et al.

    User awareness of security countermeasures and its impact on information systems misuse: A deterrence approach

    Inf. Syst. Res.

    (2009)
  • T. Daengsi et al.

    Cybersecurity Awareness Enhancement: A Study of the Effects of Age and Gender of Thai Employees Associated with Phishing Attacks

    Educat. Informat. Technol.

    (2021)
  • E. Dincelli et al.

    Choose your own training adventure: designing a gamified SETA artefact for improving information security and privacy through interactive storytelling

    European J.Informat. Systems

    (2020)
  • S. Egelman et al.

    Behavior ever follows intention? A validation of the Security Behavior Intentions Scale (SeBIS)

  • A. Farooq et al.

    Information security awareness in educational institution: An analysis of students’ individual factors

    2015 IEEE Trustcom/BigDataSE/ISPA

    (2015)
  • D.L. Floyd et al.

    A meta-analysis of research on protection motivation theory

    J. Appl. Soc. Psychol.

    (2000)
  • Fornell, C., 1985. “A second generation of multivariate analysis: Classification of methods and implications for...
  • Gjertsen, E.G.B., Gjære, E.A., Bartnes, M., Flores, W.R., 2017. “Gamification of Information Security Awareness and...
  • Naurin Farooq Khan: has done her MS in Computing and is currently working as a Senior Lecturer at Riphah International University. She has more than 10 years of teaching and research experience. Her research has been published in computing top journals such as Information and Software Technology and Artificial Intelligence Review. Her research areas are software engineering, cybersecurity, human behavior, cyberbullying and artificial intelligence. She is leading Cybersecurity Behavioral Research Group (CSBRG) at Riphah International University.

    Naveed Ikram: has more than thirty years of research and teaching experience. He is currently serving as a full Professor of software engineering at Riphah International University. His-research interests are evidence based software engineering, agile practices, global software development and requirements engineering. He is a senior member of IEEE and ACM. His-research work has been published in software engineering top journals such as Requirements Engineering, Journal of Software: Process and Evolution and IEEE Access.

    Hajra Murtaza: is working as a senior lecturer in the Faculty of Computing at Riphah International University Islamabad, Pakistan, since 2013. She earned her MS in Computing from M.A.J.U (now CUST) in 2004 and is now Ph.D. Scholar at Riphah International University. Her research interests include Machine Learning, Health Informatics and Pattern Recognition.

    Mehwish Javed: has done her MS in Software Engineering and is currently working as a Riphah junior Lecturer at Riphah International University. Her research interests involve software engineering and cybersecurity.

    View full text