Keywords

1 Introduction

The concept of digital maturity of educational institutions is becoming increasingly important due to rapid development of ICT in education. The European Commission has also recognized the significance of digital maturity and offers support throughout its policies and programmes (see for example [1, 2]). This paper defines digitally mature schools as schools with a high level of ICT integration, and with a systematic approach towards using ICT in school management and educational processes. The usage of ICT in school no longer depends on individuals – it requires a systematic approach, planned and implemented by a school and relevant stakeholders such as the Ministry, etc. Therefore, it is mandatory to identify the areas and levels that define the digital maturity of a school in order to plan improvement in the integration and use of ICT (see for example [3].

The problem analysis of the Croatian education system from the perspective of ICT integration revealed several major constraints [4]:

  1. 1.

    Lack of a systematic approach to introducing information and communication technology into education and other processes in schools;

  2. 2.

    Lack of guidelines/a system of development of digitally mature schools;

  3. 3.

    Low level of use of the opportunities information and communication technology offers in the education system; and

  4. 4.

    Lack of an officially accepted framework and strategy for the development of digitally mature schools.

With respect to this, the e-Schools programme, co-financed from the Structural Funds of the European Union [5], was initiated in 2015 in order to support the development of digitally mature schools in Croatia. Its aim is to introduce ICT into the school system in Croatia in the period 2015–2022 in order to help primary and secondary schools to become digitally mature schools. The e-Schools programme consists of the pilot project, which started in 2015 and the major project, which will be implemented in the period 2019–2022, based on the results of the pilot project. The pilot project involves 151 primary and secondary schools, which accounts for about 10% of all the schools in Croatia. It is expected that an additional 700 schools will be included in the major project [5].

The main deliverables of this programme are the Framework for Digitally Mature Schools and the Instrument for Measuring the Digital Maturity of Schools, based on the framework. The results are used to plan activities for schools that would enable them to make progress regarding the level of digital maturity.

2 Structure of the Proposed Model

In order to support schools in the development of digital maturity, a model for digitally mature schools in Croatia was developed. The model consists of the following elements: a framework for digitally mature schools, an instrument for measuring the digital maturity of schools, the initial self-evaluation of schools, the initial external evaluation of schools, as well as the final self-evaluation and external evaluation of schools (see Fig. 1).

Fig. 1.
figure 1

Elements of the model for digitally mature schools

Full size image

2.1 Framework for Digitally Mature Schools

The framework for digitally mature schools is a document that defines the areas, elements and levels of the digital maturity of schools. The methodological approach taken in the development of the framework was for the most part qualitative and included: (1) a comprehensive literature review on compatible frameworks and toolkits; (2) a meta-analysis of selected frameworks and toolkits; and (3) testing and feedback through a series of expert consultation, workshops, focus groups and semi-structured interviews. A complete and in-depth procedure of framework development is available in [10]. Here we provide only a brief overview.

Two frameworks provide the basis for the development of the Croatian framework: The European framework for digitally mature organizations – DigCompOrg [1], and eLearning Roadmap [6]. The Croatian framework therefore consists of five key areas: (1) planning, management and leadership; (2) ICT in learning and teaching; (3) development of digital competences; (4) ICT culture; and (5) ICT infrastructure. Each key area is described by several indicators. In total, thirty-seven elements are defined across all key areas. For example, the elements used to describe key area (2) ICT in learning and teaching are: awareness, planning, use, digital content, evaluation of students, student experience; and Special educational needs.

In addition, five levels of maturity are also defined with respect to [7, 8]: basic, initial, e-enabled, e-confident and e-mature. It is important to stress that different maturity stages have been established for schools to plan their journey: where are they now and where they would like to be in the future. The different levels should not be read as ‘judgmental’, but as the stages of a maturation process.

Next, each of the elements is described in terms of the five different maturity levels. This resulted in a rubric that summarizes and graphically represents the areas, elements and levels of the digital maturity of schools. For further details on the framework, see [9, 10].

Based on the description of each element through the 5 different maturity stages, a general description for each maturity level can be given [10]:

Level 1: Basic.

The school is not aware of the possibility of using ICT in learning and teaching or in its management processes. Therefore, the school does not take ICT into consideration in planning its growth and development. ICT is not used in learning and teaching. The educational staff (teachers) do not develop their digital competences. Online communication with the school is generally not possible. ICT infrastructure has not been provided yet and computers are used only in a few classrooms.

Level 2: Initial.

There is awareness of the possibility of using ICT in learning and teaching and in school management processes, but it has not yet been implemented. A small number of teachers use ICT in learning and teaching. There is an awareness of the need to enhance the digital competences of teachers and students. However, a system for the professional development of digital competences still does not exist. The school is still inactive in the online environment and access to its own ICT resources is limited. The ICT infrastructure is generally undeveloped and computers with internet access are only available in a few classrooms.

Level 3: e-Enabled.

The school is aware of the possibility of using ICT in all its activities, guides the development of its strategic documents and integration of ICT into these documents. ICT is used for working with students with special educational needs. The teachers advance their digital competences, develop digital content and have started introducing innovative teaching methods. The school participates in small ICT-focused projects. Access to different ICT resources is provided in most classrooms. Special attention is given to equipment maintenance and to controlling software licensing. The school is active online in terms of content presentation and communication.

Level 4: e-Confident.

The school very clearly recognizes the advantages of ICT in its activities and integrates ICT into strategic documents, as well as into its everyday activities. The teachers use ICT for advanced teaching and assessment methods, as well as develop their own content and protect it by copyright. There is also a shared content repository which can be used by teachers and students. The continuous professional training of teachers for the purpose of acquiring digital competences is planned and performed. Students are encouraged to develop those competences. Access to different ICT resources is provided in most classrooms, and the procurement and maintenance of ICT resources is planned. The school is active with respect to ICT projects. The school is also very active online in terms of content presentation and communication. Software licensing is controlled and the security aspects of ICT resource use are taken into consideration.

Level 5: e-Mature.

In strategic documents and school development plans, the school very clearly recognizes and requires the use of ICT in all its activities. Management practice relies on the integration and obtaining of data from all the school information systems. The approach to the development of digital competences of teachers and students is systematic, and professional training for the teachers and additional course activities for the students are available. The teachers use ICT for advanced teaching methods, for the development of new course content and the assessment of student accomplishments. Teachers and students regularly protect digital content by copyright. There is also a shared content repository available for use by teachers and students. Access to ICT resources from own devices is provided in all classrooms and other rooms in the school. The school independently plans and acquires ICT resources, which are available in nearly all classrooms and other rooms in the school. The entire school has a developed network infrastructure. An information security system has been developed and software licensing is systematically controlled and planned. The school is characterized by varied ICT project activities, cooperation between teachers and students, as well as between school and other stakeholders. This is achieved through the use of online communication tools and school e-services.

Although the Framework for Digitally Mature Schools was developed within the e-Schools pilot project, it can be applied in all Croatian schools. Schools can use the framework as a guide for planning and integrating ICT in teaching and learning, as well as in other processes. Policy makers and decision makers in the education system can use the framework to develop policies and initiatives leading to the successful integration of ICT into the education system.

The Framework for Digitally Mature Schools is the basis for the development of other elements of the model for digitally mature schools in Croatia.

3 Methodology

3.1 Instrument for Measurement of Digital Maturity of Schools

Since the main success criterion of the e-Schools programme is defined as an increased digital maturity of schools, it is necessary to accurately measure maturity at the beginning and at the end of the project in order to make a comparison between the initial and final level of digital maturity. For this purpose, an instrument for measuring the digital maturity of schools based on the framework has been developed.

The instrument for measuring the digital maturity of schools comprises an evaluation rubric consisting of 5 key areas and 37 elements described across 5 maturity stages and an accompanying questionnaire which main purpose is to offer a much simpler interface to the research participants than the rubric. During the pilot testing of the rubric, the research participants, in this case the school representatives, complained that filling the rubric was too complicated and therefore the questionnaire was developed to facilitate the data gathering from the research participants. The rubric is afterwards filled based on the answers of the school representatives from the questionnaire.

Once the rubric has been filled in based on the answers of the school representatives, the result algorithm calculates the maturity level for the school applying a taxicab geometry method. It shows the maturity level for each element, then a calculated level for all key areas, and in the end, the calculated digital maturity level of a school (see Fig. 2).

Fig. 2.
figure 2

Result display of instrument for measurement of digital maturity of schools – graph and text in respondents’ native language (comparison between self-evaluation and external evaluation)

Full size image

The instrument is accessible in cloud on a designated web server and will be used both in the self-evaluation and external evaluation of schools.

3.2 Initial Self-evaluation of Schools

An initial self-evaluation of schools is carried out in order to establish the baseline level of digital maturity before the project makes its impact. Another important reason is to educate principals and other school staff, and to raise awareness of digital maturity among the participating school staff members. The most important reason for self-evaluation in the e-Schools programme is its contribution to school improvement, to be achieved through reflection on their practice and identification of areas for action All 151 schools in the pilot project participated in the initial self-evaluation. School principals completed the questionnaire with help of other members of the school staff in June and July of 2016.

3.3 Initial External Evaluation of Schools

The external evaluation was carried out in the 151 schools participating in the pilot project in October 2016 by 23 external evaluators [10]. All external evaluators were trained in a workshop in order to ensure objective evaluation based on common criteria. The evaluators interviewed school principals and other members of school staff using the same instrument that was used in the self-evaluation. Evaluators also gathered some evidences supporting answers in the instrument, such as school curriculum and plans. Schools were asked to prepare the evidence to support their responses in advance. Additionally, the evaluators inspected the existing infrastructure and resources.

The validity of the questionnaire was assessed by calculating Cronbach’s alpha to estimate the raters’ consistency. All constructs exceeded the threshold of 0.60, which is considered to be adequate with respect to the very early stage of research and the context [11]. Therefore, the data could be analyzed with confidence.

4 Results

As is evident from Table 1, following the self-evaluation, 50% of the schools were found to be at the initial maturity level. A further 45% of the schools are e-enabled. Only 3% of the schools are e-confident, and only 1% are e-mature. There are no schools at the basic digital maturity level.

Table 1. Results of self-evaluation and external evaluation of schools (N = 151)
Full size table

Table 1 also shows the distribution of digital maturity levels as calculated in the external evaluation of schools. According to the external evaluation, the majority of schools (82%) are at the initial level, and 18% of schools are at the e-enabled level.

Besides displaying the results for each school, the online system offers several other features. Namely, besides the in-depth analysis of the current level of digital maturity of a school, it is also possible to compare the results of self-evaluation and external evaluation. Additionally, each school is able to compare its performance against that of other schools, choosing between all schools, those from the same county, the same level, etc.

Moreover, based on the current maturity level of each element, the school receives a set of recommendations from the system on how to move to the next level. In other words, the schools receive information about the elements that require attention in order to be able to repair them.

Therefore, the instrument can be used as a tool to evaluate the school’s digital maturity level, but it can also serve to identify areas for school improvement that will result in raising the school’s level of digital maturity as well as its overall reputation and academic results.

5 Discussion

The results of self-evaluation showed higher levels of digital maturity in comparison with results of external evaluation as can be seen in Table 1. While self-evaluation resulted in 45% of e-enabled schools, external evaluation records only 18% of the e-enabled schools. Higher levels of digital maturity (e-confident and e-mature) were displayed by 4% of schools in the self-evaluation, as opposed to no schools in the external evaluation. The external evaluation recorded a majority of schools (82%) at the initial level, while the self-evaluation recorded 50% of schools at the initial level.

There are many factors that contributed to notable differences between the results of self-evaluation and external evaluation. First of all, the school staff participating in self-evaluation lack experience in conducting self-evaluation since most schools in Croatia are not included in any self-evaluation practice. Secondly, the concept of digital maturity in general is a new concept in the local context, and specifically some areas and elements of digital maturity are unfamiliar. For instance, there are no requirements for school principals related to their management and leadership competences and as a result most of them lack management and leadership competences. Thirdly, while the external evaluators were trained in a workshop for performing external evaluation, school staff were provided only with written guidelines on how to conduct self-evaluation. And finally, except for explanations for the most novel terms in the instrument, the respondents did not have any kind of support during the self-evaluation.

The next step is to refine and upgrade the instrument based on the comments and suggestions gathered during the external evaluation and self-evaluation. The main aim is to minimize the difference in the results between the self-evaluation and external evaluation due to the instrument itself.

Refinements and upgrades will be performed in two basic directions. The first direction involves modifying the online system hosting the instrument and the feedback based on user feedback. The introductory page has been completely re-written to introduce the evaluation procedure and its intention to the end-users in greater detail. Some questions/elements have been rewritten and better explained, taking into account the different backgrounds of the end-users. A dictionary of ICT terms has also been added and a hover option is available whereby the users can get an explanation by just hovering over the item. The radar chart showing the difference between self- and external evaluation has been added to the result display of instrument shown in Fig. 2. Additionally, with a detailed insight into the state of each of the 37 elements at the digital maturity level, the school also gets guidelines for improvement. The online system points to the elements that need to be improved to help the school plan its own improvement.

The second direction involves an introductory webinar which has been prepared for all end-users in order to better explain the process, the aims of the e-Schools programme and to motivate them for answering the questions in the instrument. The webinar participants will be able to ask and receive answers to all relevant questions. Participation in the webinar will raise the capacity of the school staff for conducting self-evaluation.

The final self-evaluation and final external evaluation of schools will be performed towards the end of the pilot project. The comparison between the results of the initial and final evaluation of the digital maturity of schools will hopefully show an increase in digital maturity, which is one of the main objectives and the main success criterion of the project. It will also show the effectiveness of measures taken to minimize the difference in the results between the self-evaluation and external evaluation.

In respect of the results obtained in the initial evaluation of schools, it is necessary to raise the level of digital maturity in Croatian schools by implementing a series of measures and activities. The measures and activities implemented within the pilot program address all the areas identified in the Framework for Digitally Mature Schools, including raising awareness of the importance and opportunities of using ICT in learning and teaching, ICT governance in, raising the level of digital competence of teachers, and modernizing the existing ICT infrastructure. The following resources and services have been provided: educational opportunities for teachers and school principals, document templates for the strategic planning of the introduction and use of ICT, digital educational contents, teaching scenarios, adequate ICT infrastructure, etc.

6 Conclusion

The main conclusion was that in spite of significant differences in the results between self-the evaluation and external evaluation, the identified general levels of digital maturity in Croatian schools are closer to the lower end of the 5-level scale, which indicates that it is necessary to raise the level of school maturity in Croatia by implementing a series of measures and activities within the school system. The final self-evaluation and external evaluation will be conducted towards the end of the pilot project with the aim of measuring the difference between the levels of digital maturity of schools at the beginning and at the end of the pilot project. It is hoped that the measures and activities implemented within the pilot program addressing all the areas identified in the Framework for Digitally Mature Schools will lead to increased levels of digital maturity. In case there is no significant increase in digital maturity, the Framework for Digitally Mature Schools, the instrument for measuring the digital maturity of schools and the project activities aimed at the development of digital maturity should be further explored, refined and revised.

The second important conclusion is that there is a notable difference between the results of the self-evaluation and external evaluation. This paper discusses the reasons for the difference identified by the researchers. A number of measures will be taken based on these reasons, with the aim of minimizing the difference in the results between self-evaluation and external evaluation. The comparison of the final results of self-evaluation and external evaluation will show whether the reasons were identified correctly and the measures implemented effectively. If a significant difference persists, the reasons and the measures will be additionally explored.