Abstract
This study examines the impact of lifelong learning on the professional lives of employed and unemployed individuals. Lifelong learning is a crucial factor in securing employment or enhancing one’s existing career prospects. To achieve this objective, this study proposes the implementation of a multi-criteria decision support system for the evaluation of training courses in accordance with their capacity to enhance the employability of the students. The methodology is delineated in four stages. Firstly, a ‘working life curve’ was defined to provide a quantitative description of an individual’s working life. Secondly, an analysis based on K-medoids clustering defined a control group for each individual for comparison. Thirdly, the performance of a course according to each of the four predefined criteria was calculated using a t-test to determine the mean performance value of those who took the course. Ultimately, the unweighted TOPSIS method was used to evaluate the efficacy of the various training courses in relation to the four criteria. This approach effectively addresses the challenge of using extensive datasets within a system while facilitating the application of a multi-criteria unweighted TOPSIS method. The results of the multi-criteria TOPSIS method indicated that training courses related to the professional fields of administration and management, hostel and tourism and community and sociocultural services have positive impact on employability and improving the working conditions of citizens. However, courses that demonstrate the greatest effectiveness in ranking are the least demanded by citizens. The results will help policymakers evaluate the effectiveness of each training course offered by the regional government.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Lifelong learning is a core concept that describes individual learning during the entire lifecycle from early socialisation and pre-school education to retirement age in terms of employment [13]. The term lifelong learning is broadly defined and refers to ‘all learning activity undertaken throughout life, to improve knowledge, skills, and competencies within a personal, civic, social and/or employment-related perspective’ [11]. This concept addresses three fundamental objectives of education: i) personal fulfilment and development throughout life (cultural capital); ii) active citizenship and inclusion (social cohesion); and iii) employability and economic growth (human capital). In this study, we focus on the impact of lifelong learning on the third objective, namely, the working life of both employed and unemployed people, which is considered essential either for obtaining a job or for improving the job one already has. In this regard, many national and regional governments in Europe dedicate part of their policy budgets to providing training courses that can offer new skills to their citizens.
The importance of lifelong learning for improving employability and the quality of employment is evident. Numerous studies have confirmed the positive effect of lifelong learning education on improving employability. One example is the study by Järlström et al. [16], which asserts that a highly skilled and knowledgeable worker is an asset to any organisation, and skillsets are often associated with promotion, salary increases, and career success. Babos et al. [2] also concluded that most individuals who participated in lifelong learning recognised it as a positive contributor to their employability. The work of Sharma et al. [37] uses micro-credential courses for upskilling and reskilling, demonstrating their positive impact on enhancing employability. To achieve this, interviews were conducted with 65 participants from India, Nigeria, the United Arab Emirates, and the United Kingdom to explore how micro-credentials can be a valuable addition to the higher education ecosystem. Therefore, to provide quality training, it is crucial to have the necessary tools to measure the effect of these educational training courses on employment.
The accelerating pace of scientific and technological advancements and the resulting societal and economic (or labour market) changes at any given time necessitate lifelong learning. This is why policymakers continually face the problem of ensuring that individuals acquire the relevant skills and knowledge to improve employability. Hence, there is a need to design and select education training courses that adapt to the requirements of the labour market. To address this issue, the public employment service of the region of Extremadura (Spain) provided historical data and requested a scientific analysis of the impact that the realisation or not of different continuous training courses has on the working lives of those enrolled. The key contribution of this work is the development of a multi-criteria decision support system that will help policymakers evaluate each training course offered by the regional government and rank them according to improvement in the labour life of students. This is important because only with precise data can the regional government determine where more resources should be invested and which courses are less efficient. The proposed methodology combined the challenges of using large amounts of data in the system and the application of a multi-criteria unweighted method.
One of the most commonly used multi-criteria methods is the TOPSIS method, which is based on distances to ideal and anti-ideal solutions. In TOPSIS, as in other multi-criteria decision (MCDM) methods, the objective is to help decision makers choose the best option among several alternatives based on various criteria that may conflict with one another. These criteria may also have different levels of importance. Therefore, assigning criteria weights is a crucial step in any MCDM method. The weights can significantly affect the final decision even with slight changes. A common method of determining weights is the use of expert opinion. Experts in the field can define the preferences of the criteria using specific methodologies, such as reaching an agreement (Delphi method [33]) or comparing criteria in pairs (AHP [32]). However, these expert-based methods have a major drawback: they introduce subjective views that can bias the process.
Many researchers in the field of multi-criteria analysis have been interested in finding more objective methods for determining criteria weighting. Examples of these methods include the Entropy method [36], the LINMAP method [38], and recent approaches such as the IDOCRIW method [44], the Bayesian approach [42] and the FUCOM method [25].
A different approach to address the issue of weight assignment in the TOPSIS method was proposed by Liern and Pérez-Gladish [20] and Benítez and Liern [3]. This alternative approach does not require decision makers to specify the exact values of the criteria weights. Instead, it only requires providing reasonable ranges for them, leading to an unweighted TOPSIS (uwTOPSIS) method.
This remainder of this paper is structured as follows. In Section 2, we review related works on the criteria selected to measure the effect of training courses and the selection of the control group to measure that effect, and other studies using uwTOPSIS as multi-criteria decision analysis method in different scientific domains. In Section 3, we describe the database and the courses considered in this study. In Section 4, we present the methodology, define the working life curves, describe how the control groups are defined and explain how the effectiveness of the courses is assessed. In this section, we also briefly describe the uwTOPSIS method used for ranking the courses. The results are presented in Section 5, and we provide the conclusions in Section 6.
2 Related works
Analysing the effect of a certain factor or treatment on a sample is common in many scientific fields. In such analyses, it is essential to define a control group to contrast the results. For example, in pharmacy, a control group helps determine whether a new medication is effective and in agronomy, it is used to study the effects of different fertilisers. In these scientific fields, it is common for researchers to have control over both the sample they are studying and the control group that they use to compare results. In this way, it is possible to select homogeneous samples with similar individuals so that the comparisons make sense and the possible differences can be attributed only to the factor or treatment being analysed.
However, in social sciences, the situation is often much more complicated. For example, when assessing the effects of participating in a certain activity on individuals, researchers may have limited control over the sample being analysed, which can lead to heterogeneity among participants. In addition, in some studies, the control group has not been previously defined, making the comparison of results with those of individuals who have not participated in the activity a nontrivial process.
In this study, we present a methodology that allows us to assess the impact that the realisation or not of different continuous training courses, offered by the public employment service of the region of Extremadura (Spain), has on the working life of the people enrolled in them. To evaluate the effectiveness of training courses, it is necessary to determine measurable criteria. Many studies have based the evaluation of the criteria on the subjective perceptions of actors such as students, teachers and organisers [12, 18, 37]. In contrast, the proposed methodology is entirely data-driven. This involves constructing a database that includes information from various departments of the regional government, such as education, labour and social security
When assessing the usefulness of a training course, one might be tempted to analyse the quality of working life before and after taking the course and to check whether there has been improvement, according to predefined criteria. However, this can be misleading for two main reasons: first, the quality of working life is likely to improve over time, and second, it is crucial to compare these improvements with those of individuals who have not completed the course. In other words, how much (if any) improvement can be attributed to the training course, or can it all be explained by the natural improvement in the working life of a person?
To accurately determine whether the obtained results were due to the course, it is necessary to compare the results with those of a control group consisting of individuals who have not completed the course. Several authors have addressed the problem of determining the control group for this type of study in various ways. For example, [31]studied the evaluation of employment programmes in the Netherlands, with the control group defined as individuals in the same age range as the study group who did not participate in any employment programme in 2008. This study was limited to a specific period (only 2008 was considered). In contrast, another study [10] considered the effectiveness of vocational training and defined the control group as individuals who enrolled in the training course but did not attend (no-shows). In other cases [40], the analysis focused on a specific type of course in which learning certain skills was evaluated. In these instances, the determination of the control group is simpler because it is sufficient to consider individuals who took similar courses in which that skill was not taught. Similarly, in another study [34], the effectiveness of audiovisual learning programmes was analysed using an experimental design with a post-test-only control group. The study involved two classes: the experimental group that received the specific treatment and the control group that did not.
Thus, determining the control group is not a simple task. This presents two major challenges. The first is how to define a control group in a manner that ensures that similar individuals are compared and that the comparison is sensible. This is difficult because, in our case, the students taking the course are not a homogeneous group; they have different backgrounds, ages, and other characteristics. Consequently, the construction of the control group should be individualised, meaning that a specific control group should be defined for each individual taking the course. The second challenge is even more complex. If the people in the control group have not taken the course (by definition), how can we establish a valid ‘before’ and ‘after’ comparison?
To address these challenges, we first define a ‘working life curve’ (WLC) that allows us to quantitatively describe the working life of an individual. Using these WLCs, we can measure the similarity between two distinct individuals. In addition, using a methodology based on K-medoids clustering [26], we define a control group for each individual to facilitate comparison. The performance of each course, in relation to the four predefined criteria, is calculated as the average performance of the individuals who took the course, as assessed using a t-test comparison. Finally, we propose the uwTOPSIS method to rank the effectiveness of different training courses according to four criteria [3, 20].
The reason for selecting the uwTOPSIS multi-criteria method over the existing methods was the challenge of determining precise subjective weights from experts. The proposed multi-criteria method ranks decision alternatives based on the classical TOPSIS approach; however, this method does not require the introduction of fixed prior weights. Instead, it uses lower and upper bounds to express the varying importance of the criteria, thereby allowing for more objective assessment without the bias introduced by subjective weighting. Developed in 2020, this innovative multi-criteria method differs from other known multi-criteria techniques, such as VIKOR, PROMETHEE or MOORA, as it does not assign fixed weights a priori to a criterion; instead, upper and lower bounds are set.
Recent studies related to the case study have applied the uwTOPSIS method for the reasons explained above. For example, Blasco-Blasco et al. [5] proposed an academic performance indicator for science and engineering students at the Industrial University of Santander (Colombia). Similarly, López-García et al. [23] used the uwTOPSIS method to develop a methodology for the early detection of student failure. Other studies in different fields, such as sustainability and tourism, have also demonstrated the efficiency of the uwTOPSIS method in multi-criteria decision-making processes [4, 21, 22, 27].
3 Data
The data used in this study are a subset of the data stored in a data warehouse that we built in collaboration with the regional government of Extremadura to support a data-driven strategy aimed at reducing unemployment in the Extremadura region [8, 9].
All details regarding the construction of this data warehouse, including the identification of the data sources, the description of the data collected, the design of the data warehouse schema and the creation of an automated data collection process, are detailed in Section 3 of [9].
In summary, this data warehouse contains information on 120,927 citizens of the Extremadura region. Specifically, this information for each citizen in the data warehouse includes the following:
-
Educational from lower and upper secondary, vocational, and official language schools
-
University degrees (bachelor’s, master’s and doctorate) awarded by the University of Extremadura
-
Contracts, social security and visits to the Employment Office
-
Training courses offered by the Employment Office, aimed at helping individuals easily enter and remain in the job market.
3.1 Datasets
In this study, a subset of the information stored in the aforementioned data warehouse was used. In particular, the information was obtained from the following four different datasets, with their fields described in Table 1:
-
(a)
DS1 This dataset includes citizens who have not participated in any training courses. It contains a row for each regulated study level of each citizen who has not taken a training course. For instance, a citizen with lower secondary education and bachelor’s, and master’s degrees will appear four times in this view. This table contains 196,120 rows corresponding to studies out of 112,638 different citizens.
-
(b)
DS2. This dataset includes citizens who have participated in training courses. It contains a row for each regulated study level of each citizen who has taken at least one training course. For example, a citizen with lower secondary education and three training courses will appear five times in this view. This table contains 21,970 rows corresponding to studies of 8,289 different citizens.
-
(c)
DS3. This dataset includes personal and aggregated data of all the citizens considered. It contains one row for each of the 120,927 citizens that appear in the two previous views.
-
(d)
DS4. This dataset includes the contract history for each citizen. It contains one row for each contract held by the 120,927 citizens. This table contains 820,985 rows. It should be noted that the number of contracts differs significantly between citizens with and without training courses. Thus, out of the 8,289 citizens with training courses, 8,151 appear at least once with a contract, whereas 138 do not. Conversely, of the total of 112,638 citizens without training courses, 67,253 appear at least once with a contract, whereas 45,385 do not.
3.2 Courses
For this study, 113 training courses offered by Extremadura’s employment service (SEXPE) in Spain were considered. Because we set a study horizon of 1 year and were interested in assessing the impact of training courses, we filtered the data to keep only those citizens who completed a training course more than one year before data collection, making a total of 6748 citizens considered. The number of students per course ranges from 10 to several hundred students depending on the course. The basic descriptive statistics of the number of students in each course are presented in Table 2. In addition, all courses are assigned to a professional family that each job contract is classified into. Assignments were completed according to the proximity of the course description to the corresponding professional family. The list of professional families and their corresponding codes are presented in Table 3.
4 Methodology
To assess the impact of a training course on a citizen’s employment track, we propose a measure to compare the working track of different citizens: the WLC. Based on this measure, we then define a method to compute a control group for each citizen by clustering the citizens according to their WLCs. Finally, we discuss how to assess the impact of a training course according to multiple criteria. The methodology is summarized in four stages, as shown in Fig. 1.
4.1 Working life curves
We define the WLC of person i in the database as a function \(WLC_i: [0,T_{\max }]\longrightarrow [0,1]\) that assigns to each day t of the working life of person i the quotient between the number of days that a person has been employed and the number of days spanned since the beginning of their working life. That is, if N(t) is the number of days under employment until day t, then we have the following:
An example of a typical WLC is shown in Fig. 2.
When constructing the WLC, two points are convenient to emphasise:
-
Determination of the date of origin of working life. Normally, the beginning of working life is defined as the earliest between the start date of the first labour contract and the end date of the last regular study taken (not including the possible SEXPE training courses taken).
-
In many cases, the end date of a contract is not recorded and appears in the database as NULL. More than 25% of the contracts do not have an appropriately assigned end date. Therefore, a mechanism to assign an end date to the contract must be established. In this case, we proceed as follows:
-
1.
We consider as adjusted end date the midpoint of the quarter following the start of the contract. That is, if month denotes the month in which the contract started, then we have the following:
-
If month \(\in \{\text {January, February, March}\}\), then date_adjusted\(=\) 15th of May.
-
If month \(\in \{\text {April, May, June}\}\), then date_adjusted\(=\) 15th of August.
-
If month \(\in \{\text {July, August, September}\}\), then date_adjusted\(=\) 15th of November.
-
If month \(\in \{\text {October, November, December}\}\), then date_adjusted\(=\) 14th of February of the following year.
-
-
2.
The beginning date of the next contract (if any) is also considered.
-
3.
The final date is considered the earliest between the two above dates: the earliest date between date_adjusted and the date at the beginning of the next contract.
It may happen that reality does not conform to this methodology and that there are reasons why no contract end date was communicated. However, in our opinion, this methodology is a systematic and sensible way to assign end dates.
-
1.
4.2 Definition of the control group
The next step is to compare the WLC of person \(p_i\) who has completed a particular training course with that of other people who have not. In particular, we want to determine whether from the completion of a course, at instance \(t_i\) (measured in days), the working life of an individual improves in horizon h, that is, in the interval \([t_i, t_i+h]\), when compared with a group of people who did not take that course and, up to moment \(t_i\), had a labour behaviour similar to that of the individual in question.
Assume that from a course \(C_j\) under study, we take an individual \(p_i\) who has taken the course at instance \(t_i\) (measured in days since the beginning of his or her working life). Note that, in addition to \(t_i\), for each person in the database, we know the sex, age and educational level at each instance. The control group for \(p_i\) (denoted \(CG_i\)) is obtained in three stages.
First, we filter the database by considering only individuals of the same sex as \(p_i\), age within 5 years of the age of \(p_i\), the same educational level at time \(t_i\) as person \(p_i\), and working life at least as long as that of \(p_i\) plus horizon h (in days). This defines the ‘initial control group’ for \(p_i\) (denoted\(CG_{i0}\)).
However, given the massive size of the database, the initial control group \(CG_{i0}\) is expected to be large. Therefore, a second filter is required. The aim is to select from \(CG_{i0}\) individuals whose WLCs are most similar to the WLC of the person under investigation. The next problem is determining the number of ‘nearest neighbors’ to \(p_i\) because this depends heavily on the course and the individual in question.
To address these issues, we propose a methodology based on K-medoids clustering [26]. The general idea is to consider, given person \(p_i\), the WLCs corresponding to \(CG_{i0}\cup \{p_i\}\), all of them in the interval \([0, t_i]\). Then, the control group of \(p_i\) (denoted \(GC_i\)) is the optimal cluster containing \(p_i\) for K-medoids clustering. The optimality is determined using the GAP statistic [39], and the PAM algorithm [35] is used for the clustering procedure.
Nevertheless, we must note that the size of the initial control group is in the thousands, or even in the tens of thousands. Thus, obtaining the GAP statistic for each person is computationally extremely expensive. To overcome this challenge, pilot tests were conducted, and the optimal number of clusters did not vary greatly from one individual to another. Therefore, instead of determining the optimal number of clusters for each participant who took a specific training course, we computed such an optimal number for a randomly selected sample and took the mode as the optimal number of clusters for the given training course. This number was used to determine the control groups of all participants in the training course.
Figure 3 summarises the three stages involved in determining the control group for a specific person \(p_i\) taking course \(C_j\). Algorithm 1 describes the steps to be followed to obtain the control group of all individuals who have completed a given course.
4.3 Post-course assessment
Once we have devised a methodology for comparing individuals, we are now in a position to measure the extent to which the training course has had a positive effect on a particular person in some respect, or not.
Now, let us consider a particular course \(C_j\), and a particular person \(p_i\in C_j\) with their control group \(CG_i\). Assume that \(p_i\) has completed the training course at time \(t_i\) (measured in days since the beginning of the WLC). We now want to determine whether the labour conditions of \(p_i\) improved after the course, within some horizon h (i.e. in the interval \([t_i, t_i + h]\)), compared with the improvement in the persons in the control group \(CG_i\) during the same period.
The improvement above will be measured according to four criteria. Namely,
-
C1: Total number of days employed
-
C2: Total number of days under permanent contract
-
C3: Number of days in a position related to the course taken
-
C4: Average number of days between contracts
Criteria C1, C2 and C4 analyse the employability of a training course in general, regardless of its professional family, whereas criterion C3 specifically analyses the employability within the professional family of the course under consideration. With this criterion, we want to analyse the effectiveness of the courses as a facilitator of improvements in working conditions in their own professional family. We include both general and specific conditions because we note that, in many cases, it may happen that courses, although related to one professional family, provide students with resources and transversal skills that are useful in other sectors.
To determine the degree to which a course has been effective for an individual, for a given criterion, we calculate the probability with which we can be sure that the individual has performed better on that criterion than his or her control group. To this end, we perform a t-test to determine whether the value obtained from the individual on a criterion was statistically superior (i.e. higher for criteria C1, C2 and C3 and lower for criterion C4) than those of the control group.
We use the P-value of the corresponding t-test as the performance measure. This implies that the criteria are of cost type-that is, the higher the value, the worse the performance. Finally, the course performance is calculated according to the criteria as the average performance of the participants.
4.4 Unweighted TOPSIS
In this section, we briefly describe both the classical TOPSIS method and the uwTOPSIS formulation proposed by Liern and Perez-Gladish in [20] and the implementation described in [3].
Consider a multi-criteria problem with n alternatives \(A_i\), \(i = 1,\ldots , n\), and m criteria \(C_j\), \(j = 1,\ldots ,m\). Each criterion may be beneficial (i.e. ‘the more the better’) or a cost (i.e. ‘the less the better’). Let \(J_+\) and \(J_-\) denote the sets of indices j corresponding to benefit and cost criteria, respectively.
The classical TOPSIS algorithm proceeds as follows [15, 41]:
-
Step 1:
Determine decision matrix \(X=[x_{ij}]_{n\times m}\) where element \(x_{ij}\) is the performance rating of alternative \(A_i\) at criterion \(C_j\).
-
Step 2:
Construct the normalised matrix \(\hat{X}\) as follows:
$$\begin{aligned} r_{ij} \frac{x_{ij}}{\sqrt{\sum _{k = 1}^n x_{kj}^2}}\in [0,1],\quad 1\le i \le n, \, \,1\le j\le m. \end{aligned}$$ -
Step 3:
Determine the positive ideal solution, \(A^+=(A^+_1,\)\( A^+_2, \ldots ,A^+_m)\), and the negative ideal (or anti-ideal) solution, \(A^-=(A^-_1, A^-_2, \ldots ,A^-_m)\), as follows:
$$\begin{aligned} A^+_j = {\left\{ \begin{array}{ll} \displaystyle \mathop {\text{ max }}_{1\le i \le n} r_{ij}& \text {if } j \in J_+\\ \displaystyle \mathop {\text{ min }}_{1\le i \le n} r_{ij}& \text {if } j \in J_- \end{array}\right. } \qquad A^-_j = {\left\{ \begin{array}{ll} \displaystyle \mathop {\text{ min }}_{1\le i \le n} r_{ij}& \text {if } j \in J_+\\ \displaystyle \mathop {\text{ max }}_{1\le i \le n} r_{ij}& \text {if } j \in J_- \end{array}\right. } \end{aligned}$$ -
Step 4:
Given a weight vector \(\omega \in \Omega \), where
$$\begin{aligned} \Omega =\left\{ (\omega _1, \omega _2, \ldots , \omega _m)\in \mathbb {R}^m, \quad 0\le \omega _j\le 1, \quad \sum _{j = 1}^m\omega _j=1\right\} , \end{aligned}$$we calculate the weighted normalised matrix, \([w_j{r}_{ij}].\)
-
Step 5:
Determine the weighted distances between each alternative \(A_i\) and the ideal and anti-ideal solutions as follows:
$$\begin{aligned} d_i^+({\omega }) = \sqrt{\sum _{j = 1}^m (\omega _j{r}_{ij} - \omega _jA^+_j)^2}, \quad d_i^-({\omega })= \sqrt{\sum _{j = 1}^m (\omega _j{r}_{ij} - \omega _jA^-_j)^2}. \end{aligned}$$ -
Step 6:
Compute the score for each alternative as follows:
$$\begin{aligned} R_i ({\omega }) = \frac{d_i^-({\omega })}{d_i^-({\omega })+d_i^+({\omega })},\quad i = 1,\ldots ,n. \end{aligned}$$(2) -
Step 7:
Rank alternatives according to scores \(R_i({\omega }) \).
Remark 1
In this study, following the original proposal in [15], we use vector normalisation in Step 2. However, other normalisation procedures have also been successfully applied [6, 24]. Likewise, the Euclidean distance in Step 5 can be replaced by many other distances [24].
TOPSIS is very easily implemented and applied; however, weight determination (Step 4) is a significant concern because a small change in any of them can lead to different final rankings. Inspired by [20], the proposed method attempts to avoid subjective weight assignment by treating weights as decision variables. For this purpose, we consider vector \(\omega \) (Step 4) as a vector of decision variables. Therefore, for each alternative, expression (2) defines a function as follows:
Given the extreme values of the function \(R_i\) in (3) for each alternative, we can define a ranking. Thus, we must to compute the following:
Note that parameters \(l_j\) and \(u_j\), \(j =1\ldots , m,\) are lower and upper bounds for weights, respectively. All \(l_j\) values should be positive; otherwise, some criteria will not be considered. However, the upper bounds should not be very large because assigning a large weight to one criterion has similar consequences to ignoring the other criterion.
From (4) and (5), we define the following interval for each alternative:
Finally, to rank the different alternatives, Step 7 should be extended to ordering intervals rather than real numbers.
A wide range of methods for ordering intervals \(A=[a_1, a_2]\) and \(B=[b_1, b_2]\) is available in the literature [14, 30]. However, in this study, we follow the proposal in [7] as follows:
where \(k_1\) and \(k_2\) are two pre-established positive constants such that \(k_1 + k_2 = 1\).
Definition 1
Given the alternatives \(\{A_i\}_{i=1}^n\) and the set of interval-valued proximities to the ideal solution \(\{\bar{R}_i\}_{i=1}^n\) given by (6), we conclude that alternative \(A_i\) is preferable to alternative \(A_k\) whenever \(\bar{R}_i\succ \bar{R}_k\). In addition, we define the uwTOPSIS indicator \(R^{uw}\), of alternative \(A_i\) as follows:
The \(k_1\) and \(k_2\) (or \(1-k_1\)) values can be considered a measure of the decision maker’s propensity to consider a more pessimistic (or conservative) or optimistic scenario. The choice of one or another value of \(k_1\) depends on the problem in question and, above all, on the existence of external constraints that may incline the decision-maker towards one of the extremes of the interval obtained in (6). In our case, in the absence of such constraints, we rank the intervals by taking \(k_1=k_2=0.5\) in (7). Specifically, we use as the uwTOPSIS indicator the midpoints of the intervals \([R^-_i, R^+_i]\) (see (8)).
The main drawback associated with this method is the computational difficulty of solving 2n nonlinear optimisation problems. To overcome this problem, we used R statistical software [29] together with the package nloptr [43] which is an R port of the open-source nonlinear library developed by S.G. Johnson [17]. The minimising algorithm used to solve problems (4) and (5) was the Constrained Optimisation by Linear Approximations algorithm developed in [28].
5 Results
5.1 Control groups
After the first filtering (sex, age, educational level, and length of working life), the potential size of the control groups had a very skewed distribution to the right, with a mean of 2170 persons and a median of 1662 persons Fig. 4.
In all these potential control groups, the Euclidean distance between the individual’s labour curve and the labour curves of the persons included in the corresponding potential group was calculated. The potential group was then ordered in increasing order.
We then proceeded to find the final control groups by k-means clustering and determined the optimal number of clusters using the GAP statistic. Because of the high computational complexity of calculating the GAP statistic, we decided to simplify the calculation by first reducing the potential size of the control group to 500 individuals (i.e. the 500 individuals closest to the person under study). Since it was observed that in almost all cases the optimal number of clusters was between 4 and 5, we decided to perform a random sampling of 200 individuals and found that 5 was the most repeated optimal number of clusters; thus, that value was used for all clusters. Figure 5 presents four examples of the outcome of this methodology for defining the control groups.
After performing clustering and defining the control group of each individual as the cluster to which that individual belongs, Fig. 6 shows that the sizes of the control groups followed a similar distribution to that of the previous group, with a mean of 105.32 individuals and a median of 71 individuals. In this distribution a high value (specifically 496 individuals) was observed. This is because by limiting the number of individuals to 500 and forcing clusterings of five groups, in some cases in which the potential group was very large (several thousand individuals), the 500 closest people were already very similar to the person under study, so the optimal grouping was to make a group with 496 individuals and then another four groups with a single individual Fig. 6.
5.2 Criteria assessment
The general distribution of course performance for different criteria is shown in Fig. 7. It is worth noting that individual scores were, in general terms, quite extreme. Notably, they were either close to 1 or close to 0. Hence, the performance of a course at a certain criterion was highly and positively correlated to the percentage of students of that course that would have not passed a t-test at a standard significance level (e.g. 5% or 10%).
As shown in Fig. 7, criteria C2 and C3 were rarely met by respondents. The C2 criterion (‘total number of days with a permanent contract’) is very restrictive in Spain and even more so in regions such as Extremadura, where employment is characterised by high levels of temporary employment. Therefore, the results obtained for different courses under criterion C2 were as expected.
In contrast, the poor performance results for criterion C3 (‘number of days in a position related to the course taken’), provided us some useful information. The fact that the results for this criterion were also very poor indicates that, in general, SEXPE training courses do not provide specific training relevant to the Extremaduran labour market.
5.3 Course ranking
Once the different alternatives were evaluated for each of the four criteria and the performance matrix was obtained, we ranked them using the uwTOPSIS method described above. In this case, we used \(l_j = 0.1\) and \(u_j = 0.6\) for \(j = 1,\ldots , 4\) as the lower and upper bounds for different criteria, respectively. The rationale behind this selection of limits is twofold. Firstly, the wide range of values reduces the influence of subjective decision-making. Secondly, the ranges must be sufficiently narrow to exclude situations in which some weights approach 0 or 1. In such cases, significant issues may arise during the decision-making process. In the former case, a criterion can be eliminated; in the latter case, the problem is no longer multi-criteria.
Table 4 presents the ranking results obtained using the uwTOPSIS method. Column ‘uwTOPSIS’ shows the final TOPSIS score, and columns ‘Min’ and ‘Max’ contain the minimum and maximum values, respectively, of the TOPSIS score obtained in the optimisation procedure described by (4) and (5).
The aggregated ranking analysis revealed a significant drawback. An analysis of the number of students who took courses reveals that only approximately 6% of the students took courses in the first quartile, whereas more than 60% of all students took courses in the third and fourth quartiles Table 5.
This might be of relevance for policymakers, as it may be an indicator of where public resources should be increased to optimise the effectiveness of training courses. It is possible that some of the most demanded courses are not performing as effectively as expected. With better targeting or more publicity for other courses, citizens would have better information when choosing how to pursue their training to improve their work quality.
Furthermore, because of the great diversity of courses, they can be conveniently grouped by professional families to determine which types of professional activities perform better in the labour market. Therefore, Fig. 8 illustrates courses grouped by professional families, appearing in the first two quartiles of the ranking.
The figure highlights three professional families that stood out from the rest: community and sociocultural services (CSS), administration and management (ADM), and hotel and tourism (HOT). The first category is a good indicator of the ageing of the Extremaduran society (and the Spanish society in general, mostly in the inner regions of Spain) because many of the courses within this professional family are related to assistance for the elderly and dependent people.
In contrast, ADM is generally related to business management; thus, it is not surprising that it exhibited good performance according to the defined criteria. Meanwhile, HOT is linked to hospitality and tourism, which have recently gained a great boost in the region because of the rise in inland tourism and the proximity of Madrid, a major source of tourists.
5.4 Sensitivity analysis
One of the main advantages of using uwTOPSIS over other weighted methods is that, in the absence of expert information, the decision maker only has to provide some rough lower and upper limits for the weights of each criterion, thus decreasing subjectivity during the decision-making process. However, this does not mean that the results are free from subjectivity because they depend on the upper and lower limits defined above and on the value of parameter \(k_1\), which defines the inclination of the decision maker towards one extreme or the other of the interval obtained by uwTOPSIS in (6), and whose usual value is \(k_1 = 0.5\). It is thus important to analyse the effect that changing the definition of these parameters has on both the uwTOPSIS score and final ranking.
To this end, we conducted a simple analysis starting from the values of the parameters used in the previous section (i.e. \(l_j = 0.1\) and \(u_j = 0.6\), for \(j = 1, \ldots , 4\), and \(k_1 = 0.5\)). We varied these values to a certain extent and measured the difference between the obtained and original solutions. It is essential to carefully consider and maintain objectivity when determining the degree of uncertainty to be assigned to the input factors (in our case, the criteria weights or the parameter \(k_1\)) to assess the variance of the output (i.e. score or the position in the ranking). If the degree of variation attributed to input factors is excessive, the model exhibits an unacceptable degree of variability, rendering it ineffective [19]. In accordance with the methodology proposed by other authors, such as [1], the parameters were varied within the following range: \(l_j\in [0.05, 0.15]\), \(u_j\in [0.55, 0.65]\) for \(j = 1,\ldots ,4\), and \(k_1\in [0.3, 0.7]\).
To analyse the variability of the results, three different measures were used. These included the mean absolute percentage difference, which was used to measure the difference in uwTOPSIS scores, and the Kendall-tau distance and average ranking position difference, which were used to measure the difference between the rankings produced by uwTOPSIS.
The results of the sensitivity analysis with respect to the lower and upper limits of the weights are presented in Figs. 9 and 10. Figure 9 shows the impact of varying the limits on the uwTOPSIS score. The range of variation of the upper limit (u) was slightly extended to enhance the graphical representation. In the most unfavourable scenario, the score variation was approximately 10%. Figure 10 shows the difference in rankings. Both the Kendall-tau distance and the average ranking position difference revealed comparable patterns, albeit with different scales.
With regard to the sensitivity analysis of the \(k_1\) parameter, the corresponding results are shown in Figs. 11 and 12. The outcomes obtained are analogous to those obtained for the limits of the weights. Sensitivity analysis demonstrated the transparency and robustness of the proposed methodology, because the changes observed in both the uwTOPSIS score and ranking positions were consistent with the modifications made to the parameters.
6 Conclusions
In this study, we proposed a methodology to analyse the effect of the performance of a certain activity (training courses) on a population. The proposed methodology is completely data-driven (contrary to many studies on the same topic, which are based on opinion surveys), and bottom-up. In particular, the proposed methodology starts by analysing the performance of each participant who took a course. Then, these performances can be aggregated to define the performance of the given course. Finally, courses are ordered using the uwTOPSIS method.
The control group was selected based on the definition of the WLC, which parameterises the working life of individuals and allows comparisons with other subjects. In this way, by means of a clustering process, we can define the control group with which to compare the results of each participant who took each course. This novel methodology permits the definition of control groups in experiments that use pre-existing databases, which can be expanded over a long period of time with heterogeneous individuals. Nevertheless, it requires access to a substantial quantity of data from disparate databases that must be cross-referenced. This requires significant collaboration and coordination between the various institutions.
With regard to the proposed multi-criteria method, uwTOPSIS has an advantage over other weighted methods. In the absence of expert information, the decision maker should only provide some rough lower and upper limits for the weights of each criterion to reduce subjectivity during the decision-making process. This innovative multi-criteria method cannot be compared with other known multi-criteria techniques because fixed weights are not assigned a priori to the criteria; instead, the upper and lower bounds are set. The development of other multi-criteria methods with the introduction of weight bounds could prove an interesting avenue for future research, potentially enabling a comparison of the results of this methodology with those of new approaches.
A sensitivity analysis was conducted to analyse the effect of changing the definition of subjective parameters used in uwTOPSIS on the final score and ranking. This analysis demonstrated the transparency and robustness of the proposed methodology because the observed changes were consistent with the modifications made.
Results of the multi-criteria uwTOPSIS method demonstrated that training courses related to the professional families of ADM, HOT and CSS have a positive effect on employability and improve the working conditions of citizens. Nevertheless, the most efficient courses in the ranking were those that were least demanded by citizens.
Policymakers can optimise the benefits of lifelong learning by strategically addressing challenges and leveraging opportunities presented in this study. By investing in data quality, engaging stakeholders, building technical capacity, ensuring system flexibility, securing funding and integrating existing systems, policymakers can create robust frameworks that enhance the effectiveness of training programmes. These findings may inform policy decisions on the allocation of public resources, the identification of less efficient training courses for discontinuation and the promotion of courses with a high impact on the employability of citizens. This will ultimately result in enhanced educational outcomes, enhanced employability and a more adaptable and proficient workforce.
Data Availability and Access
The data supporting the findings of this study are available from the Council of Education, Science, and Vocational Training of the Government of Extremadura (Spain). However, access to these data is restricted due to privacy limitations; they were used under license for the current study and are not publicly available. The data can be obtained from the authors upon reasonable request to the corresponding author, with permission from the Council of Education, Science, and Vocational Training of the Government of Extremadura (Spain).
References
Annoni P, Kozovska K (2010) EU Regional Competitiveness Index (RCI) 2010. Tech. Rep. EUR 24346 EN, Publications Office of the European Union, Luxembourg (Luxembourg). https://doi.org/10.2788/88040, https://publications.jrc.ec.europa.eu/repository/handle/JRC58169
Babos P, Lubyova M, Studená I (2015) Lifelong learning is a growing factor in employability. policy brief, proceedings of lllight’in’europe research project. https://www.lllightineurope.com/fileadmin/lllightineurope/download/LLLight_LLL_is_growing_factor_in_employability_policybrief_C7_20150922.pdf, accesed 2023/09/21
Benítez R, Liern V (2021) Unweighted topsis: a new multi-criteria tool for sustainability analysis. Int J Sustain Develop World Ecol 28(1):36–48. https://doi.org/10.1080/13504509.2020.1778583
Blasco-Blasco O, Liern-García M, López-García A, et al (2021) An academic performance indicator using flexible multi-criteria methods. Mathematics 9(19). https://doi.org/10.3390/math9192396
Blasco-Blasco O, Liern-García M, López-García A et al (2021) An academic performance indicator using flexible multi-criteria methods. Mathematics 9:1–19. https://doi.org/10.3390/math9192396
Cables E, Lamata M, Verdegay J (2016) RIM-reference ideal method in multicriteria decision making. Inf Sci 337–338:1–10. https://doi.org/10.1016/j.ins.2015.12.011
Canós L, Liern V (2008) Soft computing-based aggregation methods for human resource management. Eur J Operat Res 189(3):669–681. https://doi.org/10.1016/j.ejor.2006.01.054
Conejero J, Preciado J, Prieto A et al (2021) Applying data driven decision making to rank vocational and educational training programs with topsis. Decision Support Syst 142:113470
Conejero JM, Preciado JC, Fernández-García AJ et al (2021) Towards the use of data engineering, advanced visualization techniques and association rules to support knowledge discovery for public policies. Expert Syst Appl 170:114509
Elena R (2014) Effectiveness evaluation of training programmes for disadvantaged targets. Procedia - Social and Behavioral Sci 141:1239–1243. https://doi.org/10.1016/j.sbspro.2014.05.213, https://www.sciencedirect.com/science/article/pii/S1877042814036374, 4th World Conference on Learning Teaching and Educational Leadership (WCLTA-2013)
European Comission (2001) Making a european area of lifelong learning a reality. communication from the commission. com (2001) 678 final. http://aei.pitt.edu/42878/
Farjad S (2012) The Evaluation Effectiveness of Training Courses in University by Kirkpatrick Model (Case Study: Islamshahr University). Procedia - Social Behavioral Sci 46:2837–2841. https://doi.org/10.1016/j.sbspro.2012.05.573
Gal J, Janowszky S, Juhasz-Fodor T (2007) Lifelong learning in the hungarian education system. In: Gal J (ed) MERLIN- A történetmesélés, mint az élethosszig tartótanulás egyik formája. IM Informatikai Magániskola Kft, pp 125–130
Gil-Aluja J (1999) Elements for a Theory of Decision in Uncertainty, 1st edn. Applied Optimization 32, Springer US
Hwang CL, Yoon K (1981) Multiple Attribute Decision Making, vol 186. Springer, Berlin Heidelberg
Järlström M, Brandt T, Rajala A (2020) The relationship between career capital and career success among finnish knowledge workers. Baltic J Manag 15(5):687–706. https://doi.org/10.1108/BJM-10-2019-0357
Johnson SG (2022) The NLopt nonlinear-optimization package. http://github.com/stevengj/nlopt, accessed September 30, 2022
Kirkpatrick DL (2006) Evaluating Training Programs: The Four Levels, 3rd edn. Berrett-Koehler Publishers
Leamer EE (1983) Let’s take the con out of econometrics. Am Econ Rev 73(1):31–43. http://www.jstor.org/stable/1803924
Liern V, Pérez-Gladish B (2022) Multiple criteria ranking method based on functional proximity index: un-weighted topsis. Annal Operat Res 311(2):1099–1121. https://doi.org/10.1007/s10479-020-03718-1
Liern V, Pérez-Gladish B, Rubiera-Morollón F et al (2021) Residential choice from a multiple criteria sustainable perspective. Annal Operat Res. https://doi.org/10.1007/s10479-021-04480-8
López-García A, Liern V, Pérez-Gladish B (2023) Determining the underlying role of corporate sustainability criteria in a ranking problem using uw-topsis. Annal Operat Res. https://doi.org/10.1007/s10479-023-05543-8
López-García A, Blasco-Blasco O, Liern-García M et al (2023) Early detection of students’ failure using machine learning techniques. Operat Res Perspect 11:1–11. https://doi.org/10.1016/j.orp.2023.100292
Ouenniche J, Pérez-Gladish B, Bouslah K (2018) An out-of-sample framework for topsis-based classifiers with application in bankruptcy prediction. Technol Forecasting Soc Change 131:111–116. https://doi.org/10.1016/j.techfore.2017.05.034
Pamucar D, Stevic Z, Sremac S (2018) A new model for determining weight coefficients of criteria in mcdm models: Full consistency method (fucom). Symmetry 10(9). https://doi.org/10.3390/sym10090393, https://www.mdpi.com/2073-8994/10/9/393
Park HS, Jun CH (2009) A simple and fast algorithm for k-medoids clustering. Expert Systems with Applications 36(2, Part 2):3336–3341. https://doi.org/10.1016/j.eswa.2008.01.039, https://www.sciencedirect.com/science/article/pii/S095741740800081X
Pérez-Gladish B, Ferreira FAF, Zopounidis C (2021) Mcdm/a studies for economic development, social cohesion and environmental sustainability: introduction. Int J Sustain Develop World Ecol 28(1):1–3. https://doi.org/10.1080/13504509.2020.1821257
Powell M (1994) Advances in Optimization and Numerical Analysis, vol 10.1007/978-94-015-8330-5, Susana Gomez and Jean-Pierre Hennart, chap A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation, pp 51–67. https://doi.org/10.1007/978-94-015-8330-5_4
R Core Team (2019) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, https://www.R-project.org/
Ramík J, ímánek J, (1985) Inequality relation between fuzzy numbers and its use in fuzzy optimization. Fuzzy Sets Syst 16(2):123–138. https://doi.org/10.1016/s0165-0114(85)80013-0
Rotar LJ (2021) Evaluation of the effectiveness of employment programmes on young unemployed people. Eng Econ 32(1):60–69. https://doi.org/10.5755/j01.ee.32.1.23276
Saaty T, Vargas L (2012) Models, methods, concepts & applications of the analytic hierarchy process. Springer
Sackman H (1974) Delphi Assessment: Expert Opinion, Forecasting, and Group Process. RAND Corporation, Santa Monica, CA
Sanulita H, Hendriyanto D, Lestari NC, et al (2024) Analysis of the effectiveness of audio visual learning media based on macromedia flash usage on school program of increasing student learning motivation. J Educ 6:12641–12650. https://doi.org/10.31004/joe.v6i2.5121
Schubert E, Rousseeuw PJ (2019) Faster k-medoids clustering: Improving the pam, clara, and clarans algorithms. In: Amato G, Gennaro C, Oria V et al (eds) Similarity Search Appl. Springer International Publishing, Cham, pp 171–187
Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27(3):379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
Sharma H, Jain V, Mogaji E et al (2024) Blended learning and augmented employability: a multi-stakeholder perspective of the microcredentialing ecosystem in higher education. Int J Educ Manag 38. https://doi.org/10.1108/ijem-12-2022-0497
Srinivasan V, Shocker AD (1973) Linear programming techniques for multidimensional analysis of preferences. Psychometrika 38(3):337–369. https://doi.org/10.1007/BF02291658
Tibshirani R, Walther G, Hastie T (2001) Estimating the number of clusters in a data set via the gap statistic. J Royal Stat Soc: Series B (Statistical Methodology) 63(2):411–423
Towler A, Kemp RI, Mike Burton A et al (2019) Do professional facial image comparison training courses work? PLoS ONE 14(2):1–17. https://doi.org/10.1371/journal.pone.0211037
Tzeng GH, Huang JJ (2011) Multiple Attribute Decision Making: Methods and Applications, 1st edn. Chapman and Hall /CRC
Vinogradova I, Podvezko V, Zavadskas EK (2018) The recalculation of the weights of criteria in mcdm methods using the bayes approach. Symmetry 10(6). https://doi.org/10.3390/sym10060205, https://www.mdpi.com/2073-8994/10/6/205
Ypma, J. (2013) Introduction to NLoptr: An R interface to NLopt. http://cran.r-project.org/web/packages/nloptr/vignettes/nloptr.pdf, accessed December 23, 2019
Zavadskas EK, Podvezko V (2016) Integrated determination of objective criteria weights in mcdm. Int J Inf Technol Decision Making 15(02):267–283. https://doi.org/10.1142/S0219622016500036
Acknowledgements
This work was developed with the support of Grant PID2021-127412OB-I00, and Grant CPP2021-008491, funded by MCIN/AEI/10.13039/501100011033 and “European Union NextGenerationEU/PRTR”. We would like to express our gratitude to the reviewers for their dedication and commitment in evaluating the manuscript. We are grateful for all constructive feedback, which has enabled us to enhance the quality of the manuscript.
Funding
Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.
Author information
Authors and Affiliations
Contributions
Conceptualization: M. C. Bas, V. J. Bolós, A. E. Prieto. Methodology: M. C. Bas, V. J. Bolós. Data acquisition and curation: A. E. Prieto, R. Rodríguez-Echeverría, F. Sánchez-Figueroa. Formal Analysis and Investigation: M. C. Bas, V. J. Bolós, A. E. Prieto. Writing - original draft preparation: M. C. Bas, V. J. Bolós, A. E. Prieto, R. Rodríguez-Echeverría Writing - review and editing M. C. Bas, V. J. Bolós, A. E. Prieto, R. Rodríguez-Echeverría, F. Sánchez-Figueroa Funding acquisition: A. E. Prieto, R. Rodríguez-Echeverría, F. Sánchez-Figueroa
Corresponding authors
Ethics declarations
Conflict of interest
The authors have no competing interests to declare that are relevant to the content of this article.
Ethical and informed consent for data used
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Bas, M.C., Bolós, V.J., Prieto, Á.E. et al. A multi-criteria decision support system to evaluate the effectiveness of training courses on citizens’ employability. Appl Intell 55, 57 (2025). https://doi.org/10.1007/s10489-024-05967-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-05967-0