Keywords

1 Introduction

The number of web applications has grown very rapidly in recent years [1]. With an increasing number of web applications, there is also an increase in the number of web developers [2]. Developing web applications is a very complex process that demands highly skillful personnel. Its complexity is due to the fact that it involves many different information technologies, information systems, programming languages and databases, among others [3]. In this context, Web Application Frameworks (WAFs) have emerged as a solution to develop web applications. WAFs are powerful techniques for large-scale reuse promoting developers to improve quality, save costs and time. Examples of WAFs used in development today include: Zend framework for PHP, Spring framework for Java, .NET Framework (ASP.NET MVC), and Django for python among others [4]. Despite of the benefits of using WAFs, the initial effort invested in learning to use them can be very high [4, 5], especially for novice developers [6]. This problem is due to the large amount of WAF components and the increasing number of documents. Sometimes, developers face the reading of hundreds of documentation pages with information they will never use.

Good documentation is crucial for effective framework reuse; several authors have proposed different framework documentation and learning techniques including: patterns [7], cookbooks [8], example-based learning [6], and collaborative learning environments [9] among others. Nevertheless: (i) the true impact of these techniques on framework understanding is still fuzzy [9], (ii) there are a few studies that deal with issues around effective framework reuse and understanding [9], (iii) generating good documentation is often hard, costly, and tiresome, and comes with many issues [6]; and (iv) currently, when a developer must use a specific WAF, he/she has to invest considerable effort on understanding it [10].

Based on some of the previous issues, and trying to improve the WAF learning experience, we previously presented two contributions in this field [10, 11]. However, they presented an incomplete learning technique. This paper finalizes the new learning technique by involving concepts of micro-learning and example-based learning, which improves the new learning technique (presented in Sect. 3).

The main contribution of this paper is the new WAF learning technique that we call CME (Concerns, Micro-learning and Examples). This technique defines a customized learning path that a novice WAF developer should follow to learn to use a WAF. The technique and the learning path were built as a mix of concerns, micro-learning and example-based learning concepts. The second contribution of this paper is the development of a CME WAF learning tool; this tool allows to create documentation and examples for different WAFs; therefore, it allows to design the learning paths. This way developers could use the tool, and access a customized learning environment. The third contribution is the comparison of two WAF learning techniques: CME and a WAF cookbook, through a quasi-experiment.

The remainder of this paper is structured as follows. Section 2 presents the related work. Section 3 describes the new WAF learning technique (i.e., CME). Section 4 presents the WAF learning tool. Section 5 presents a quasi-experiment comparing the use of CME and a WAF cookbook. Finally, Sect. 6 summarizes the contributions and presents future research directions.

2 Related Work

2.1 Framework Learning

Flores and Aguiar [9] present a platform called DRIVER, a collaborative learning environment for framework users to share their knowledge. The platform provides the documentation on a wiki, where the learning paths of the community of learners can be captured, shared, rated, and recommended, thus tapping into the collective knowledge of the community of framework users. Aguiar and David [7] present a small set of patterns addressing problems related to the framework documentation. These patterns provide guidance on choosing the kinds of documents to produce, how to relate them, and which contents to include. Hou [6] analyzed the data gathered from a framework-based course project where students were taught about the architectural designs of the framework. Based on that, Hou defined a set of hypotheses, along with supporting evidence, that characterize the various effects of framework designs on project outcome.

2.2 Concerns, Example-Based Learning, Micro-Learning

Caeiro-Rodríguez et al. [12] present a separation of concerns approach to Educational Modeling Languages (EMLs). Their main idea was not to attain the modeling of educational materials as a whole problem, but to divide it in several separated concerns that could be solved part by part.

Shull et al. [13] presented a study about processes that developers would engage in when learning and using object-oriented frameworks. They found evidence that learning by example (as opposed to gaining familiarity with the framework itself first) is useful for helping beginning learners produce working systems quickly (even for non-trivial developments).

Sun et al. [14] implemented micro learning in massive open online courses (MOOC); they organized the micro learning environment with the use of a Software as a Service (SaaS). They employed a data mining technique to understand learner’s learning behaviors and recognize learning resource features in order to identify potential micro learning solutions. Job and Ogalo [15] analyse and highlight the significance of micro learning in the education industry. They presented a quantitative research methodology based on a questionnaire survey; the results highlight the importance and need of micro learning in the education industry.

Other learning techniques include the use of documents, games and adaptive e-learning systems [16], among other. Nevertheless, most of the previous studies were made for framework learning in general or other areas such as MOOCs, EMLs and teaching software engineering concepts [17]; and none of them has been applied to WAF learning specifically. The most similar study was made by Flores and Aguiar [9], they presented a quasi-experiment in which students were separated in different groups, and they were requested to build an information system within 4 iterations. Nevertheless, none of the scores about the development process produced any relevant statistical results.

Some of the previous studies and approaches presented important advantages, we decided to take some elements of them in order to create a new approach.

3 A New WAF Learning Technique Based on Concerns, Micro-Learning and Examples

CME is a new WAF learning technique based on the following concepts: (i) A list of Concerns, (ii) Micro-learning and (iii) Example-based learning. The development of this technique was carried out through five stages as presented in Fig. 1 and described thereafter. It is important to highlight that Sects. 3.1 to 3.4 describe the process that we followed to complete the definition of the new learning technique; Sect. 3.5 describes the consolidated CME learning technique, including the process that novice WAF developers should follow in order to apply the CME learning technique.

Fig. 1.
figure 1

CME development stages (UML activity diagram).

Full size image

3.1 Definition of WAF Components and Tasks

Initially, each WAF could be seen as a completely different tool; usually, we tend to think that the difference could be even bigger for WAFs that are developed in different software languages (i.e., comparing CodeIgniter framework for PHP and Spring MVC framework for Java). However, several WAF comparison studies show many similarities between them [18, 19], especially for MVC WAFs. In a previous work, we defined a list of 13 WAF components based on these similarities. We divided the learning process for each component in a set of fundamental tasks, each task guides developers in what they should learn in order to use each component [11]. Table 1 shows an example of one WAF component with its respective tasks. The complete list of WAF components and tasks can be found on the Internet [20].

Table 1. Excerpt of WAF components and tasks
Full size table

The previous tasks provide an insight about specific elements that a developer should learn in order to use a specific WAF component. However, it is not necessary to learn how to use all components of a specific WAF, even it is not necessary to learn about all the tasks of a specific component [10]. The reason is that each developer has her/his own needs, and wants to develop very different web applications. The next section shows how to guide developers according to her/his own needs.

3.2 Definition of Web Developers Concerns

As aforementioned, developers use WAFs for different reasons: developing a software project, acquiring more knowledge, applying for a job position, accessing the training about tools in organizations, etc. No matter the reason, the final goal for learning a WAF usage is to develop specific web applications; and these applications could vary a lot from one to another. In other words, different developers are driven by different interests or concerns. “Separation-of-concerns is a long-standing idea that simply means that a large problem is easier to manage if it can be broken down into separate parts [21].”

In a previous work, we analyzed some web application projects in order to find common web developers concerns. This analysis showed that no matter how different each web application seems from one another, web developers presented similar concerns. As a result of this analysis, we proposed a list of 29 basic concerns, and a connection between the concerns list and WAF components list [10]. Table 2 shows an example of one web developers’ concern; it also includes a suggestion that the developers should read to know if that concern matches with his/her needs.

Table 2. Example of common web developers’ concern
Full size table

In web applications, concerns are codified with the use of different components [22]. Due to the WAFs features and taking advantage of WAF components separations, we connected these concerns to WAF components and their tasks. Table 3 shows an example of one concern linked to different components and their corresponding tasks. It is important to highlight that the connection between concerns and WAF components and tasks is not an ultimate one; a senior WAF developer could make adjustments as he/she considers. This connection gives the possibility to know, for each concern, what are the specific components and tasks that the developers should learn; thus, if a developer chooses some concerns based on her/his needs, then a list of specific WAF components and tasks will be provided to her/him (providing a customized learning process).

Table 3. Excerpt of web developers’ concerns linked to WAFs components and tasks
Full size table

3.3 Definition of Examples

Example applications have been identified as an effective learning aid for an application framework [6]. According to Hou [6], example applications are useful in framework learning because: (i) point out features that the framework provides, (ii) contain concrete solutions that are useful to developers (which could be reused to develop applications), and (iii) allow developers to focus more of their attention on analysis and modification, rather than search and construction. However, “examples alone are not sufficient for the effective learning of application frameworks [6].”

After recognizing the importance of example-based learning, we decided to create a list of examples connected to the previous concerns. This is important because it provides an alternative material for novice developers; one example could show in a practical way how to use the WAF components, and developers could recognize what they learnt from the component tasks. Furthermore, most examples are transversal to the WAF architecture; a simple “Display information on screen” example could cross through different WAF layers and components. Subsequently, these examples are also useful to identify the relationship between different components. Table 4 shows one concern example, and the instruction to be coded for a WAF senior developer. The complete list of examples can be found on the Internet [20]. These examples should be developed by a WAF senior developer.

Table 4. Excerpt of examples for each concern
Full size table

To complete the learning strategy and connect the previous elements (WAF components and tasks, concerns and examples) we take advantage of Micro-learning, as detailed in the next section.

3.4 Definition of a Micro-Learning Strategy

No matter how learning is conceptualized, in all cases there is the possibility of considering it in terms of micro, meso and macro aspects, levels or tasks [23]. For example, in the context of linguistics learning, one might think in micro level aspects in terms of single letters, the meso level aspects in terms of words and sentences, and the macro level aspects in terms of texts or conversations.

Taking advantage of micro-learning characteristics, we decided to separate the new WAF learning technique into micro, meso and macro tasks as explained in the following.

Micro-tasks correspond to follow specific documentation that is attached to each task of each WAF component. The micro-task documentation could be a link to a website, forum, blog, video or a specific explanation text.

Meso-tasks correspond to follow examples that are designed for each concern. In this case, meso-tasks refer to pieces of pre-elaborated code, which are designed based on the concern’s descriptions.

Macro-tasks correspond to develop a small web application, or a part of an application; all the knowledge gathered in the previous levels is required to execute this task. The applications to develop could vary, could be a part of a job development application or a teacher activity among others.

3.5 Development of the New WAF Learning Technique

The combination of the previous learning approaches allows us to define the new CME learning technique. Figure 2 shows how the CME concepts have been related to each other; and Fig. 3 shows the CME process, the process that WAF learners should follow in order to learn to use a WAF. This process is presented thereafter.

Fig. 2.
figure 2

CME concepts with their relationships (pre-conceptual schema).

Full size image
Fig. 3.
figure 3

CME process (UML activity diagram).

Full size image

Extraction of the Application Requirements:

The learning process begins with a developer who wants to develop an application (macro-task). In the first step, the developer extracts the requirements of this application. She/he could extract just an excerpt of the requirements.

Selection of WAF:

In the second step, the developer chooses a specific WAF, that she/he will use to develop the application.

Selection of Concerns:

A list of concerns is presented to the developer. The developer selects one or multiple concerns based on her/his needs.

Use of Documentation:

Depending on the previous selected concerns, a list of micro-tasks (micro documentation) and meso-tasks (examples codified) are presented to the developer. This documentation must be previously completed by a WAF senior developer. This documentation is used by the developer to acquire knowledge about the WAF elements.

Development of the Application:

Finally, the developer develops the application (macro-task).

Supporting the CME technique is crucial to allow developers (learners) and WAF senior developers to access the WAF documentation. The following section presents a tool that allows completing the WAF documentation and accessing it.

4 Tool Support

In order to provide learners with the WAF documentation and allow them to access to the customized learning paths, we developed a web application called “driving-learning application”. This application also provides an admin section that is used by WAF senior developers to complete the WAF documentation. The application can be found on the Internet [18]. Figure 4 presents the driving-learning application home page; here the WAF developer selects the WAF and selects her/his concerns. Then, two options appear to the learner: (i) get examples (which presents the meso-tasks) and (ii) get documentation (which presents the micro-tasks).

Fig. 4.
figure 4

Home page of driving-learning application.

Full size image

Suppose that a WAF developer wants to learn and develop applications with CodeIgniter. The developer needs to learn about how to capture information from the view layer, and how to send information from the controller layer to the view layer. Therefore, the developer selects the “capture and data assignment” concern; and then clicks on “get examples” and “get documentation”. Figure 5 shows how the driving-learning application presents the respective documentation to the developer, including: (i) micro-tasks (cf. Fig. 5.a) and (ii) the meso-tasks (cf. Figure 5.b).

Fig. 5.
figure 5

Excerpt of CodeIgniter documentation related to the “capture and data assignment” concern, presented through driving-learning application.

Full size image

With the purpose of analyzing how CME and the driving-learning application improve the WAF learning process, we designed a quasi-experiment as presented in the next section.

5 Quasi-Experiment

The use of empirical studies with students (ESWS) in software engineering helps researchers gain insight into new or existing techniques and methods [24]. ESWSs are useful in obtaining preliminary evidence in support of, or against research hypothesis. There are different types of ESWSs: controlled experiments, quasi-experiments, correlation studies, case studies, or surveys [25]. For a preliminary evaluation, we selected a quasi-experiment study, because randomization was not possible with the students of undergraduate courses. We carried out the quasi-experiment in order to (i) obtain preliminary evidence about how CME and the driving learning application works, and (ii) to evaluate the following hypotheses.

  • H0 (null): Providing novice WAF developers with the CME technique makes them more operational to develop small web applications in a reduced period of time, in a similar way, than through the use of a WAF cookbook.

  • H1 (alternative): Providing novice WAF developers with the CME technique makes them more operational to develop small web applications in a reduced period of time, in greater proportion, than through the use of a WAF cookbook.

5.1 Quasi-Experiment Execution

Before the execution of the quasi-experiment, we had to select the specific WAF to be used in the experiment. The WAF had to fulfill the following requirements: (i) had to be unknown for all participants, (ii) the WAF official documentation had to be in Spanish (participants language) and be available online, (iii) had to be an open-source and developed in PHP, and (iv) had to be known for the authors of this paper, in order to be able to complete the documentation. Based on these requirements, we selected CodeIgniter [26], a PHP web framework. Afterwards, the quasi-experiment was divided into different phases, which are represented in Fig. 6 and described thereafter.

Fig. 6.
figure 6

Quasi-experiment phases.

Full size image

A. Participants’ Selection:

The experiment’s participants were 15 undergraduate students from the “System and Informatics Engineering” program at the Universidad Nacional de Colombia; students attended an optional course on “Design and Construction of Software Products,” a course which introduces PHP programming and architectural software patterns, like the MVC pattern; nevertheless, the course does not introduce the use of WAFs.

B. Pre-questionnaire:

The second phase of the quasi-experiment was to distribute a 15 min pre-questionnaire to previous students. The questionnaires were designed using a Likert scale, which had a five-point format: (1) strongly disagree, (2) somewhat disagree, (3) neither agree nor disagree, (4) somewhat agree, and (5) strongly agree. The pre-questionnaire was used to ascertain the participants’ background and general profile in order to screen out possible differences amongst the participants regarding their basic skills. It also served to confirm the students’ lack of knowledge with CodeIgniter.

C. Group Formation:

In quasi-experiments, it is important to assure that the participants are similar, and that their base skills do not pose a significant threat to the validity of the results. Therefore, students were scrutinized based on their academic track, by analyzing their grades on a selected subset of courses. These courses were deemed relevant to the outcome of the experiment, namely: (I) Programming Fundamentals, (II) Data Structures, (III) Object Oriented Programming, (IV) Software Engineering, (V) Databases I, (VI) Logical and Functional Programming, and (VII) Requirements Engineering.

The participants were divided into three groups; each group had its own characteristics. Therefore, an independent sample t-test was conducted to compare the average participants’ grades (cf. Table 5).

Table 5. Participants grades group statistics.
Full size table

Group 1 (G1) – Baseline: this group served as the control group. Its subjects used the CodeIgniter official documentation or cookbook.

Group 2 (G2) – Experimental: this group used the driving-learning application by only accessing the meso-tasks or examples material.

Group 3 (G3) – Experimental: this group used the driviling-learning application by accessing the micro-tasks and meso-tasks material (the complete CME learning material).

D. Treatments:

the participants were submitted to a treatment phase, where each group was introduced to their own experiment environment.

Treatment A: treatment only applied to G1. Participants were provided with the Codeigniter Spanish official documentation [27].

Treatment B: treatment only applied to G2. Participants were provided with the driving-learning application, but limited only access to the examples or meso-tasks.

Treatment C: treatment only applied to G3. Participants were provided with the driving-learning application, with complete access to the application, including access to the micro and meso-tasks information.

E. Macro-task:

At this point, the subjects were ready to develop the main task. The macro-task was designed with the intention of developing a part of an application (about managing information of different coffee stores). This macro-task was divided into four iterations, the main idea was that students could incrementally build this application. After each participant finished one iteration, a supervisor verified the task and documented the time the participant spent on its development. An experiment document and a SQL (Structured Query Language) file including a list of coffee stores and employees was provided to all the participants; two hours was the maximum allowed time to develop the macro-task.

F. Post-questionnaire:

At the end of the experimentation, the participants were submitted to the post-questionnaire, which included questions about: (i) the execution environment, which asked the participants about if they got distracted and if they found the environment intimidating, among others; (ii) the use of the tools, which asked the participants about if they found the documentation sufficient and if they felt the need to have access to more information, among others; and (iii) questions about CodeIgniter concepts, which asked the participants about CodeIgniter controllers, routes and models, among others.

5.2 Quasi-Experiment Results

The results and files of the quasi-experiment can be found on the Internet [18]. We compared each answer from each questionnaire using the nonparametric, two-sample, rank-sum Wilcoxon-Mann-Whitney test. The significance level for all tests was set to 5%, so probability values of p ≤ 0.05 are considered significant, and p ≤ 0.01 considered highly significant. We did not find significant differences in the pre-questionnaire results; however, we found five significant differences and three highly significant differences in the post-questionnaire results between the G3 and G1, and the G2 and G1. Specifically, in the “overall satisfaction” (OS) question results, which questioned the participants about their performance, comfort and feel of the presented learning environment and material. All these differences provided support in favor of the CME technique; G3 and G2 presented better results than G1 (cf. Table 6 – OS results).

Table 6. Summary of the quasi-experiment results, between the G2 and G1, G3 and G1, including the values of the non-parametric significance Wilcoxon-Mann-Whitney test.
Full size table

The previous results were supported with the results of the number of participants who completed each iteration as shown in Fig. 7.a. G3 (M = 2,40) and G2 (M = 3,40) completed more iterations than G1 (M = 1,20).

Fig. 7.
figure 7

(a) Number of subjects of each group who completed each iteration, (b) Average of correct answers about CodeIgniter concepts for each group.

Full size image

On average, the reported time of the completion of the iteration 1 after the use of the CME learning technique, by the experimental groups G3 (M = 31,2) and G2 (M = 30,2); was significantly lower than after the use of a Cookbook, by G1 (M = 76,0), (p = 0.007) in both comparisons (cf. Table 6 – time result). Other iterations were not analyzed with this method, because G1 participants did not complete most of the iterations (cf. Fig. 7.a). On average, the number of completed iterations after the use of the CME learning technique, by the experimental group G2 (M = 3,4); was significantly higher than after the use of a Cookbook, by G1 (M = 1,2), p = 0.004 (cf. Table 6 – tasks result). Finally, G3 (M = 5,00) presented the best results in the average of correct answers about CodeIgniter concepts; nevertheless, we did not find significant differences in this aspect (cf. Fig. 7.b; cf. Table 6 – QNS result).

Previous results provide evidence that the presented new WAF learning technique helps developers not only with their first contact with the framework, but also in the application development process. G2 and G3 were driven by their own needs and the documentation presented matched with their needs. On the other hand, G1 had to manually look over a large amount of documentation and contents, trying to determine how to use it to satisfy its needs. Finally, we found preliminary evidence that allowed us to reject the H0 (null hypothesis) and accept the alternative one (H1).

5.3 Threats to Validity

With the results of both the pre-questionnaire and post-questionnaire, we discarded some threats to the validity of the quasi-experiment, including: (i) insufficient skills to execute the tasks, (ii) experiment-related factors, (iii) environment factors, and (iv) lack of motivation. Although we conducted our evaluation with care, we could not completely discard some threats. First, we only applied the quasi-experiment with the use of one WAF (CodeIgniter); more experiments over different WAFs should be applied. Secondly, we compared participants using CME and a WAF cookbook. However, we cannot generalize our findings to other approaches such as design patterns, collaborative environments, among others. Third, we analyzed the successful execution of the iterations and the code integration; nevertheless, the measures of code quality, cohesion and complexity were not addressed. Fourth, it is difficult to prove if experimental groups were able to develop more iterations than baseline group because of the similarity between the experiment tasks, and the meso-tasks documentation. However, the meso-tasks were designed in a previous study, establishing the web developer’s common concerns; therefore, developers should understand how the meso-tasks code worked, otherwise they could not reuse it to develop the experiment iterations. Finally, it is important to highlight that the main objective was to analyze the applicability of CME over a WAF learning environment. Rigorous experiments addressing the aforementioned aspects should be developed in a future work.

6 Conclusions and Future Work

Developers usually face the need for developing an application by using a specific WAF (perhaps an unknown one); consequently, they need to learn how to use the WAF for developing the application. Currently, when a developer has to use a specific WAF, he/she has to invest considerable effort and time in understanding it. In order to deal with these problems, we defined a strategic combination: concerns, micro-learning and example-based learning concepts. Afterwards, we developed a new WAF learning technique called CME and a supportive tool. Then, we developed a quasi-experiment with the use of CME and a WAF cookbook. Experimental groups who used the CME technique reported better results than the baseline group; we found preliminary evidence, that supports the idea, that novice WAF developers who are provided with the CME technique are more operational in developing small web applications in a reduced period of time, compared to novice WAF developers that used a WAF cookbook. As a future work, we plan to develop more rigorous experiments, including the use of different WAFs; we plan to implement the learning technique in other software areas, and we plan to improve the learning tool.