Keywords

1 Introduction

Interaction is nowadays a cornerstone aspect for knowledge acquisition and a revulsive that constantly creates a sense of engagement within a learning ecosystem. The discussion forums in virtual courses are well known activities that strategically planned, can generate spaces for the construction of knowledge and if they are supported by tools that allow the monitoring, measurement, collection, analysis, and visualization of interactions between students and tutors, thus then can provide inputs for the pedagogical reorientation of the teaching and learning action. It is important to mention that Learning Analytics (LA) is highlighting as an emerging discipline, used in educational research, which makes use of technologies to understand the digital contexts in which learning takes place and is being used to provide pertinent information to improve the teaching and learning process. In this sense, virtual tutors can benefit from the application of LA to know the interactions that are being developed when they propose a forum for the presentation of the participants, discussion, reflection or debate of ideas. In this sense, the authors in [1] presents that there is strong evidence that in well-structured activities, knowledge-building processes reach higher levels of critical thinking, where students are able to establish and sustain cohesive groups. This justifies the need for a broader Social Network Analysis (SNA), taking into account the discussion forums as the first kind of social interaction activity that is evolving into external tools as the different social networks applications available today. For this there is a need to have tools that are capable of providing real-time analysis for tutors and teachers from different external tools.

In this sense, it is worth mentioning that both concepts of Visual Learning Analytics and Social Network Analysis are key factors to the success in Learning Ecosystems. Learning Ecosystems are known as a complex group of elements that interact and coexist with each other to enhance the teaching and learning process with the help of technology. The factor related to interaction is quite important based on the fact that the main actor will be the student. In [2], the authors emphasize that the human factor has a main role in the definition and development of the whole process inside a learning ecosystem and should be taken into account constantly in order to engage the participants in each of the learning activities.

This work presents a study that was carried out at the Faculty of Humanities, University of San Carlos at Guatemala. The purpose of this study is twofold. On the one hand the aim was to stimulate engagement of participants in a virtual learning environment through the invitation to write an application essay on their motivation to be part of the learning experience. This first experience presented a positive reaction in terms of participation in a Massive Open Online Course (MOOCs). The second objective within a subset of the participants in the study was to evaluate the importance of using visual learning analytics to improve and enhance interaction in learning ecosystems. For this, the work presents an analysis of the interaction developed between students and tutors in several types of forums implemented in a virtual course: presentation, doubts, debate, reflection, analysis.

This work is organized as follows: In Sect. 2 the related works on Learning Ecosystems and Visual Learning Analytics are presented. Next, Sect. 3 describes the experience, then Sect. 4 shows the most significant results that are complemented with a discussion and analysis. Finally, Sect. 5 presents the general conclusions and some proposals for future work.

2 Related Work

Learning Ecosystems are known as a complex group of elements that interact and coexist with each other to enhance the teaching and learning process with the help of technology. The factor related to interaction is quite important based on the fact that the main actor will be the student. In [2], the authors emphasize that the human factor has a main role in the definition and development of the whole process inside a learning ecosystem and should be taken into account constantly in order to engage the participants in each of the learning activities. Additionally, authors in literature [3] defined a learning ecosystem metamodel as a platform-independent model to define learning ecosystems based on an architectural pattern [4, 5].

Among the different components of a Learning Ecosystems it is important to mention the process related to Learning Analytics (LA). In a broader sense, one of the leading institutions in the field of Learning Analytics (LA) is the Society for Learning Analytics Research - SoLAR - which since 2011 is organizing events to disseminate scientific research related to the field of engineering and education in subjects related to LA and knowledge. In 2015, the 5th International Conference on Analysis and Knowledge of Learning (LAK 2015) was held. Nested to this event, the First International Workshop on Visual Aspects of Learning Analytics was organized in which a series of questions were discussed and analyzed in an interactive way, such as: What type of data are being visualized? What tools were used to clean up the data (if any)? Who are the visualizations for (student, teacher, manager, researcher, other)? How are the data visualized? What interaction techniques are applied? What tools, are libraries, data formats are being used for technical implementations? What workflow and what strategies are used to develop visualization? These questions represents and active research topic nowadays to provide dashboards and useful representations for the teachers to make decisions and improve the interaction among the participants. In this sense it is possible to affirm that Learning Analytics tools are key players in any learning ecosystem to provide valuable information to decision makers.

The term Learning Analytics has been discussed by several researchers; Siemens expresses that it is the use of intelligent data, data produced by the student and analysis models to discover information and social connections, and to predict and advise on learning [6]; Ferguson considers that LA is the use of Big Data to provide actionable intelligence for students and teachers [7] and Pardo et al. indicates that the LA allows tracking the fingerprints generated in virtual environments to develop practical recommendations that can support student learning [8]. The Society for Learning Analytics Research (SOLAR) at the first International Conference on Learning Analytics and Knowledge (LAK-11) concluded that LA can be considered as the measurement, collection, analysis and reporting of data about learners and their contexts in order to understand and optimize learning and the environments in which it occurs [9]. Also in the Horizon Report in 2016, LA was defined as an educational web analytics application aimed at a learner profile, a process of collecting and analyzing data on the individual interaction of learners with online learning activities [10].

It is evident from literature that in summary Learning Analytics aim to improve the learning processes that are developed in contexts mediated by technology, empowering pedagogical interventions in order to measure, compile, analyze and understand the actions of the tutor and student in their contexts. In other words, LA are based in a systematic process that makes use of digital learning registers to determine the pedagogical and didactic actions required in educational contexts and thus improve learning [11].

Authors in [1] present a model of analysis of educational data based on visual analytics, learning analytics and academic analytics. By means of a system called VeLA that allows analysis of exploratory and confirmatory data, in interaction with the information obtained from a typical learning management system and present them through different graphs. In the scope of a Learning Ecosystem, LA represented by dashboards or visual aids to the stakeholders enhance the overall learning experience. Additionally, authors in [1] highlight that the goal of this techniques is to help users discover unexpected and surprising anomalies, changes in patterns and relationships that are then evaluated to develop new ideas. Overall these visual features make it possible to obtain a mental model of complex data and new knowledge [1].

Considered as a branch of learning analytics, social network analysis is the actual procedure allowing the researchers to study interactions and relations between individuals in a social network [12]. Despite the importance of SNA in learning environments, authors emphasize [12] that there is a clear lack of this feature in most of the current Virtual Learning Environments that make part of a Learning Ecosystem. Related to social networks it is possible to identify the discussion forums in the virtual courses as key elements that are activities that strategically planned can generate spaces for the construction of knowledge and supported by tools that allow the monitoring, measurement, collection, analysis, and visualization of interactions between students and tutors. Forums can provide inputs for the pedagogical reorientation of the tutorial action. Authors in literature describe interesting experiences using LA to identify the influence of leadership on the academic performance of work teams [13], perfectly applied to the scenario of forums. Authors also presented a Comprehensive Training Model of the Teamwork Competence with positive findings [14, 15]. In this sense, virtual tutors can benefit from the application of LA to know the interactions that develop when they propose a forum for the presentation, discussion, reflection and debate of ideas.

In terms of engagement, authors [16] have proposed several strategies to create a sense of belonging and motivation to take part of Massive Open Online Courses (MOOCs), that in the last year are being part of a complex Learning Ecosystem with high drop-out rates and a beginning work to be accessible to all [17, 18].

Within the described works it can be valued that systems and tools have been created for the application of LA in virtual courses learning activities; complex systems supported in the mining of educational data and tools that can be considered complements of existing educational platforms. In addition, it can be determined that these tools can facilitate the work of the virtual tutor in terms of his pedagogical intervention, providing information to be visualized and analyzed in a timely manner.

3 Experience Description

Virtual Learning Environments, have opened new possibilities for the collection of digital records of learning through user interactions in environments, forums, resources, activities, content, among others. In formal online learning contexts, it is well known that the interaction, participation, social exchange and discourse-based knowledge building processes occur essentially in the course forums as the main communication system [19].

There are many advantages to using forums in a virtual course, but one of them is that it is a resource that provides a space for constant feedback from a group as knowledge is exchanged. Unlike an individualized task, forums constitute opportunities for the development of critical thinking through collaborative learning [19]. It is worth to notice the work of Salmon [20] describing a five-stage model for a structured framework that gradually constructs on participants’ previous experience. In the different stages the tutor makes use of different activities and tools to guide the student to achieve the learning objectives. It is possible to mention the following stages [20]: Access and motivation, online socialization, information exchange, knowledge construction and development. In this sense, it is important to mention that the social interaction plays a worthwhile role in the whole process in a learning ecosystem.

Thinking about the first stage: “Access and motivation”, the scenario that the Massive Open Online Courses is presenting nowadays represents high drop-out rates and some strategies should be prepared to overcome this issue. For this, the first objective of this work envisages to ask to the students a motivation essay in order to be enrolled in the course. This experience will compare the results from two groups.

The Learning Analytics applied to the forums makes it possible to identify the interactions that occur in a virtual course [19], the frequency of participation, as well as to detect patterns associated with social exchange and knowledge management. These digital learning records can be obtained with the use of LA tools focused on data visualization.

The purpose of the second part of the study was to analyze the interaction developed between students and tutors in the types of forums implemented in a virtual course and for this purpose, the Forum Graph tool was applied, which allowed innovation through visualizations and highlighted situations to be improved in virtual tutoring.

As mentioned above, the application of LA tools in virtual course forums can provide relevant information for pedagogical moderation and intervention and peer participation. To carry out the study, two virtual courses were selected, implemented in the Virtual Learning Environment and structured in five learning modules that were developed during five weeks. Each course was prepared with learning content and presented three discussion spaces: technical forum, social forum and academic forums. For this experience Forum Graph was applied as a Learning Analytics - LA - tool for the visualization of intervention actions from the participants.

The research method used was formulated under a quasi-experimental design; because it allowed observing the variables and describing them as they are presented in their natural, cross-sectional environment; because it was done in a certain time and space, with a mixed approach; because digital records of learning were obtained from the Virtual Learning Environment where qualitative and quantitative information was retrieved from the educational event through the forums, and with a descriptive scope; because it allowed describing the context and situations based on the application of LA tools in the types of forums used in the virtual course. On the other hand there was a descriptive scope; because it allowed describing the context and situations based on the application of LA tools in the types of forums used in virtual courses. The process was carried out through six phases: preparation, design, development, analysis, intervention and evaluation, based on the principles of the ADDIE Instructional Design Model [21].

For this experience, regarding the typology of forums, forums have been developed for presentation, doubts, debate, reflection, analysis, cafeteria, etc. [22]. Table 1 presents the classification of forums in three types: Technical, Social and Academic.

Table 1. Virtual forums classification
Full size table

The process was carried out through six phases: preparation, design, development, analysis, intervention and evaluation, based on the principles of the ADDIE Model [21] as follows:

Preparation:

A review of the internal and external tools of the Virtual Learning Environment to apply Learning Analytics in the forums was carried out, as well as the course planning, the determination of the moments for the development of forums, the bases for the call and the registration form.

Design:

In this phase the authors worked on the instructional design of the course, the writing of the forums, the prototyping, graphic elements, multimedia and interactive resources. An initial training was also given to all interested parties who had pre-enrolled in the course, on the use of the educational platform and forms of communication.

Development:

With the design of the elements of the course the authors proceeded to the implementation and evaluation of the activities and educational resources in the environment, as well as to the configuration and installation of the plugin selected for the application, in this case Forum Graph. In this phase the course started.

Analysis:

After each week of the course, the Forum Graph tool was applied to visualize the interventions of the participants in the support forum, social forum and academic forums. After having the graphs of the forums, the authors proceeded to identify the nodes [23] where the participation took place and to detect the participants with little participation, for decision making.

Intervention:

After analyzing the graphics of the forums, the next phase was to make pedagogical interventions through external tools to mediate learning and motivate the participants, in addition some academic forums were adapted to achieve greater interaction.

Evaluation:

Finally, in the last week of the experience, the tutors proceeded to verify the trajectory in the participation of the forums and the identification of the forums with greater participation and pedagogical intervention.

4 Results and Discussion

For the first part of the study an A/B testing process was conducted. The course related to soft skills with a focus on technology was open for enrollment. A first phase with a regular enrollment was conducted and then for the second phase the students had to write a short essay expressing the intention and interest to take the course as well as the motivation and time that they will devote to the course.

Using the data of the two phases, information related to the following variables was compiled for each learning experience:

  • EP Enrolled participants: indicates the number of participants registered for the learning experience.

  • P0 Students that never logged in: indicates the number of the enrolled participants that did not visit the virtual learning environment to begin with the learning experience.

  • P1 Students with more than one log in: indicates the number of participants that started the online training course.

  • P2 Active students with at least one learning activity submitted.

  • P3 Active students that completed half of the learning experience.

  • AP Total of approved students.

Another set of variables was defined based on the above data sets in order to have rates for comparisons between groups and to expand the scope of the study:

  • P1vEP: This variable indicates the relation expressed from the number of enrolled participants and the number of participants with more than one login. This rate indicate the amount of participants that really started with the learning experience. The aim of this rate is to have rough estimate of real participation in order to have a better resources planning and to encourage student’s commitment.

  • APvP1: This variable indicates the relation expressed from the number of participants that finished and approved the learning experience divided by the number of students that started with the learning experience.

  • APvP2: This variable indicates the relation expressed from the number of participants that finished and approved the learning experience divided by the number of students with at least one learning activity completed. In MOOC experiences, authors in literature affirm that a group of students enroll to a course for curiosity reasons and to have a sneak-peek of the learning contents but they do not have the intention to complete the course. The aim of this rate is to identify sustained engagement from students, based on the completion of at least one learning activity.

  • APvP3: This variable indicates the relation expressed from the number of participants that finished and approved the learning experience divided by the number of students that completed half of the learning experience.

The identified variables are: P1vEP (rate between students with more than one login and enrolled participants); APvP1 (rate between total of approved and students with more than one login); APvP2 (rate between total of approved and students with at least one activity); APvP3 (rate between total of approved and students that completed half of the training).

From the comparison presented in Table 2, specifically from the P1vEP rate, it is interesting that in average, 13.9% of the participants never started the course. On the other hand, it is really positive that the approval rate of students that completed half of the training is 61.9%. Overall, the approval rate of the first experience was 19.20%.

Table 2. Comparison between identified variables for the first experience
Full size table

Comparison rate between identified variables is presented Table 3 this group represented the students that completed an essay to be enrolled in the course showing a positive increase in the amount of students that finished and approved the course. Finally students had different opportunities to provide additional feedback about their perceptions and attitude towards the learning experience. In the anonymous survey there was an open question in which students were asked to provide any comment or suggestion. In general we get numerous positive responses.

Table 3. Comparison between identified variables for both learning experience
Full size table

The second part of the study was conducted by students from the Faculty of Humanities who represented twenty cities in the country of Guatemala: Sololá, Guatemala, Petén, Sacatepéquez, Alta Verapaz, Baja Verapaz, Jutiapa, Jalapa, Retalhuleu and Chiquimula. This was a subset of the second group of participants.

The total number of participants in this particular experience were 55. The group of participants was made up of 76% women and 24% men, ranging in age from 18 to 25 years (17%), 26 to 35 years (52%), 36–45 years (25%) and 46–55 years (6%), who voluntarily decided to participate in the course. The moderation of the course was led by five virtual tutors.

For the Technical Forums developed to raise and resolve doubts related to the use of the platform the Forum Graph presented single node answers. Figure 1 presents the visual analysis from the Learning Analytics tool showing the participation of the tutors as an active player solving particular questions. In this forum the presence of singular nodes is shown and it is evident that there was no need of a further discussion related to the technical support.

Fig. 1.
figure 1

Visual analytics of technical forum

Full size image

The Social Forums, representing the online socialization phase was depicted as a Coffee Shop forum developed to generate a space for free dialogue and the creation of social links among the participants.

Figure 2 presents a visual representation of the active interaction among participants. In this experience more than 80% of the participants took part of the social threads. It is worth noting that the independent comments are also considered as significant contributions within the forum, however, not with the expected transcendence.

Fig. 2.
figure 2

Visual representation of social forum

Full size image

The academic forums developed to raise doubts, debate, reflect and give solution to a particular case presented a high volume of participation throughout the course.

Figure 3(a) shows a degree graph indicating that half of the participants in the course raised doubts about a topic during the course. The interaction between the participants is interesting because they were collaboratively helping each other. The graph shows the number of participants and the diameter of each node is proportional to the number of interventions. In this case the central node represents the leadership of the tutor in front of the statements made by the students.

Fig. 3.
figure 3

(a) Visual representation QA forums (b) Discussion forum representation

Full size image

Figure 3(b) shows the participation of more than 85% of the learners who answered the question presented by a tutor, it is possible to identify the horizontal interaction and debate from the students with other participants generating communication nodes. Figure 4(a) shows the participation of the 100% of the participants given that it was an assessment to contribute their opinion. Finally, Fig. 4(b) shows the active participation of the students. It is also visualized that the participation of tutors is lower than in previous forums all based on the type of learning activity designed.

Fig. 4.
figure 4

(a) Thoughtfulness forums (b) Representation resolution forums

Full size image

The tutors of the courses expressed a positive evaluation on the importance of having the visual aid to help them to make smart decisions to reintroduce learners with troubles and have a daily screenshot of the status of the course.

5 Conclusions and Future Work

The study discussed in this paper had two principal objectives. The first objective was to stimulate engagement of participants in a virtual learning environment through the invitation to write an application essay on their motivation to be part of the learning experience. This first experience presented a positive reaction in terms of participation in a Massive Open Online Course (MOOCs). The second objective within a subset of the participants in the study was to evaluate the importance of using visual learning analytics to improve and enhance interaction in learning ecosystems, where the participant plays a main role and a visual aid could provide to the teachers different mental models and early alerts related to change of patterns or really low participation. It can be concluded that the experience of innovation applying Learning Analytics in forums was satisfactory, it allowed to identify that the academic forums - reflexive, the social forums and the academic - resolutive forums, were the ones that developed greater participation in the students of the virtual course. It is pertinent to mention that the intervention of the tutors was demonstrated more in the forums of doubts than in the others; therefore, as a future work an investigation related to the competences that the virtual tutors must possess to carry out a work of follow-up and appropriate moderation in a virtual course can be raised. The use of LA tools applied to forums that are developed in virtual courses can highlight the interventions of students and tutors in different learning scenarios.

The results in this study provide evidence that despite the inherent openness behind MOOCs, participants should be encouraged to have a commitment to finish the courses and the standalone discussion forums will not be the only element to consider. In fact the Social Network Analysis will play an important role in a learning ecosystem.

The future work envisages the inclusion of a strong component based on different social network tools, not only discussion forums.