Keywords

1 Introduction

The Internet of Things (IoT), as a realm that enabled connection and interaction of digital and physical entities, represents global network of smart objects along with the set of supporting technologies and combination of applications and services utilizing such technologies into new functional possibilities and opportunities [12]. As rapid advances in underlying technologies are constantly opening new doors for novel applications that promise the improvement in quality of our lives [24], such global information-based network of uniquely addressable heterogeneous objects and sensors that seamlessly blend with the environment [6], quickly found its application in all aspects of human activity. Education is no exception as there are already IoT supported teaching and learning environments. On one hand, ubiquitous learning environments have been explored throughout specification of their technical framework and the system architecture [19, 25], and on the other hand by exploration of the IoT environment effectiveness in lifelong learning [3]. However, to our best knowledge, there are no reports of using a combination of software, hardware and liveware objects (which we call IoT ecosystem) with the aim of students’ activities tracking and success prediction.

User experience (UX) represents an essential part of every software lifecycle that is user-centered, especially in stages of its design and evaluation. In the IoT context, user experience is not only related to active interactions with the tangible product, but also to passive confrontations with “invisible” elements in the IoT ecosystem (e.g. sensors outside the user’s sight) that may provoke various perceptions of the system and resulting emotions. As such, anticipation of hedonic quality of the system plays an important role in the design of the future system. In this paper we examine anticipated UX in the context of IoT ecosystem prototype we developed for students’ activities tracking. The paper capitalizes on our prior work [16] in which we introduced a set of factors that affect the academic success of students and proposed an architecture and an interactive prototype of the IoT ecosystem meant for students’ activities tracking and success prediction. More specifically, it investigates relevant pragmatic and hedonic UX facets with respect to the nature of the aforementioned IoT ecosystem.

This paper is structured in six chapters, starting from introduction and chapter on research background where we bring the results of our literature review regarding the use of IoT in learning environments with the focus on evaluation of its user experience. In the third chapter we shortly introduce the design and features of an IoT ecosystem employed in students’ activities tracking and success prediction. The subsequent chapters present the methodology and results of the study aiming to examine the relevant UX dimensions of the proposed prototype of the mobile component of the ecosystem.

2 Research Background

Advancements in information and communication technologies (ICT) and their integration into learning environments bring a lot of changes into the way people learn. The main purpose of the ICT usage in learning is to provide better learning experience through aligning students’ interests, preferences, objectives and previous knowledge that would lead to effective knowledge acquisition and retention. While mobile devices and technologies have enhanced learning portability and mobility, IoT is contributing to construction of ubiquitous learning environments. In such environments, learners and technologies, real world and virtual worlds, are interconnected, and IoT serves as a backbone that enables recognition and identification of environmental objects, and retrieves network information to facilitate their adaptive functionality [25].

In the context of learning environments, we still do not have a clearly prescribed guidelines for development of the IoT ecosystem. Joyce et al. [8] designed the IoT ecosystem which was piloted in eight schools in England, but although it had the data collection mechanisms, its main purpose was to enable discovery-based learning with the help of supporting mobile and web technologies. Authors concluded that their learning/studying ecosystem, where environmental data is readily available, provided a new learning experience enabling students and teachers to observe the real time data and to discuss it. On the other hand, Georgescu and Popescu [5] discuss that IoT, if introduced to eLearning in person-centered design process, could open different possibilities in enhancing learning experience and learning process itself. Even more, Bandara and Ioras [1] have been using IoT to enhance gamification techniques in eLearning environments. Furthermore, the same authors state that “IoT has the ability to improve upon the learning experience by allowing for real-time and actionable insights into student performance” thus drawing a clear line connecting IoT and user experience in learning environments.

User experience (UX) consists of wide range of factors that can be classified into three categories: context around the user and system (social, physical, task, technical and information context), the user’s state (e.g. motivation, emotions, expectations) and systems properties (e.g. functionalities, aesthetics, responsiveness, sustainability, etc.) [17]. Although the concept of UX was introduced in the late 80’s, the consensus on its scope and definition has not been reached yet. Law et al. [11] argue this is because UX is a dynamic and subjective concept that results from interaction with a piece of software thus depending on the context of use and perceptions of individuals. When the nature of UX is taken into account, literature distinguishes two streams of researchers and practitioners. While the first one considers UX as a synonym for usability focused on the pragmatic (utilitarian) facets of software design and evaluation, the second one believes UX is a distinct concept, more recently referred to as quality of experience (QoE), dealing with hedonic (emotional) dimensions in the aforementioned respect. As an outcome of the comprehensive literature review, Bargas-Avila and Hornbæk [2] uncovered generic UX, affect and emotion, enjoyment and fun, aesthetics and appeal, hedonic quality, engagement and flow, motivation, enchantment, frustration, values, and spontaneity as the most common UX dimensions. Current studies (e.g. [9, 20, 21]) in the field indicate that learnability, usefulness, availability, attractiveness, and reliability of software as well as users’ efficiency, effectiveness, and productivity are relevant antecedents of UX.

Regarding various aspects of user experience present in learning environments, Simic et al. [19] and Xue et al. [25] focused their work only on technical aspects of IoT-based learning environments development, while Cheng and Liao [3] besides that examined quality aspects of the system that includes usability. Considering complexity of ubiquitous environments, in a new product development it is essential to evaluate design concepts to “ensure predetermined design goals and to obtain design feedback” [4]. Design concepts enable users to form user experience before the user actually interacts with the real product, application or the system (anticipated UX as identified in [17]), in addition to experiences that can take place during or after the item usage. In that respect, early design concepts and actual design decisions could be drawn not only from requirements identification, but also from UX research about users’ expectations and anticipated feelings that could be provoked by the future product or system. In addition, early system representations in the form of mock-ups and prototypes could be used in later design stages to evaluate proposed design.

While there is a plethora of UX evaluation methods that can be applied in various development phases [23], UX evaluation design methods for IoT ecosystem are scarce. One method can be borrowed from the product design, e.g. UX evaluation method of design concept based on multi-modal experience scenario [4], which employs semantic questionnaire and can be applied in the earlier design stages, but requires development of product usage scenarios and physical prototypes. Another approach presented in [18] is built upon integration of the quality of service (QoS) and the QoE from a user-centric perspective since IoT is provided through some form of service. The QoE model [18] gives good predictions on users’ quality experiences in the use of IoT services, but might be cumbersome to use in practice due its quantitative and qualitative approach.

In that context, our approach was to create a measurement tool for evaluation of IoT ecosystem that would be feasible to use in early design stages and enable assessment of relevant UX dimensions.

3 Evaluated IoT Ecosystem

From human-computer interaction (HCI) perspective, the development of ubiquitous environments that incorporate IoT, calls for the flexibility in the design process. Main characteristics of user-centered design of new media products are: a) the design process that is based on various prototypes, from low-level to high-level, b) the design is feature-driven, where most important features are implemented first, and c) the design process is incremental and iterative [4].

Those pillars were present in the approach we took while designing IoT ecosystem for learning environment. Our main goal was to design a prototype of the IoT ecosystem meant for students’ activities tracking and success prediction, as well as evaluate relevant UX dimensions in that respect. The design was materialized as an interactive prototype of mobile application as the most tangible part of the future IoT ecosystem.

A user-centered design approach, which is further elaborated in [16], consisted of the following:

  1. (1)

    identification and analysis of the student needs relevant for achieving academic success by employing questionnaire, focus group, and the design of personas and storyboards,

  2. (2)

    requirements definition based on needs analysis in order to propose IoT ecosystem functionalities and architecture, and

  3. (3)

    design of wireframes and interactive prototypes with an aim to evaluate IoT ecosystem idea.

In the first phase, a preliminary study was conducted to effectively determine what students’ features were mostly connected to academic success. The study employed two techniques for data collection, one being the focus group with university professors, and the other a student-involved questionnaire. Both techniques revealed features that were deemed important to achieve student success, but also identified means that could be used to detect or measure some of those features like motivation, goal centeredness, work related obligations, time management, etc. In addition to the results of the questionnaire and the focus group, five personas were created in order to identify student groups relevant for the learning environment. Two personas were chosen to further elaborate students’ needs throughout sketching the storyboards. Outputs identified by aforementioned techniques helped in forming the idea of an IoT enabled system employed in tracking students’ behavior in order to improve students’ success by informing them of recognized patterns and suggesting the ways to improve their lifestyle.

In the second research phase, we built on the above mentioned study results and identified basic features that the system should implement. Thus, software requirements specification document included the requirements description of the following main components: My classes, My Behaviours and Habits, Study Marketplace, Ambience and Analytics. The purpose of the ‘My classes’ feature was to enable student to have an insight on his/her own courses’ obligations but also on the level and quality of their fulfilment. Special attention was paid to the option of enabling student to be able to compare his/her own results with the results of other groups of students without intruding their privacy. ‘My Behaviours and Habits’ section described functionalities grouped either as academic (like tardiness or study time) or leisure (like sleeping habits or other routines). ‘Study companion’ section includes the options of offering or seeking help in any aspect of student’s life. On the other hand, section on ‘Ambience’ would give the student a possibility to observe the suitability of ambient for learning in it. Finally, the overall academic lifestyle analytics were put together into specific feature named ‘Analytics’.

The usual activities of defining the system architecture and structure design followed and topological view on the system proposed in [16] is presented in Fig. 1. Faculty building areas (library, auditoriums, and teaching staff offices) and students’ dormitory public areas are equipped with sensors able to track some aspects of students’ behavior. Thus, the IoT ecosystem can draw conclusions based on data collected physically (via sensors) and data available on faculty services (scheduling, library records, event management and learning management system (LMS)) and finally present the data to the user via API used by the mobile application.

Fig. 1.
figure 1

IoT ecosystem overview

Full size image

In the third phase, based on outputs from the first and the second phase, wireframes and an interactive prototype for an Android mobile application were created. Wireframes presented the structure and initial content of the application screens (Fig. 2) and its design was qualitatively evaluated by two members of the design team who were not involved into their creation. The interactive prototype was redesigned based on the feedbacks from the design team, and included graphic design, more elaborated content and interactivity among the screens (Fig. 3). The aforementioned one didn’t have the interactive tabs on every screen in order to identify whether the students would reveal the flaws in the prototype.

Fig. 2.
figure 2

Examples of a mobile application wireframes

Full size image
Fig. 3.
figure 3

Examples of screens of an interactive prototype

Full size image

4 Methodology

Procedure.

The study was carried out during the winter semester of academic year 2017/18. and was composed of two parts: (1) scenario-based interaction with IoT system prototype and (2) its evaluation with respect to pragmatic and hedonic facets of UX by means of the questionnaire. Upon arriving to the lab, the participants were welcomed and briefly informed about the purpose of the study. To ensure high accuracy of the gathered data, study participants were given detailed written and oral instructions on the implementation of each particular step of the research. At the beginning of the scenario performance session, every respondent received URL with the interactive prototype and URL of on online questionnaire. Participants were asked to browse throughout the prototype screens following various paths using interactive elements on the screens, and thereafter to fill out the questionnaire. At the end of the study, respondents were debriefed and thanked for their participation.

Apparatus.

Data were gathered through the questionnaire which was administered online by means of the Google Forms. The questionnaire comprised 4 five-point bipolar items related to participants’ demographics, 4 five-point bipolar items aimed for exploring respondents’ awareness of and experience in interaction with IoT systems, one open-ended item where the respondents could leave their remarks and 86 items with bipolar verbal anchors meant for measuring pragmatic and hedonic dimensions of user experience. The sum of responses to items assigned to corresponding attribute was used as a composite measure which reflects particular facet of user experience.

Framework.

Based on the literature review that included measuring instruments designed for evaluating user experience (e.g. [7, 10]), studies (e.g. [13,14,15]) in which various facets of quality were examined as well as international standards for quality assessment (e.g. [22]), a novel framework meant for measuring 17 pragmatic (accessibility, advantageousness, availability, context coverage, compatibility, customizability, dependability, ease of use, effectiveness, efficiency, familiarity, manageability, minimal workload, perspicuity, privacy, scalability, and trust) and 5 hedonic (attractiveness, connectivity, novelty, satisfaction, and stimulativeness) dimensions of user experience with respect to the IoT ecosystem prototype was designed.

Accessibility refers to the extent to which the IoT ecosystem prototype can be used by students with the widest range of characteristics and capabilities. Advantageousness denotes the degree to which students perceive the IoT ecosystem prototype as beneficial. Availability examines the extent to which features of the IoT ecosystem prototype are continuously reachable to students. Context coverage measures the degree to which the IoT ecosystem prototype is usable within and beyond initially intended contexts of use. Compatibility explores the extent to which the IoT ecosystem prototype operates properly with different types of devices and among different environments. Customizability refers to the degree to which the IoT ecosystem prototype can be personalized to meet students’ needs and suit the characteristics of the task at hand. Dependability denotes the extent to which the IoT ecosystem prototype is perceived by students as unfailing, predictable, and secure. Ease of use examines the degree to which it is easy for students to become skilled in interaction with the IoT ecosystem prototype and is easy for them to memorize how it is used. Effectiveness measures the extent to which the IoT ecosystem prototype enables students to perform tasks accurately and completely. Efficiency explores the degree to which interaction with the IoT ecosystem prototype saves students’ resources (e.g. time). Familiarity refers to extent to which interaction with the IoT ecosystem prototype is similar to systems previously used by students. Manageability denotes the degree to which the IoT ecosystem prototype is well structured and governable. Minimal workload measures the degree to which interaction with the IoT ecosystem prototype requires a small amount of physical and mental effort. Perspicuity examines the extent to which interaction with the IoT ecosystem prototype is unambiguous. Privacy explores the degree to which the IoT ecosystem prototype protects students’ data and artefacts from unauthorized use and disclosure. Scalability refers to the extent to which the IoT ecosystem prototype is capable to operate under an increased or expanding workload. Trust denotes the degree to which the IoT ecosystem prototype is receptive to the students’ needs and has all resources required to successfully perform its activities.

Attractiveness measures the extent to which the IoT ecosystem prototype has visually appealing user interface. Connectivity examines the degree to which the IoT ecosystem prototype is integrating, inclusive, and brings students closer to each other. Novelty signifies the extent to which the IoT ecosystem prototype is distinctive among other ones. Satisfaction indicates the degree to which interaction with the IoT ecosystem prototype has met students’ expectations and arouses positive emotional responses in them. Stimulativeness represents the extent to which interaction with the IoT ecosystem prototype is focused and encourages students’ creativity.

5 Results

Participants.

A total of 50 respondents took part in the study. They ranged in age from 19 to 36 years (M = 22.44, SD = 2.557). The sample was composed of 76% male and 24% female students. At the time the study was conducted, the majority (50%) of participants was in the third year of the undergraduate study, 38% of them was in the first year of the graduate study, 8% of them was in the first year of the undergraduate study whereas remaining 4% was in the second year of the graduate study. All respondents were enrolled to study programs in the field of information and communication sciences. The majority (94%) of participants were full-time students. When awareness about IoT systems was considered, 62% of students reported they are familiar with the concept of IoT while 32% of them stated they are acquainted with IoT technologies. On the other hand, when experience in interaction with IoT systems was examined, 52% of study participants reported they used at least two IoT systems while 28% of them took part in the development of at least one IoT system.

Findings.

The results of data analysis revealed that 72% of study respondents reported that the IoT ecosystem prototype could be used by their peers regardless their capabilities and characteristics. More specifically, 56% of students think that the IoT ecosystem prototype can be used by everyone, while 88% of them think that the IoT ecosystem prototype is accessible. It was also found that 83.33% of study participants find the IoT ecosystem prototype advantageous. Namely, majority of students perceived the IoT ecosystem prototype as useful (88%), practical (88%), suitable (86%), functional (80%), usable (80%), and valuable (76%). Study findings also indicate that 81% of respondents agree that the IoT ecosystem prototype is continuously reachable because 82% and 80% of students find it available and disposable, respectively.

As much as 49.50% of study participants agree that the IoT ecosystem prototype is usable beyond initially intended contexts of use. More concretely, 68% of students think that the IoT ecosystem prototype has variety of possible applications, 58% of them believe that the IoT ecosystem prototype has very specific purpose, 44% of students agree that the IoT ecosystem prototype is usable only within aimed context, while 70% of study participants find it technical in nature. The analysis of gathered data also uncovered that 77% of individuals involved in the study agree the IoT ecosystem prototype operates well among diverse environments because 78% of students find it compatible and 76% supported in that respect. The same holds for the extent to which the IoT ecosystem prototype is customizable in general (77%) as well as the degree to which students’ can personalize it (78%) and the IoT ecosystem prototype is adjustable to the context of its employment (76%). It also appeared that 73.60% of respondents perceived the IoT ecosystem prototype as dependable which is due to most students consider it reliable (82%), stable (80%), bug-free (56%), predictable (72%), and secure (78%).

According to the results of data analysis, 93% of study participants agree that is easy to use the IoT ecosystem prototype. Namely, 94% of students believe that is easy to become skillful in interaction with the IoT ecosystem prototype while 92% of them think that is easy to recall how to use the IoT ecosystem prototype functionalities. It was also discovered that 75.50% of study participants stated that the IoT ecosystem prototype enhances their performance in executing tasks which is because majority of students find it effective (92%), productive (90%), complete (60%), and whole (60%).

Study findings are implying that 78.50% of students believe that the IoT ecosystem prototype improves their efficacy in performing assignments owing to majority of them think that the IoT ecosystem prototype has acceptable response time (92%), is fast (86%), efficient (84%), and saves their resources (52%). As much as 51% of individuals that took part in the study reported that interaction with the IoT ecosystem prototype does not differ much from previously used applications and systems. More specifically, 64% of students found the IoT ecosystem prototype familiar whereas only 38% of students agree the IoT ecosystem prototype is similar to other systems they previously employed. The IoT ecosystem prototype was perceived by 86% of students as governable because the same proportion of them find it organized, controllable, and manageable.

A total of 80.67% of respondents agree that the interaction with the IoT ecosystem prototype requires minimal workload. Namely, it was found that majority of students reported that employing the IoT ecosystem prototype does not require a lot of physical (82%) neither mental (78%) activity nor makes students tired (82%). Moreover, 90.50% of study participants agree that interaction with the IoT ecosystem prototype was unambiguous which is due to majority of them find it understandable (96%), clear (94%), easy (92%), and consistent (80%). Only 38.67% of students think the IoT ecosystem prototype preserves their data and artefacts from unauthorized use. More concretely, 28% of respondents believe the IoT ecosystem prototype takes care about their privacy, 38% of them is convinced that the IoT ecosystem prototype would not share their personal data with third parties, while half of the study sample think that the IoT ecosystem prototype protects the privacy of their artefacts.

Study findings also indicate that 67.33% of respondents found the IoT ecosystem prototype scalable. Namely, majority of students reported that the IoT ecosystem prototype enables simultaneous work of a large number of users (80%), supports simultaneous work on a large number of tasks (66%), and allows the execution of complex tasks (56%). As much as 81% of students agree that the IoT ecosystem prototype is receptive to their needs and has all resources required to successfully perform its activities which is due to 82% of study respondents find it trustworthy while 80% of them think the IoT ecosystem prototype is competent.

All the set forth implies that in the context of IoT ecosystem prototype advantages, the most relevant pragmatic UX dimensions are ease of use, minimal workload, and perspicuity while IoT ecosystem prototype disadvantages could be mostly related to privacy, context coverage, and familiarity. Study findings related to pragmatic UX dimensions are summarized in Fig. 4.

Fig. 4.
figure 4

Summary of study findings related to pragmatic UX dimensions

Full size image

It appeared that 74.75% of study participants consider the IoT ecosystem appealing. More specifically, majority of students perceived the IoT ecosystem prototype as pleasing (88%), attractive (84%), friendly (78%), professional (76%), beautiful (74%), classy (74%), and presentable (74%) while half of the sample believe the IoT ecosystem prototype is valuable. Moreover, 78% of respondents reported the IoT ecosystem prototype encourages students networking because majority of them stated the IoT ecosystem prototype is integrating (90%), inclusive (76%), and brings students closer to each other (68%).

The results of data analysis revealed that 67.67% of study respondents reported that the IoT ecosystem prototype is distinctive among other ones. Namely, most of students find it creative (76%), innovative (72%), leading edge (70%), inventive (70%), original (64%), and unique (54%). As much as 84.29% of individuals involved in the study is happy with the IoT ecosystem prototype which is because majority of the sample agree the IoT ecosystem prototype is good (88%), likeable (88%), enjoyable (86%), pleasant (84%), meets students’ expectations (84%), makes good impression (82%), and meets students’ needs (78%). Finally, 72.44% of study participants agree the IoT ecosystem prototype provides stimulating effects. Namely, most of students believe the IoT ecosystem prototype encourages their concentration on task execution (90%), is supportive (86%) and interesting (86%), successfully retains students’ attention (82%), is exiting (72%), motivating (72%), courageous (58%), and challenging (54%), and stimulates students’ creativity (52%).

Taking into account all above, advantages of IoT ecosystem prototype with respect to hedonic UX facets are most commonly colored with satisfaction and the most sparsely with novelty. Study findings on hedonic UX aspects in the context of IoT ecosystem prototype are presented in Fig. 5.

Fig. 5.
figure 5

Summary of study findings related to hedonic UX dimensions

Full size image

When the qualitative data are considered, 38% of study participants provided 66 comments of which the same proportion (50%) were related to the benefits and to the flaws of the IoT ecosystem prototype. In the context of benefits, the IoT ecosystem prototype was perceived by study participants as all-in-one solution that includes all necessary functionalities an academic citizen requires thus representing very good idea and initiative with a potential to be employed for educational purposes and help students to achieve as much success as possible in their study. Comparison among students, monitoring sleeping pace and the quality of air, graphical representation of results, intuitive navigation and understandability, visually appealing user interface, study time average, my classes categories, and a social aspect that enables students to be better connected are the most relevant features the students reported in terms of advantages of the IoT ecosystem prototype.

On the other hand, the students expressed their concerns with respect to the privacy of data that is going to be stored in the IoT ecosystem as well as related to the functionality of comparing results with other students that could result in the opposite effect and demotivate students even further or even cause misbalance in their life habits. In addition, some of the respondents are not convinced that built-in feature of social interaction will be often used. Study participants also determined several bugs related to study marketplace, my posts, and my classes which should be addressed. Inconsistencies related to the calendar, lack of labels in active lifestyle display and graphical display of data, boring and common interface design, lack of tag search, and too much information at one place were the most often IoT ecosystem features study participants reported in terms of its disadvantages.

Finally, students suggested some additional features such as module for providing reviews on courses, faculty staff, and teaching materials, displaying each post in separate tab, schedule of lectures, digital post-it for taking notes, simplified calendar for planning learning activities, etc. All the aforementioned indicates that pros of the IoT ecosystem prototype are most commonly related to advantageousness, satisfaction, stimulativeness, and attractiveness whereas cons in the same respect are most often in relation with effectiveness, attractiveness, dependability, perspicuity, and stimulativeness. The most relevant UX facets with respect to reported advantages and disadvantages are presented in Fig. 6.

Fig. 6.
figure 6

Summary of study findings with respect to qualitative data obtained from students

Full size image

6 Concluding Remarks

Although the IoT quickly found its application in all aspects of human activity, including education and learning, in our literature review we found no reports of using a combination of software, hardware and liveware objects (i.e. IoT ecosystem) with the aim of students’ activities tracking and success prediction. Thus, in our previous research, we identified the needs, defined features and designed the architecture and structure of such system. By following the user-centered design process we also defined storyboards, wireframes and interactive prototype of Android mobile application as the only part of the ecosystem which is exposed to the end users.

As presented in this paper, the main objective of this research was to determine and examine both pragmatic and hedonic attributes that constitute UX assessment framework in the context of the introduced IoT ecosystem prototype.

Regarding evaluation of an IoT ecosystem prototype, as an outcome of literature review which included relevant studies, measuring instruments, and international standards in the field, questionnaire composed of 86 items with bipolar verbal anchors and one open-ended item was created. As a follow up, an empirical study was carried out in which students served as participants. Although there is still room for improvements, results of data analysis have shown that proposed IoT ecosystem prototype was well received by students. Taking into account together quantitative and qualitative study findings, it appears that advantages of IoT ecosystem prototype are most commonly related to 3 pragmatic (ease of use, minimal workload, and perspicuity) and 3 hedonic (attractiveness, satisfaction, and stimulativeness) UX dimensions whereas disadvantages in the same respect are associated with 6 pragmatic (context coverage, dependability, effectiveness, familiarity, perspicuity, and privacy) and 3 hedonic (attractiveness, novelty, and stimulativeness) UX facets.