Keywords

1 Introduction

Student-faculty formal and informal interactions are positively related to student persistence, satisfaction, and learning. Nevertheless, students who engage less with faculty members and institutional agents are more likely to feel disconnected to the institution and unsatisfied with the campus environment [1]. Students’ social connections with peers and faculty may also enhance students’ sense of belonging to the institution, ultimately leading to greater classroom engagement in reasoning activities [2]. The students consider that a high-quality student-faculty interaction occurs when faculty care and support them [3]. The Federal University of Ceará, in Brazil, supports low-income students with the University Restaurant (UR), a unit designed to provide balanced meals to students and staff. The UR has the capacity for 3,000 meals/day and serves breakfast, lunch and dinner. The students constantly discuss the taste and quality of each meal, the staff service, and the usually crowded lines. However, they are unable to expose their opinion for a massive group of students or give feedback to the institutional agents in a simple manner. As a result, the students may be uninspired to interact with faculty members and institutional agents, hence increasing the feeling of dissatisfaction with the campus environment.

To help to overcome this situation, the mobile social application Eai? (Brazilian Portuguese for “Hey, what´s up?”) was developed. In this application, students can follow the weekly menu of the UR, qualify, and share opinions and reviews about the daily meals. The students can use the Android application Eai? to qualify the state of the UR lines and to know, in real time, what other users are saying about the lines, and to know when is the best time to go to the UR. Also, Eai? provides a news feed about the University, and other also miscellaneous news, where the students can also interact. This work presents the evaluation of usability and user experience (UX) of Eai?, which used a multi-method approach aiming to understand the system usage and to explore the quality of user’s interaction and perceptions of the system. Our research hypotheses are: (1) Eai? can positively impact the students’ behavior, increasing their social connections with peers; (2) Eai? is useful to assess the quality of the UR service, increasing the students-faculty interactions; (3) To provide tools like forum and news related to the University positively impacts student’s sense of belonging to the institution. The results show that overall the students find Eai? useful, trustable, stimulating and easy to use, although they pointed some necessary improvements. We also discuss how the answers to hypotheses 2 and 3 vary according to the student’s experience with the UR, and their use of the application.

2 Eai?, a Mobile Social Application

Eai? is a place-centric mobile social application, where college students can share a common interest and constantly interact, benefiting each other with the knowledge and experience related to their current location, the University Restaurant (UR). The application provides diverse facets of social interactions, such as social feedback mechanisms, personal identification, and identity expression [5]. Features such as rating, reviewing, discussion forum and customizable profiles support the possibilities of social interaction in Eai?. The application also allows students to share at any time, their opinions about each one’s personal experience in the University Restaurant. In this manner, Eai? enables a real-time exchange of location-related experiences of locally dispersed participants, so the students became part of a mobile community [4]. Currently, the application has over 2600 downloads in the Google Play platform, counts 194 reviews (183 are 5 or 4 stars), and its rating is 4.7 in the Store. The main functionalities of the application are listing the weekly meal menu of the UR, sharing reviews about the meals, showing the states of UR lines, and providing news feed and a discussion forum (Fig. 1).

Fig. 1.
figure 1

(A) List of meals of the day, (B) Details of a selected meal, (C) Pop up for rating and commenting about a meal, (D) Home Screen showing the state of the restaurant lines and the News feed

Full size image

The Fig. 1A illustrates the list of meals organized from Monday to Friday, according to the meal time. Figure 1B shows the details of a selected meal, as side dishes, and juice and desert options. Figure 1C demonstrates the reviewing process, where the user can rate and comment about a meal. At the Home Screen (Fig. 1D) the user can see the real-time state of the UR lines according to the interaction of users located at the UR. For this purpose, the application has a function where the users can classify the state of the line. The home screen also shows a short version of the news feed, which intends to increase the experience for a common user of the restaurant. The news is distributed into two categories: general news and news about the University campuses. This news is sorted by the user’s interest, calculated in an algorithm focused on variables of continuous use. The students can ‘like’ or ‘dislike’ the articles in General News, while in each item about the University they can comment and interact with each other. These commentaries may become a bigger discussion about the subject. In this case, it can be discussed at the Forum named Boca no Trombone (Brazilian Portuguese for “Blow the whistle”) where the students can expose their concerns and give feedback about the UR and other University services.

3 Evaluation Process

The assessment process of Eai? considered two of the challenges of deploying a mobile social application, utility and usability [6], while assessing the user experience as well. The evaluation planning was based on the DECIDE framework. To understand the system usage and to explore the quality of user’s interaction and perceptions of the system, we applied quantitative and qualitative methods. First, we collected quantitative data in the form of a questionnaire, adapted by the authors from the Computer System Usability Questionnaire (CSUQ) [7], and from the Questionnaire for User Interface Satisfaction (QUIS) [8]. It was administered to 109 users, among students and university staff, in September 2015. Due to the context of the evaluation (in the restaurant, when users were coming for the lunch or leaving after it), its purpose was to obtain an overview of the user feedback on the application. The questionnaire consisted of 14 closed-ended questions, answered in 5-point Likert scales, plus three open-ended questions. The questions acquired information about the application’s impact on the behavior of the users, the utility of the application to evaluate the UR service, and the user’s expectations.

The second phase of the evaluation occurred from October to November in the same year, to refine the data and to expand the understanding of the user experience for new users. We gathered qualitative data with semi-structured contextual interviews, observation of usage and informal conversation with 34 participants, among users and potential users, allowing a depth research. We performed the evaluations in group sessions with students belonging to three categories: students who currently do not take meals at the University Restaurant; students who regularly have lunch or dinner at the UR, but are not Eai? users; and students who regularly have the same meals at the UR and currently are Eai? users. During this evaluation, users completed a pretest questionnaire about their experience with technology and with the University Restaurant. Then, each user performed eight tasks in the application, selected from possible usage scenarios. For each activity performed, we observed and took notes of the reactions and both correct and incorrect user interactions. After this, in a structured contextual interview, users gave their opinion about the clarity, the efficiency and the importance of the functionalities, in the context of use of the UR. Users were also asked about the mistakes, difficulties, and experience (regarding emotions and feelings) while performing the tasks. After finishing the activities, the users filled out a post-test questionnaire, adapted from the Final User Questionnaire for Usability Evaluation of websites [9] and QUIS. Each session lasted 30 min on average. After the individual evaluations, the users were divided into two focus groups, where they shared and discussed impressions, opinions, and suggestions on the application.

4 Results and Discussion

In the first evaluation, the average age of the 109 Eai? users (63 female) was 23.5 years for women and 23.7 years for men. Among these, 73 % were students of the university and 11 % were employees of the University or outsourced. Overall, the application was considered useful (95 %), trustworthy (90 %), easy to use (92 %), stimulating (67 %) and worthwhile (81 %). Regarding the organization of information and the sequence of screens during the interactions in the application, most users gave positive feedback, considering Eai? well-Organized (87 %), and with a clear sequence of screens (86 %).

The second phase of the evaluation consisted in a deeper investigation with 34 students (9 female), remaining the same age of the first evaluated group. We identified that 80 % of the participants used to interact with their smartphones while having meals at the UR. They mainly use applications of instant messaging (66.7 %), social networking (47.6 %), music and games (28.6 %), and reading (19 %). It is important to point out that the functionalities of social networking and reading are also present in Eai?. It combines in a single application both the capabilities the students need for a better UR service and the ones they would like to have as pastime activities while at the UR. For this group of students, the overall evaluation presented slightly lower percentages, due to the number of students using Eai? for the first time (potential users). They considered the application useful (84 %), trustworthy (69 %), easy to use (66 %), and worthwhile (56 %).

The user experience assessment revealed some problems in the navigation, organization, and labeling which made the new users felt confuse and sometimes frustrated. It happened mainly in the activities of qualifying the lines, accessing the breakfast menu options, adding a meal as favorite and commenting on the news, and in the students’ forum. These results evidenced that Eai? was not as easy to learn and efficient as it should be, although even the new users acknowledged the utility of its functionalities. The discussions on the focus groups showed that the students like the overall idea of the application. They were excited about the new possibilities and pleased with the initiative of the University to improve the UR service by creating a social application. They expressed the desire to use an improved version of the application. Moreover, they expect Eai? to become an essential application for any student of UFC. The students suggested that Eai? should integrate other features relevant to the UR, such as buying UR credits, as well as possibilities of direct interaction among people in the UR.

4.1 Research Hypothesis#1: Eai? Can Positively Impact the Students’ Behavior

To find out if the current users were using application features according to the expected behavior, we evaluated how often they use the following functions: (1) to send information about the status of the line; (2) to access news; (3) to choose a dish based on application reviews, and (4) to send notes and reviews about the meals. The data collected from the questionnaires showed that Eai? can increase the students’ perception of social connections with peers, through the rating and comments about the meals. As Table 1 summarizes, the users would like to consume the information Eai? provides, but so far they do not show the same eagerness to provide information and collaboratively feed the application.

Table 1. Summary of the answers about the impact of Eai? on the habits and behavior of the user
Full size table

According to the questionnaires, 69 % of the students choose their meals based on the other users’ comments about the food quality on Eai?. Although for 31 % of them, it is not a daily practice yet. In the second phase of the evaluation, 77 % of the participants declared that most of the times they choose a meal just when they take a look at it, and not previously. It has been the predominant behavior pattern on the UR because there was no other trustful way to choose a good meal. The number of daily accesses to Eai? shows that this behavior is beginning to change. We believe that illustrating the menu with actual pictures of the meals available on the day, and allowing users to send their own pictures of the UR food will help to increase the number of students who choose their meals previously, based on the comments available on Eai?. These were suggestions the users gave to improve their sense of reliability on the application. However, to support sending pictures in this context is a very sensitive matter and must be planned carefully, to avoid problems of deceit, embarrassment, pranks, and violation of privacy.

The questionnaires’ data showed that 63 % of the Eai? users access the News feature (Table 1). The second investigation explained that users access the News while waiting on the lines, or when they are having meals by themselves. However, among the new users, 26.5 % think that this feature is out of context in the application, and 52.9 % said they would not use it often. Sending information about the lines, and rating/commenting about the meals are crucial functionalities for the proper working of Eai?. While 57 % of the users send information about their meals, 63 % declare that they do not send information about the lines (Table 1). The observations and contextual interviews showed that 91,2 % of the users found such features useful. However, they do not have the commitment to feed the application frequently. For the success of the application, it is necessary that more users will feed it with both information about the lines and the food, on a daily basis. A gamification approach could be used to stimulate the behavior of feeding the application every time a user goes to the UR. This strategy associated with real prizes and UR credits.

4.2 Research Hypothesis #2: Eai? Is Useful to Assess the Quality of the UR Service, Increasing the Student-Faculty Interactions

To analyze the perception of the students about the impact that Eai? brings to the student-faculty interactions, we gathered their opinion about whether the application improved: (1) the way they use the UR; (2) the collaboration among students; and (3) the communication with the University. In general, the users feel that the application has benefited them in the context of interactions with the University. As shown in Table 2, all the results are positive, and the users found Eai? useful to analyze the quality of the service and the food of the UR. They considered that Eai? increases the student-faculty interactions by improving the way how they use the UR (76 %), their collaboration with one another (68 %), and their communication with the University (53 %). To promote a better communication with the University, the application can aggregate a feature for feedback from the UR staff about changes in the menu based on the students requests and opinions, showing how the University listens and support them.

Table 2. Summary of the answers related to the expectations and perceptions of the users
Full size table

4.3 Research Hypothesis #3: To Provide Tools like Forum and News Related to the University Positively Impacts Student’s Sense of Belonging to the Institution

Eai? provides a forum that intends to enhance students’ social connections with peers, while the News feed is supposed to complement their connection with the faculty. A valuable and healthy social connection between student peers and with faculty may enhance students’ sense of belonging to the institution [2]. During the focus groups, we discussed the quality of these tools from the students’ point of view, and their perception of the benefits of using both the forum and the News. The answer varies according to the student’s experience with the UR, and their use of the application. Overall, the forum and the News to enhance the students sense of belonging to the institution. However, the students declared that after fixing some interaction problems, the forum and the News would be able to help to reduce the feeling that the faculty board is far from the students. Also, they affirmed that whenever their discussion topics would become a subject of interest to the alumni and faculty staff, they would feel empowered, and an active part of the community. Some students suggested that the application should also provide an easy way for the students to get in touch with the ombudsman’s office of the University.

The forum named Boca no Trombone is a virtual space where the students can expose their concerns and give feedback about the UR and other University services. However, during the evaluations, we realized that neither the users nor the new potential users were able to find this functionality in the application interface. It occurred because the forum was misplaced as if it was a category of News. This error explained the few comments existing until then in the forum. When introduced to the forum and explained about its purpose, the students were unanimous stating that is a feature they would certainly use, especially if they would receive any feedback from the faculty staff.

There are two categories of news available in the application: general and university-related. The users can comment and like the news using the same profile they use to comment and rate the meals. Although 76 % of the students classified it as easy to use, they suggested unifying the news, not to limit the number of characters of a comment, to offer options to edit and delete, and to provide clearer feedback that the comment was submitted. There were discrepancies about the News functionality, according to the user profiles. Among the current users, 63 % uses the News feature regularly. However, 52.9 % of the potential new users said they would not use this function often, despite they are very interested in using the other features of Eai?. We believe that the News tend to become more important to the users as they incorporate the use of Eai? to their daily routine.

5 Conclusion and Future Work

In this paper, we introduced Eai?, a mobile social application aiming to empower college students and to increase the student-faculty interaction. It allows the students to expose their opinion on the service provided by the University Restaurant, and to discuss other relevant topics. We also reported and discussed an evaluation of usability and user experience of this application. The results indicate that, although some improvements are necessary, Eai? is useful to assess the quality of the UR service, and can positively impact the students’ behavior, and sense of belonging to the institution. The application can evolve according to the needs of users and to provide a pleasant and efficient use. A future work should include correcting the usability issues, improving the News and Forum features, incorporating a gamification strategy, and developing versions to other mobile operating systems.

The sense of belonging to a greater community can improve people’s motivation, health, and happiness. In the case of students, a sense of belonging to the institution ultimately can lead to greater classroom engagement in reasoning activities [2]. However, to build a sense of belonging requires active effort and practice. There must exist a cooperation between the students and the University. While the institution provides and maintains an application carefully developed to enhance student’s daily routine, the students have to commit to feeding the application frequently, instead of just expect to consume the information. In this sense, the success of the application depends on the interaction within the user community.