Keywords

1 Introduction

Smartphones are being increasingly utilized for mobile learning (m-learning) now-a-days [1]. Many m-learning apps do exist for learning, e.g., m-learning app for higher education studies [2], learn English-Sindhi app [3], English vocabulary learning app [4, 5], learn English language [6], learn English tenses [7], app for English language and literature [8], Bengali language learning app [9], app for learning idioms and collocations [10], app for learning Chinese characters [11], and learn Chinese language courses [12].

Learning a new language is both fun and beneficial for a learner. Chinese language is one of the most spoken languages of the world. China shares borders with Pakistan; many Pakistani businessmen are doing business with their Chinese counterparts. Recently, a multi-billion-dollar project called China-Pakistan Economic Corridor has been initiated in Pakistan with the help of China. This is one of the important projects linking China to Arabian Sea via Gwadar port in Pakistan. Many Chinese engineers are working on this project as well as on other projects in Pakistan. With regard to this, the education department of provincial government of Sindh has signed the memorandum of understanding (MOU) with the Chinese officials for teaching the Chinese language in the secondary schools of Sindh province in Pakistan. Therefore, learning Chinese for Pakistani people, particularly the students would be beneficial. Although few apps do exist for learning Chinese in English [12] or learning Chinese in other languages do exist on the Google play store and other app stores. To the best of our knowledge, an app for learning Chinese in Sindhi for the people who speak and read Sindhi language does not exist yet. Moreover, usability is one of the crucial aspects for the acceptance of software applications, specifically the mobile apps, among the masses [3]. So, there was a need to develop such type of app having good usability. Due to this reason, we developed the Learn Chinese in Sindhi app. We also evaluated its usability through cognitive walkthrough, usability testing and a questionnaire. The details are given in the subsequent sections.

2 Related Work

Wuttikamonchai developed a mobile application for Android operating system to learn English tenses. Learning outcomes of the students were assessed and users’ satisfaction was also evaluated. Sample of population was the students of information technology department. The results revealed that the students were satisfied with the English tenses learning mobile app [7]. Rockahr et al. have discussed the e-learning app economy. They conducted interviews both from teachers and pupils; both of them regard m-learning apps to be a worthwhile supplement in learning [13]. Ng et al. developed an Android based mobile app for beginners to learn fundamental Japanese language. The app provides basic knowledge about Japanese language so as the beginners may gain self-learning proficiency in Japanese. The results of the evaluation reveal that the app was effective. More than 60% learners were agreed or strongly agreed that the app was useful in reading, writing, listening, and speaking simple Japanese [14]. Hussain et al. evaluated the usability of Urdu learning mobile apps for children. Fun Toolkit was also used in this study [15]. In another research study, an Android based app was developed for the Sindhi speaking students to learn English. Its usability testing was conducted. The results show that the app was effective and equally efficient for all the participating children [3]. Arain et al. developed a mobile learning app for the students of higher education institutes. The usability of the app was evaluated through a test and system usability scale (SUS). The results show that the app was effective and efficient; most of the students were satisfied in this regard [2]. García-Peñalvo et al. conducted the usability test of WYRED Platform: a project for children and young people regarding digital society. SUS was used to evaluate the usability and learnability of the system. Two groups were formed for usability test: group 1 (users who tested the platform for earlier version) and group 2 (users who tested the improved version). According to the results, the SUS score of group 1 was 64.67 and the SUS score of group 2 was 66.54. Both SUS scores were under the threshold (which is 68) but can be considered as decent score. The general perception of the participants about the system was positive [16]. Kortum and Sorber evaluated the usability of 10 mobile apps based on iOS and Android operating systems for mobile phone and tablet PC. They conducted two experiments with 3575 users using SUS. The results revealed that the apps on mobile phones were more usable than tablet platforms [17]. Many researchers have extensively explored other aspects of m-learning, e.g., Sánchez-Prieto et al. worked on m-learning [18], more specifically on extending technology acceptance models [19, 20]. As the effective, efficient and user-friendly app for learning Chinese in Sindhi was not available, so we developed the app and evaluated its usability.

3 Learn Chinese in Sindhi App

The Learn Chinese in Sindhi app has been developed at Quaid-e-Awam University of Engineering, Science and Technology, Nawabshah, Pakistan. The app is developed for the people who speak and read Sindhi language which is mostly spoken in Sindh province. The app is designed for such people who want to visit China or want to speak a little bit Chinese language. Owing to the MOU signed by the two governments, the target audience also includes the students of grade VI to grade X in the age group of 11–17 years. The app provides three translation opportunities to the users: from Sindhi to Chinese language, Urdu to Chinese language and English to Chinese language. However, Internet connection is required for this translation feature . Additionally, the app contains various sentences and features of daily use, e.g., Conversation (حال احوال), Tour (سیاحت), Emergency (هنگامي حالت) and Business (ڪاروبار). Moreover, the app has also useful basic information regarding the names of colors, fruits, animals, days, months and numbers from Sindhi to Chinese language. Figure 1 shows few screenshots of the app.

Fig. 1.
figure 1

Screenshots of the app: (a) Main menu (b) Name of fruits (c) Business sentences

Full size image

4 Methodology

The app was evaluated through a variant of cognitive walkthrough [21], usability tests and the SUS questionnaire. For the initial version, the cognitive walkthrough was conducted and the usability testing along with the questionnaire was also conducted. The problems found during these evaluations were fixed. After few weeks, another usability test along with the questionnaire was conducted for the improved version of the app. Twenty students of grade X in the age group of 16–17 years of a local secondary school participated in the usability tests. All the students had their own smartphones. 75% students were male and 25% were female. All the participants were native speakers of Sindhi language. During the second usability evaluation conducted on the improved version of the app, the same three tasks were performed by the same students. However, the order of the tasks was counterbalanced.

5 Results

The subsections present the results of the cognitive walkthrough and the usability tests along with the questionnaires.

5.1 Cognitive Walkthrough

The variant of cognitive walkthrough [21], a usability inspection method, was performed by two of the authors on the initial version of the app. According to the results, the prototype of the app was relatively simple. The amount of interfaces was appropriate; the information structure was clear; and the interactive mode was simply tapping. Hence the app was relatively easy to understand and easy to use. Above all, in terms of structure, the content category was used as navigation in the home page, and at the secondary page users could view the specific words under the category. However, the secondary page title was the same as the first page, and there were no back buttons. Users could only return using the Android system return key, which may cause a trek. Moreover, visual style of the icons was inconsistent.

The information on the first interface consisted of two parts: The Sindhi text on the left and the icons on the right (as shown in Fig. 2(a)). Overall it was clear and without any visual noise. However, few problems were found. The icons for “name of days” and “ name of months” were not accurate enough to express their meanings. Without reading the text, users cannot fully understand what these two icons mean and cannot tell the difference between them.

Fig. 2.
figure 2

Screenshots of the initial version of the app (a) and (b)

Full size image

There was no dividing line between two rows; and the distance between texts and icons was too far (as shown in Fig. 2(a)). It may visually cause a misunderstanding for the middle column to be a whole, and the right column to be a whole. This can create difficulty for users to correspond the text to the icon.

Usually on mobile devices, users focus on what the center of screen displays. In this version of the app, the current center of the screen contained the Sindhi texts while the right side was the icons. However, the texts were smaller than the icons. While the icons are supposed to assist the texts, thus they should have been smaller to highlight the texts. Moreover, the left margin was much broader than the right margin (as shown in Fig. 2(b)), which causes visual imbalance. Overall, visual style of the icons was inconsistent.

The icons of Arabic numerals were more like podium icons; moreover, the icons for fruits needed to be redesigned. Regarding the content layout, although the vocabulary was quite basic and common, there was no content of social aspects in the current version. Moreover, the current version lacked information of airports and tourism.

The second interface was a words learning page which covered four main contents: Chinese and Pinyin, icons, Sindhi, and pronunciation buttons. The overall layout and information were not complicated. However, the layout of the page needed to be rearranged. The visual focus of a mobile device should be the center of screen, and on this screen the icons were at the center. Moreover, since this is an app for learning Chinese, so it would have been better to place the Chinese text at the most conspicuous position. Apart from this, placing the pronunciation button next to the Sindhi text could cause a misunderstanding as the pronunciation was in Sindhi; thus the button should have been placed next to the Chinese text. Therefore, the order should have been: icon → Sindhi text → Chinese text → pronunciation button as shown in Fig. 3.

Fig. 3.
figure 3

Screenshots of numbers

Full size image

The first letter of Chinese Pinyin did not need to be capitalized. Pronunciation is a very important and frequently used function for language learning apps; hence the icon should be intuitionistic. The pronunciation icon in this interface was too ambiguous as a pronunciation button. Moreover, the buttons were small and did not have any feedback when being tapped which may get users confused. There was also a small flaw: the icon depicted a male, while the voice was of female which could lead to inconsistency. Furthermore, what makes Chinese text differ from Western languages is that it consists of single block characters, thus adding a pronunciation button for each single Chinese character may be more in line with Chinese learning habits. There was no obvious distinction between the title bar and the content as shown in Fig. 4.

Fig. 4.
figure 4

Screenshot of numbers

Full size image

The title bar of the Arabic numerals page showed “Counting”, while the other pages only had Sindhi written on them. According to the Chinese language habits, the Pinyin labels of each character should be separated with spaces, otherwise it is easy to be understood as just for one character, especially for westerners. Some pages were in accord with this habit, some were not (as shown in Fig. 5). Besides, it was better to have a one-to-one correspondence between the text and the Pinyin.

Fig. 5.
figure 5

Screenshot of the name of colors

Full size image

The label of the word “ڳاڙهو (Red)” lacked “色” after “红” (as shown in Fig. 6), which was inconsistent with other labels, such as “蓝色”, “黄色”. Also, the label of “نارنگي (Orange)” (“橙色”) was wrong (as shown in Fig. 6). The label of the word “سومر (Monday)” lacked “一” after “星期”. The labels of “ڍڳي (Cattle)” (“牛”) and “شينهن (Lion)” (“狮子”) (as shown in Fig. 6) were incorrect; the word “ٻلي (Cat)” (“猫”) was labeled in traditional Chinese, which was inconsistent with other words. Pinyin of the word “نومبر (November)” (as shown in Fig. 6) was wrong too.

Fig. 6.
figure 6

Four sub-screens of the app (Color figure online)

Full size image

5.2 Usability Testing

Besides cognitive walkthrough, the usability testing along with the questionnaire was also conducted for the initial version of the app. The problems found through the cognitive walkthrough and the usability testing methods were fixed in the newer version of the app. After few weeks, another usability testing along with the questionnaire was conducted for this improved version of the app. The compared results are discussed here. Table 1 shows the tasks that were performed during both of the usability tests. The order of the tasks was counterbalanced.

Table 1. Usability tasks
Full size table

According to the results of the usability tests, all the students completed all the tasks successfully on both versions of the app. So, the total task completion rate was 100% for both versions of the app which shows that the app was effective and easy to use.

Table 2 shows descriptive statistics of the tasks completion time (in seconds) of three usability tasks conducted on the initial version and the improved version of the app, clearly indicating that the students took lesser time on the improved version.

Table 2. Descriptive statistics of the tasks completion time
Full size table

The students took on average 27.40 s to perform all the tasks on the initial version of the app. While for the improved version of the app, the students took on average 19.25 s only to complete all the tasks. Table 3 shows statistical results for the initial version and improved version of the app, regarding the total tasks completion time.

Table 3. Paired samples t-test regarding the tasks completion time
Full size table

The results show that there was statistically significant difference (p < 0.05) across the groups on the total tasks completion time. The participants took lesser time to perform the usability tasks on the improved version of the app; this shows that the improved version of the app was more efficient than the initial version.

5.3 System Usability Scale

After performing the tasks, SUS questionnaire [22] was immediately administered in both of the usability tests. A paired-samples t-test was conducted to compare the SUS scores for the initial version and the improved version of the app. The result shows that there was a statistically significant difference in the SUS scores for the initial version of the app (mean = 71.87, SD = 5.248) and for the improved version of the app (mean = 78, SD = 5.104); t(19) = −4.307, p < 0.001. The result clearly indicates that the students were more satisfied with the improved version of the app. Although for the initial version of the app, the SUS score is also above the threshold value. This shows that the students were satisfied with the both versions; however, they were more satisfied with the improved version of the app. For the improved version of the app, besides SUS, the adjective rating scale was also filled by the students. The range of the adjective rating scale is from 1 (Worst Imaginable) to 7 (Best Imaginable). According to their responses, 75% participants rated the app as Excellent, 15% rated the app as Good and 10% participants rated the app as OK. After the second usability test, the students were also asked to give their preference for both versions of the apps. Their results show that all the students preferred the newer version of the app. In addition, three more questions adapted from [23] were also included for getting the general perception of the students regarding the improved version of the app. The results in Fig. 7 show that all the students responded as either strongly agree or agree for the statement that they are satisfied with the app. Most of the students (94.10%) perceived that it was fun to use the app. Moreover, 96.30% students responded that they would recommend the app to their friends.

Fig. 7.
figure 7

General perception of the students regarding the improved version of the app

Full size image

6 Conclusions

The usability study regarding the evaluation of the newly developed app for learning Chinese in Sindhi language was conducted. For the initial version, the usability was assessed using two methods: cognitive walkthrough and the usability testing along with the questionnaire. The cognitive walkthrough revealed many significant problems. Moreover, few problems were also found during the usability testing. These problems were fixed. Later for the improved version of the app, another usability testing along with the questionnaire was conducted. The results show that the total task completion rate was 100% for both versions of the app which shows that both versions of the app were effective and easy to use. Regarding the efficiency of the app, the results show that there was statistically significant difference across the groups on the total tasks completion time. The participants took lesser time to perform the usability tasks on the improved version of the app; this shows that the improved version of the app was more efficient than the initial version. Regarding the assessment of the perceived usability, the SUS score for the improved version of the app was greater than the initial version. The results show that there was statistically significant difference across the groups on the SUS scores, clearly indicating that the students were more satisfied with the improved version of the app. Moreover, all the students preferred the newer version of the app. It is recommended that the usability of the apps, specifically the m-learning apps, may be assessed with various usability methods, whenever possible; as good usability leads to more satisfied users. The combination of these methods helps in finding more usability problems that need to be fixed. Moreover, as the education department of provincial government of Sindh has already signed the MOU for teaching the Chinese language in the secondary schools of Sindh province so it is recommended that the app may be introduced in the schools where Chinese would be taught in the traditional classroom setting for better learning outcomes of the students. As this app can be used anywhere and at any time so by complementing this mobile learning app with the traditional classroom learning, the students can gain better learning experiences.