UnlockLearning – Investigating the Integration of Vocabulary Learning Tasks into the Smartphone Authentication Process

2.1 Mobile & Micro-Learning

The ubiquity of mobile devices fosters the shift from learning being a classroom-based activity to an activity that is embedded into our daily life. By applying the micro-learning paradigm, learning content is broken down into small bits of information and presented to learners using simple interaction concepts [7]. Hereby, the learning task is spread out over time. Especially for vocabulary acquisition, a higher number of short interactions should be favored over one long learning streak [12]. By spacing out the learning content, prior work has demonstrated that learners could memorise vocabulary content better than immediate successive presentation [14]. The concept of “spaced-repetition” suggests that repeating learning content according to a repetitive schedule helps consolidate the memory of it [1], [43]. This schedule targets the transient forgetting of information that naturally occurs after a certain amount of time of the information is not rehearsed or applied [40]. As MLL apps require self-directed learning, it needs to be taken into account that the success of autonomous learning and the adherence to rehearsal schedules depends on the consideration of various dimensions (i. e. context of learners, learner interest and motivation, language proficiency, etc.) [26], [28]. Thus, the overall goal of micro-learning is to maximise the degree of exposure with the learning content due to increased presentation and, thus, frequent repetition of content. In particular for the teaching of languages, the application of the micro-learning approach has proven to improve vocabulary acquisition and recall [10], [16], [45]. Prior work presented different ideas for realizing micro-learning in mobile learning. For example, Dearman and Truong (2012) [13] implemented a mobile language learning application that can display learning tasks (i. e. vocabulary translations with three multiple-choice options) on the lockscreen. The authors show that users frequently interacted with the application and improved their knowledge of the language over the course of this study. Dingler et al. (2017) [15] achieved similar results by implementing and evaluating a vocabulary application that presents tasks in push-notification. Their app “QuickLearn” detects opportune moments for the presentation of the learning content, in particular, when users are bored and use their smartphone without a specific purpose. Dingler and colleagues concluded that their application increased the number of “quick” learning sessions on the go, which was appreciated by their participants.

2.2 Smartphone Usage

Users’ interactions with their mobile devices can range in their level of intensity. Prior work has shown that half of our interactions with our smartphones take less than 30 seconds, with only one in ten interactions exceeding four minutes [48]. In particular, many interactions do not even reach the threshold of fifteen seconds, what Ferreira et al. (2014) [17] defined as so-called “micro-usage” situations. For example, users briefly reply to a message, check their phone for new push notifications, or want to know the time. For many of those interactions, users do not unlock their device [20], [21] or only use one application after unlocking [29]. However, unlocking the mobile device is still an action that happens frequently throughout the day. Mahfouz, Muslukhov and Beznosov (2016) [29] report that users unlock their phone on average 46 times, while other studies report an average of between 25 [20] and 47 unlocks [19] per day per participant. As this interaction has no other purpose than the authentication, prior work suggests combining the authentication event with a micro-interaction task, in particular, when the slide-to-unlock authentication is used. Truong, Shihipar and Wigdor (2014) [44] emphasised that authentication gestures serve no purpose other than to unlock the phone and outline their potential to serve other tasks such as data collection. Therefore, it has been proposed as a quick and easy method to enter journaling data [49], nutrition tracking [23], or sleep tracking [11].

2.3 Authentication Methods and Their Prevalence

In general, smartphone authentication mechanisms can be divided into four categories according to Wang et al. (2020) [47]:

1. Knowledge-based: Require the user to remember certain information, either text-based or graphical. Example: Passwords, PINs, patterns, or graphical passwords

2. Physiological Biometric-based: Makes use of unique identifying biological traits of users such as their fingerprint or facial features. This form of authentication can be performed upon explicit prompt or implicit assessment. Example: Fingerprint, voice, and face recognition

3. Behavioral Biometric-based: Captures users’ unique behavioral patters or characteristics. Example: Gait recognition, tapping behavior, hand gesture

4. Mixed Models / Multi-factor: Combination of two or more approaches to increase security. Example: Integrating knowledge factors with biometric authentication

For increased security, knowledge-based mechanisms can be included in multi-factor authentication schemes. Thus, even with implicit authentication such as fingerprint input or FaceUnlock becoming more common, PINs, passwords, and swipe-to-unlock do not become obsolete. Findings from studies such as by Qiu et al. (2019) [37] show diverse user preferences and authentication behavior, thus, strengthening the argument that offering multiple different unlocking mechanisms for smartphones is necessary.

Therefore, the first part of this chapter will present an exploration and evaluation of embedding different learning tasks into a diverse set of smartphone authentication methods, i. e. PIN, pattern, swipe-to-unlock, or fingerprint input using fingerprint sensor gestures.

2.4 Summary & Research Gap

In summary, micro-learning applications for vocabulary-based language learning have proven to increase vocabulary recall [10], [16], [45] and have shown great potential for pervasively embedding learning into everyday smartphone usage [13], [15]. Yet, it remains unclear how best to embed the learning task into everyday device use. This motivates our idea and investigation of our new concept of integrating learning directly into the authentication process, by displaying a vocabulary task every time the phone is unlocked using any form of authentication mechanism. Furthermore, to the best of our knowledge, no comparative analysis has been performed yet to investigate potential differences between the proposed pervasive presentation of vocabulary and a “standard” in-app learning application that requires the user to actively initiate learning. For example, while Dearman and Truong (2012) [13] showed a knowledge gain through their wallpaper app, they do not implement a “standard” app as a control condition.

3.1 Authentication Interaction

We characterised the authentication methods from an interaction perspective. Thus, it was less important what the user actually inputs (characters, numbers, etc.) but rather what type of interaction is required. We distinguish the following four types of interactions found in common smartphone authentication processes:

1. Single discrete input, aka. “tap” – An individual tap that can be used for any button when applied on the touchscreen, to unlock the phone when applied on the unlock hard key or to use on the fingerprint sensor.

2. Multiple discrete inputs, aka. “sequence” – A set of consecutive taps that can be used to input a PIN or password on the touchscreen.

3. Uni-directional continuous input, aka. “swipe” – A continuous gesture that can be used as swipe to unlock on the touchscreen or as gesture input on the fingerprint sensor (current sensors recognise four swipe directions)

4. Multi-directional continuous input, aka. “pattern” – A continuous gesture that connects swipes in multiple directions without lifting the finger from the touchscreen, as it is known from common pattern authentication formats.

5. Implicit input – This format includes all authentication methods that do not require the user to perform an action on the smartphone such as FaceUnlock or voice recognition.

3.2 Learning Task Interaction

We selected three different tasks for the interaction with the language learning content. All tasks are of simple design and focus on vocabulary recognition. They require only micro-interactions with the aim to be feasible for solving the task on the lockscreen and during the authentication process. We included the commonly used Multiple-Choice question format, in which the correct answer is presented accompanied by a number of incorrect answers (also called “lures”). The user has to pick the correct answer from the set. While this format can be used to present multiple correct answers, we decided to implement the simplest version of only one correct answer so only one click (single tap) will be needed to solve the task. With the simple interaction, this task is specifically feasible to be used for single touch authentication methods such as swipe-to-unlock or fingerprint authentication (using fingerprint sensor gestures, see details in subsection 3.1). The number of presented “lure” answers can further be adapted considering the limited space on the lockscreen but needs to be chosen carefully as too many answer options can increase the number of incorrect answers in later testing situations [6], [8] (i. e. “Negative Suggestion Effect” (NSE), for summary see [41]). For new learning content, Roediger and Marsh (2005) [39] recommend to use fewer alternatives in the beginning. Thus, we will present only one to two lure options as addition to the correct answer. Furthermore, at this prototypical stage, it is possible to present either the native word (L1) and ask the user for the second-language translation (L2) or present the L2 word and ask for the L1 translation.

As a second format, we implemented a “Check Word” task along the vocabulary test yes/no format, for example, applied in the Eurocentres Vocabulary Size Test [32], [33]. This task format aims to present a large number of words while requiring minimal time and effort by the user. In our examples, the L1 words are presented along the L2 translations while the user has to state if he/she knew the translation or not (single tap on a yes or no button). Similar to the the multiple-choice task, the yes/no answer format can be implemented also for single interaction authentications. To control for an overestimation of knowledge caused by users marking unknown words as known, the later application could include phonetically or orthographically similar pseudowords. In contrast to the yes/no test by Meara and Jones (1988) [33], this format (with included translation) tests not only the recognition of words but also checks if learners are aware of the word’s meaning.

As a third task, we include a Sentence-Building exercise. Similar to the multiple-choice task design, the user is presented with a set of words from which they are asked to select those that make up a semantically and syntactically correct sentence. Due to the limited space of a lockscreen during authentication, we chose to present simple sentences with three words that need to be chosen out of a set of six words (three correct words, three lure answers). For example, a task could look as follows:

A sentence-building task can thus combine vocabulary recall with applying the correct grammatical construct. In the first example above, picking the correct article (a/an) and remembering the meaning of the words green/yellow and lemon/orange. In the second example, choosing the correct combination of pronoun (he/they) and verb form (drink/drinking) and remembering the meaning of the words water and wood.

4.1 Mapping of Learning Tasks and Authentication Methods

In Table 1 we show how our different prototypes address combinations of learning task and authentication method. All interaction concepts were implemented as high-fidelity interactive prototypes (PT1 to PT12) using the Justinmind^[1] prototyping tool for mobile applications. For example, PT6 is depicted in Mockup 1 (see Figure 1a). The users enter their PIN and tap the screen once more to select one out of three translations for the L2 word presented below. Prototype PT3 implements Mockup 2 (see Figure 1b) by combining a check word task with a uni-directional continuous input, i. e. a swipe gesture starting at the on-screen fingerprint sensor to either of two buttons to indicate if the translation is known or unknown. This prototype is also applicable for other uni-directional continuous input types such as slide-to-unlock. Since sentence-building inherently requires connecting multiple words, it is not applicable for single input (discrete or continuous) authentication methods such as swipe-to-unlock. A sentence can be built with a multi-directional continuous input as visualised in Mockup 3 (PT12) see Figure 1c). Here, the user enters their predefined authentication pattern and, without lifting the finger, continues the gesture through connecting the words presented below to form a sentence.

4.2 Exploratory Evaluation of Prototypes

To investigate users’ preferences for our different prototypes, we performed a user experience evaluation with ten in-depth semi-structured interviews. We showed the participants the interactive prototypes of the 12 concepts on a smartphone, asked them to interact with them, and collected their feedback. The focus of this user study was to gain insights into the usability and user experience of the concepts.

4.2.1 Procedure

At the beginning of each session, we welcomed the participants and informed them about the purpose of the study. After agreeing to the data protection policies along the GDPR regulations, participants signed a consent form. A short questionnaire was presented asking for demographic information such as age, gender, degree of education, current occupation, their experience with mobile learning, their current authentication technique of their smartphone, and its operating system.

Figure 1

Three mockups^[2] to visualise our concept for integrating learning into the authentication process. Mockup 1 outlines a multiple choice task after PIN entry. Mockup 2 Asks the user to indicate if the word is known (green button with arrow) or unknown (red button with cross). Mockup 3 shows how sentence-building can be integrated into pattern authentication.

Afterward, we introduced the participants to our prototypes and varied the order of presentation between the participants. The prototypes were shown on a Sony Xperia Z smartphone with a 5-inch screen through the Justinmind app. People interacted with one prototype after another. We asked them to follow the think-aloud protocol, verbalizing their thoughts about the interface, interaction, and usability as well as problems of understanding. If the user was not able to perform the required interaction, we explained and/or demonstrated it and the user was asked to retry the interaction. We explicitly advised all participants to not input any authentication PIN or Pattern during the study that is similar to their own.

For each prototype, the interview was guided by the following user experience metrics: Ease of use, Enjoyment, Interaction Speed, and Willingness to use the prototype as a learning method in the future. Those facets were posed as yes/no questions but with the collections of further comments. The participants were furthermore encouraged to state any concerns regarding the prototype as well as potential improvements that come to their mind. Overall, the study took around 60 minutes depending on the extent of comments users had about the prototypes. We transcribed people’s comments and applied a thematic analysis approach for comments and suggestions. This method is commonly used for identifying, analyzing, and reporting themes found within a qualitative set of data such as interview transcripts [5], [35].

Table 2

Users’ assessment of the prototypes along the dimensions ease of use, enjoyment, interaction speed, and their willingness to use the prototype in their daily lives. Questions were posed in a yes/no format and the answers in the table are visualised as “+” when 8 or more users agreed; similarly, a “–” indicates two or less users agreeing.

Task Type	Action	Prototype	Easy to use	Enjoyable	Fast	Would use it
Check Word	SD	PT1	+	+	+	+
	MD	PT2	+	o	+	+
	UdC	PT3	+	+	+	+
	MdC	PT4	+	o	+	o
Multiple-Choice	SD	PT5	+	o	o	+
	MD	PT6	+	+	+	o
	UdC	PT7	+	+	o	o
	MdC	PT8/9	+	+	+	+
Sentence	MD	PT10	o	o	o	o
	UdC	PT11	o	o	–	–
	MdC	PT12	+	o	–	o

4.3 Limitations

During our study, participants expressed difficulties in rating the interaction with the embedded authentication learning task without being biased by the authentication method used. In general, several participants considered swipe-to-unlock as insufficiently secure and would not use this mechanism. We stressed that this technique could be integrated into fingerprint authentication and should be rated independently from users’ opinion on the authentication itself. However, we cannot be certain that our participants were able to avoid such bias. We further noticed that the alignment of the learning tasks below the authentication interface was not optimal. In particular, for continuous input gestures such as in PT9 and PT12, users had to interrupt the continuous gesture to have a look at the options again. The visibility of the options is ultimately defined by the end-point of the pattern input – i. e. when the pattern ends in the lower left corner and the user is right-handed, the hand will almost completely cover the answer options.

4.4 Summary

We have presented twelve prototypes for combining three learning tasks (check word, multiple-choice, and sentence-building) with four common forms of smartphone authentication mechanisms (represented through their required interaction: single discrete input, multiple discrete, single continuous, multi-directional continuous). A first usability evaluation with ten participants showed that users’ preferences vary greatly. Most prototypes were reported to be easy to use and fast. While some participants preferred discrete taps, others especially enjoyed the single and continuous swipe input schemes.

The main take-away from this evaluation is that users desire a quick and easy learning interaction. While they value the idea of combining learning with their authentication, many participants emphasised that the learning should not impede or obstruct the unlock process. In particular, the sentence-building tasks were considered fairly slow and not a good fit for the presentation on the lockscreen. We thus excluded this type of task for our further evaluation. However, participants doubted the efficiency of the check-word tasks. Thus, we will further explore the multiple-choice task as the most promising concept.

Lastly, we noticed that the speed perception varied greatly among participants. While some found the interaction slower than necessary for an authentication method, others perceived the learning task itself as rather quick. Along the glass half-full vs. glass half-empty situation, we argue that the perception strongly depends on the users’ perspective on the situation. As the proposed methods combine authentication and learning, it is clear that the interaction will take longer than authentication alone. However, when comparing the combined interaction with the action of having to open a stand-alone learning app to learn a vocabulary, it is undeniable that the newly proposed prototypes enable a faster interaction. Thus, our following evaluations include a standalone app as control condition for participants to compare the interaction fairly.

5.1 Learning Content

All three apps include vocabulary allowing for learning Spanish, French, and Swedish with translations into the native language of our University’s country. The word list comprised of 446 common nouns (originally from the British National Corpus,^[3] last accessed December 13th, 2021) and have been translated to the other languages using Google Translate while removing inaccurate translations and highly ambiguous words, similar to the procedure of Cai et al. (2014) [10] and Dingler et al. (2017) [15].

In the background, all three applications are connected to a Firebase database, which stores both the learning content and interaction logging data. In the database, each user is assigned a unique id to ensure anonymity. All three applications are described in more detail in the following.

5.2 UnlockApp

To integrate the learning task into the authentication action, we display the vocabulary translation interface immediately when the user unlocks the screen (see Figure 2a). As the current Android versions do not allow a direct manipulation of the lockscreen, we chose to implement a context-registered broadcast receiver. This receiver listens for the unlock event (ACTION_USER_PRESENT indicates that the user is present after the device is woken up^[4]) while running continuously in the background and places the app in the foreground when an unlock occurs. Since it is possible that applications are shut down for battery optimisation purposes, we further included a foreground service that allows for automatic restart of the application.

When an unlock event occurs, the UnlockApp asks the user to translate one word from the native language to one of the three languages of choice – Spanish, French, or Swedish. To keep the learning task as short as possible, the interface only presents the translation task, two potential translation options in the form of buttons. Further, we include a “skip” option to keep learners from guessing the answer, as guessing increases the risk of experiencing the negative testing effect [31] (i. e. remembering the false answer one gave to a multiple-choice question rather than the correct solution).

After the user selects one of the two translation options, the UnlockApp displays corrective feedback by highlighting the answer option in either red or green. Providing feedback in multiple-choice learning tasks is essential and counteracts the negative testing effect [9]. After a short delay for users to perceive the feedback, the app moves into the background.

We chose a translucent background to not distract the user too much from the primary task they were about to accomplish when unlocking the phone. Prior research by Hodgetts and Jones (2006) [22] has shown that when the primary task was still visible during an interruption, participants are able to resume the task faster. To ensure consistency, we used the Android material design and color palettes.

The UnlockApp further allows to use a fingerprint sensor gesture (swiping left or right on the fingerprint sensor) as input to answer the tasks. We utilise the Fingerprint Gesture Controller accessibility service.^[5] This controller registers gestures performed on the fingerprint sensor, independent from sensor location (depending on the model, the sensor can be located on the back of the phone, on the side, or on the front below or on the screen). While prior work has already shown that fingerprint sensor gestures can be a feasible tool for quick input such as unlock journaling [18], this feature is not yet supported by many Android devices. Thus, we will not evaluate the usage of it in this study.

5.3 NotificationApp

In this app, learning content is presented in a continuously displayed push notification, both in the (pulled down) status bar of the smartphone (see Figure 2c) but also on the lockscreen (see Figure 2b). This enables the user to interact with the content at any point in time and answer as many tasks as they like to, the continuous notification working as an active and constant reminder. The notification displays the task and word to be translated as well as two potential translation options via action buttons.^[6] Those buttons are frequently applied by other applications to allow immediate interaction with the app sending the notification. For example, Gmail^[7] employs action buttons to enable users to delete or reply to emails directly from the notification. We implemented a broadcast receiver to listen for the input in the notification and check the correctness. In the same style as the UnlockApp, green and red coloring of the two potential translations indicates corrective feedback. Afterward, the notification is updated and a new word is shown. Likewise, when the user dismisses a notification or restarts the app, it presents a new word. The notification itself cannot be removed by the user but remains constantly visible in the pulled down status bar.

5.4 StandardApp

In contrast to the UnlockApp and NotificationApp, the StandardApp does not show any continuous reminder of the learning task. The users have to actively open the application and engage with the learning content. Once the users open the app, they can complete as many learning tasks (i. e. request as many translation tasks) as they like. The appearance of the application’s interface is consistent with the interface of the UnlockApp: The app shows the native word, the two translation options in buttons, a dismiss button, and corrective feedback for the task (see Figure 2d). In contrast to the UnlockApp, the background is fully opaque.

6.1 Procedure

We provided participants with a detailed installation guide enabling people to download the three applications from the Google Play Store (setup as a non-public test version), after they gave informed consent. The guide also encouraged them to restart the (current) app in case they restart their phones or the app is closed for any other reason and does not restart automatically.

At the beginning of the study, people filled in a questionnaire on demographics and current authentication method, smartphone usage habits, language proficiency, and motivation to learn a new language. In this process, people selected a language to learn with the apps (Spanish, French, or Swedish). We informed participants that the vocabulary taught in the apps is on a beginner’s level and thus recommended to choose a language they are not very proficient in.

During the study, we logged data on the users’ interaction with the apps, including the number of solved vocabulary tasks, their correctness, response times and task dismissals (in the UnlockApp).

6.2 Sample

We recruited 30 participants via University mailing lists, social media channels, and word-of-mouth, nineteen identifying as female and eleven as male. Their average age was 29.8 years (SD=15.55, range 19–78 years) and the majority (18) reported to be studying or working full time in jobs such as Engineer, Physicist, Accountant, Doctor, or Dentist, while two were pensioners. Fifteen stated having a high school degree and twelve a university degree (including bachelor, master, or phd). One third (19 people) reported using fingerprint authentication on their smartphones, and seven reported using a PIN or password, two patterns, and two face recognition. All participants stated to use their device at least multiple times a day (15) if not multiple times per hour (15).

At the time of the study, ten participants confirmed learning or actively improving on a language (four Spanish, three French, two English, and one Japanese). They further stated to be proficient in at least one and up to five foreign languages (Md=3), and reported to have great interest in learning a new language (M=6.07, SD=0.85, Likert-scale from 1 = “I fully disagree” to 7 = “I fully agree”). As languages to learn in this study, thirteen people chose Spanish, twelve French, and five Swedish. For their successful participation all people received a 25 Euro Voucher for an online store, or an equivalent amount of study credit points.

6.3 Results

We report on the data of 30 people. However, one person did not complete any of the final questionnaires, and two additional people did not submit the questionnaire for the StandardApp. Therefore, we report the interaction data of 30 and the questionnaire results of 29 people for the UnlockApp and NotificationApp, 27 for the StandardApp, respectively. For the statistical analysis of the questionnaires, we performed pair-wise exclusion for incomplete data sets.

We used R [38] for significance testing; concretely, (generalised) linear mixed-effects models (LMMs, packages lme4 [3] and lmerTest [27]). The LMMs accounted for individual differences (participants) and app order via random intercepts. Note that app order was counterbalanced yet we still included it here following best practices. As fixed effects, we included app, plus the number of days since the start of the study.

Figure 3

Overview of participants’ usage of the three apps during the study: (per participant from left to right) the overall number of vocabulary tasks solved in the UnlockApp, NotificationApp, and StandardApp.

6.3.1 Overall Usage

In total, we recorded 7715 answered vocabulary tasks over all participants across the three weeks of our study. The most tasks were answered using the NotificationApp (2945), followed by the UnlockApp (2604) and the StandardApp (2166). On average per day, people used the apps to solve 10-13 vocabulary tasks. Here, the StandardApp showed the lowest mean usage (Md=2, M=10.28, SD=20.05), while the UnlockApp (Md=8, M=12.28, SD=14.73) and NotificationApp (Md=5.5, M=13.7, SD=28.43) were used more frequently. While the NotificationApp shows the highest overall usage, the Median is lower than the Median of the UnlockApp. This is due to one exceptional case in the NotificationApp usage (see P4 in Figure 3): P4 reported having used the NotificationApp extensively out of enjoyment, with over 800 answered vocabulary tasks (increasing the overall usage count but with little effect on the Median). Since this was intended use, we do not remove this as an outlier from our analysis.

For significance testing, we fitted a generalised LMM (Poisson family) on the answer count data (i. e. number of answered vocabulary questions). The model had app as a significant positive predictor (UnlockApp: β=.43, SE = .06, CI_95 % = [.31, .55], p < .0001; NotificationApp: β=.36, SE = .06, CI_95 % = [.25, .48], p < .0001): Therefore, compared to the StandardApp, using the UnlockApp was estimated by the model to result in exp(β)=1.54, that is, 54% more answered vocabulary questions. Similarly, using the NotificationApp was estimated to result in 43% more. Moreover, the model had day (since start of the study) as a significant negative predictor (β=−.15, SE = .01, CI_95 % = [−.17, −.12], p < .0001): The number of answered questions was estimated by the model to decline over the course of the study (estimated as exp(β)=0.86 i. e. −14% per day). The interaction of day and app was significant for UnlockApp (β=−.07, SE = .02, CI_95 % = [−.10, −.04], p < .0001), but not for NotificationApp (p=.176).

Figure 4

The relative distribution of vocabulary tasks solved for the UnlockApp (left), StandardApp (middle), and NotificationApp (right) in percent (0.06 = 6%). The amount of vocabulary tasks is normalised for each user with respect to their overall number of solved tasks and visualised according to the seven usage days (x-axis) and the hours of a day (midnight to midnight, y-axis).

We further plotted people’s interactions with the three applications over the course of the whole day and the week of use: Figure 4 visualises the number of tasks solved by the users, indicating that they interacted with each app most frequently on the first day of use. It has to be noted here that participants started the study on different days of the week (Monday: 5, Tuesday: 2, Wednesday: 2, Thursday: 5, Friday: 5, Saturday: 6, Sunday: 5) and the starting weekday then stayed the same for each person across app conditions. Moreover, the plots reveal that interactions with the UnlockApp are scattered more across the time of the day compared to the interactions with the StandardApp and NotificationApp. Furthermore, the plot matches the results of the LMM that the number of answered questions declined from day one to day seven.

6.3.3 Favorites, Ratings & Subjective Results

We asked participants directly which of our three applications they would prefer for everyday use. As their most favorite application, 18 out of 30 people named the NotificationApp, followed by seven mentions of the UnlockApp. Vice versa, 15 participants stated the UnlockApp as their least favorite of the three applications, followed by 13 naming the StandardApp.

Moreover, we analysed participants’ subjective impressions of the three applications (expressed in the final questionnaire as 7-point Likert scale items from 1 = “I totally disagree” to 7 = “I totally agree”). For each item we performed a Friedman test with post-hoc Wilcoxon pairwise-comparisons. There was a significant difference in how much participants liked the applications (a: χ2(2)=6.812, p<.05, see Figure 5a). Post-hoc analysis with Wilcoxon signed-rank tests showed a significantly higher score for the NotificationApp compared to both the StandardApp (Z=−2.159, p<.05) and the UnlockApp (Z=−3.100, p<.01).

We further asked participants (b) if they felt they used the app only when they had time to learn and (c) if they used the app even when they had no time to learn. Both items revealed significant differences in participants’ subjective perception of their usage among all three applications (b: χ2(2)=16.568, p<.001, see Figure 5b; c: χ2(2)=15.571, p<.001, see Figure 5c). For item (b), participants report to be more likely to use the StandardApp when they had time to learn compared to the NotificationApp (Z=−2.060, p<.05) and the UnlockApp (Z=−3.960, p<.001), and more likely to use the NotificationApp when compared to the UnlockApp (Z=−2.737, p<.01). For item (c), participants report to be more likely to use the UnlockApp even when they had no time to learn compared to the NotificationApp (Z=−2.786, p<.01) and the StandardApp (Z=−3.945, p<.001), and more likely to use the NotificationApp when compared to the StandardApp (Z=−3.945, p<.001).

We found no significant differences in participants’ ratings of the design of the three applications and their intuitiveness of use. Moreover, the Friedman test revealed no significant difference in terms of participants’ willingness to continue using the applications in their everyday life. The descriptives, however, show a slight preference in favor of the NotificationApp (M=4.37, SD=1.85) compared to the UnlockApp (M=3.70, SD=2.05) and the StandardApp (M=3.30, SD=1.73), see Figure 5d.

Figure 5

Participants’ ratings of the StandardApp (SA), NotificationApp (NA), and UnlockApp (UA) for four questionnaire items (from 1 = “I totally disagree” to 7 = “I totally agree”): (a) I liked the [SA/NA/UA]; (b) I only answered tasks with the [SA/NA/UA] when I had the time to learn; (c) I answered tasks with the [SA/NA/UA] even when I did not have time to learn; (d) I would continue using the [SA/NA/UA] frequently in my everyday life for learning a language. Asterisks indicate all statistically significant differences (* <.05| ** <.01| *** <.001).

7.1 Limitations

Our user study was conducted during the COVID-19 pandemic, probably leading to anomalies in users’ daily routines, mobility, and smartphone usage due to lockdown and work-from-home phases. All these factors potentially influence the use of mobile language learning applications; thus, we can not ensure the generalisability of our results. However, we included a control condition via the StandardApp.

The focus of our evaluation lies on the users’ experiences and users’ interaction with the three apps of varying integration into smartphone usage. Although our applications do not constitute an exhaustive representation of all MLL applications, we chose these three as a concise comparison of how different levels of embedding could be implemented. Further, with our focus on user’s interaction we can not report on actual vocabulary retention. However, as retention of vocabulary is increased by frequent interaction with the content [12], [14], we expect our applications to positively impact people’s vocabulary recall and recognition. Nonetheless, actual vocabulary retention, particularly concerning the consolidation properties of the apps over time, needs to be further evaluated in future work.

By providing people the three applications to use for each seven days during this user study, we gained interesting insights into users’ learning behavior. We decided in favor of a shorter within-subject study over a longer between-subject study to enable users to draw comparisons among their interaction with the three apps and to make the best use of the limited sample size. However, we are aware that our study only presents a narrow view onto users’ actual behavior. As the users show a great diversity in individual preferences, only a long-term evaluation with a larger sample will be able to show which behaviours will prevail in users’ daily lives. Our study data shows a more extensive usage of each application on the first day, declining over the seven days of usage. This pattern could indicate a form of curiosity or novelty effect. Educational technology research has shown that accustomisation with new learning technology can negatively impact learners’ preferences regarding technology-based learning [25] and reduce users’ motivation [24]. In contrast to the StandardApp, the UnlockApp’s concept partly counteracts this decline by continuously engaging the learner in solving vocabulary tasks with each authentication event. However, a future long-term evaluation in particular of the new UnlockApp concept is required to reveal the strength of the novelty effect in everyday usage.

7.2 UnlockApp and NotificationApp Increase Vocabulary Exposure

Our analysis confirms our initial hypothesis, showing that the NotificationApp and UnlockApp result in a higher number of solved vocabulary tasks per participant. Thus, we conclude that the learning content exposure was higher for these two apps when compared to the baseline, the StandardApp. Furthermore, the presentation of the vocabulary tasks after the authentication leads to a more spread out exposure to the tasks across the day, confirming Dingler and colleagues’ [15] conclusion that proactively triggering vocabulary learning sessions can effectively space out learning. Further, the continuous presentation of content through the NotificationApp allows for short but also extensive learning sessions (cf. the outlier P4). The similar distribution of correctly and incorrectly answered tasks across the three applications gives no indication that users might have been less focused when learning with the UnlockApp or NotificationApp. Additionally, we observed that users occasionally skip tasks with the UnlockApp. This suggests that even though the skipping requires the same amount of effort as selecting an answer (one button press), the users can judge if they have the mental capacity and/or time to engage in the learning or not.

7.3 Potential for Adaptation & Personalisation

The subjective feedback revealed individual differences regarding people’s attitudes toward the three applications: While many state to like the concepts of the NotificationApp, opinions on the UnlockApp are mixed. Specifically, some people felt distracted by the UnlockApp’s vocabulary presentation when they unlocked their phone with a specific task in mind. People do not oppose the concept in general but express their need for further personalisation. Suggested adaptation features include the definition of learning time frames for the UnlockApp or adjusting the number of tasks presented with each authentication event. Moreover, we see potential for automated mechanisms that learn from people’s interactions with the UnlockApp (in particular dismissals) and adjust the presentation accordingly. I. e. the app should stop promoting users with learning tasks after unlock events at times during the day when they are frequently dismissed.

7.4 Extending the UnlockApp Concept to Different Authentication Methods

Based on the results here, we deem it useful to discuss possible extensions of the UnlockApp concept to further authentication methods: Concretely, the majority of participants in our study used fingerprint authentication to unlock (yet with phones that do not support fingerprint gestures). Our app and concept already support fingerprint gestures to respond to vocabulary tasks (see implementation section), which can help to embed the learning task even more implicitly into the authentication (i. e. no switch from fingerprint to touch required). Prior work has found that using fingerprint sensor gestures for journaling is perceived faster and less intrusive compared to notifications [18]. Both touch input and fingerprint gestures might also be combined with unlocking methods that do not require further input themselves, such as face unlock, or when Android’s smart screen lock feature is enabled (i. e. no unlock at home). Beyond this, we already gathered experiences with knowledge-based authentication methods: One third of our sample stated to use PIN, password, or pattern authentication. In these cases, the UnlockApp also presents the task immediately after the unlock event.

7.5 Extending the Learning Features of Embedded Learning Applications

Our application focused on the presentation of vocabulary translations to reduce the complexity of the content. With this, we aim to simultaneously increase the control of potential effects among the three apps. The participants of our study strongly emphasised the demand for additional features such as grammar knowledge or pronunciation exercises in open comments, and compared our application to applications available on the market. To accommodate for people’s need for more complex learning tasks, we see three potential solutions: (1) By combining the embedding concepts with a fully-featured learning app, we could address users’ preferences for long learning streaks when they have time to spare (i. e. with the StandardApp), during which the app could also teach grammar or pronunciation knowledge. The differentiation between two complexity levels could also be realized by (2) differentiating between short and long learning sessions as suggested by Cai and colleagues [10] (e. g. by exploiting different types of waiting situations). Additionally, the UnlockApp could refresh users’ vocabulary knowledge by continuously presenting translation tasks after each authentication to engage users with a language on a daily basis. Option (3) is to investigate extensions of learning tasks that are short enough to be embedded into the UnlockApp or NotificationApp. These tasks need to be solvable with simple interactions and with low effort and time. Possible examples include fill-in-the-blank tasks offering a word in two tenses or with two suffix options.