Keywords

1 Introduction

During literacy process children face several difficulties, as long as they consolidate this process and become able to read, write and understand what they read and write. Within literacy studies, make children to understand the relation between letters and sound is a challenge [1]. This is due each letter would represent a sound and each sound a letter in a written alphabetical system. However, this correspondence is not straightforward. Within Portuguese language, as with another languages, these relations between the spoken sound with the written letters is quite complex, since there are some cases where two or more letters represent the same sound. This scenario ends up by having the students to make several orthographical mistakes during the literacy process.

Therefore, in order to deal with these orthographic drawbacks, teachers search for mechanisms and activities that would be helpful for children to make them understand the language’s orthographical subtleties and overcome these literacy challenges.

In this context, the goal of this work is to present an application (educational game) to assist the learning processes of words likely to cause orthographical difficulties. For this purpose, a natural interface (using Kinect [2]) is applied in order to motivate and attract children to use the application. In order to validate the proposed work, some usability tests were carried out with elementary school students under literacy process.

The current work is organized in the following sections: Sect. 2 presents the main concepts related to the developed work; Sect. 3 discusses some related work regarding the use of Kinect within educational applications; Sect. 4 introduces the development model applied for the proposed application; Sect. 5 discusses some tests evaluation results; and, finally, Sect. 6 presents some general conclusions and future perspectives.

2 The Human Language and Its Relation with the Written Word

We define the human language as a code inside which a set of linguistic forms is related to a given meaning, used to express a particular language task (or skill): speech, auditory comprehension, reading and writing [3].

The spoken language is acquired spontaneously as children grow up, interact and communicate with other people; as children expose themselves to the speech, they improve their acquisition process and their communication skills. In opposite, the acquisition of the written word requires an explicit instruction process, a systematic learning effort [4].

Alphabetic systems are applied to represent the abstract structures of the spoken language, such as the phonemes (smallest linguistic unit of a language's phonology). The conversion between graphemes (smallest unit in a writing system) and a phoneme is quite variable from alphabet to another alphabet, since some sounds (phonemes) can be represented by several letters, or the opposite, a letter being represented by different sounds, depending on the context of the whole word [3].

The Portuguese language alphabetic orthographic system is classified as irregular, its means that the graphemes do not possess a direct relation with the phonemes. These irregularities within the alphabetic system reflect directly on the learning drawbacks, leading to longer and larger rates of orthography misspelling [3, 5].

Cristofolini [6] claims that there is a close relation between speech and literacy, since that the literacy process relies on carrying out a mapping between a sequence of sounds and a written code. This mapping affects directly the orthography domain, since reproducing Portuguese phonetic sequences in written expression requires the utilization of graphical signals representing these sounds, or the so-called graphemes or letters. When someone speaks, this person produces phonemes and, when he/she writes, these sounds should be described as graphemes. Therefore, since there is no one-to-one correspondence between Portuguese’s graphemes and phonemes, orthography misspelling reflects this lack of correspondence between these two systems.

According to Zorzi [5], orthography has been one of the issues constantly discussed during literacy process, since learn how to write implies in understanding the relations between sounds and letters, in other words, one has to master the conventional words writing. Thus, the orthographic mistake has been a major issue, which has not been given the proper attention and understanding during writing acquisition. The author suggests a classification regarding the types of mistakes commonly made by children at elementary school:

  • Modifications or mistakes due the possibility of multiple representations: The alphabetic writing systems present as essential feature correspondence between sounds and letters. Regarding the Portuguese language, it is possible to find several types of correspondences: A stable relation, within which just one letter is always applied to write a given sound (as in the case of letter f which always describes the sound of /f/); A non-stable relation, within which the same letter can represent several sounds (the letter c, for instance can represent the sounds /k/ and /s/) and, at last, another non-stable correspondence, which is characterized by the fact that the same sound can be characterized by several letters (for instance, the sound /s/ can be represented by s, ss, c, ç, x, z, sc, sç, and xc). Therefore, it possible to make an assumption of the amount of mistakes generated by this non-stable correspondence between letters and sounds, as it is illustrated by the written form of the word “cabeça” with letter “s” or “ss”, resulting in “cabesa”, “cabeça” or “cabessa”.

  • Replacements related to the spelling of the unvoiced and voiced phonemes: some pairs of phonemes can be differentiated by their sound trace; it means that some of them are unvoiced, others are voiced. The phonemes /p/, /t/, /k/, /f/ and /s/ are considered unvoiced since they do not present vibration of the vocal cords when they are reproduced in Portuguese. In turn, the phonemes /b/, /d/, /v/ and /z/ are produced in Portuguese with the vibration of the vocal cords, being considered, therefore as voiced phonemes. The sound traces is related to an important distinction between pairs of phonemes, such as /p/x /b/; /t/x /d/; /f/x /v/; /s/x /z/. The orthographic modifications considered as “unvoiced/voiced switches” regard the words presenting replacements between the letters that describe the following phonemes p/b; t/d; q-c/g; f/v; ch-x /j-g and the set of letters that represent the phoneme /s/ when replaced by those related to the sound /z/. For instance, this type of replacement can be found in the following words: “tijolo” – “ticholo”; “filme” – “vilme” and “gato” –“cato”.

Similar letters: some letters or even a set of letters that compose words can be considered similar if we consider their graphical shape. In other words, the similar drawing of these letters can take to misspelling, for instance, as in “maçã” being written as “naçã”.

3 Related Works

Several contributions have presented the development of applications for the educational environment focusing on learning drawbacks [7]. Guerra [8] developed an application for teaching color blending using Kinect as interaction resource. The developed application proposed colorful circles where children are encouraged to select colors using natural gestures and blend them in order to observe the resulting colors.

Alves et al. [9] developed an application that uses Kinects to assist in the literacy process. The application provides a user interface that presents to the user a letter, a word and an image; each one with a particular geometric shape. The user should move these objects around screen in order to position them correctly (association).

Kawamoto and Martins [10] developed a game, also using Kinect, to automatize the Corsi test [11]. This test verifies the short-term memory. In this game, a sequence of squares is presented to the user who should select them in a given order. The sequence presented to the user is constantly modified and improved, in order to verify the user’s level of short-term memory. The tests with this application were carried out with an elderly target public.

Homer et al. [12] conducted an experiment comparing language and reading outcomes for children who had a story read to them by an adult, to those who had the same story read to them by a character in a prototype of a Kinect game, either with or without the addition of in-game activities. Their findings encourages the next generation of digital literacy games.

4 Development of the Application

The game developed provides interaction through Kinect [2], or mouse. The main concept is related to the presentation of an image and the pronunciation of the respective word represented by the image. The child should choose, with the hand if he/she is using a Kinect otherwise with a mouse, the initial letter of the word he/she has just heard. The option of the initial letters is related to the level of orthographic difficulty within Portuguese language, detailed in Sect. 2. When the child selects the incorrect option, a sound alert is triggered informing the user has made a mistake, and he/she will have another opportunity to select the correct option. Consequently, the child will learn how to write the first letter of the word.

During the development, an interactive approach was adopted, where several versions were generated. Improvements and corrections were implemented using pilot versions. The final version was tested with children under literacy process. The functional and non-functional requirements are discussed in the following sections.

4.1 Functional Requirements

For the definition of functional and non-functional requirements, there were several meetings with pedagogy and psychology professionals with large experience with learning drawbacks in childhood. The following requirements were identified:

  • The game should allow, in case the children do not listen or have doubt about the pronunciation of a word, to repeat the reproduction of the sound or the word.

  • If the children select an incorrect answer, the application should notify them with a pleasant sentence: “Try again, you are almost there” or “Keep trying”.

  • Every time the game is executed, it should generate a report with the following information: image visualized by the child; the correct initial letter; the time the child selected the correct letter; how many times the child listened to the pronunciation of the word, and; how many mistakes the child made before selecting the correct letter.

  • In order to select the options of the game, the child will use his/her hands and to select (click) he/she has to push his hand forward.

  • The application should allow the children before starting the game, to train how to interact using hands gestures.

  • The words should be presented randomly during the execution.

  • When all the words are made available and the user concludes the word successfully, a textual and audio message should be presented saying: “Congratulations, you completed all the stages of the game”.

4.2 Non-functional Requirements

The non-functional requirements identified during the requirements phase were:

  • The time to complete each task of the game, that is, to select the initial letter of the presented word, cannot be too long or demanding. It should be easy and intuitive for the children.

  • Only one click should be enough to select an answer. More than one click should be difficult for the children to interact.

  • The distance between the objects on the screen should not be large, so that user can move the mouse and click easily over the object.

  • The game should not last too long, since this can be too demanding and exhaustive for the children.

4.3 Design and Implementation

The game was implemented using the programming language C# in order to apply the platform .NET Framework [2]. Also the development kit Kinect SDK 1.7 [2] was applied, besides the Kinect device.

In this game the hands gesture recognition was implemented in order to simulate a “click” (selection). Figure 1 illustrates one of the screens of the developed application. In this picture, the user’s hand is already depicted; however the interaction was not yet available.

Fig. 1.
figure 1

A screenshot of the application

Full size image

4.4 Assessment Methodology with Final Users

Initially, a private school in the city of São Paulo (Brazil) was contacted, and the developed project was presented in order to have their permission to run an application usability evaluation with their elementary school students (k2), which are already literate (can read and write), but still can make some misspelling. After this request approved by the school board, the consent letter was sent to the parents clarifying the content of the evaluation to be carried out, and requesting their approval so that their kids would participate in the evaluation.

Out of 35 children invited to participate in the evaluation, only 16 kids were authorized. The experiment was carried out at the school’s multimedia room, which was equipped with a projector and an appropriate sound system (Fig. 2).

Fig. 2.
figure 2

(a) Child interacting through Kinect. (b) Child interacting using a mouse

Full size image

Before starting the tests, a short interview was carried out with the users in order to characterize their profile, where some questions were made such as sex, age, previous experience with computers and if they were already familiar with the Kinect motion sensor.

Children were invited to interact with the application using Kinect (Fig. 2a) and also interact using the computer’s mouse (Fig. 2b). The tests were executed randomly, using Kinect or mouse, so that kids were not suggested to think that one or another interaction solution was better than the other.

After the interactions, children were invited to answer a quiz in order to provide further information related to usability and to their acceptance regarding the proposed application. In the quiz the Likert scale of 3 points [13] was applied with some images support to facilitate children’s understanding. The quiz proposed after the interaction using a mouse and Kinect was composed of the following questions: (a) Do you think the size of the image was good?; (b) Do you think it was easy to click on the image?; (c) Did you think the images were beautiful?; (d) Do you think the size of the letters was good?; (e) Did you like the sound?; (f) oud you play this game again using mouse/Kinect?

Running a pilot test, it was possible to verify that it would be necessary, in average, 1 min for explanations and profile interview, 3 min for executing the application using mouse interaction, 2 min for the quiz after test with mouse, 5 min for the execution of the application with Kinect, 2 min for the quiz after test with Kinect and 2 min of the interview after test.

5 Results

The usability evaluation was carried out with fourteen children from the school, eight girls and six boys, all of them with six years old. The result of the profile quiz indicated that all the participants had previous contact with computers, from which eight informed that are used to use the computer up to twice a week; four informed that they use the computer up to four times a week and two children everyday.

As for the utilization of the computer for entertainment purposes (games), thirteen children informed that they play with the computer, from which nine answered that they play with the computer up to twice a week; two informed that they play up to four times a week and two play everyday with the computer. Regarding Kinect, the profile quiz indicates that only six children had already previous experience with this type of interaction, mainly with Xbox Kinect console [14].

5.1 Results of the Usability Tests Using Mouse for the Interaction

Based on the answers provided by the children on the quizzes, it was possible to verify the usability of the application interacting with the interface with the mouse. The quiz revealed that only one child agreed partially about the size of the images in the application, and the remaining agreed that the size of it was good. Two children thought it was difficult to click on the image using the mouse, while the remaining considered this action straightforward. Also, it was possible to verify that twelve children thought the images beautiful and two partially agreed.

As for the size of the letter, thirteen kids informed that the size was good; one child answered that it was not good. Regarding the sound of the application, eleven children answered that it was good; one informed it was partially good, and two thought the sound was bad.

When they were questioned if they were likely to play again using mouse, just one kid informed negatively, the remaining answered yes.

As for the interaction with the mouse, we could conclude that the application met children’s expectations. As for the kid that informed that probably would not play again, it was not possible to observe the reason that caused this reaction since he/she did not inform the reason for his/her answer.

5.2 Results of the Usability Tests Using Kinect for the Interaction

After interacting with the application using Kinect, children answered usability assessment quiz, where it was possible to verify that thirteen kids agreed that the size of the image was good, and just one thought it was partially good.

As for clicking, nine children thought it was easy, two partially easy and three considered it difficult. We could observe that the children that did not considered it easy, or partially easy, had never interacted before with Kinect. Nevertheless, these kids claimed that they enjoyed using this kind of interaction.

Regarding the images, thirteen kids considered the images beautiful and only one answered that he/she though the image partially beautiful. All the kids answered that the size of the letter on the answers alternatives was good. Thirteen kids considered the sounds of the application good and only one informed that the audio quality was reasonably good.

In the question related to the application approval, twelve children liked to apply Kinect to play; one child liked partially and one didn’t like it. As for the question if they would ever play again the game using Kinect, twelve children answered affirmatively; one was undecided and one answered that would not play again using Kinect.

During the tests with Kinect, all the kids demonstrated interest and joy to be executing this activity. No child required assistance to use the application with Kinect. After explanations and the initial training, all the children concluded the activity successfully.

In addition, it was not possible to observe if any child felt irritated or tired with the game, however, we verified the successful conclusion of the activity by all the children.

During the tests, we could observe that six children selected any answer by mistake, that is, they selected a letter with a non-intentional push movement.

5.3 Comparative Between the Obtained Results

By the utilization of Kinect, children needed in average six minutes, and with the mouse, just three minutes. Since it was for the kids, in general, their first experience with a gesture-based interface, a longer duration to conclude the activity could be justified. Children that already had previous experience with a mouse required less time to conclude the activity.

After having used the game with two different forms of interaction - conventional with a mouse and gesture-based with Kinect, children answered an interview in order to verify which form was more attractive for them. Seven children preferred Kinect; five enjoyed both forms and they could not define which one pleased more and two children enjoyed most using the mouse to interact with the application.

When questioned what they liked most about the application, children replied that Kinect as the interaction solution since they were using their own hands to play with the computer. Just two children claimed that they got tired with the motion required by their arms and hands in order to interact and play.

6 Conclusions

The application developed in this work applied words which children under literacy process most likely have misspelling difficulties with. These words are in general related to unvoiced-voiced switches and multiple representations. Besides this, the developed game provided students with focus when studying, the first experience with natural interface for the interaction with the computer.

Through the usability tests carried out with K2 children in a private school of the city of São Paulo (Brazil), we were able to verify that children indeed make misspelling in this stage (in average, 19 % of mistakes). Therefore, we were able to conclude that the proposed application can be very useful during literacy process since help children in the written acquisition process.

As for the gesture-based interface with the utilization of Kinect, it was possible to verify children’s acceptance, demonstrating that it is not only viable but also useful to apply this kind of technology within classroom, helping teachers to make their classes more attractive, interactive and motivational within learning process.

Nevertheless, a minor issue was raised regarding Kinect which had difficulties to recognize children’s small hands. Therefore, kids needed to have their hands wide opened, with fingers away one from the other, so that Kinect could recognize the open hand gesture for the selection of objects on the screen. With adults this drawback could not be verified.

As for future works, it is interesting to add in the application other common misspelling mistakes that children usually make during literacy process, such as accentuation mistakes and spelling verification of the whole word, not only the first letter. It is also interesting if the application recognizes other motions and gestures, once that through the usability tests we could verify how motivated children were to interact with the computer though natural gestures. Another possibility would be to adapt the application to work with other idioms.