Keywords

1 Introduction

Neurological findings on the plasticity of the human brain and its ability to acquire and improve cognitive skills (much more than once thought possible) [1] have produced what Owen et al. [2] called the “multimillion-pound industry” (pp. 775) of computerized systems aimed at training the brain and improving cognitive functions. The explosive growth of this industry is also fueled by the rising number of elderly people and their desire to maintain their cognitive abilities [3], as well as advancements in our knowledge about learning deficits and attention disorders such as ADHD ([4, 5]).

An important question that must be addressed when designing computerized training systems concerns the provision of feedback to trainees during the training. Feedback is information provided by an agent – whether a teacher, a book, a program, etc. – about aspects of a learner’s performance or understanding. In a sense, then, feedback should be a straightforward reflection of the individual’s performance [6]. However, the effects of feedback are not straightforward. Feedback can improve performance by providing the trainee with information that can serve as a basis for error correction [7]. On the other hand, several studies have suggested that learners may become dependent on the feedback they receive, and in consequence fail to take in or effectively use other sources of information. The dependence on feedback creates over-confidence and blocks the development of learners’ intrinsic processing capacity, so that they become unable to detect errors themselves, or (for example) to process proprioceptive information in motor tasks – a phenomenon known as the guidance hypothesis ([7, 8]).

In addition, with the advance in technology, designing systems can take advantage of the opportunity to add complementary feedback channels for the performance various visual-spatial tasks. Adding complementary feedback channels to a visual-spatial task, and especially an auditory feedback, is supported by evidence from several studies which demonstrate that using both the visual and auditory channels for presenting information improves performance compared to the use of a single modality ([9–13]). However, although adding a complementary audio feedback to a visual-spatial task has the potential to be beneficial for performance and learning, several studies have demonstrated the in some cases this is not helpful, and sometimes it is even distractive. Audio feedback of earcones was found to be detrimental to response time in a drag-and-drop task [14]. Additional audio and haptic feedbacks in a menu-selection task harmed performance because of overloading the users with superfluous and distractive information [15]. An auditory feedback in a virtual reality insertion task deteriorated completion time, because it made participants to pay extra, unnecessary attention to collisions [16]. It seems that the benefit of a complementary auditory feedback is questionable.

In order to address some of these concerns, two studies were performed in computerized systems for puzzle completion tasks. The first study addressed the issue of training executive functions. Executive functions are the high-level cognitive skills – e.g., attention, self-control, working memory, and abstract thinking – involved in planning, organizing, and executing lower-level cognitive tasks. The first study questioned whether it may be possible to harness the motivational effects of feedback without risking the potential negative effects explained by the guidance hypothesis. This could be the case if the motivational power of feedback is derived simply from its being available, as distinct from any effect of actually drawing on feedback for guidance during training. The result should be improved performance among trainees who are given the opportunity to receive feedback compared with those for whom feedback is not available, even if the former rarely or never take up the feedback offer. We tested this hypothesis among 76 undergraduate students using a puzzle replication task.

The second study evaluated the effect of a complementary audio feedback in a 50-piece puzzle completion task. During the puzzle completion task, each time the participants composed a piece correctly he got a visual feedback, since the piece was stuck together with the other pieces. The complementary auditory feedback was a “beep” sound when the piece was composed correctly. Trainees were invited to two consecutive sessions, in each of them they were asked to complete three puzzles as quickly as possible. Fifty-three undergraduate students were randomly assigned to the two training groups: Complementary Auditory Feedback (CAF) group, who received the auditory feedback during the task; and a Control group, who did not receive the auditory feedback. We hypothesized that the CAF group will develop different performance strategies compared to the Control group.

2 Study 1: Feedback Availability

2.1 Method

Design and Experimental Task.

Several tasks were evaluated from a computerized training system developed by Mindri (http://www.mindri.com). The chosen task was a simple puzzle replication task using a variety of geometric shapes (see Fig. 1). Trainees were invited to two training sessions, held a week apart, in each of which they were asked to complete four 4-piece puzzles and eleven 9-piece puzzles. In the second session they were also given a transfer test, which required them to complete five 16-piece puzzles using a set of identical geometric shapes (see Fig. 2). To ensure that trainees would focus on improving their executive function skills, they were instructed to complete each task as efficiently as possible (i.e., using the smallest possible number of rotations and moves). They were also told that their payout for the session would depend on their performance across the entire set of tasks.

Fig. 1.
figure 1

The experimental puzzle completion task

Full size image
Fig. 2.
figure 2

The transfer test task

Full size image

Participants were randomly assigned to two between-participants groups: with and without the option of receiving feedback (the Feedback and No Feedback groups respectively). The feedback option showed participants the most efficient way to complete their most recent move. Because the task was relatively simple, most trainees with the feedback option chose to use it only once, or not at all. The data of trainees who used the feedback option more than once were excluded from the analysis (see below).

The experiment initially followed a 2X2 factorial design that would have compared the effects of two types of feedback: specific feedback on individual moves (Feedback/No Feedback) and more general feedback about the participant’s overall strategy (Strategy/No Strategy). Half the participants in each feedback condition were told that an expert trainer had evaluated their performance in the first session and had prepared strategies that would help them improve their performance. The strategies were very short and very general (e.g., “Try to think through your moves and carefully consider whether a piece matches before placing it in the frame. This will improve your efficiency”). A manipulation check showed that the strategies produced no significant effects. Hence, we decided not to include this manipulation as an independent variable.

Participants.

Participants initially included 88 undergraduate students (59 males, 29 females) from ORT Braude College, Israel. Forty-three participants (31 males, 12 females) were randomly assigned to the Feedback group, and 45 participants (28 males, 17 females) to the No Feedback group. Twelve participants in the Feedback group were excluded from the analysis because they used the suggested system’s feedback at least twice in at least one of the two sessions. Hence, the final sample comprised 76 participants, of whom 31 (21 males, 10 females) were in the Feedback group.

2.2 Results

The main dependent measures in both the main task and the transfer test were the number of excess rotations and moves performed when completing the puzzles. These were calculated by subtracting the minimum required rotations and moves for each puzzle from the number actually performed. Task duration was also calculated, although participants were not instructed to complete the task as fast as possible.

To analyze the results for the main tasks, a repeated-measures ANOVA was conducted, with feedback condition (Feedback/No Feedback) as the between-participants independent variable and the training session (first or second) as the within-participants independent repeated measure. Separate ANOVAs were conducted for the transfer test and the pencil-and-paper visual perception test, with feedback condition as the between-participants independent variable.

Main Task.

With respect to excess moves in the main task, the effect of training session was significant (F(1,74) = 4.74, p = 0.03, partial eta squared = 0.06): participants made fewer excess moves in the second session (M = 17.12, SE = 3.57) compared to the first session (M = 21.26, SE = 3.97). The effect of feedback condition was also significant (F(1,74) = 6.23, p = 0.02, partial eta squared = 0.08): participants in the Feedback group made fewer excess moves (M = 10.08, SE = 5.62) compared to the No Feedback group (M = 28.30, SE = 4.66). The interaction between training session and condition was not significant (F(1,74) = 0.65, p = 0.42, partial eta squared = 0.01).

The question arises whether the results would have been similar had we included those participants who were dropped from the analysis because they took advantage of the feedback option twice or more in either session (suggesting that they might have been weaker performers). We therefore ran a second repeated measures ANOVA with the addition of these 12 participants. The results confirmed the previous findings. Again, the effect of training session was significant (F(1,86) = 5.40, p = 0.02, partial eta squared = 0.06), with fewer excess moves made in the second session (M = 17.59, SE = 3.16) compared to the first (M = 21.37, SE = 3.46). The effect of condition was also significant (F(1,86) = 7.54, p = 0.01, partial eta squared = 0.08), with fewer excess moves made in the Feedback condition (M = 10.66, SE = 4.59) compared to the No Feedback condition (M = 28.30, SE = 4.49). As before, the interaction between training session and condition was not significant (F(1,86) = 1.36, p = 0.25, partial eta squared = 0.02).

With respect to excess rotations, the effect of training session was significant (F(1,74) = 33.87, p < 0.001, partial eta squared = 0.31): fewer excess rotations were performed in the second session (M = 17.56, SE = 2.65) compared to the first (M = 28.97, SE = 3.30). The effect of condition was not significant (F(1,74) = 2.42, p = 0.12, partial eta squared = 0.03), nor was the interaction between training session and condition (F(1,74) = 0.29, p = 0.66, partial eta squared = 0.003).

The pattern of results for mean duration is similar to the pattern for mean extra rotations. The effect of training session was significant (F(1,74) = 125.96, p < 0.001, partial eta squared = 0.63): participants needed less time to complete the task in the second session (M = 505.49 s, SE = 16.70) compared to the first (M = 667.26 s, SE = 23.64). The effect of feedback condition was not significant (F(1,74) = 3.01, p = 0.09, partial eta squared = 0.04), nor was the interaction between training session and condition (F(1,74) = 0.27, p = 0.61, partial eta squared = 0.004).

Transfer Test.

In the transfer test, the effect of feedback condition on excess moves was very close to significance (F(1,74) = 3.75, p = 0.057, partial eta squared = 0.05): fewer excess moves were made in the Feedback condition (M = 6.16, SE = 4.08) compared to the No Feedback condition (M = 16.42, SE = 3.39). As before, to evaluate whether excluding the participants who took advantage of the feedback option twice or more in either session affected the results, we ran a second ANOVA with the addition of these 12 participants. In this analysis the effect of feedback group was even more significant (F(1,86) = 6.46, p = 0.01, partial eta squared = 0.07), with fewer excess moves in the Feedback condition (M = 4.98, SE = 3.22) compared to the No Feedback condition (M = 16.42, SE = 3.15).

The effect of feedback condition on mean excess rotations in the transfer test was not significant (F(1,74) = 0.17, p = 0.68, partial eta squared = 0.002).

The effect of feedback condition on mean duration was not significant (F(1,74) = 1.91, p = 0.17, partial eta squared = 0.02).

2.3 Discussion and Conclusions

Overall, the current findings suggest that in computerized training systems for executive functions, simply making feedback available, even if most trainees are likely to make little use of it, can have strong influence on both training and performance. This finding should be evaluated further using other tasks and in other domains, to examine its robustness for future computerized training system design recommendations and guidelines.

The current findings are interesting in light of research about the extent to which priming – i.e., exposing individuals to a particular stimulus – can affect participants’ behavior and performance in different situations. For example, Bargh et al. [17] found that “participants whose concept of rudeness was primed interrupted the experimenter more quickly and frequently than did participants primed with polite-related stimuli,” while “participants for whom an elderly stereotype was primed walked more slowly down the hallway when leaving the experiment than did control participants” (p. 230). Steele and Aronson [18] showed that African Americans who were primed with a negative stereotype about their intellectual ability performed more poorly in intellectual tests than similar black participants who were not so primed. Likewise, Dijksterhuis and Van Knippenberg [19] reported that priming the stereotype of a professor or the trait “intelligent” improved participants’ performance on a general knowledge test, while priming the stereotype of soccer hooligans or the trait “stupid” reduced their performance. In a similar manner, the mere knowledge that feedback is available may prime trainees with the motivational effect of feedback, creating a psychological state of mind that may improve training and transfer.

3 Study 2: Complementary Auditory Feedback

3.1 Method

Design and Experimental Task.

Participants were randomly assigned to two between-participants groups: CAF group, who received an auditory feedback during the puzzle completion task: a “beep” sound when the piece was composed correctly; and a Control group, who did not receive the auditory feedback. Both groups got a visual feedback, which was the appearance of a piece as stuck together with the other pieces when it was composed correctly.

The experimental task entailed computerized puzzles composed of various pictures. The site used for the task was http://thejigsawpuzzles.com/Waterfalls-jigsaw-puzzle (see Fig. 3). Participants had to select a piece with the mouse and move it to the correct place in the puzzle. No rotations were needed. Clicking on the upper right icon displayed the entire picture, and clicking on it again closed the picture. A watch displaying the time elapsed from the beginning of each puzzle was located in the bottom right corner. When the puzzle was completed successfully the watch stopped. Participants in the CAF group used headphones. Participants faced a 20-pieces puzzle for practice (without headphones for the both groups) and 6 50-pieces puzzles (3 in the first day of the experiment and 3 in one week afterward). A transfer task with an additional 50-pieces puzzle (without headphones for the both groups) was given to both groups. The 8 puzzled used in the study were identical for all participants. Participants were told to complete the puzzles as quickly as possible.

Fig. 3.
figure 3

The experimental puzzle completion task

Full size image

Participants.

Participants included 53 undergraduate students (33 males, 20 females) from ORT Braude College, Israel. Twenty-seven participants (16 males, 11 females) were randomly assigned to the CAF group, and 23 participants (17 males, 9 females) to the Control group. The average participants’ age was 25.4, with a range of 20 to 30. All participants had normal or corrected-to-normal visual acuity.

Performance Strategies.

During the two session, participants’ actions when completed the 50-pieces puzzles were recorded and analyzed. Specifically, six performance strategies were analyzed. For each session, each participant scored “0” or “1” for every strategies, “1” if he used it and “0” if not. Below is the list of strategies and the measures:

  1. 1.

    Displaying Picture: Displaying the picture by clicking the icon. Participants who clicked the icon to display the picture at least twice got the score of 1.

  2. 2.

    Separating the frame: Separating the frame parts from the other parts. Participants who separated at least 5 parts of the frame in at least one puzzle got the score of 1.

  3. 3.

    Connecting the frame: Starting from connecting the frame parts before connecting the body of the puzzle. Participants who connected no more 5 parts of the body before completing connecting the frame in at each of the puzzles got the score of 1.

  4. 4.

    Connecting to the frame: Connecting puzzle parts to the frame. Participants who connected at least 5 parts to the frame before connecting the other puzzle parts in at least one puzzle got the score of 1.

  5. 5.

    Gathering parts: Gathering parts according to colors or patterns. Participants who gathered at least 5 parts in at least one puzzle got the score of 1.

  6. 6.

    Trial and Error: Trying to connect parts by trial and error. Participants who tried to connect at least one part to at least 5 places in the puzzle in at least one puzzle got the score of 1.

3.2 Results

The main dependent measures for the main task were the mean completion time of the puzzles and the use of each of the 6 strategies. In addition, it was decided to attach to each participant, in addition to his group (CAF or Control), a rating of his achievement. Based on the mean completion times in the first session, each participants scored as high achiever (belongs to the quickest half of the participants in his group) or low achiever (belongs to the slowest half of the participants in his group). A repeated-measures ANOVA was conducted, with group condition (CAF or Control) and level (high or low) as the between-participants independent variable, and the task session (first or second) as the within-participants independent repeated measure. Separate ANOVA was conducted for the transfer task, with group and level conditions as the between-participants independent variables, and the mean completion time as the dependent measure.

Due to technical problems while recording, the data on 7 participants’ strategies usages from the Control group was not available for analysis. However, the mean completion time in the main task for these 7 participants was 15.90 min compared to 16.00 min for the rest 19 participants, hence it was decided to continue with the analysis also these data were missing.

Main Task.

Mean completion time (in minutes). The effect of session was significant (F(1,49) = 82.95, p < 0.001, partial eta squared = 0.63): mean completion time was shorter in the second session (M = 14.86, SD = 3.67) compared to the first session (M = 17.86, SD = 4.83). The effect of group was not significant (F(1,49) = 1.47, p = 0.23, partial eta squared = 0.03). As expected, the effect of level was significant (F(1,49) = 91.11, p < 0.001, partial eta squared = 0.65): mean completion time for the high level (M = 13.25, SD = 1.80) was shorter compared to the low level (M = 19.47, SD = 3.08). The interaction between group and level was significant (F(1,49) = 6.68, p = 0.01, partial eta squared = 0.12): the difference between higher achievers (M = 12.80, SD = 1.69) and lower achievers (M = 20.71, SD = 3.01) in the CAF group was greater than the difference between higher achievers (M = 13.70, SD = 1.86) and lower achievers (M = 18.23, SD = 2.71) in the Control group, see Fig. 2. In addition, the interaction between session and level was also significant (F(1,49) = 21.49, p < 0.01, partial eta squared = 0.31): the difference between higher achievers (M = 14.00, SD = 1.94) and lower achievers (M = 21.74, SD = 3.56) in the first session was greater than the difference between higher achievers (M = 12.51, SD = 2.08) and lower achievers (M = 17.21, SD = 3.21) in the second session. The other interactions were not significant: the interaction between session and group (F(1,49) = 0.03, p = 0.86, partial eta squared < 0.001), and the triple interaction session X group X level (F(1,49) = 0.42, p = 0.52, partial eta squared = 0.01).

Mean strategies usage. Below are the main analysis of the strategy usage.

Strategy #2: Separating the Frame. The effect of session was not significant (F(1,42) = 0.57, p = 0.46, partial eta squared = 0.01). In contrast, the effect of group was significant (F(1,42) = 3.91, p = 0.06, partial eta squared = 0.13): 63.0 % (SD = 48.7 %) from the participants in the CAF group used this strategy, compared to 89.5 % (SD = 31.1 %) from the participants in the Control group. The effect of level was not significant (F(1,42) = 0.82, p = 0.37, partial eta squared = 0.02). The interaction between group and level was not significant (F(1,42) = 0.001, p = 0.92, partial eta squared < 0.001), nor the interaction between session and level (F(1,42) = 0.23, p = 0.64, partial eta squared = 0.01), the interaction between session and group (F(1,42) = 0.51, p = 0.48, partial eta squared = 0.01), and the triple interaction session X group X level (F(1,42) = 0.31, p = 0.58, partial eta squared = 0.01).

Strategy #6: Trial and Error. The effect of session was not significant (F(1,42) = 0.19, p = 0.66, partial eta squared = 0.005). In contrast, the effect of group was significant (F(1,42) = 4.59, p = 0.04, partial eta squared = 0.10): 59.3 % (SD = 46.9 %) from the participants in the CAF group used this strategy, compared to only 36.8 % (SD = 48.9 %) from the participants in the Control group. The effect of level was not significant (F(1,42) = 0.81, p = 0.37, partial eta squared = 0.02). The interaction between group and level was not significant (F(1,42) = 2.39, p = 0.13, partial eta squared = 0.05), nor the interaction between session and level (F(1,42) = 0.94, p = 0.34, partial eta squared = 0.02), the interaction between session and group (F(1,42) = 0.50, p = 0.59, partial eta squared = 0.01), and the triple interaction session X group X level (F(1,42) = 3.83, p = 0.06, partial eta squared = 0.08).

Transfer Test.

Mean completion time (in minutes). The effect of group was not significant (F(1,49) = 0.63, p = 0.43, partial eta squared = 0.01). The effect of level was significant (F(1,49) = 38.43, p < 0.001, partial eta squared = 0.44): mean completion time for the high level (M = 3.91, SD = 0.68) was shorter compared to the low level (M = 5.21, SD = 0.88). The interaction between group and level was significant (F(1,49) = 5.00, p = 0.03, partial eta squared = 0.09): the difference between higher achievers (M = 3.76, SD = 0.58) and lower achievers (M = 5.52, SD = 0.95) in the CAF group was greater than the difference between higher achievers (M = 4.06, SD = 0.77) and lower achievers (M = 4.89, SD = 0.70) in the Control group.

3.3 Discussion and Conclusions

The differences in performance between the CAF and Control groups, though the effect of group was not significant, we did find that, both for the main task and the transfer task, the difference between the higher achievers and the lower achievers was larger in the CAF group compared to the Control group. It seems that the CAF helped the better performers but deteriorated the performance of the weaker ones. Most interestingly, this effect was dominate also in the transfer task, when the CAF was not supplied. It can be assumed that participants who got CAF developed different performance strategies, which persisted also without its existence.

Indeed, a closer look at the strategies used by the two groups reveals some differences between the groups. While the Control group used the strategy of separating the frames parts from the other puzzle parts more, the CAF group used trial and error strategy more. It can be speculated that since the CAF emphasized immediate, short-term result, it caused participants to lower their level of pre-planning (for example, by separating the frame parts) and focus more on trying to maximize short-term achievements (for example, by trial and errors). In general, this behavior was less helpful for the weaker performers, maybe because it distracted them from the long-term goal of completing the entire puzzle.

To sum, the conclusion from the current study are that CAF which is in the low-level of feedback is not recommended for the weaker performers, as it caused them to adopt less effective strategies which are focused more on the short-term goals. Designer of modern system should be, therefore, careful when augmenting the visual-spatial task with CAF. It should be noted that the task which was chosen for this study, a puzzle task, requires planning, and it is possible that with simpler tasks this finding will not be evident. Future research should evaluate the robustness of this finding.