1 Introduction

Smart rooms, smart offices, and smart homes are now a rising trend. Ensuring their user acceptance requires understandability [16, 26, 27]; i.e., users must be able to understand the functionality of each device in the environment [5]. Systems that ensure understandability and user satisfaction are more likely accepted and adopted by users [29]. In turn, ensuring satisfaction requires continuous evaluation during the user centric engineering process. Usually, evaluations are conducted under laboratory experiments, which is time and energy consuming. In addition, recruiting the necessary number of participants is difficult when conducting real world experiments, because subjects usually are selected from a geographically and culturally restricted area. Different solutions have been proposed to address these problems. Outsourcing, for example, helps researchers to outsource their testing tasks to other countries or laboratories with lower labour costs. However, Khaan shows disadvantages in such outsourcing [13]. Trust and security are big concerns, and many organizations hesitate sharing their critical information with others. Also, there is a threat of not getting the exact output that one expects, wasting a lot of money and time without having the desired outcome.

As an alternative, focus-group or group-thinking is proposed, giving the advantage of gathering various people and asking them to discuss different aspects of the system. Although proven to be efficient in collecting ideas, this still poses challenges [21]. For instance, it may lead to bias due to the vocal members influencing others and preventing them from expressing their viewpoints [15, 19]. Online experiments provide a way out of this; however, often participants lack necessary previous experience with such smart environments and do not have sufficiently developed mental models, leading to misunderstandings and wrong assessment and responses during the experiments.

In the past, traditional methods such as watching videos [25] or using visual narratives [3, 14] have been used to introduce novice users to smart environments and familiarize them with basic concepts of such systems. However, these instruments may lack interactivity and immersiveness, which are critical for a realistic experience that would lead to proper mental model development. In contrast, Interactive Realistic Virtual Reality Based Simulations (IRVRS) can successfully address this problem [12]. First, IRVRS facilitate easy and inexpensive access to a wide range of population, because anyone with a smartphone or other computing device can download and run a 3D environment. Second, IRVRS are multimedia enriched and create immersive user experiences, enabling users to feel and virtually touch the simulated environment, leading to the development of the necessary mental models. Although such virtual simulations cannot replace real world experiences, they have proved to be sufficient in communicating basic system concepts and principles [8], thus provide valid responses when surveyed. In other words, the interaction with IRVRS is hypothesized to support the development of mental models of novice users. The main drawback of using videos or IRVRS is the challenging task of multimedia storage, streaming, processing, and rendering. Dealing with these tasks is required because very large amount of multimedia content and experimental data will be produced and need to be managed, streamed or processed when conducting large-scale experiments with massive amount of users. However, these facets are not within the scope of this paper. Several effective solutions can be found in the literature [20].

In this paper, our contribution is the study of the effect of short-time experience with IRVRS on mental models of novice users of smart environments. We used a web-based IRVRS to study smart home users’ mental models, and we showed that IRVRS can potentially be effective in communicating basic system concepts and principles and create correct mental models.

The remainder of this paper is organized as follows. In Sect. 2, we describe the background and basic concepts in this topic, while in Sect. 3 we provide a brief literature review of related work. Section 4 describes our method to elicit the user’s mental model. Our results and analysis are shown in Sect. 5, and finally Sect. 6 concludes the paper.

2 Background

2.1 Smart Home Definition

According to Balta-Ozkan et al. “A smart home is a residence equipped with a communications network, linking sensors, domestic appliances, and devices, that can be remotely monitored, accessed or controlled, and which provide services that respond to the needs of its inhabitants” [2].

2.2 Mental Model Definition

According to Jonassen “Mental models are the conceptual and operational representations that humans develop while interacting with complex systems” [12]. In this research we aim to use IRVRS to develop the users’ mental model in order to match the conceptual model of smart home designers. The more this match is provided, the more we can rely on IRVRS in building the right mental model for users [18].

2.3 Interaction Conflict Definition

In the literature, conflicts happen when an agent or a group of agents cannot perform a task that they intend to do [27]. Shirehjini classifies human-automation conflicts to four major categories: wrong automation, inappropriate automation, over automation and interaction conflicts. In this paper, interaction conflicts are the type of conflicts that happen in the system, where the output of the automation process opposes user’s desires. For example the smart environment would turn off a light that the user has turned on because of energy consumption concerns [24]. In this research we aim to figure out the user’s understandability about conflict in order to evaluate their mental model development about this concept.

3 Related Work

In this section, we first discuss the existing work on the creation of mental models. Then, we discuss research on the elicitation of mental models.

3.1 Mental Model Development Tools and Methods

The open literature includes several methods for creating and capturing mental models. The work by Greenberg et al. investigates different methods that can create and develop mental models [9]. Wizard of Oz has been used by many researchers in developing mental models. This method is particularly helpful when an animated sketch has no real backend to understand users’ inputs and commands. A solution for the mentioned situation is to make a human a Wizard to do the backend side of the tasks which means that instead of the real system, the wizard would perform the backend tasks to be able to respond to the user.

Traditional methods such as watching videos or using visual narratives also support developing mental model for novice users. Diana L.M. Sharp et al. explore the impact of video on the language comprehension of children when they are listening to short stories [25]. They aim to improve the children’s comprehension and retelling of the story by using three types of video which are Helpful video, No video and Minimal Video.

In another research by Guttman et al., children listened to short stories that include pictures in some parts of the story which is named “partial pictures” [10]. By using videos and visual narratives to describe a story or an environment, the users can build mental models through which they can understand the environment in a better way. However, these methods cannot provide interactivity and immersiveness compared to 3D virtual environments. When no interactivity is provided, the quality of experience could be poor or not exist all. This means that novice users within a study are asked to comment on concepts they possibly have never experienced before, neither in the real world, nor in a virtual environment. As a result, subjects’ responses would be biased. Consequently, to address such biases it is necessary to provide a short term experience with the system under study. In this sense, IRVRS could be used.

Another related study by Nazari Shirehjini et al. investigates the effectivity of 3D virtual environments as a tool for rapid prototyping [17]. The work presented in [17] contributed a method to represent smart meeting spaces and prototyping a number of components (such as lights, blind, etc.) by using 3D-based virtual environments.

Other methods that support mental model development are reading and narrating stories. G.H. Bower and Morrow D.G. explain how readers and listeners construct mental models while they are reading or listening to what is described [3]. Creative drama is another method that supports mental model development. According to Arieli “Role-play has the potential to assist students to develop and create their own mental models” [1]. In this research, creative drama is used in order to build deeper understanding for the students in school.

3.2 Mental Model Analysis or Evaluation

In another study by A. Faiks, a Card Sorting method was deployed on a comprehensive online system for its digital library at Cornell University. It is implied in the research that Card Sorting is a beneficial method in discovering users’ mental models [6]. After collecting data from their 12 participants, they recorded the data in a dissimilarity matrix and then a cluster analysis was run on the data. In [6], the Card Sorting method and an approach for quantifying recorded data are explained in detail. The Card Sorting approach proved to be useful in incorporating users’ input into a system, and by doing so, increased the probability of user satisfaction with the final system. Also another research by K.A. Smith-Jentsch uses the Card Sorting method to assess similarity among subjects’ mental models of teamwork [28].

In terms of the elicitation of mental models, the work by Greenberg et al. explain the Think Aloud approach [9]. This approach works by asking users to think aloud while they are interacting with a system. By analyzing the users’ thoughts, one is able to determine their expectations of the system, and one can realize which parts of the system need to be redesigned in case the users are having difficulty in their interaction process. The method is easy to learn and it shows great potential in detecting usability flaws in the system. For examining the relationship between users’ mental model and trust, Sack et al. focus on analyzing mental model development in terms of anthropomorphic understanding of a smart home [22]. In their research, the mental models were evaluated by different methods such as teach-back [23, 30] and Think Aloud [9]. In the teach-back method users are asked to explain some metaphors and analogies of the mentioned simulated environment.

In another research, Wood et al. propose card sorting as a means of mental model capturing [31]. There are two main approaches to Card Sorting:

  1. 1.

    Open Card Sorting project is a method in which users should categorize a list of items that are representative of the content that it being studies.

  2. 2.

    Closed Card Sorting project is a method in which users should put some items that are given to them in some predefined categories.

In our research, we evaluated the mental models by Card Sorting and Think Aloud. We implemented a simulated smart home environment which users could interact with to get a sense of how the system works. After users were done interacting with the system, an interview was conducted to evaluate the users’ mental models using the Think Aloud method. We also used the Card Sorting method to evaluate the effectiveness of 3D web-based virtual environment in creating correct mental models.

3.3 Virtual Environment Evaluation Methods

In the area of evaluating virtual environments, Gabbard et al. propose a four-step approach [7], as follows:

  1. 1.

    User task analysis

  2. 2.

    Expert guidelines-based evaluation

  3. 3.

    Formative user-centered evaluation

  4. 4.

    Summative comparative evaluation.

In the first step, a complete list of tasks and subtasks in the virtual environment is documented. Also, the required resources that both the users and the system should have to be able to perform the tasks is identified. In the second step, based on the error and usability violation detected by an expert in user interaction design, the recommendation provided by the expert needs to be implemented in the next iteration of virtual environment development. In the third step, by including users, we try to evaluate and improve the user interaction design. Users utilize the Think Aloud approach [9] while performing tasks defined in the previous stage. Designers and evaluators record both qualitative and quantitative data using the output from Think Aloud, and fix the detected error for the next iteration. In the fourth step, the interaction design is compared to other matured and complete interaction designs having the same user’s tasks. The result of this stage is usually quantitative.

In [7], a limited number of users are marked as representative users which could be a potential disadvantage of the approach. It is mentioned in the paper that the third and fourth steps which require subjects are the most expensive steps, so in practice only one to three users were included for each cycle of the evaluation.

In another study, Bowman collects different types of evaluation techniques for virtual environments [4], including the following approaches:

  • Cognitive Walkthrough: In his approach, a list of tasks is given to the user, and while the user is performing the tasks, the interface ability in supporting those tasks is evaluated.

  • Post-hoc questionnaire: After the users have interacted with the environment, a set of questions is given to them to collect subjective data.

  • Interview/Demo: In this approach the interviewer directly talks to the subjects. One of interview’s differences with questionnaire is that, with interview, it is possible to go into details. Also in cases that it is important to have the users’ opinion or reaction to a particular situation, interviews seem like a more viable option to evaluate the virtual environment.

4 Our Research Method

In this study, to evaluate the effectiveness of web-based 3D virtual smart homes (IRVRS) in developing correct mental models, a user-centered design (UCD) framework was used. We started by asking subjects to perform tasks and we used the output data to both enhance the user interface design and to evaluate the system’s efficacy in developing mental models. In this section, the details of the tasks and experiments which subjects had to perform are explained. 17 subjects participated: 14 participants (aged 22–24) were undergraduate students from the Computer Engineering Department and Industrial Engineering Department of Sharif University of Technology, and 3 participants were middle-aged and non-student. Overall, 6 of the participants were female. Experiments were conducted within one week. Participants were randomly assigned to pre-arranged time slots. The experimentation took place within the lab, library, and the CE and IE departments’ lobbies. Our research method shown in Fig. 1 consists of a number of steps. Four of these steps, namely Think Aloud, Open Card Sorting, Analyzing the Results, and Closed Card Sorting are explained with details next.

Fig. 1.
figure 1

Research methodology

Full size image

4.1 Think Aloud

In the first stage of the Think Aloud method, we introduced ourselves to the participants and prepared the equipment which were needed, such as audio recording tools. Afterwards, it was explained to the participants that the recorded data would not be used in any other contexts, and the only reason the voice is being recorded is that we would be able to review the session in the future to do further analyzing. In the second stage, the purpose of the observation was described to the participants. Then we emphasized on the fact that we are trying to detect the flaws and problems of the system, so if the user has trouble interacting with the system, the user would know that it is the system’s fault, and not his/hers.

In the third stage, the 3D virtual system was given to the users. This 3D virtual system was a smart home with three major devices which are television, air conditioner and lights. The system has three modes of operation and depending on that the user has to turn on different devices in order to complete the given scenario.

  1. A.

    The “simple mode” included 2 tasks, one of which led to a conflict:

    1. 1.

      Turning on the television

    2. 2.

      Turning on the lights.

  2. B.

    The “normal mode” included 3 tasks and two of them led to a conflict:

    1. 1.

      Turning on the television

    2. 2.

      Turning on the lights

    3. 3.

      Turning on the air conditioner.

  3. C.

    The “complex mode” included the same 3 tasks as the second mode and the only difference was that all the tasks could lead to conflicts.

The simple mode was given to 7 users, 8 users were given the normal mode, and 2 users were given the complex mode.

In the fourth stage, after conducting the experiment with all the participants and recording the facts, we detected the difficulties in user-system interaction and fixed them iteratively before conducting further studies.

4.2 Open Card Sorting

In Open Card Sorting, the same subjects participated. They were given a list of items representing important entities of a smart home. Participants were then asked to categorize the items in a way they thought was the best organization, as shown in Fig. 2.

Fig. 2.
figure 2

Participants categorizing the items in way the thought was best

Full size image

4.3 Analyzing the Results

In the third phase of our research we analyzed the results in an expert focus group in which the authors and experts of smart systems engineering participated. The research instrument “expert focus group” is described in detail in [8].

Based on this analysis, four categories of major concepts of smart systems were identified: “users control over smart home”, “users control over smart home devices”, “users’ understandability of conflict” and “categorizing the conflict with relevant cards”. In other words, with our research results from steps 1 to 3, the expert focus group concluded that a mental model of a smart system under study must include 24 entities, as shown in Table 1, which can be sorted using the 4 aforementioned major categories.

Table 1. Cards which were used in the experiment.
Full size table

4.4 Closed Card Sorting

In the fourth phase of our research, the 4 categories mentioned in Sect. 4.3 were used by the same 17 participants for sorting the items. The first category is “users’ control over smart home” which contains two sub-categories: “has control”, and “does not have control”. The users should group the Smart Home and User cards within the mentioned sub-categories. In the second category which is “users control over smart home device”, the user should put the home devices in one of the two sub-categories which are “under smart home control” and “under users’ control”. For the third category which is “users understandability of conflict”, the user has to put the conflict card in one of the two sub-categories of “error is caused by technical flaws” and “error is caused by smart home configuration”. In the fourth category which is “categorizing the conflict with relevant cards” the user is expected to change the place of the conflict card to a group which its devices has led to conflict. Closed Card Sorting [6, 31] was especially useful for achieving more detailed adjustments.

5 Results and Analysis

We analyzed our qualitative results and identified 4 major concepts to convert them to quantitative results, which are shown in Fig. 3.

Fig. 3.
figure 3

The results of mental model development evaluation

Full size image

The first concept is the understandability behind the concept of conflict. Those users who figured out a conflict as an error caused by the smart home configuration were able to develop the correct mental model. Approximately 76 % of the users understood the concept behind conflict and 24 % of them did not figure out the mentioned concept correctly. This result proves the effectiveness of using IRVRS.

The second concept is the user’s ability to group the “Conflict” card with the devices that had conflict in them rather than categorizing it with some irrelevant cards. After working with the system and performing different tasks, the result of the open card sorting unveiled that 47 % of the users grouped the “Conflict” card with the devices that had conflict in them and 53 % of them did not group the card correctly with the related devices. So 53 % of the users were wrong, which does not prove the effectiveness of using IRVRS for this specific case.

Participant’s conceptualization about control migration between system and its users was the third concept under study. Again, the majority of the users, 59 %, believed that only the smart home has control over the tasks, which is wrong, while 41 % of users were correct and believed both smart home and themselves have the ability to control the environment. The results here also does not prove the effectiveness of using IRVRS for this specific case.

But the 4th concept, user’s control over smart home devices, proves that IRVRS are effective in communicating basic system concepts and principles. 82 % of the users found themselves having control over devices which had no conflict and they successfully created the correct mental model about this concept. On the other hand, 18 % of them realized they have control on almost all the devices including the ones having conflict.

As the results of the study show, 2 of the major concepts (users’ understandability of conflict and users’ control over smart home devices) proved that mental models were developed adequately, however, the other 2 concepts did not have the same output. By enhancing the quality of IRVRS and also by using methods that are more precise in mental model capturing, we can improve the quality of the experiment to get more promising results.

6 Conclusion

In this paper, we discussed the crucial need of assessing users’ mental models before the construction of the real systems. By using the Think Aloud and Card Sorting methods which are two efficient approaches for capturing mental models, we were able to evaluate mental model development of web-based 3D virtual environment users. The experiment included 17 participants who were asked to sort the cards before and after interacting with the 3D environment. By analyzing the results, the research showed that virtual environments have a great potential in creating adequate mental models. There are number of aspects which can be more developed in future work. First, by developing more advanced 3D virtual environments we would be able to develop mental models more accurately. Another factor that possibly challenged the presented experiment is the usability of the IRVRS we used. Another future work is to redesign the system. Also we aim to use other approaches of mental model capturing. By doing so we would able to get a more accurate assessment of mental model development.