Keywords

1 Introduction

Many families are separated, with family members being unable to see one another often, for various reasons. This paper terms such family members as distant family members. Distant family members can easily communicate with one another using current technologies that provide communication services, including text messaging and voice chatting. Although these services are convenient to use, we found that they require an accurate and timely response and active participation from both parties. Such communication methods also have a certain degree of intrusiveness into people’s daily life because they drive users to view information notifications frequently, and even cause a form of psychological addiction [1], of which a prominent feature may be anxiety arising from a separation from communication devices [2].

Existing research on nonintrusive communication and emerging techniques offer us new possibilities for connecting distant family members in a more natural and less intrusive way than current communication methods. Former studies on the emotion and information expression by actuators with limited degrees of freedom provided reasonable elements for constructing nonintrusive interaction, while artificial intelligence (AI) provided more interaction possibilities, such as emotion recognition and facial identification, and offered users a better experience when integrated with the Internet of Things (IoT) [3].

We therefore propose the form of tacit interaction that supports nonintrusive communication in a natural way, and develop QianLi on the basis of this form of interaction to connect distant family members. In our work, tacit interaction refers to ambiguous, simple, and periphery interaction that requires a certain degree of empathy but not demands timely responses. We apply the concept of tacit interaction to the QianLi system so that QianLi is developed as an AI and IoT (AIoT) system comprising a central base and several modular devices that connects distant family members without excessive intrusion while fully respecting the wishes of both parties. Because tacit interaction is not alike in different families, QianLi has a modular design that aims to connect distant families by providing customized communication channels. Adopting emerging AI and IoT technology, QianLi provides more diverse interaction possibilities between family members.

In this paper, we first define the concept of tacit interaction and provide design guidelines for this type of interaction. We then demonstrate the implementation methods, technical components, and interaction possibilities of the QianLi system. At last, we recruit several participants to create interactions between distant family members with the help of the system. The created design cases and the collected feedback demonstrate that our system can support the exploration of tacit interaction for connecting the distant family members.

The main contributions of the paper are as follows:

First, we propose the form of tacit interaction and develop the QianLi system as an implementation of the concept. The QianLi system applies AI and the IoT to connect distant family members, convey emotions or evoke precious memories in a tacit and nonintrusive way, and generate a feeling of “being together”.

Second, we let the user decide how to map their input information to the reaction displayed on the other side, which makes QianLi a potential research material that allows users to contribute their preference in expressing such tacit interaction in a real-life scenario.

2 Related Work

2.1 Connecting Distant Family Members

Family activities, including celebrations, traditions, and patterned family interactions, are important in family lives and are called family rituals [4]. Petrelli et al. [5] constructed interactive machines for family Christmas rituals to reinforce the family’s feeling for the annual celebration. Mynatt et al. [6] introduced the digital family portrait that provides qualitative visualizations of a family member’s daily life and regarded the use of the portrait as a family ritual.

In recent years, studies have investigated distant family members who live separately owing to their jobs and other reasons [7]. Several design projects [8, 9] investigated the rituals of family members who live in different places. Chatting et al. [9] built playful tangible objects that help distant family members be involved in ritualized activities, such as drinking together or anticipation of time together, which constitute being a family [8].

However, Wolin et al. [4] proposed that all families struggle for finding a suitable role for rituals in their collective lives and their actual rituals vary greatly, which is consistent with the result of Family Rituals 2.0 [9], in which there are five different interactive machines for characteristics of different families.

Previous work on connecting family members with tangible objects [5] and phatic technologies [10, 11] led us to design networked things with tacit, nonintrusive, and natural interaction in this area. Former methods provide interesting devices for connecting distant families, but it seems more suitable for family members to have the opportunity to find their own way to express themselves because the actual solution for connecting families varies greatly across different families.

2.2 Nonintrusive Communication

Multiple communication services and technologies support a variety of social communication, but most communication methods, such as email, twitter, text messaging, and video chatting, are often accurate, and real-time. Such communications require us to express ourselves as accurately and timely as possible, which calls for active participation.

We classify the existing communication methods according to the complexity of the information, to what extent they work in real-time, and the demand for a response. Common communication methods are shown in terms of three dimensions (see Fig. 1). We generally consider a communication method that is simple, not instant, and does not require a response to be a type of nonintrusive communication.

Fig. 1.
figure 1

Three-dimension space of communication methods and nonintrusive communication

Full size image

Nonintrusive communication can be regarded as an attempt to design calm technology systems that inform but do not demand our focus, or attention [12]. Efforts have been made to accomplish such calm technology or nonintrusive communication. These studies explored one-way [13] and two-way [14] information channels as well as the interaction involving different information types, such as haptic [15] and light [16, 17].

The study of Olivera et al. [16] and García-Herranz et al. [18] showed new forms of interaction that restrictively express limited information based on the common context that conversational partners share and will possibly support intrusive forms of communication. However, such researches do not notice how people in daily activities reach tacit consensus for nonintrusive communication, which reminds us to leave more space to the users themselves to explore.

3 Tacit Interaction

Compared with communication methods that call for the attention of users, less intrusive ways of communicating are more commonly used between people who are familiar with each other. Those who know each other often do not need two-way or accurate communication and only need, for example, facial expressions and gestures to achieve a tacit consensus and interact. Such interactions include the conversational and vocal ways of moving, engaging, and touching, and play an important role in structuring the forms of “being together” throughout life [19].

On the basis of those interactions achieving a tacit consensus and former research on nonintrusive communication, we propose the concept of tacit interaction that supports nonintrusive communication further.

Tacit interaction usually refers to ambiguous, simple, and periphery interaction that requires a certain degree of empathy but not demands timely responses. Several characteristics of tacit interaction are summarized and shown below.

No Response Required:

Tacit interaction can be two-way but most tacit interaction often does not require an exact and immediate response, because the intention of such interaction is not to officially start a dialogue or communication. If an interaction is set to be real-time or must be responded to, then the receiver of the message will, to some extent, feel pressure of having to reply to the message.

Ambiguous:

Tacit interaction cannot express and convey particularly accurate information, and is often used to express ambiguous information, such as complex feelings and different emotions. For example, it is difficult to accurately express information, such as the exact time and place of a meeting, through a simple gesture that is sufficient to express some emotions like joy or sad.

Simple:

The information that tacit interaction conveys cannot be too complicated because excessive information may result in a cognitive load that is too high and even stress the user. For instance, an accident may occur if a user sends and receives messages while driving, but a quick glance at the indicator light on the dashboard usually does not increase the cognitive load.

Periphery:

Both parties engaged in tacit interaction often realize such interaction by noticing something in their periphery of their attention. Indeed, most of these interactions are not deliberately prepared but simply express a type of thought inadvertently. The user may be disturbed by a pop-up notification when using a computer, but a vibrating alert of notification may not disrupt the user’s concentration.

Empathy:

Often, both parties of the interaction need to develop empathy with each other to better carry out tacit interaction. Playing the same song may generate different feelings in different users whose life experiences differ. On the contrary, the users may better understand the feeling of others if they have shared life experiences or memories related to the song.

It is difficult to establish tacit interaction because such interaction relies on previous consensus or similar prior experiences, and a lot of information expression methods that easily interferes with people are not suitable anymore. When designers want to design tacit interaction, they need to strike a balance between the possibility of users resonating and the possibility of annoying users. We thus propose several guidelines for the design of such interactions.

Scenario:

Because tacit interaction is ambiguous, the lost detailed information needs to be reasoned and complemented by the receiver using contextual knowledge and information. Designers therefore consider engaging interactions in a scenario where it is easy to promote empathy among users or where a cultural or social consensus exists. As an example, a simple flashing light could mean anything, but if a light flashes in a specific scenario where both parties have reached a consensus in advance, such as a traffic scenario, its meaning may be clear.

Customized:

Tacit interaction has much to do with the user’s previous experiences and it is thus impossible to arouse a tacit feeling in the user by providing only one specific design to all users. It is more reasonable to give users the opportunity to customize such interaction according to their own ideas.

4 QianLi System

We intend to apply the concept of tacit interaction to the system for connecting distant family members because the distant family members are familiar with each other, which makes it easier to establish tacit interaction. Meanwhile, distant family members are in urgent need of nonintrusive communications rather than communication that can overload the user.

In Chinese, QianLi is the pronunciation of both “kilo miles away” and “hold your hands”, which presents our sincere wish that QianLi can connect distant family members. In contrast with other systems of connecting distant families, the QianLi system lets users reserve the right to design their own expression because we assume that the interaction between family members differs from family to family.

To offer appropriate design space for users to explore tacit interaction, we look into unobtrusive ways of expression with limited information in previous research (see Table 1). We then take those reasonable expression methods as the expression elements of our system. We subsequently adopt user evaluations to verify whether users experience a satisfactory interaction if we provide them with space for exploration.

Table 1. Categories of expression with limited information
Full size table

4.1 Expression with Limited Information

For there to be no intrusion, it is necessary to use limited and simple information or unobtrusive ways in the expression process. In fact, several researchers are studying how to use actuators with limited degrees of freedom can be used for the expression in mobile phones or wearable devices. We summarize these methods of expressing emotion or information, including the use of light, sound, motion, and vibration, in Table 1.

The above research and findings, especially the different methods of expression, provide us with the expression elements of the QianLi system. Although previous research has obtained some ground truth about emotional feelings and various expression (see Table 1), we leave the design space to the users to customize their own interactions because each family has its own ways of communication.

4.2 Description of QianLi

Our system has a minimalist design and fits well with various styles of interior design. It has simple shapes, such as a triangle and hexagon, all of which are formed using simple straight lines or arcs (see Fig. 2).

Fig. 2.
figure 2

Composition of the QianLi system

Full size image

Central Base:

The base consists of a Raspberry Pi and Movidius Neural Compute Stick [30] (NCS) (see Fig. 3). The NCS is used to deploy an AI application and the Raspberry Pi performs information processing and transmission between modules and QianLi platforms. To handle the amount of information, we tapped into Alibaba Cloud’s server for data transmission.

Fig. 3.
figure 3

Components of the central base.

Full size image

Modules:

We designed nine modules in total, comprising five input and four output modules. Each module consists of a NodeMcu [31], sensor or actuator, switch, and battery (see Fig. 4). The NodeMcu is used for wireless communication with the base and control of the sensor or actuator. All the input modules and output modules are shown in Table 2.

Fig. 4.
figure 4

Components of the modules.

Full size image
Table 2. Summary of nine modules
Full size table

The camera module and sound module are empowered by AI technology and equipped with an NCS. The camera module has facial identity, facial emotion and object recognition abilities respectively afforded by different AI models, such as Inception model and FaceNet model [32]. The sound module can detect simple voice commands with the help of an NCS and AI models. To protect the user’s privacy and to reduce the feeling of surveillance, no footage taken is recorded by these modules or the base. Users can customize the output depending on the voice, facial identity, facial emotion, or object detected. For instance, different output modules on the other side will be activated when the camera detects different people.

Interaction Possibilities:

Distant family members would have individual sets of the QianLi system. Each communication is one way, with one set identified as the sender and the other as the receiver (see Fig. 5). Users can decide where QianLi is placed and how it is used. Users have the flexibility to mix and match the modules according to their preferences.

Fig. 5.
figure 5

Working process of the QianLi system

Full size image

When one input module detects that a condition is fulfilled (e.g., a PIR sensor is triggered), it sends a signal to the central base and the QianLi platform (cloud server). The receiver base can then produce a customized output through a module, which could be the spinning of a rotation module or lighting up of a LED.

Furthermore, the users themselves can decide how to present their output signal to enable actuators, such as those of an LED or rotation servo. For instance, the AI camera module can detect different faces and facial emotions, so that users can freely map their emotions to different colors of LED, rotation angles of servo, or vibration frequencies and durations (see Fig. 6).

Fig. 6.
figure 6

Mobile application of customizing the mapping between input and output modules

Full size image

4.3 Evaluation

To evaluate QianLi from the user’s point of view, we recruited four participants who lived separately from their family to create tacit interaction they want for their family and received feedback for further development.

Procedure:

Initially, participants were introduced to the system for around 15 min to ensure they knew how to set up the system. They were then invited to the formal trial of using the QianLi system to communicate with their own distant family. We recorded the design process of the participants and only offered help when they had trouble constructing an interaction between modules.

After several successful attempts, the participants were asked to give feedback on QianLi in terms of its usefulness and usability.

Participants:

The four recruited participants (P1–P4) were carefully selected to cover different genders, ages, and familiarities with the relevant technology knowledge and are thus representative of the majority of distant family members. Table 3 summarizes the participants in the evaluation.

Table 3. Summary of participants in the evaluation
Full size table

Results:

We recorded each participant’s main design processes, design ideas, and major problems encountered. To protect the privacy of the participants, images presenting four design cases are taken with two female models.

P1 is an undergraduate student studying industrial design, who has basic knowledge of the IoT and learned how to use QianLi rapidly. P1 used the camera module to share her smile with her parents through the output of a motion module connected to a sunny doll because “a sunny doll is an object in our favorite cartoon used to pray for sunny days, and I (P1) hope the doll induces a happy mood in my parents”. The parents know that their child is smiling from the motion of the sunny doll (see Fig. 7). P1 was pleased that our system gave her the chance to stay connected with her family using the valued memory. P1 also looked forward to using other modules that provided interesting feedback. P1 believed that more modules could be created to provide more possibilities such as odors. At the same time, she hoped for more control channels of the current outputs such as the flashing frequency or color change of lights.

Fig. 7.
figure 7

Design case of P1

Full size image

P2 is a foreign student whose family now lives in Spain. He is new to the IoT but receptive to new technology, and he thus smoothly became familiar with the system. In the design case of P2, a color sensor can be placed on the table to detect the color of fruit that he picks to eat. The corresponding LED will light up on his parents’ side (see Fig. 8). This idea originated from his parents who has been always reminding him to eat fruit to obtain the required vitamins when he first came to China. Additionally, the different time zones between the two countries calls for an expression, such as that conveyed by lighting, to last longer. The parents can place a photograph of P2 under the light and see him when the light shines. P2 felt that expressions would be constrained when distant family members were in different time zones and hoped to record and replay interactions that have already occurred through the mobile app.

Fig. 8.
figure 8

Design case of P2

Full size image

P3 is the mother of a 23-year-old girl and lives apart from her daughter because of work. P3 often uses a smart speaker and owns many smart home devices and she therefore learned how to use and set up the QianLi modules quickly. P3’s proposal is shown in Fig. 9, where the sound module and vibration module are connected. When P3 uses a smart speaker or listens to music, the vibration module on the other side will shake and a toy bird will slide down a pole. P3 created this expression because she wanted to share her relaxing moments with her family and the motion of the funny toy bird makes her family laugh. P3 was highly appreciative of the system because she believed that being involved in the creation of the interaction with her children and husband allowed her to better express her love, and that the noninvasiveness would not bother her children. However, she wondered if the modules could be designed to connect to other IoT devices, like a smart speaker at home, so that she could play a song for her family even when she is absent.

Fig. 9.
figure 9

Design case of P3

Full size image

Figure 10 illustrates the design case of P4, showing a windmill turning on the other side when the child leaves the house or returns (i.e., the PIR sensor is placed near the door). The parents then know that the child is leaving for school or has returned from the motion of the windmill. This interaction involves a windmill, with P4 stating that “I taught my daughter how to make a windmill when she was a child, and the windmill is therefore the most representative symbol for us.” P4 is not a fan of fancy technology and is tired of becoming familiar with smart phones and different applications. P4 spent much time getting familiar with the QianLi system with our help. P4 was happy to see the system helping him to connect with his daughter without using a mobile phone that is a little complex for him. P4 acknowledged the practicality of such system, but he felt that there were still difficulties in setting up the modules and hoped “that the children can help me set up the different modules or that I can get a customized interaction initialized when buying the modules”.

Fig. 10.
figure 10

Design case of P4

Full size image

5 Discussion

The evaluation showed that our system works well in providing a platform for customizable tacit interaction between family members. The interactions created by family members indeed conveyed emotions or evoked precious memories in a tacit and nonintrusive way, and generated a feeling of “being together”. However, the interviews and records of the design process showed the shortcomings of the system and provided insights for improving the system.

First, QianLi is fully functional but not stable owing to time constraints in its design and it is still at the prototyping stage. The current QianLi system is not mature, and many deficiencies and defects need to be addressed to improve the ease of usage of the system; e.g., adding a way of recording and replaying the interaction. In addition, we are also considering expanding the functionality to connect with other IoT devices and improving all the modules with the usage of AI.

Second, hints and interesting interaction examples of how our modules can be used need to be provided because we found that participants were easily inspired by the designs of others or the provided examples to achieve interesting results. We also found that participants who were unfamiliar with IoT technology were reluctant to try the most complex AI input modules. Therefore, we may provide an interface that allows separated family members to set up modules for their parents.

Furthermore, we found that although participants were willing to associate their family memories and previous experiences with interaction designs, they used to present these special memories using additional objects. Previous studies found patterns of emotional expression, such as red being used for happiness and blue for unhappiness [21]. However, existing design cases have not paid much attention to emotional expression but rather sent a message in what can be considered a family ritual [4]. Perhaps it is because this is the first time that users have used QianLi and they only made simple attempts. However, it is also possible that we did not provide enough space for users to express their emotions. Therefore, we are considering providing different modules or more control channels of the current module to encourage exploration.

6 Conclusion

Inspired by the idea of nonintrusive communication, this paper proposes the concept of tacit interaction and presents an application, QianLi, a modular AIoT system comprising a base and several modular devices. Qianli is designed to connect distant family members by providing tacit interaction channels that can be customized by users. An evaluation demonstrated that users can indeed create their own way of nonintrusive communicating with their family tacitly with the help of the QianLi system. It also demonstrated that tacit interaction can make people feel connected in a nonintrusive way.

The modules of QianLi currently can only communicate within the system. However, we are interested in expanding the functionality to connect with other IoT devices and looking into the usage of AI to improve the functions and features of all the modules. In future work, we may explore other interactions that are created in the practical application of QianLi. So far, we have built a platform to invite participants to take part in the program. We hope to gain insights about tacit interaction from users’ preferences in expressing themselves in addition to their attitudes and feelings towards the system.