1 Introduction and Background

The technological revolution thoroughly changed the way people communicate. There has been a shift from the physical to the virtual world and, nowadays, most of the human interaction is computer mediated. A consequence of this shift towards a more and more important role of the human-to-computer-to-human interaction (HCHI) is that all the nonverbal part of a normal face-to-face communication, such as paralanguage and kinesics, is totally missing. In particular, kinesics is the use of body motion communication such as facial expressions and gestures. The use of facial expressions is particularly important not only to convoy richer information in human communication but it also plays a crucial role in people’s mental wellbeing. Indeed, in psychology, it is well known the principle that performing facial expressions directly influences our emotional state [11]. In fact, the James-Lange theory of emotion states that life experiences produce a direct physiological response via the human nervous system and the emotions occur as a consequence of these changes, rather than being the cause of them. In particular, Kleinke et al. conducted an experiment that produced interesting results confirming the aforementioned theory [13]. In fact, they found out that the test subjects reported an increased positive mood while performing positive facial expressions and a less positive experience while performing negative facial expressions. This was true even if the experiment participants were only mimicking these facial expressions. Moreover, the effects were enhanced when participants could see their reflection in a mirror. These findings become particularly relevant when associated to the benefits that smiling provide. In fact, smiling activates the release of neuropeptides that can help in fighting stress [18]. Moreover, smiling is also associated to the release of dopamine, endorphins and serotonin, which function as natural pain-reliever [15] and as anti-depressant [12].

In this paper, we present an application for smartphones that enables users to communicate in Social Awareness Streams (SAS) using facial expressions to share their emotional state. The main contribution of this work is twofold: (1) the introduction of kinesics in affective HCHI; (2) empowering users for the self-monitoring of their emotional states in order to support the expression of positive emotions. Through these two axes, the application allows augmenting the human expressivity for the communication of emotional states in CMC. This could also augment users’ social skills in off-line interaction and, finally, their happiness. This interface has been implemented in a mobile environment because currently a big part of the population owns and uses the smartphone, which also provides ubiquitous interaction and continuous connection. Indeed, the smartphone can be considered a pervasive technology that deeply penetrated the current society and that, therefore, can effectively influence users’ daily life.

The next section presents the related work; in the third section, the prototype interface is described in detail. The fourth section is dedicated to the observation study we conducted, while the last section presents the conclusion and future work.

2 Related Work

Positive technology aims at creating technologies that contribute to enhance happiness and psychological wellbeing. In [16], Riva et al. proposed a framework to classify positive technology according to their effect on three main features of personal experience:

  • Hedonic: technologies used to induce positive and pleasant experiences;

  • Eudaimonic: technologies used to support individuals in reaching engaging and self-actualizing experiences;

  • Social/Interpersonal: technologies used to support and improve social integration and/or connectedness between individuals, groups, and organizations.

The smartphone application introduced in this paper integrates the Hedonic and Social/Interpersonal features. For this reason, we present the related works concerning these features and using facial expressions as main means of emotion communication.

2.1 Social/Interpersonal Feature

Facial expressions provide an important channel for emotional and social display and some works tried to introduce this nonverbal interaction in computer-mediated communication. For example, FAIM captures users’ spontaneous facial expressions via a video and displays them as dynamic avatars in instant messaging software [7]. Similarly, Kuber and Wright developed an instant messaging application that recognized facial expressions through the Emotiv EPOC headset and displayed the users’ emotional states with smileys [14]. This study showed that this approach improved affective interaction compared with normal text-based communication. Caon et al. used the same technology for the facial expression recognition but they applied this concept to the communication over the SASs and the feedback was displayed in the environment in a context-aware multimodal manner [4].

2.2 Hedonic Feature

The notion that inducing a person to perform a smile can make her happier encouraged some researchers to propose novel interfaces that aim at promoting smiles. For example, Tsujita and Rekimoto presented a variety of digital appliances, such as refrigerators, alarm clocks, mirrors et cetera that require the user to smile in order to function properly [20]. In [19], they described two field tests that they conducted with the refrigerator. This refrigerator integrated a camera to recognize and count the user’s smiles, and only when the user smiled, the system facilitated opening the refrigerator door. These field tests showed that this system motivated users to smile more frequently, provided a tool for self-reflection and fostered communication in a couple. Hori et al. proposed a system composed of a smartphone and a wearable device, called communication pedometer, which applied the gamification approach to promote smiling during interpersonal communication [10]. The “Moodmeter” is a system that aimed at encouraging smiles in public spaces [9]. It was composed of a camera that recognized by-standers smiles and displayed it on a public display. This set up was replicated in many spots of the campus and it was possible to display a heat-map to show which spot was the “happiest”. Another interesting work in this field is the mirror that manipulates the user facial expression [22]; the authors of this work demonstrated that it is possible to manipulate emotional states through feedback of deformed facial expressions in real time.

The smartphone application presented in this paper proposes an interface that enables users to use facial expressions to share emotional information. At the same time, like the work made by Tsujita and Rekimoto [20], aims at fostering smiles.

3 Prototype

The smartphone application presented in this paper has been designed to support the first and third features of the positive technology framework [16]. Indeed, it allows the user to use facial expressions to communicate her emotional state in SASs. Since this application aims at improving the computer-mediated interaction, it could also enhance the feeling of connectedness between the elements of a social group. At the same time, this application provides some feedback about the performed facial expressions and reminds the user to smile in order to induce positive emotions and increase happiness.

3.1 Interface for Affective Interaction

This part of the interface aims at enhancing the computer-mediated communication of emotions and can be classified in the Social/interpersonal feature of the framework of the positive technologies. The SASs, such as Twitter, do not allow a variety of different modalities and have also a limited number of characters per message; therefore, the expressivity of emotions in SASs is quite limited. We developed this app in order to provide ubiquitous sharing of emotions in SASs (i.e., Twitter) through text, images and emotions (Fig. 1(b)). This app induces the user to communicate her emotional state performing the associated facial expression using the front camera embedded in the smartphone. In particular, when a user wants to share a message in Twitter, she can open the app where she will find the interface to enter the text, to browse the pictures and, before sending the message, the app asks the user if she wants to share an emotion in this message. If she agrees, the app will show the view from the front camera in order to allow the user to focus on her face and on the emotion detected by the system, as depicted in Fig. 1(a). The algorithm for the user’s facial expression recognition is able to recognize four different facial expressions associated to four emotions: smiling for happiness, frowning for sadness, scowling for anger, and winking for trust. The facial expression recognition has been implemented with the OpenCV library [1]. We used a supervised learning approach, where the user trains the system before the first utilization performing three times each expression. For the recognition phase, a similarity algorithm compares the ORB feature descriptors [17] for the mouth and eye regions. These regions are extracted using the Haar feature-based cascade classifiers proposed by Viola and Jones [21]. The app provides also an overview of the sent Tweets with the text, image and a smiley to represent the shared emotional state (Fig. 1(b)).

Fig. 1.
figure 1

Screenshots of the app (a) interface for emotion recognition; (b) overview of the sent Tweet with associated emotion; (c) the hedonometer; (d) the emotion pie chart.

Full size image

The choice of giving to the user the opportunity of communicating her current emotional state using the facial expression aims at reintroducing the use of the kinesics typical of the nonverbal communication. In fact, the facial expressions and the gestures are very meaningful means of expression in the face-to-face communication and they add more information to the spoken sentences. In particular, affection not only allows experiencing a richer interaction but also helps to disambiguate meaning, facilitating a more effective communication [2].

3.2 Interface for Mental Wellbeing

This part of the interface aims at inducing positive emotions to increase the user’s happiness and can be classified according to the hedonic feature of the positive technology framework. This application provides some statistics as feedback for user empowerment; indeed, they help users explore and understand information about themselves to support self-reflection and to identify opportunities for behavior change [6]. These statistics take into account the number of facial expressions performed during each day. Then, it is possible to show the number of smiles per day as a graph (Fig. 1(c)) and the percentage of all the performed facial expressions in a pie chart (Fig. 1(d)). Moreover, this application integrates a persuasive interface that solicits the user to smile when the smile counter detects a low level of happiness. In this case, the application triggers an alarm shown in the notification bar (Fig. 2(a)) and when the user acknowledges it, an interface showing funny or positive images coming from an RSS feed chosen by the user is displayed (Fig. 2(b)). In this screen, a progression bar shows how much the user should smile in order to accomplish this task. However, the user can access the RSS feed whenever she wants and when she smiles, the images are saved in the smartphone to allow the user to watch them again (Fig. 2(c)). This persuasive interface aims at inducing the user to smile in order to make her happier, as the works presented in [10] and [20]. In fact, as explained in Sect. 1, it is not important if the smile is unconscious or voluntary to increase positive mood, and the possibility to see the user’s face smiling like in a mirror can even increase this effect. Indeed, as for the hedonic feature of positive technology, this interface aims at enhancing happiness. Moreover, this tool aims at supporting self-reflection to train the self-regulation of emotions, which also contributes to mental health [8].

Fig. 2.
figure 2

Three screenshots to show the persuasive interface to motivate the user to smile: (a) the alarm in the notification bar; (b) the app shows an image to motivate the user to smile; (c) the user can browse the RSS feed in every moment and the app saves in the smartphone the images that make the user to smile.

Full size image

4 Observation Study

We performed the observation study during a live demonstration held during the conference on Affective Computing and Intelligent Interaction (ACII 2013) at the International Conference Centre of Geneva in Switzerland on 4 September 2013. We set up a prototype composed of multiple input and output interfaces for the communication of emotional states [5]. In particular, the setup comprised two main input interfaces: a touch-enabled Smart TV using the Microsoft Kinect for the facial expression recognition and the Android smartphone running the application we developed. We conducted an observation study both on the field and with the recorded video; other findings were presented in [3]. Watching the video, we calculated the number of conference attendees involved in the interactive demonstration: 85 people stopped to watch the demo and 29 of them actually interacted with the prototype. The people attending the conference were of different ethnics, genders and ages; although all of them were interested in affective computing, there were researchers and practitioners with very heterogeneous backgrounds, providing a varied pool of subjects for our first public test. The testers generally demonstrated a large interest in the proposed prototype. In particular, the interaction through facial expressions was appreciated, even if considered too artificial. In fact, this kind of interaction implies that the user has to express her emotional state through voluntary facial expressions that in this case can be seen more as a gesture rather than an unconscious physiological reaction; for this reason, some participants suggested to integrate an automatic facial expression recognition that detects the unconscious expressions and to propose to the user to share the detected emotional status on the SAS. The interaction with the persuasive interface to motivate the user to smile provided the same result: the users felt of being “faking” a smile in order to save the images they liked. However, as explained in Sect. 1, mimicking positive facial expression is an effective means to increase the user’s positive mood.

The most important research question we were trying to answer in this interactive event was to understand if performing facial expressions could be a problem in terms of user acceptability in a public context. In fact, using facial expressions to communicate with the mobile phone could be seen as awkward by the user, in particular in a social context like this event, where users were surrounded by a crowd that could be composed of friends, colleagues and strangers. The result after this first acceptability test is quite encouraging; indeed, nobody refused to perform facial expressions in front of the mobile phone for the communication of emotional states. This makes us think that in the current society, where “selfies” are a main trend invading the social networks, the use of a camera as means of communication is considered acceptable even in public spaces. We can consider these preliminary results encouraging, although we are aware that further tests are needed.

5 Conclusion and Future Work

In this paper, we presented a smartphone application integrating two features of positive technology: a Social/Interpersonal interface that aims at introducing a part of kinesics, i.e., facial expressions, in the computer-mediated communication of emotions. Moreover, this prototype takes advantage of the recorded emotions shared by the user in order to provide some statistics to enable self-reflection about the shared emotions; this aims at improving mental health and if the user did not smile enough during the day, a persuasive interface inspired by Tsujita and Rekimoto’s work [19], motivates the user to smile.

As future work, we are integrating a gamification mechanism, like in Hori et al.’s work [10], to sustain user’s engagement and to create an active community of people willing to improve their affective interaction and mental wellbeing.