A Social-Aware Assistant to support individuals with visual impairments during social interaction: A systematic requirements analysis

Highlights

• A socially-aware assistant, consisting of both perceptive and feedback components.

• Our approach is based on a wearable platform.

• Our system has been tested with visually impaired people.

Abstract

Visual impairment affects the normal course of activities in everyday life including mobility, education, employment, and social interaction. Most of the existing technical solutions devoted to empowering the visually impaired people are in the areas of navigation (obstacle avoidance), access to printed information and object recognition. Less effort has been dedicated so far in developing solutions to support social interactions. In this paper, we introduce a Social-Aware Assistant (SAA) that provides visually impaired people with cues to enhance their face-to-face conversations. The system consists of a perceptive component (represented by smartglasses with an embedded video camera) and a feedback component (represented by a haptic belt). When the vision system detects a head nodding, the belt vibrates, thus suggesting the user to replicate (mirror) the gesture. In our experiments, sighted persons interacted with blind people wearing the SAA. We instructed the former to mirror the noddings according to the vibratory signal, while the latter interacted naturally. After the face-to-face conversation, the participants had an interview to express their experience regarding the use of this new technological assistant. With the data collected during the experiment, we have assessed quantitatively and qualitatively the device usefulness and user satisfaction.

Introduction

The visual impairment affects almost every activity of daily living, including mobility, orientation, education, employment, and social interaction. Despite these limitations, blind people reinforce their other perceptual mechanisms: hearing (Kolarik et al., 2016), tact (Voss et al., 2016) and olfact (Araneda et al., 2016), as a form to compensate for their impairment. Some blind people are even capable of developing echolocation, which is the ability to detect objects in their environment by actively creating sounds, e.g., by tapping their canes, lightly stomping their foot, snapping their fingers, or making clicking noises with their mouths (Secchi et al., 2017), and sensing echoes from sound bouncing on the objects. A skilled blind pedestrian can approach an intersection, listen to the traffic, and from audible information alone, judge the spatial layout of intersecting streets, the width of the road, the traffic volume, and the presence of pedestrian islands or medians  (Liu and Sun, 2006). Likewise, blind people tend to develop a highly sensitive tactile sense that allows them to read Braille system (Russomanno et al., 2015), and to scan their surroundings (Goldreich and Kanics, 2003). However, even though blind people tend to compensate their lack of sight by augmenting their other senses, the sensory bandwidth of the visual channel is orders of magnitude greater than that of auditory and touch  (Loomis et al., 2012).

Nowadays, we witness an unprecedented effort of the scientific community to develop solutions which could restore, even partially, the sense of sight. As a result, several assistive systems using computer vision have been developed for different applications: access to printed information  (Cutter, Manduchi, 2013, Cutter, Manduchi, 2015), object recognition  (Gallo, Manduchi, 2011, Guida, Comanducci, Colombo, 2011, Winlock, Christiansen, Belongie, 2010), navigation  (Flores, Manduchi, Zenteno, 2014, Vera, Zenteno, Salas, 2013) and social interaction  (Krishna, Bala, McDaniel, McGuire, Panchanathan, 2010, Krishna, Panchanathan, 2010). A comprehensive survey of these applications could be found in Manduchi and Coughlan (2012). Of all these areas, the least exploited one is social interaction (Loomis et al., 2012). The importance of the visual channel is reflected in the type of information exchanged with our counterparts during social interactions. According to Ambady and Rosenthal (1992), nearly 65% of the generated volume is through non-verbal cues, i.e., eye gaze, facial/hand/body gestures. Thus, a visually impaired person suffers a serious limitation in accessing this rich flow of information and in consequence this deprivation could lead to social isolation, depression, loneliness, and anxiety  (Hinds et al., 2003).

In this work, we develop a wearable technological assistant to provide visually impaired people with cues that may enhance their social interaction during face-to-face conversations. The system architecture is depicted in Fig. 1(a). It consists of a perceptive component (represented by smartglasses which have embedded a video camera shown in Fig. 1(b)) and a reactive component (represented by a haptic belt shown in Fig. 1(c)). The video stream captured by the camera is sent to a computer, where a specialized software runs face detection, tracking, and head gestures algorithms. When a head nodding is detected, this information is sent to the vibratory belt informing the user, who, on his turn, could replicate the gesture. By closing this gesture loop, a mirroring event is triggered.

The main contributions of our approach are:

• The introduction a Socially-Aware Assistant (SAA), consisting of perceptive, and feedback components.

• The implementation of a computer-vision-based application for a social assistant on a wearable platform. The use of smartglasses confers the system a first-person perspective, which is ideal for this type of applications when compared with other assistive systems that offer a third-person perspective.

• The development of new insights obtained by testing the SAA with visually impaired people. We validated our system in a simulated tourism office environment, where we envisioned a face-to-face conversation taking place between a blind person (acting as a tourist guide) and a sighted person (acting as tourists). As a result, a custom data corpus has been recorded, annotated, and will be made available to the research community upon request.

Our application integrates work we have previously done on head-gesture recognition (Terven, Raducanu, Meza-de Luna, Salas, 2016, Terven, Salas, Raducanu, 2014) and social interaction (Meza-de Luna et al., 2016), and new results which include the development of a real-time technology assistant for visual impaired social interaction, and the use of a vibratory belt for haptic feedback.

The rest of the paper is structured as follows: we dedicate Section 2 to review the related work in the area of assistive technology for social interaction. In Section 3, we describe the system’s architecture. In Section 4, we explain the design of the experimental study. In Section 5, we report our results and discuss the findings of our study. Finally, we draw our conclusion and provide some guidelines for future work.