Shared Autonomy Perception and Manipulation of Physical Device Controls

Abstract

As robots begin to enter our homes and workplaces, they will have to deal with the devices and appliances that are already there. These devices are invariably designed with human perception and manipulation abilities in mind. Unfortunately, this often makes them hard for robots to interact with autonomously. Control elements, such as buttons, switches, and knobs, are often hard to identify with current sensors. Even when they are found, it is often not clear how they should be manipulated to achieve a specific goal without extensive background knowledge of the specific device and task.

In this paper, we describe a shared-autonomy approach to the identification and operation of these device controls. A human operator provides assistance with perception and high-level planning, while the robot takes care of the low-level actions that depend on closed-loop sensor feedback. We demonstrate our approach by controlling a consumer electronics device, despite not being able to autonomously sense some of its control elements, and give the results of some initial evaluations suggesting that the shared autonomy interface is both easier to use and more efficient than a direct teleoperation interface.