Graphical instruction for a garment folding robot

This project proposes to use interactive graphical editing interface for an end user to give instructions to intelligent robots to complete a real world object manipulation task. Natural language is often considered as an ideal communication method for robots, but it not intuitive at specifying tasks that require visual (geometry) information. Learning from demonstration can be useful, but it is not easy to generalize a provided example into a working program. Our approach is to provide a specialized graphical editor that abstracts the target task and to have the user specify how to complete the task by performing simple editing operations (clicking and dragging). We show the effectiveness of our approach by building and testing an example application based on this concept, which is a graphical editor for teaching garment folding to a robot. This example shows that our approach is particularly effective for an end user to configure the robot behavior to satisfy their own needs, which cannot be covered by a single, pre-programmed solution for general audience.