Generative Attention Learning: a “GenerAL” framework for high-performance multi-fingered grasping in clutter

Abstract

Generative Attention Learning (GenerAL) is a framework for high-DOF multi-fingered grasping that is not only robust to dense clutter and novel objects but also effective with a variety of different parallel-jaw and multi-fingered robot hands. This framework introduces a novel attention mechanism that substantially improves the grasp success rate in clutter. Its generative nature allows the learning of full-DOF grasps with flexible end-effector positions and orientations, as well as all finger joint angles of the hand. Trained purely in simulation, this framework skillfully closes the sim-to-real gap. To close the visual sim-to-real gap, this framework uses a single depth image as input. To close the dynamics sim-to-real gap, this framework circumvents continuous motor control with a direct mapping from pixel to Cartesian space inferred from the same depth image. Finally, this framework demonstrates inter-robot generality by achieving over $92\%$ real-world grasp success rates in cluttered scenes with novel objects using two multi-fingered robotic hand-arm systems with different degrees of freedom.

Appendix

1.1 Mechanical structure of robotics hands in experiments

1.1.1 Staubli–Barrett

The finger joint angles of the Barrett Hand range from 0 (open) to $2.44 \text { rad}$ (close) for finger-1, finger-2 and finger-3 and from 0 to $\pi \text { rad}$ for the lateral-spread. We used the original hand throughout all experiments.

1.1.2 UR5-Seed

For the anthropomorphic Seed hand, flexion joints curl the fingers toward the palm of the hand between 0 (open) to $1.48 \text { rad}$ (close), while adduction joints spread the fingers apart from 0 to $\pi /2 \text { rad}$. Therefore, only the thumb on the Seed hand may spread. The wrist-rotation, wrist-flexion and wrist-adduction DOFs, in addition to the finger flexion and adduction DOFs, contribute to a total of 8 DOFs for the Seed hand. However, these three wrist-DOFs belong to arm-DOFs $\varPsi _{\mathrm{arm}}$ because they do not actuate any fingers. Each robotic finger consists of 3 joints, all of which are controlled by a single dyneema tendon (i.e. a Kevlar fiber string). The ring and pinky fingers are coupled such that a single actuator is responsible for the flexion of both fingers, while all other fingers are controlled by their own actuator, respectively.

1.2 Mechanical modification to the RH8D Seed hand

As shown in Fig. 6, fingers demonstrate under-actuated behavior during control. As the actuator moves through its full range of motion, the tendon pulls on the distal, intermediate, and proximal joints of a given finger in three stages:

• 1. Initially, the finger is fully open before closing (Fig. 6a). As the finger starts to close, the distal joint rotates almost completely before the intermediate joint even begins to move.

• 2. The distal joint, reaching its maximum displacement, ceases to rotate while the intermediate joint continues to move (Fig. 6b).

• 3. The proximal joint reaches its limit right after the intermediate joint stops moving (Fig. 6c).

Fig. 6

The finger-closing behavior of the Seed hand before a–c and after d–f modification, using the index finger as an example

While the fingers are anthropomorphic in design, human fingers would move differently to grasp an object. In particular, the proximal joint of the human finger will traditionally curl before any other joint at a greater rate than the distal joint. Fingers of the human hand will also rarely ever settle in a hook-like position as depicted in Fig. 6b—the fingers must sweep through a greater volume in order to sufficiently contact and grasp objects beyond thin cylindrical geometries. With these observations in mind, tape is applied around the distal joint (Fig. 1b) to inhibit it from rotating (Fig. 6d–f). This configuration effectively reduces the number of under-actuated joints on each finger by one. Figure 6f depicts the resulting grasp form. Also, the fingers of the Seed hand are composed of very low-friction thermoplastic material. To increase friction on the fingers, we added small rubber caps to the fingertips (Fig. 1b).