Human-human collaboration is characterized by a back-and-forth, where an action of one agent elicits the response of the other. This interaction is inherently multimodal and includes both high-level modalities such as language and low-level ones such as force exchanges. In this work, we investigate human collaborative manipulation: we show distinct patterns that can be identified in low-level physical data and that can be interpreted as primitives used by humans to negotiate about various aspects of the motion and to execute the motion. These primitives provide a high-level interpretation of the interaction and can be used to connect low-level behavior to language. We describe the human study used to collect the data, the data analysis process, and discuss how the identified primitives could be used by a robot’s interaction manager to mediate physical Human-Robot Interaction (pHRI).