This paper identifies five human-robot interaction (HRI) challenges for physical object manipulation by small unmanned aerial systems (UAS). A survey of small unmanned aerial vehicles (UAVs) was used to categorize physical interaction into three configurations (ventral, frontal, and dorsal). Following this, anticipated individual and shared responsibilities for physical object manipulation are discussed for two previously established human team roles for small UAS (Pilot and Mission Specialist) using the Shared Roles Model. For each physical interaction configuration, the two human team roles are evaluated to determine: 1) what types of human-robot interactions are likely, 2) how the Pilot and Mission Specialist could interact with existing user interface technology, and 3) what are the perceived consequences of these physical interactions in the context of the human-robot team. The HRI challenges identified through this work suggest that Pilot interfaces will need to support a mixed view (exocentric and egocentric) and that interfaces for both roles will require shared visual common ground.