This paper explores the haptic interaction between various users collaborating in performing a manipulation of an environment through telerobotic systems. In this type of interactions, mechanical energy is simultaneously exchanged between the users and the object being manipulated. Moreover, agents can be physically located at high distances to each other as well as objects, resulting in non-neglectful time delays. A framework based on passivity is presented to allow stable designs and implementations of multi-user/robot interactions. The common Position-Force (PF) and Position-Position (PP) can be applied as links between two agents in the multilateral system in a modular way.