We propose a two-layer control architecture for bilateral teleoperation with communication delays. The controller is structured with an (outer) performance layer and an (inner) passivity layer. In the performance layer, any traditional controller for bilateral teleoperation can be implemented. In the passivity layer, the output of the performance layer is modified to guarantee that, from the operator and environment perspective, the overall teleoperator is passive: the amount of energy that can be extracted from the teleoperator is bounded from below and the rate of increase of the stored energy in the teleoperator is bounded by (twice) the environment and operator supplied power. Passivity is ensured by modulating the performance layer outputs and by injecting a variable amount of damping via an innovative energy-based logic that follows a principle of energy duplication and takes into account the effects of the delays. In contrast to the traditional teleoperation approach, where the master and slave controllers implement an as-stiff-as-possible coupling between the master and slave devices, our scheme is specifically designed for direct force-reflecting bilateral teleoperation: the slave controller mimics the operator action, while the master controller reflects the slave-environment interaction. We illustrate the effectiveness of the approach in a challenging simulation example with a round-trip delay of 100 ms, while making and breaking contact with a static environment having a stiffness of 50.000 N/m.