The handling of contacts between virtual objects is a fundamental problem of 6 DOF haptic rendering in virtual environments. Especially, the haptic rendering of the contact between rigid and deformable objects is challenging because the physics simulation with high computational burden must be calculated in real time with good stability. In this paper, we introduce a fast and stable contact handling method that can be used for 6 DOF haptic rendering. The two-way contact response is formulated based on a penalty method. To enhance the numerical stability in large time step, the penalty-based contact force is integrated using implicit time integration. The method was evaluated by experiments using a haptic device combined with a physics simulator based on finite element method.