Studies indicate that haptic interaction with a computer generated virtual scene may become more intuitive by aligning (co-locating) the visual and haptic workspaces so that the visual and haptic feedback coincide as they do in the real world. Co-located haptics may gain importance when more advanced haptic interfaces, such as high-fidelity whole hand devices, become available. We describe a user study that investigates the pros and cons with physically co-located versus non-collocated haptics on two different display types: a commercial half-transparent mirror 3D display with shutter glasses and a prototype autostereoscopic display based on a Holographic Optical Element (HOE). We use two accuracy tasks with spatial accuracy as the dependent variable and one manipulation task with time as the dependent variable. The study shows that on both displays co-location significantly improves completion time in the manipulation task. However, the study shows that co-location does not improve the accuracy in the spatial accuracy tasks.