Depth sensors in general, but especially consumer level sensors, have noise characteristics that are difficult to generalize due to external factors such as material specific properties or varying lighting conditions. The quality of the reconstructed surface from fused depth data may therefore vary and some surface segments may be more accurately reconstructed than others. High average accuracy is important but it must be supported by an estimation of the local quality if subsequent algorithms, such as robotic grasping, use the surface reconstruction. We propose a novel approach which models sensor noise depending on local object properties and incrementally refines surface estimations using Bayesian updates. Experiments demonstrate that our algorithm reconstructs challenging scenes of the ROBI dataset (Yang et al., 2021) with comparatively high accuracy and significantly fewer outliers. We additionally provide a reliable, local estimation of the surface uncertainty.