The objective of this paper is to propose an approach which combines the multi view 3D reconstruction process of a building's facade with semantic information obtained by a deep neural network on the 2D observations of the considered scene. The object detector was used to annotate precise 2D features with semantic labels based on the object proposals. In a next step, spatially connected 2D features were evaluated to extend the initially computed labels to the whole facade element. The set of annotated 2D precise features was used to derive a classification of reconstructed 3D features. The a priori probabilities of the object detection model were used to compensate for biased predictions. The described approach was evaluated thoroughly with a view to different strategies for the annotation correction in 2D. Several failure patterns occurring in the considered application of facade reconstruction were observed and suggested solutions were given.