3D object reconstruction from images involves two important parts: object identification and object modeling. Human beings are very adept at automatically identifying different objects in a scene due to the extensive training they receive over their lifetimes. Similarly, machines need to be trained to perform this task. At present, automated 3D object identification process from aerial video imagery encounters various problems due to uncertainties in data. The first problem is setting the input parameters of segmentation algorithm for accurate identification of the homogeneous surfaces in the scene. The second problem is deterministic inference used on the features extracted from these homogeneous surfaces or segments to identify different objects such as buildings, and trees. These problems would result in the 3D models being overfitted to a particular data set as a result of which they would fail when applied to other data sets. In this paper, an algorithm for using probabilistic inference to determine input segmentation parameters and to identify 3D objects from aerial video imagery is described. Bayesian networks are used to perform the probabilistic inference. In order to improve the accuracy of the identification process, information from Lidar data is fused with the visual imagery in a Bayesian network. The imagery is generated using the DIRSIG (Digital Imaging and Remote Sensing Image Generation) model at RIT. The parameters of the airborne sensor such as focal length, detector size, average flying height and the external parameters such as solar zenith angle can be simulated using this tool. The results show a significant improvement in the accuracy of object identification when Lidar data is fused with visual imagery compared to that when visual imagery is used alone.