Data-to-decision systems must fuse information from heterogeneous sources to infer a high-level understanding of a situation. A high degree of confidence in the inferred knowledge is necessary for appropriate actions to be taken based upon the assessment of a situation. This paper presents an extensible Semantic Web compatible framework that uses rich ontological descriptions for the autonomous and human-aided fusion of heterogeneous sensors and algorithms to create evidence-based hypotheses of a situation under persistent surveillance. Raw data acquired from profiling sensors is combined with the output of visualization and classification algorithms, yielding information with a higher degree of confidence than what would be obtained without the fusion process. The framework can readily accommodate other data sources and algorithms into the fusion process.