Prior to committing personnel to investigate a building or suspicious site such as a cave, it is imperative to determine the importance and current danger of the site. To this end, sensors on a robotic platform can interrogate the site prior to sending in personnel. This paper investigates methods to exploit multiple sensor modalities in order to automatically 1) detect human presence, and 2) detect human infrastructure and recent human activity. The paper describes 10 experimental scenarios to support these two tasks, demonstrates what type of inference each modality can make, and shows how to fuse the information from all sensors. Experimental results are also provided for the detection of the presence of humans.