Autonomous mobile robots exploring areas in a search and rescue (SAR) context benefit greatly from the usage of thermal imaging cameras (TICs). In unknown environments, the interpretation of the temperature values provided by the camera is a challenge. This is due to the fact that temperature calculation based on thermal images depends strongly on the regarded surface emissivity. In this work, we show how to improve the temperature interpretation in thermal images making use of the diverse characteristics of the emissivity at varying viewing angles. Using several thermal images taken from different points of view, we are able to do a basic material classification. In combination with an algorithm that maps temperature values on the environment's 3D structure, we estimate the regarded surface emissivity and temperature. The presented approach is evaluated by a single camera setup using nonlinear optimization techniques.