This article introduces a framework to generate three-dimensional models for augmented reality (AR) including semantics. The semantics of 3D models have been studied previously in the field of intelligent virtual environments, but the process of creating real- world models containing such information have received little attention. We introduce a method for the creation of semantic, 3D models of the environment using an AR application named InventAry. Assisted by an ontology, InventAry enforces the creation of combined geometric and semantic environment model.