In this paper, we propose a novel method for scene recognition using video images through the analysis of human activities. We aim at recognizing three kinds of things such as human activities, objects and environment. In the previous method, locations and orientations of objects are estimated using shape models, which are often claimed to be dependent upon individual scene. Instead of shape models, we employ conceptual knowledge about function and/or usage of objects as well as that about human actions. In our method, the location and usage of objects can be identified by observing interaction of human with them.