This paper proposes a novel framework to estimate the amount of physical activity at a place where people stayed, by utilizing facility information and user's acceleration data. The total amount of physical activities, energy expenditure of a physical activity is a good scale. Our framework provides a physical activity scale based on typical energy expenditure of the activity given by existing researches already. To estimates the energy expenditure, we use typical value of metabolic equivalents to task (MET) which is used as practical scale. Unlike the other studies for monitoring physical activity, we estimate the type of user's activity using facility information which are obtained from road map and land-use/land-cover map. To confirm the feasibility of our approach, we have conducted a long-term experiment on monitoring the activity of elderly people living in less-populated area. As a result, our framework provides good summarization of daily activities of participants.