Conventional user behavior detection relies on a large amount of sensors and expensive monitoring devices. Moreover, the systems are usually intrusive and may suffer from deployment problems. In this paper, we design and implement an energy management system (EMS) consisting of a non-intrusive load monitoring (NILM) meter, gateway, server and mobile device. The NILM meter provides a non-intrusive and low-cost solution to recognize the states of appliances and to disaggregate the energy consumption of appliances in a house/building. Based on the proposed EMS, we further implement a data mining scheme to detect users' behaviors based on the usage patterns of appliances. A prototype system verifies our design concept and the simulation results show that the detection accuracy of users' behaviors is more than 80% for most of the activities.