In order to prevent the construction injuries effectively, it is essential to fully understand the accident causation in construction. Worker action detection and recognition can be treated as the initial step of further productivity and risk factor analysis. With the development of computer vision and machine learning techniques, monitoring worker activity automatically and continuously using camera becomes feasible and promising. In this paper, we focus on worker activity recognition problem and propose an automate recognition system based on an unconstrained worker activity video dataset, in which both coarse-grained and fine-grained actions coexist. Videos are segmented by graph cuts energy minimization. Neural network technique is integrated with principle component analysis for recognizing workers' activities automatically. Discussion on different scenario settings and comparison to the state-of-the-art method are provided. Experimental results show that the average accuracy outperforms the state-of-the-art results.