To allow collaborative robots to work efficiently and effectively with their human partners, one of the critical functions they need is to precisely and robustly recognize human intentions, i.e., what action they will perform next. In this paper, we present a Recurrent Convolutional Neural Networks (RCNN)-based system that is capable of recognizing a human intention much earlier than the intended action takes place. The system consists of two main components, a Deep Convolutional Neural Networks (DCNN) component that extracts spatial patterns of human activities and a Long Short-Term Memory (LSTM) component that extracts temporal patterns of human activities. We demonstrate the power of our proposed system to data of humans manipulating objects. The results show that our system has superior performance than many existing algorithms in term of recognition accuracy. Moreover, our system can achieve a quite high intention prediction accuracy (about 80%) provided with only the first 80% of the data sequence.