This paper presents a multi-channel continuous-wave near-infrared spectroscopy (NIRS) system, which is applied to classify different cortical activation states of the prefrontal cortex (PFC). Mental arithmetic, digit span, semantic task and relax were selected as four mental tasks. A deep forest algorithm is employed to achieve high classification accuracy. With employing multi-grained scanning to NIRS data, this system can extract the structural features and result in higher performance. The proposed system with proper optimization can achieve 85.7% accuracy on the self-built dataset, which is the highest results compared to the existing systems.