The rolling bearing prognostics holds a great potential in improving maintenance actions and promoting reliability for the operation of the machinery. This paper proposes a novel direct bearing remaining useful life (RUL) prediction approach based on the newly developed temporal convolutional networks (TCN). Unlike many exist data-driven approaches which apply complex feature engineering to achieve efficient results, such as time-frequency analysis and feature selection, etc., the proposed end-to-end prediction approach focus on performing the feature learning more directly and lightly from the raw vibration signals. For the first, signal segmentation is conducted and some statistical features can be attained. Then, these features are fed into the TCN model for RUL prediction. Numerical experiments based on practical rolling bearing dataset show that the proposed approach can not only achieve competitive prediction accuracy, but also require much less time for training in comparison with several baseline data-driven approaches.