The GPS trajectories are rich with potential information that could be used to explore the regulation of traffic to serve the public. While that past approaches for short-term traffic prediction have existed for some time, emerging smart transportation technologies require the traffic prediction capability to be both fast and scalable to full urban networks. In this paper, we propose a novel neural network, named L-CNN based on CNN and LSTM, and develop an effective real-time prediction model to forecast the most likely potential passenger for taxi drivers. It is noteworthy that our model can be easily extended to other real-time traffic prediction problems, such as road traffic and flow prediction. Finally, we test our method based on GPS trajectories generated by Cheng Du taxi. The method presented provides passenger prediction over 15-min intervals for up to 1 h in advance and the results prove the efficiency of our predicting system.