In autonomous driving, state-of-the-art methods detect pedestrians from their appearances in 2-D spatial images. However, these approaches are typically time-consuming in terms of the algorithm complexity to cope with the large variations in shape, pose, action, and illumination. Capturing motion needs even more efforts. In a completely different approach, this work recognizes pedestrians along with their motion directions in a temporal way. By projecting a driving video to a 2-D temporal image called Motion Profile (MP), we can robustly distinguish pedestrian in motion against smooth background motion. To ensure non-redundant data processing of deep network on a compact motion profile further, a novel temporal-shift memory (TSM) model is developed to perform deep learning of sequential input in linear processing time. In experiments containing pedestrian motion from various sensors such as video and LiDAR, we demonstrate that, with the reduced data size around 3/720th of a video volume, this motion-based method can reach the detecting rate of pedestrians at 90% in near and mid-range on the road. With the super-fast speed and good accuracy, this method is promising for intelligent vehicles.