Predicting individual vehicles’ future locations is of great significance in location-based services. Regularity and preference are two predominant features of individual vehicles for location prediction. However, these two features cannot be adequately captured with existing models based on raw trajectories, due to their oversimplified view. Modeling regularity and preference entails the following challenges: 1) how to design features to accurately reflect regularity and preference with raw trajectories; and 2) how to capture regularity and preference effectively based on sparse travel behaviors. To that end, first, we design four features to represent travel regularity and preference by conducting an in-depth correlation analysis on individual vehicle trajectories. Then, we incorporate these four features in context and propose a deep neural network to jointly capture regularity and preference via explicitly retrieving the context. Specifically, our proposed model extends LSTM with memory to store all the output hidden states, which provides long-term information for retrieval to overcome the vanishing sequential dependency in sparse scenarios. To fully capture regularity and preference for prediction, a backtracking attention mechanism is designed to aggregate all the relevant historical hidden states in memory with different weights based on the regularity and preference similarity. The weighted aggregation produces a new hidden state, which is used for the final prediction. Experiments on three real-world vehicle trajectory datasets containing over 10,000 individual vehicles show that our proposed model outperforms the state-of-the-art models by 7%–10% in terms of prediction accuracy.