Public buildings such as shopping arcades and railway stations are environments in which pedestrian movement is of significance to many smart building applications. The data-driven approach of pedestrian trajectory prediction is effective in learning a reliable model that can represent complex human movement. Pedestrian trajectories are highly linked to the locations of facilities and services inside a building as pedestrians move towards these destinations for engagement. This paper suggests that the notion of destination is a strong predictor of pedestrian trajectories and proposes a novel enhancement of the data-driven approach for pedestrian tracking in public buildings. The method of destination-driven pedestrian trajectory prediction (DDPTP) first evaluates the most likely destinations of the pedestrian using the destination classifier (DC) and then predicts the future trajectories with the destination-specific trajectory model (DTM). The proposed solution has been evaluated on the NYGC and the ATC datasets and found to outperform state-of-the-art models. The notion of destination can be further developed into a region of interest of which the within-region and out-of-region features can be factored out for more effective learning.