In this paper, we propose the use of semantic segmentation to incorporate scene information for better understanding of human motion in crowded environments. Our proposed SSeg-LSTM method leverages SegNet, which is a semantic segmentation encoder-decoder architecture, to extract semantically meaningful scene features. We then train the Social Scene LSTM (SS-LSTM) model with the contextual information regarding dynamics, social neighborhood, and scene semantics to predict future trajectory points of pedestrians. Experimental evaluation on public datasets show better performance for SSeg-LSTM than SS-LSTM which highlights the utility of semantic encoding for trajectory prediction.