Semantic labels are crucial parts of many location-based applications. Previous efforts in location-based systems have mostly paid attention to achieve high accuracy in localization or navigation, with the assumption that the mapping between the locations and the semantic labels are given or will be done manually. In this paper, we propose a system called Deep-Crowd-Label that automatically assigns semantic labels to locations. We propose a novel transfer learning method that leverages deep learning models deployed on many crowd-workers to assign semantic labels to locations by classifying associated visual data. Deep-Crowd-Label uses the power of the crowd to aggregate the individual predictions done by the model across the crowd-workers visiting the same location. Our preliminary experiments with 26 different types of locations show that, our method and our prototype system is able to find the right label for the locations i.e., coffee shop to the Starbucks.