Pollen recognition has been shown to be important for a number of areas ranging from criminal investigations to paleoclimate studies. However, these palynology studies rely on highly qualified professionals to analyze pollen grains, which have become scarce and costly. Therefore, the automation of this task using computational methods is promising. Deep learning has proven to be the ultimate technique in computer vision tasks, but is very difficult to build a pollen data set with size enough to train such networks from scratch. This study investigated the use of transfer learning from pre-trained deep neural networks for pollen classification and compared their results with training from scratch and with promising predesigned features. Additionally, we introduced the biggest data set of pollen to the date. Experimental results achieved up to 96.24% of classification accuracy, suggesting that the fine-tuned deep learning architectures can be successfully applied to pollen classification.