Machine learning has been a major driver for improving semiconductor laser chip manufacture process. The virtual metrology system was used to enable the manufacturers to conjecture the wafer quality and deduce the causes of defects without performing physical metrology. However, building the virtual metrology system required a large amount of classified chip images. Therefore, a fast, accurate, portable image classifier was needed to fit modern flexible semiconductor laser manufacture setup, even without Internet connection. Based on a few pre-trained deep learning modes(AlexNet, ZFNet, and GoogLeNet), we use transfer learning to train the classifier on semiconductor distributed feedback (DFB) laser chip images. The GoogLeNet was identified to outperform the other two, and a portable image classifier was built. This paper has two main contributions: (1) A GoogLeNet-based semiconductor laser chip defect detection and classification network was developed with better than 97% accuracy in manufacturing production test. (2) The inference network is implemented on single board computer with an Intel Movidius Neural Compute Stick and USB digital microscope to form a low-power off-line handheld laser chip defect image classifier.