Automatic vision-based defect detection on the mobile light guide plate (LGP) is a challenging task due to the low contrast between the defect and the background, uneven brightness, and complex gradient texture. An end-to-end multitask learning network architecture for the defect detection of mobile phone LGP is proposed. First, the main structure of the multitask learning network is designed. The encoder part uses a similar U-Net encoder structure to obtain multiscale features, and the feature fusion part adopts feature fusion to interact with multiscale features. Second, the segmentation head is designed to complete the precise location of each defect in an image by using the multiscale feature fusion, which prepares it for the quantification of defect characteristics. Combining the multiscale features and the output mask of the segmentation head, the classification head is designed to accurately detect the defects of mobile phone LGP. Finally, the defect detection data set has been constructed based on the mobile phone LGP images collected on the industrial site, and a lot of experiments are performed on the mobile phone LGP data set and Kolektor surface-defect data set (KolektorSDD). The experimental results show that the F1-score on the two data sets can reach 99.67% and 96.77%, respectively, which verifies the effectiveness of the method proposed in this article.