Since the traditional gear pitting detection technique based on artificial observation has low accuracy and efficiency, an accurate vision detection method based on a new development of Mask R-CNN is explored. In order to solve the issue of small pitting samples, the generated adversarial network is firstly employed to enlarge the training samples with multilevel pitting. Then, we put forward a multipath fusion Mask R-CNN with double attention (DAMF Mask R-CNN) to implement the simultaneous segmentation of tooth surface and gear pitting. This new type of Mask R-CNN enhances the ability of target segmentation by embedding two attention modules, which can strengthen the feature expression. Considering the difference in the grayscale texture shapes of different-level pitting images, a multicascade multipath feature extraction network is constructed by designing a multicascade multipath pyramid module and adding a dual attention module, which significantly improves the generalization ability and segmentation performance of small pitting. Finally, we segment the gear pitting and tooth surface by the DAMF Mask R-CNN neural networks, and then compare it with Mask R-CNNs, Mask Scoring R-CNNs, U-Net, and Deeplabv3+. It can be known from the comparative results that the proposed DAMF Mask R-CNN has higher detection accuracy and convergence performance than the traditional segmentation networks.