Accurate low-contrast defect detection has become a common bottleneck to further improve the performance of automated visual inspection (AVI) instruments. Inspired by visual crypsis, a novel concept of camouflaged defect has been proposed to assist surface defect detection, and then, a camouflaged defect detection network (CDDNet) was proposed. To be specific, a new inception dynamic texture enhanced module (IDTEM) was proposed to aggressively strengthen the indefinable boundaries and deceptive textures. To further explore spatial information over long distance, a lightweight recurrent decoupled fully connected attention (RDFCA) is designed with cost-effective computation. Finally, a new adaptive scale-equalizing pyramid convolution (ASEPC) was designed to achieve cross-scale feature fusion by exploiting the inter-layer feature correlation. The proposed CDDNet obtained competitive mean average precision (mAP) of 84.2%, 96.7%, and 67.1%, respectively, on three public datasets of NEU-DET, DAGM, and CAMO, when compared with state-of-the-arts.