Illegal construction should be detected as early as possible as it can damage the environment and economy. However, the existing methods for detecting illegal construction can be improved in terms of their detection cycles, accuracy, and speed. Moreover, there are relatively few valuable real-world image datasets for detecting illegal construction. To address these issues, a high-precision real-time detection model named YEMNet and a new large-scale dataset for detection of illegal construction objects (ICOS) are proposed herein. Our YEMNet is based on the You Only Look Once v3 object detection model; this model adopts a lightweight convolutional neural network called “EfficientNet” as the backbone for feature extraction. Then, YEMNet employs a new activation function Mish outside the backbone to achieve efficient optimization and strong generalization, thereby improving the recognition accuracy for ICOS in complicated scenes. Our proposed dataset comprises 15 categories and 13,701 photographs of ICOS captured under different conditions concerning weather, lighting, and natural scenes. Extensive experiments on the proposed dataset show that YEMNet achieves a mean average precision of 91.41% with fewer parameters, thereby outperforming state-of-the-art object detectors. Our dataset and code are available at https://github.com/king-king-king/ICOS-Dataset.