Deep learning-based automatic recognition network of agricultural machinery images

Highlights

• Construction of image annotation datasets for agricultural machinery.

• A network called AMTNet was designed and trained for machinery recognition.

• We designed the C-DCGAN of agricultural machinery for dataset enhancement.

Abstract

Due to the massive amount of data generated by the mobile Internet and the development of large-scale computing devices and technologies, the deep learning algorithm has experienced a breakthrough in terms of image recognition technology. Traditional image recognition requires the complex extraction of image features, whereas deep learning technology can automatically learn image features through multi-layer nonlinear transformation, which is especially proficient at extracting complex global features. An image annotation dataset containing the images of seven types of machines and six types of abnormal images was constructed in this study from the large number of machine images in the agricultural machinery operation supervisory service system. To improve the Inception_v3 network, a network called AMTNet was designed and trained for automatic recognition of agricultural machinery images. Under the same experimental conditions, AMTNet achieved recognition accuracies of 97.83% and 100% on validation sets Top_1 and Top_5, respectively, demonstrating better performance than the classic networks ResNet_50 and Inception_v3. To further test the performance of AMTNet, 200 images of each of the 13 types of machine images were selected as test sets. The average area under the curve and F1-score of the network for image recognition of various machines reached 92% and 96%, respectively. According to the test results, AMTNet shows good robustness to illumination, environmental changes, and small area occlusion, which meets the practical application requirements of intelligent supervision over agricultural machinery operation.

Introduction

In recent years, China’s mode of agricultural production has undergone rapid transformation, with the degree of agricultural mechanization continuing to increase (Yang, 2011). In 2017, the Ministry of Agriculture issued the Thirteenth Five-Year Plan for the Development of Agricultural Mechanization in China (Ministry of Agriculture and Rural Affairs of the People's Republic of China, 2017), proposing a strategic goal of achieving a comprehensive mechanization level of crops of more than 70% by 2020 and setting up approximately 500 demonstration counties to take the lead in realizing the mechanization of agricultural production. As the level of agricultural mechanization continues to grow, the demand for agricultural machinery clustering and intelligent management has promoted the integration of new information technology and agricultural machinery technology represented by the Internet of Things (Shaonong et al., 2015, Kang et al., 2018). Beijing Engineering Research Center for Intelligent Agricultural Machinery developed an agricultural machinery operation supervision service system (Yin et al., 2018) that consists of a vehicle-mounted monitoring terminal, Global Navigation Satellite System (GNSS) positioning sensor, machine operation monitoring sensor, vehicle-mounted waterproof camera, and machine recognition sensor. The location, operation status, speed, and image of the agricultural machinery are uploaded to the central server through the General Packet Radio Service (GPRS) network so that the supervisory department can check the real-time trajectory, historical trajectory, area, images, and other regulatory indicators of agricultural machinery operation through a Web browser.

The recognition of agricultural machinery is a daily task performed by supervising personnel. It is also an important index that governments use to issue subsidies for the use of agricultural machinery. At present, the agricultural machinery operation supervision service system has provided data management services for hundreds of thousands of users. Due to the quantity of daily real-time transmission of image data and the inefficiency, cost, subjectivity, and error rate of manual recognition, developing automatic recognition systems for agricultural machinery is of great significance.

In many cases, traditional image processing technology is characterized by acquiring features of targets. There features include color, shape, and graining. Image processing systems then classify the images through methods such as artificial neural network (Roffman et al., 2018) and support vector machine (Thanh Noi and Kappas, 2018). Because of the great differences in color and brightness of images of different types of agricultural machinery and the great similarity between certain agricultural machinery, it is difficult and inefficient to extract image features using the traditional image recognition method. Therefore, there is a profound need for a simple, efficient, and accurate automatic agricultural machinery recognition method.

Due to the massive amount of data generated by the mobile Internet and the development of large-scale computing devices and technologies, the deep learning algorithm (LeCun et al., 2015) has seen a breakthrough in terms of image recognition technology. Instead of manually extracting features based on prior knowledge, the algorithm automatically learns image features through multi-layer nonlinear transformation under the data-driven design. The deep learning algorithm is especially proficient at extracting complex global features, with a strong robustness to the judgment of the transformation and rotation of objects.

Agricultural machinery recognition is quite different from flower recognition (Xia et al., 2017) and plant recognition (Lee et al., 2015). The main manifestations are as follows: (1) image data acquisition methods are different. Agricultural machinery images are mainly captured by the main vehicle terminal camera. Images are easily affected by illumination, temperature, humidity, wind, precipitation, and bumpy roads. Special preprocessing operations are needed for the collected agricultural machinery images; (2) the distribution structure of agricultural machinery image is more complex. A mature deep convolutional neural network can extract all the features of the image. However, the complex network structure, large number of parameters, and long training time are not conducive to the deployment and application of the network. Furthermore, there may be a certain degree of over-fitting, resulting in a slightly lower recognition accuracy (Dandan and Dongjian, 2019); (3) Image recognition technology based on deep learning requires a training dataset that covers a large number of annotated images. Current open-source datasets, such as cifar10, cifar100, and ImageNet, provide data regarding general objects, such as plants, cars, faces, and flowers, but there are no open-source data set of agricultural machinery. Therefore, it is necessary to perform studies on agricultural machinery recognition.

Currently, there are few studies on the recognition of agricultural machinery using convolutional neural networks, especially on the recognition of various types of agricultural machinery and abnormal images. Yang et al. used a convolutional neural network for subsoiler recognition (Yang et al., 2018), and its recognition rate reached 98.5%. However, due to the relatively simple processing of abnormal images, only a few types of agricultural machinery can be recognized by the network, and the accuracy tends to be less than 90%, which is insufficient for the requirements of practical applications of agricultural machinery operation supervision service systems.

In this study, an agricultural machinery image annotation dataset was constructed for 7 types of agricultural machinery images and 6 types of abnormal images. According to the actual needs of the supervisory system for automatic image recognition and the characteristics of agricultural machinery image, an AMTNet network was designed for the automatic recognition of agricultural machinery images. The feasibility and effectiveness of the AMTNet network were assessed by comparing it to ResNet_50 network and Inception_v3 network using the same validation set. To further test the performance of the AMTNet network, test sets were selected for verification experiments. The results showed the average values of AUC area under the curve and F1-score of AMTNet network to be 92% and 96% respectively, which does meet the requirements of practical applications of agricultural machinery operation supervision service system for the automatic recognition of agricultural machinery images.