Developing thermal infrared de-ghost and multi-level nested conglutinated segmentation algorithm for detection of rice seed setting rate

Highlights

• Rice seed setting rate detection was achieved and compared with deep learning models.

• The detection performance of thermal infrared image was better than RGB image.

• Multi-layer nested conglutinated segmentation algorithm segmented cohesive grains.

• Thermal infrared de-ghost algorithm obtained the best detection accuracy.

Abstract

The seed setting rate (SSR) of rice is not only a key component of yield, but also an important parameter in rice phenotypic analysis. Fast and accurate detection of SSR is of great significance for yield prediction. The purpose of this research was to detect the SSR of rice quickly and automatically. A thermal infrared–visible light dual imaging system was built to obtain thermal infrared images and RGB images of rice grains. This paper proposed image registration method, thermal infrared de-ghost method and multi-layer nested conglutinated segmentation algorithm to detect SSR. Compared with the detection accuracy of three deep learning models (Faster RCNN 96.43%, SSD 81.84%, YOLO V3 96.75%) and image registration methods (80.83%), the highest SSR detection accuracy (97.66%) was achieved by fusing thermal infrared de-ghost and multi-layer nested conglutinated segmentation algorithm. This method has the advantages of simple structure, high efficiency and competitive results, and has great potential in detecting seed setting rate.

Introduction

Rice is one of the most important food crops in the world, especially in Asia (Bellis et al., 2022, Wang et al., 2021). The quality and number of grains per panicle has a critical impact on rice yield and precision agricultural management (Lu et al., 2022). Rice grains can be divided into full grains and empty grains according to their fullness. In agronomy, grains with grouting degree greater than one-third are full grains, and grains with grouting degree less than one-third are empty grains. Full grains are the real component of rice yield. The seed setting rate (SSR) is defined as the percentage of full grains in all rice grains per panicle. It is an important phenotypic parameter to characterize rice yield (Zhang, 2007). Li et al. (2013) had reported that a RING-type E3 ubiquitin ligase could regulate the SSR. Xiang et al. (2019) found a new rice gene that could regulate the SSR by facilitating rice fertilization. In these breeding studies, it is necessary to calculate the SSR of experimental samples. So rapid and accurate detection of SSR is of great significance for breeding, phenotypic measurement and yield prediction (Gong et al., 2018, Guo et al., 2021).

Traditional measurement methods mainly relied on manual identification of grain plumpness, but manual visual method is time-consuming and labor-intensive(Guo et al., 2021). With the rapid development of breeding technology, hundreds of new breeding materials need to be processed every day. Therefore, there is an urgent need to develop a fast and reliable SSR measurement technology to break through the bottleneck of traditional grain phenotype measurement methods (Jiang et al., 2022, Li et al., 2022).

At present, plant phenotypic measurement technology which helps to increase the efficiency of breeding and improved cultivation has been a hot research area. Many researches mainly focused on the quality detection and grain classification by morphological, color and texture characteristics. Paliwal et al. (2004) used the color camera to collect grain images and applied them to classify wheat varieties and to identify moldy or abnormal grains. Lin et al. (2018) proposed a machine vision system based on deep convolution neural network (DCNN) structure. Compared with traditional methods, this system could improve the classification accuracy of three different groups of rice images.

However, there is little difference between the external features of full and empty grains. Therefore, it is difficult to complete the recognition of two kinds of grains and detect the SSR by image processing methods based on color and texture features. So many scholars have developed methods based on X-ray imaging and computed tomography (CT) to study the internal quality of grains. Karunakaran et al. (2004) applied soft X-ray imaging technology to grain internal quality detection. Their team also used near-infrared camera and carbon dioxide sensor to detect internal quality of grain. But these devices and techniques could not distinguish between full and empty grains. The solution of Duan et al. (2011) was that the fusion between two images obtained based on X-ray and line array camera could more accurately identify full and empty grains, but the low efficiency and high cost of X-ray made it difficult to popularize. Kong and Chen (2021)proposed a feature extraction and three-dimensional recognition method based on mask region convolution neural network (Mask R-CNN) (He et al., 2017) for rice panicle CT images. Combining CT feature characterization with Mask R-CNN algorithm, feature extraction and classification were carried out for panicles and grains of each layer of CT sequence to calculate the SSR. The difficulty and high cost of CT image acquisition, this method was not practical (Zhang and Xia, 2021).

In recent years, thermal infrared technology has developed rapidly, and the thermal infrared sensor with gradually reduced cost has become more and more popular in the detection of agricultural products. According to the temperature difference, thermal infrared technology can detect the internal quality of products. External information of agricultural products can be detected based on RGB image. Fusing thermal infrared image (TII) and RGB image (RGBI) to detect seed setting rate is a solution. Our research was aimed at exploring an automatic and accurate method to detect SSR by developing thermal infrared de-ghost, image registration and multi-layer nested conglutinated segmentation algorithm. The main objectives were: (1) to build a thermal infrared–visible light dual imaging system, and to collect TIIs and RGBIs of rice grains; (2) to propose three kinds of methods (thermal infrared de-ghost, image registration and deep learning) to detect the SSR, and to compare the performance of different methods; (3) to evaluate detection performance of the models.