Towards in-field insect monitoring based on wingbeat signals: The importance of practice oriented validation strategies
Introduction
Insects play an important role in agriculture, impacting it in both positive and negative ways. Bees and other pollinating insects are essential to the agricultural economy since they support crop production, leading to increased yield and better quality. The production of many widely used crops like apples, coffee and tomatoes, relies on insect pollinators (Klein et al., 2007), bringing the global value of insect-based pollination to €153 billion (Gallai et al., 2009). Decline in pollinator populations could significantly affect future food production (Winfree, 2008). However, not all insects have a positive impact on agriculture. Pest insects are responsible for causing significant damage to crops and reducing quantity and quality. The increase in global trade has resulted in the spread of pest insects around the globe, challenging farmers to protect their crops against these new pests. An example of this evolution is the invasion of Drosophila suzukii in Europe from east Asia (Cini et al., 2012). According to a study focused on small fruit cultivation in the province of Trento (Northern Italy), the overall economic impact of Drosophila suzukii damage in 2011 was estimated at €3-4 million (Ioratti et al., 2011). In an attempt to minimize losses from pests, insecticides are used increasingly to control their populations. However, this creates serious negative effects on the environment and the public health (Wilson and Tisdell, 2001). In order to avoid the excessive insecticide use and have an objective overview of their fields’ health, farmers could gain valuable insights from monitoring specific insect populations so that they can detect declining pollinator activity or increased pest activity promptly.
Traditional monitoring techniques for pests or pollinator insects rely on the manual inspection of insect traps (Amjad Bashir et al., 2014, Woodcock et al., 2013), or flight cages and quadrats (Garratt et al., 2014, Ono et al., 2008, Yamaji and Ohsawa, 2016). Traps can lure target insects by using known attractants like color (Westphal et al., 2008), wavelength-specific illumination (van Langevelde et al., 2011), pheromones (Clare et al., 2000), or sounds (Walker, 1988). Flight cages and quadrats surround a controlled region in the field that trained observers visit and inspect. Maintenance and inspection of all these systems is a tedious and labor intensive procedure, especially for larger production areas. In the case of traps, a farmer or a worker is required to visit the field(s) periodically to identify and count the trapped insects or replace the attractants. Similarly for flight cages and quadrats, an observer usually spends hours determining seed visitation rates by pollinator insects. These are complex tasks that require specialized knowledge. Therefore, they are usually carried out by trained experts, which can make these procedures costly or inaccessible to the average farmer. Besides, extensive use of traps or cages can also negatively affect the active pollinator insect populations.
Limitations of manual monitoring systems can be addressed by the use of automation, which can make these processes more efficient, precise and economical (Jha et al., 2019). Automatic monitoring systems have the potential to map not only the distribution of various insect species present, but also the evolution of their activity in target regions. For instance, invasive pest species that arrive through global trade could be traced fast so that farmers can take appropriate action. Also, they could create prompt warnings for the farmers when critical levels of beneficial or pest insect activity are reached. This would allow them to have a better overview of the expected yield as well as the general health of their fields’ ecosystem (González et al., 2020). Moreover, these improvements could be made accessible to all farmers thanks to the low cost and adaptability of devices like Arduino or Raspberry Pi and the advancement of open source software (Muangprathub et al., 2019).
Currently, a series of (semi-)automatic monitoring technologies is already implemented by employing technological advancements in software and hardware. Most approaches try to enhance components of existing manual methods. For instance, while former attracting and trapping methods are still used, the identification and counting of insects is done by computational algorithms. In some cases, it is based on visual features of insects extracted from camera images inside traps. In the past, image-based attempts used “hand-crafted” features of insect images (Arbuckle et al., 2001, Solis-Sánchez et al., 2011, Weeks et al., 1999), whereas the use of Deep Learning (DL) algorithms like deep Convolutional Neural Networks (CNNs) is recently becoming more popular because they have been reported to outperform older methods (Kamilaris and Prenafeta-Boldú, 2018). Although these algorithms seem very promising when used with image data, they often still struggle to deal with problems like varying illumination of images, movement behavior and crowding of trapped insects, or out of focus cameras. Because of this, they are often applied to data collected under strictly controlled lab environments which are not representative for the situation in the field.
Other approaches to detect and identify insects are based on the analysis of bioacoustic signals of insects. Calling or courtship sound signals from different species are recorded using a microphone or similar audio recorders and they are analyzed in the time (Chesmore, 2001, Zamanian and Pourghassem, 2017) or the frequency domain (Potamitis et al., 2006, Zamanian and Pourghassem, 2017). Zamanian and Pourghassem (2017) reported a high accuracy of 99.13% when classifying among cicada species using their calling songs and a combination of a genetic algorithm and multi-layer perceptron for classification. However, these methods are challenging to implement in the field since they can be severely affected by environmental noise and weather conditions (Chesmore, 2001).
Another method for insect classification is based on the analysis of their wingbeat signal. Originally investigated by Chadwick (1939), the wingbeat frequency can be measured by recording the sound with a microphone (Mankin, 1994, Raman et al., 2007) or using an optical system (Richards, 1955, Unwin and Ellington, 1979, Moore and Miller, 2002, Li et al., 2005). Optical systems have advantages over microphones since 1) they are invariant to wind noise or ambient sounds and 2) they can record insect flight from a large distance, even for small insects (Unwin and Ellington, 1979). Using optical sensor data, Chen et al. (2014) achieved 79.44% classification accuracy for 10 different insect classes (of flies and mosquitoes) with a Bayesian classifier. van Roy et al. (2014) achieved a classification rate of 88.37% for discriminating between two bumblebee species (B. terrestris and B. ignitus) using decision trees. One disadvantage of these optical sensors was that optical measurements could still be influenced by flickering artificial light which made them hard to adopt for indoor use (e.g. in a greenhouse). This was solved by the application of a light (de-)modulation technique (Potamitis and Rigakis, 2015, Potamitis and Rigakis, 2016). The sensors developed in this way apply modulation to the emitted light while the demodulation is employed at high frequencies (Potamitis and Rigakis, 2015). This makes them robust against ambient light noise and, thus, good candidates for an insect monitoring device that is deployed in the field. Fanioudakis et al. (2018) used this sensor to classify six different mosquito species (Aedes aegypti, Aedes Albopictus, Anopheles arabiensis, Anopheles gambiae, Culex pipiens, Culex quinquefasciatus) of three different geni which forms a relevant challenge for several insect monitoring applications, due to their high anatomic similarity. With deep learning methods they obtained classification accuracies of up to 96% using the CNN “DenseNet121” with wingbeat spectrograms as input. Similar to most recently published studies that use machine and deep learning techniques to model insect data, they did not employ any strict validation for their models and split their data randomly (Silveira and Monteiro, 2009, Wang et al., 2012, Wang et al., xxxx, Espinoza et al., 2016, Potamitis et al., 2017, Potamitis et al., 2018, Fanioudakis et al., 2018, Sun et al., 2018, Gutierrez et al., 2019, Ismail Fawaz et al., 2019, Lu et al., 2019, Thenmozhi and Srinivasulu Reddy, 2019). It is hypothesized here that these results may give an over-optimistic view on the accuracy which can be obtained in practice, as signals from the same insect may both have been included in the validation set and the test set.
Therefore, the aim of this study was to critically examine the robustness of a wide range of established algorithms from the fields of Machine Learning and Deep learning using data that was collected by noise-robust optical sensors with the aim of revealing their “true” performance in field conditions. Both classical, and state-of-the-art models were chosen to fit the 3 different types of wingbeat data that were modelled (raw wingbeat signal, power spectral density, and spectrograms).
To benchmark the algorithms used in this study, we use a well-controlled validation procedure instead of the random one which is typically used to get a better idea on the “real” in-field performance that can be expected. In this way, we want to support the creation of practical and precise monitoring devices for flying insects.
Section snippets
Sensor design
The sensors used to collect the data consist of 3 core components: a sensing head for recording wingbeats, a microcontroller device for basic pre-processing of the signals and an SD storage card where wingbeat signals are saved. The sensing head consists of a light emitting and a light capturing board. It retrieves the light intensity variation imprinted on the light-capturing board as an insect flies through the gap between the emitter and receiver, occluding the light with its body and wings.
Results and discussion
The histograms of the main wingbeat frequencies for all mosquito species (Fig. 5) show that there is overlap of the main wingbeat frequencies and their harmonics within and across genera. This makes it difficult to classify insect species using their main wingbeat frequencies alone (Chen et al., 2014, Genoud et al., 2018). Besides, mosquitoes are highly sexually dimorphic and thus, the wingbeat frequencies produced by the two sexes differ (Genoud et al., 2018), which is also illustrated in Fig.
Conclusions
In this paper, we benchmarked intelligent algorithms from the fields of Machine Learning and Deep Learning on wingbeat data from 6 mosquito species. The resulting models claimed very high accuracies when they were tested in a random data splitting setting.
The obtained classification accuracies reduced substantially when the models were trained and tested based on distant collection date ranges, which are relevant for the practical application of insect detection in the field. Hence, it is
CRediT authorship contribution statement
Ioannis Kalfas: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization. Bart De Ketelaere: Conceptualization, Methodology, Resources, Writing - review & editing, Project administration, Funding acquisition, Supervision. Wouter Saeys: Conceptualization, Methodology, Resources, Writing - review & editing, Project administration, Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
We would like to thank the MeBioS technical staff which helped materialize many of the ideas for this project, more specifically Frank Mathijs and Niels Wouters. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. This research is conducted with the financial support of VLAIO (Flanders Innovation & Entrepreneurship) (project HBC.2016.0795) and the Horizon 2020
References (64)
- et al.
Combination of image processing and artificial neural networks as a novel approach for the identification of Bemisia tabaci and Frankliniella occidentalis on sticky traps in greenhouse agriculture
Comput. Electron. Agric.
(2016) - et al.
The identity of crop pollinators helps target conservation for improved ecosystem services
Biol. Conserv.
(2014) - et al.
Generative adversarial network based image augmentation for insect pest classification enhancement
IFAC-PapersOnLine
(2019) - et al.
Automatic recognition and measurement of butterfly eyespot patterns
BioSystems
(2009) - et al.
Automatic in-trap pest detection using deep learning for pheromone-based Dendroctonus valens monitoring
Biosyst. Eng.
(2018) - et al.
Optical identification of bumblebee species: Effect of morphology on wingbeat frequency
Comput. Electron. Agric.
(2014) - et al.
A new automatic identification system of insect images at the order level
Knowledge-Based Syst.
(2012) - et al.
Crop flower visitation by honeybees, bumblebees and solitary bees: Behavioural differences and diversity responses to landscape
Agric. Ecosyst. Environ.
(2013) - et al.
Effectiveness of sticky traps in monitoring insects
J. Environ. Agric. Sci.
(2014) - et al.
Biodiversity informatics in action: identification and monitoring of bee species using ABIS
EnviroInfo
(2001)
Mosquito (Aedes aegypti) flight tones: Frequency, harmonicity, spherical spreading, and phase relationships
J. Acoust. Soc. Am.
Optimal division of data for neural network models in water resources applications
Water Resour. Res.
A Simple stroboscopic method for the study of insect flight
Psyche (New York)
XGBoost: A scalable tree boosting system
Flying insect classification with inexpensive sensors
J. Insect Behav.
Application of time domain signal coding and artificial neural networks to passive acoustical identification of animals
Appl. Acoust.
A review of the invasion of Drosophila suzukii in Europe and a draft research agenda for integrated pest management
Bull. Insectology.
Pheromone trap colour determines catch of nontarget insects
New Zeal. Plant Prot.
Economic valuation of the vulnerability of world agriculture confronted with pollinator decline
Ecol. Econ.
Optical remote sensing for monitoring flying mosquitoes, gender identification and discussion on species identification
Appl. Phys. B Lasers Opt.
Distribution of ecosystem services within oilseed rape fields: Effects of field defects on pest and weed seed predation rates
Agric. Ecosyst. Environ.
A benchmarking of learning strategies for pest detection and identification on tomato plants for autonomous scouting robots using internal databases
J. Sensors
A comprehensive foundation
Neural Networks
Drosophila suzukii, Matsumura, una nuova specie invasiva dannosa alle colture di piccoli frutti
I Georg.
Deep learning for time series classification: a review
Data Min. Knowl. Discov.
A comprehensive review on automation in agriculture using artificial intelligence
Artif. Intell. Agric.
Deep learning in agriculture: A survey
Comput. Electron. Agric.
The importance of choosing the right validation strategy in inverse modelling
J. Near Infrared Spectrosc.
Cited by (5)
An introduction to artificial intelligence in machine vision for postharvest detection of disorders in horticultural products
2023, Postharvest Biology and TechnologyTowards automatic insect monitoring on witloof chicory fields using sticky plate image analysis
2023, Ecological InformaticsOptical Detection of Winged Insects Utilizing Wireless Sensor Network with Ant Colony Optimizer
2023, 2023 8th International Conference on Business and Industrial Research, ICBIR 2023 - ProceedingsOptical Identification of Fruitfly Species Based on Their Wingbeats Using Convolutional Neural Networks
2022, Frontiers in Plant ScienceA fresh look at computer vision for industrial quality control
2022, Quality Engineering