Towards in-field insect monitoring based on wingbeat signals: The importance of practice oriented validation strategies

https://doi.org/10.1016/j.compag.2020.105849Get rights and content

Highlights

  • Successful classification of flying insects from their optical signatures using machine learning and deep learning.

  • Practice oriented validation is key in developing insect classification models.

  • Random data splitting can lead to over-optimistic classification performance estimates.

  • Classification at the insect genus level is a more feasible option than on the species level.

Introduction

Insects play an important role in agriculture, impacting it in both positive and negative ways. Bees and other pollinating insects are essential to the agricultural economy since they support crop production, leading to increased yield and better quality. The production of many widely used crops like apples, coffee and tomatoes, relies on insect pollinators (Klein et al., 2007), bringing the global value of insect-based pollination to €153 billion (Gallai et al., 2009). Decline in pollinator populations could significantly affect future food production (Winfree, 2008). However, not all insects have a positive impact on agriculture. Pest insects are responsible for causing significant damage to crops and reducing quantity and quality. The increase in global trade has resulted in the spread of pest insects around the globe, challenging farmers to protect their crops against these new pests. An example of this evolution is the invasion of Drosophila suzukii in Europe from east Asia (Cini et al., 2012). According to a study focused on small fruit cultivation in the province of Trento (Northern Italy), the overall economic impact of Drosophila suzukii damage in 2011 was estimated at €3-4 million (Ioratti et al., 2011). In an attempt to minimize losses from pests, insecticides are used increasingly to control their populations. However, this creates serious negative effects on the environment and the public health (Wilson and Tisdell, 2001). In order to avoid the excessive insecticide use and have an objective overview of their fields’ health, farmers could gain valuable insights from monitoring specific insect populations so that they can detect declining pollinator activity or increased pest activity promptly.

Traditional monitoring techniques for pests or pollinator insects rely on the manual inspection of insect traps (Amjad Bashir et al., 2014, Woodcock et al., 2013), or flight cages and quadrats (Garratt et al., 2014, Ono et al., 2008, Yamaji and Ohsawa, 2016). Traps can lure target insects by using known attractants like color (Westphal et al., 2008), wavelength-specific illumination (van Langevelde et al., 2011), pheromones (Clare et al., 2000), or sounds (Walker, 1988). Flight cages and quadrats surround a controlled region in the field that trained observers visit and inspect. Maintenance and inspection of all these systems is a tedious and labor intensive procedure, especially for larger production areas. In the case of traps, a farmer or a worker is required to visit the field(s) periodically to identify and count the trapped insects or replace the attractants. Similarly for flight cages and quadrats, an observer usually spends hours determining seed visitation rates by pollinator insects. These are complex tasks that require specialized knowledge. Therefore, they are usually carried out by trained experts, which can make these procedures costly or inaccessible to the average farmer. Besides, extensive use of traps or cages can also negatively affect the active pollinator insect populations.

Limitations of manual monitoring systems can be addressed by the use of automation, which can make these processes more efficient, precise and economical (Jha et al., 2019). Automatic monitoring systems have the potential to map not only the distribution of various insect species present, but also the evolution of their activity in target regions. For instance, invasive pest species that arrive through global trade could be traced fast so that farmers can take appropriate action. Also, they could create prompt warnings for the farmers when critical levels of beneficial or pest insect activity are reached. This would allow them to have a better overview of the expected yield as well as the general health of their fields’ ecosystem (González et al., 2020). Moreover, these improvements could be made accessible to all farmers thanks to the low cost and adaptability of devices like Arduino or Raspberry Pi and the advancement of open source software (Muangprathub et al., 2019).

Currently, a series of (semi-)automatic monitoring technologies is already implemented by employing technological advancements in software and hardware. Most approaches try to enhance components of existing manual methods. For instance, while former attracting and trapping methods are still used, the identification and counting of insects is done by computational algorithms. In some cases, it is based on visual features of insects extracted from camera images inside traps. In the past, image-based attempts used “hand-crafted” features of insect images (Arbuckle et al., 2001, Solis-Sánchez et al., 2011, Weeks et al., 1999), whereas the use of Deep Learning (DL) algorithms like deep Convolutional Neural Networks (CNNs) is recently becoming more popular because they have been reported to outperform older methods (Kamilaris and Prenafeta-Boldú, 2018). Although these algorithms seem very promising when used with image data, they often still struggle to deal with problems like varying illumination of images, movement behavior and crowding of trapped insects, or out of focus cameras. Because of this, they are often applied to data collected under strictly controlled lab environments which are not representative for the situation in the field.

Other approaches to detect and identify insects are based on the analysis of bioacoustic signals of insects. Calling or courtship sound signals from different species are recorded using a microphone or similar audio recorders and they are analyzed in the time (Chesmore, 2001, Zamanian and Pourghassem, 2017) or the frequency domain (Potamitis et al., 2006, Zamanian and Pourghassem, 2017). Zamanian and Pourghassem (2017) reported a high accuracy of 99.13% when classifying among cicada species using their calling songs and a combination of a genetic algorithm and multi-layer perceptron for classification. However, these methods are challenging to implement in the field since they can be severely affected by environmental noise and weather conditions (Chesmore, 2001).

Another method for insect classification is based on the analysis of their wingbeat signal. Originally investigated by Chadwick (1939), the wingbeat frequency can be measured by recording the sound with a microphone (Mankin, 1994, Raman et al., 2007) or using an optical system (Richards, 1955, Unwin and Ellington, 1979, Moore and Miller, 2002, Li et al., 2005). Optical systems have advantages over microphones since 1) they are invariant to wind noise or ambient sounds and 2) they can record insect flight from a large distance, even for small insects (Unwin and Ellington, 1979). Using optical sensor data, Chen et al. (2014) achieved 79.44% classification accuracy for 10 different insect classes (of flies and mosquitoes) with a Bayesian classifier. van Roy et al. (2014) achieved a classification rate of 88.37% for discriminating between two bumblebee species (B. terrestris and B. ignitus) using decision trees. One disadvantage of these optical sensors was that optical measurements could still be influenced by flickering artificial light which made them hard to adopt for indoor use (e.g. in a greenhouse). This was solved by the application of a light (de-)modulation technique (Potamitis and Rigakis, 2015, Potamitis and Rigakis, 2016). The sensors developed in this way apply modulation to the emitted light while the demodulation is employed at high frequencies (Potamitis and Rigakis, 2015). This makes them robust against ambient light noise and, thus, good candidates for an insect monitoring device that is deployed in the field. Fanioudakis et al. (2018) used this sensor to classify six different mosquito species (Aedes aegypti, Aedes Albopictus, Anopheles arabiensis, Anopheles gambiae, Culex pipiens, Culex quinquefasciatus) of three different geni which forms a relevant challenge for several insect monitoring applications, due to their high anatomic similarity. With deep learning methods they obtained classification accuracies of up to 96% using the CNN “DenseNet121” with wingbeat spectrograms as input. Similar to most recently published studies that use machine and deep learning techniques to model insect data, they did not employ any strict validation for their models and split their data randomly (Silveira and Monteiro, 2009, Wang et al., 2012, Wang et al., xxxx, Espinoza et al., 2016, Potamitis et al., 2017, Potamitis et al., 2018, Fanioudakis et al., 2018, Sun et al., 2018, Gutierrez et al., 2019, Ismail Fawaz et al., 2019, Lu et al., 2019, Thenmozhi and Srinivasulu Reddy, 2019). It is hypothesized here that these results may give an over-optimistic view on the accuracy which can be obtained in practice, as signals from the same insect may both have been included in the validation set and the test set.

Therefore, the aim of this study was to critically examine the robustness of a wide range of established algorithms from the fields of Machine Learning and Deep learning using data that was collected by noise-robust optical sensors with the aim of revealing their “true” performance in field conditions. Both classical, and state-of-the-art models were chosen to fit the 3 different types of wingbeat data that were modelled (raw wingbeat signal, power spectral density, and spectrograms).

To benchmark the algorithms used in this study, we use a well-controlled validation procedure instead of the random one which is typically used to get a better idea on the “real” in-field performance that can be expected. In this way, we want to support the creation of practical and precise monitoring devices for flying insects.

Section snippets

Sensor design

The sensors used to collect the data consist of 3 core components: a sensing head for recording wingbeats, a microcontroller device for basic pre-processing of the signals and an SD storage card where wingbeat signals are saved. The sensing head consists of a light emitting and a light capturing board. It retrieves the light intensity variation imprinted on the light-capturing board as an insect flies through the gap between the emitter and receiver, occluding the light with its body and wings.

Results and discussion

The histograms of the main wingbeat frequencies for all mosquito species (Fig. 5) show that there is overlap of the main wingbeat frequencies and their harmonics within and across genera. This makes it difficult to classify insect species using their main wingbeat frequencies alone (Chen et al., 2014, Genoud et al., 2018). Besides, mosquitoes are highly sexually dimorphic and thus, the wingbeat frequencies produced by the two sexes differ (Genoud et al., 2018), which is also illustrated in Fig.

Conclusions

In this paper, we benchmarked intelligent algorithms from the fields of Machine Learning and Deep Learning on wingbeat data from 6 mosquito species. The resulting models claimed very high accuracies when they were tested in a random data splitting setting.

The obtained classification accuracies reduced substantially when the models were trained and tested based on distant collection date ranges, which are relevant for the practical application of insect detection in the field. Hence, it is

CRediT authorship contribution statement

Ioannis Kalfas: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization. Bart De Ketelaere: Conceptualization, Methodology, Resources, Writing - review & editing, Project administration, Funding acquisition, Supervision. Wouter Saeys: Conceptualization, Methodology, Resources, Writing - review & editing, Project administration, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

We would like to thank the MeBioS technical staff which helped materialize many of the ideas for this project, more specifically Frank Mathijs and Niels Wouters. The resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government. This research is conducted with the financial support of VLAIO (Flanders Innovation & Entrepreneurship) (project HBC.2016.0795) and the Horizon 2020

First page preview

First page preview
Click to open first page preview

References (64)

  • B.J. Arthur et al.

    Mosquito (Aedes aegypti) flight tones: Frequency, harmonicity, spherical spreading, and phase relationships

    J. Acoust. Soc. Am.

    (2014)
  • G.J. Bowden et al.

    Optimal division of data for neural network models in water resources applications

    Water Resour. Res.

    (2002)
  • Breiman, L., 2001. Random forests. Mach. Learn....
  • L.E. Chadwick

    A Simple stroboscopic method for the study of insect flight

    Psyche (New York)

    (1939)
  • T. Chen et al.

    XGBoost: A scalable tree boosting system

  • Y. Chen et al.

    Flying insect classification with inexpensive sensors

    J. Insect Behav.

    (2014)
  • E.D. Chesmore

    Application of time domain signal coding and artificial neural networks to passive acoustical identification of animals

    Appl. Acoust.

    (2001)
  • A. Cini et al.

    A review of the invasion of Drosophila suzukii in Europe and a draft research agenda for integrated pest management

    Bull. Insectology.

    (2012)
  • G. Clare et al.

    Pheromone trap colour determines catch of nontarget insects

    New Zeal. Plant Prot.

    (2000)
  • Fanioudakis, E., Geismar, M., Potamitis, I., 2018. Mosquito wingbeat analysis and classification using deep learning....
  • Fawaz, H.I., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D.F., Weber, J., Webb, G.I., Idoumghar, L., Muller,...
  • N. Gallai et al.

    Economic valuation of the vulnerability of world agriculture confronted with pollinator decline

    Ecol. Econ.

    (2009)
  • A.P. Genoud et al.

    Optical remote sensing for monitoring flying mosquitoes, gender identification and discussion on species identification

    Appl. Phys. B Lasers Opt.

    (2018)
  • E. González et al.

    Distribution of ecosystem services within oilseed rape fields: Effects of field defects on pest and weed seed predation rates

    Agric. Ecosyst. Environ.

    (2020)
  • A. Gutierrez et al.

    A benchmarking of learning strategies for pest detection and identification on tomato plants for autonomous scouting robots using internal databases

    J. Sensors

    (2019)
  • S. Haykin et al.

    A comprehensive foundation

    Neural Networks

    (2004)
  • Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected convolutional networks, in:...
  • C. Ioratti et al.

    Drosophila suzukii, Matsumura, una nuova specie invasiva dannosa alle colture di piccoli frutti

    I Georg.

    (2011)
  • H. Ismail Fawaz et al.

    Deep learning for time series classification: a review

    Data Min. Knowl. Discov.

    (2019)
  • K. Jha et al.

    A comprehensive review on automation in agriculture using artificial intelligence

    Artif. Intell. Agric.

    (2019)
  • A. Kamilaris et al.

    Deep learning in agriculture: A survey

    Comput. Electron. Agric.

    (2018)
  • B.J. Kemps et al.

    The importance of choosing the right validation strategy in inverse modelling

    J. Near Infrared Spectrosc.

    (2010)
  • Cited by (5)

    • Optical Detection of Winged Insects Utilizing Wireless Sensor Network with Ant Colony Optimizer

      2023, 2023 8th International Conference on Business and Industrial Research, ICBIR 2023 - Proceedings
    View full text