Deep learning in olive pitting machines by computer vision

https://doi.org/10.1016/j.compag.2020.105304Get rights and content

Highlights

  • Testing of a neural network based on physical chips to classify olives.

  • The chipsets are able to process images with 11 × 11 pixels resolution and up to 16 × 16.

  • The system reaches to 300 olives/min, which is far below the 800–2500 olives/min.

  • The system offers an alternative to other conventional image classification systems.

  • The CM1K returns: empty pocket, normal, “boat” position and 255 for untrained cases.

Abstract

Olive pitting machines are characterized by the fact that their optimal functioning is based on an appropriate adjustment: selection of a feed plate adapted to the olive variety and its caliber, geometrical characteristics of the feed chain, etc. The first of these elements sets the optimal way for olives to enter the feed chain and, therefore, it prevents empty pockets or more than one olive to be placed in the same pocket. The second element sets the appropriate position for the olive to be pitted and prevents it to be pitted by a secondary axis.

The proposed study analyzes the appropriate placement of olives in the pockets of the feed chain by using the following items:

  • 1.

    A computer vision system with an external trigger, which is capable of taking a picture of every pocket passing in front of the camera.

  • 2.

    A classifying neural network that, appropriately trained, differentiates between four possible pocket cases: empty, normal, incorrectly de-stoned olive in any of its angles (when the olive is de-stoned transversally instead of longitudinally, also known as “boat”) and anomalous case (two olives in the same pocket, small parts of it or foreign elements, such as small branches or stones).

A preliminary analysis, carried out with the MATLAB Neural Network Toolbox, has enabled to test the viability of using a neural network to perform this type of classification.

The main objective of this paper is to illustrate the use of a physical chip with neural networks, NeuroMem CM1K (General Visions, 2016. CM1K), for sorting purposes.

Therefore, it is necessary to identify the minimum resolution required to classify the images of olives in olive pitting machines and their adequate position to be pitted considering an input vector of up to 256 bytes, which is the maximum dimension supported by NeuroMem CM1K.

As described before, a camera with an external trigger will be used for image capturing synchronized with the feed chain.

Given that the image classification speed must be higher than 15 Hz to be operatively convenient, the industrial feasibility of this system will be assessed in order to implement it in an olive pitting machine, the operating speed of which starts at a rate of 900 olives/minute.

The use of the physical chip NeuroMem CM1K, for its greater capacity and scalability, has been proven satisfactory and, therefore, it offers a great potential for sorting purposes. As stated in the obtained results contained in following pages, it has been possible to train, for the first time, an artificial neural network (ANN) implemented in a neuromorphic chip to classify the images of the olives in the feed chain of olive pitting machines. Consequently, it sets an alternative system in order to study possible cost, space and energy use reductions in contrast with traditional common computer systems or PLCs.

Introduction

Neural network is a widely used technology in the industry due to its successful way to classify or improve the production. Some examples are described below.

Parallel Neural Network Chips are used for fish inspection before filleting offshore (Menendez and Paillet, 2008). Each network chip system uses four neural network chips (accounting for 312 neurons) based on a natively parallel, hard-wired architecture that performs real-time learning and nonlinear classification (RBF).

Common tasks are related to fill bottles in factories. The process is quite simple. However, it may be necessary to use an intelligent device to inspect the process (Menendez and Paillet, 2013).

There are other similar papers related to fruits and vegetables:

  • Classification of apples using three layers of 9-6-3 neurons, 96.6% accuracy (Yang, 1993).

  • (Nagata and Qixin, 1998) developed a grading system for fruit and vegetables using neural network technologies, obtaining a high percentage of accuracy for strawberries and green peppers (94% to 98% and 89%, respectively).

  • Applied machine vision and ANN for modelling and controlling grape drying process. This paper presents a new method for predictive modelling of grape drying process for on-line monitoring and controlling of this process. (Behroozi Khazaei et al., 2013).

  • Olive Fruits Recognition Using Neural Networks. Olive fruit recognition is performed by analyzing RGB images taken from olive trees. (Gatica et al., 2013).

  • Identifying Olive (Olea europaea) Cultivars Using Artificial Neural Networks. Backpropagation neural networks (BPNNs) were used to distinguish among 10 olive (Olea europaea L.) cultivars, originating throughout the Mediterranean basin. (Mancuso and Nicese, 1999).

The combination of computer vision and neural networks is a way to perform tasks that are more complex:

  • A neural network chip is used for license plate recognition (Liu et al., 2011). The chip combines a video image-processing module with a neural network module by using equalized image processing algorithms and network classification algorithms.

  • Image recognition is simpler than image processing methods for face recognition, mainly due to the lack of a fixed pattern for comparison purposes (Sardar et al., 2011). Santu Sardar published a paper based on automated face recognition, which is a technique employed in a wide range of practical applications, including personnel access control or identification systems.

Regarding table olives, there are some techniques to classify them by using computer vision and neural network according to the papers below.

The book Computer Vision Technology for Food Quality Evaluation, 2nd Edition, (Sun, 2008) includes a specific chapter based on different technologies that analyze the quality of food. Chapters 11 to 13 deal with quality evaluations of apples, citrus fruits and strawberries, respectively.

Of particular interest to the problem, chapter 14 explains the classification and evaluation of table olives and describes how to classify them by color, shape or external defects made by insects.

A usual way to classify the olives is by using computer vision (Diaz, 2004). The paper analyzes the images captured by a camera connected to a PC. The image analyzed 66 olives per matrix. In this paper, the author used a Bayesian math model for pre-classification to perform this process. A neural network software was used with 15 sorting parameters and a hidden layer. The result was successful: the network was able to classify more than four types of olives. The results could be improved by using high-resolution images.

Other chipsets have been successfully used in several trials obtaining positive results:

Lohi is a chip manufactured by Intel. It integrates programmable synaptic learning rules. This chip can solve LASSO optimization problems. (Davies et al., 2018) presented an unambiguous example of spike-based computation using a range of different neuron models and some preliminary Lohi results.

Moran et al. (2018) presented a new work demonstrating, for the first time, spinal image segmentation using a deep learning network. It was implemented on a neuromorphic chip, an IBM TrueNorth Neurosynaptic System. The results compared to human-generated segmentations of spinal vertebrae and disks.

Moradi et al. (2018) presented a work which established a novel routing methodology to minimize memory requirements and latency, maximizing programming flexibility. The authors validated the proposed scheme in a multi-core neuromorphic processor chip prototype, DYNAP-SE.

Frenkel et al. (2019) presented a digital spiking neuromorphic processor. This chip obtains a minimum energy per synaptic operation. They demonstrated an efficient implementation of the spike-driven synaptic plasticity learning rule for high-density embedded online learning.

Google's Coral Dev (Fried, 2019) is a single-board computer based on the Raspberry Pi form factor, designed to run TensorFlow Lite models. The chip Edge TPU is implemented on this board and it is capable of up to 4 trillion operations per second (TOPS).

Lobachev et al. (2018) used the physical deployment model employed in the Intel Movidius VPU modules and Raspberry Pi microcontrollers to investigate and model the neural network integration. This network model utilizes a lower power approach and faster responses as well as it improves overall system efficiency.

Hubbard (2019) selected the Nvidia Jetson platform, specifically the Nvidia Jetson Nano, to build a neural network for detecting objects. It requires a very high-speed device with the machine learning runtime.

At the time of writing this paper, the chip selected was the NeuroMem CM1K. In 2019, the manufacturer Nepes IA in collaboration with General Vision launched a new powerful chip with 575 Neurons (NM500).

Kim (2019) proposed a system to control vehicle speed by recognizing the traffic information images marked on road with an Advanced Driver Assistance System, ADAS. This application used this newly developed neuromorphic artificial intelligent chip NM500.

The following article (General Visions, 2018 CogniPat SDK for MATLAB) proves the benefits of this chip under the MATLAB platform to learn and recognize extracted data vectors, signals and images.

There is another platform on the market to develop research projects: the NeuroMem USB Dongle (General Visions, 2019 NeuroMem USB Dongle) uses 4 NM500 chips packaged, accounting for 2304 neurons in total.

Section snippets

Neural network

A technology based on ANNs has been used due to its noticeable ability to obtain complicated and imprecise data, such as image analysis. Therefore, it can be used to deduce patterns and detect correlations which are hard to be seen by humans or other computational techniques.

The architecture of a neural network is composed of multiple elementary processors (Goodfellow et al., 2016). It is an adaptive system with an algorithm which can adjust its weight values in order to meet the performance

Results obtained using MATLAB neural network toolbox

The first one uses 1-byte depth greyscale images, 16 × 16 pixels ROI-scaled. For training purposes, a set of 9 images of empty pockets, 11 of “boat” de-stoned olives and 10 of normal olives has been used. Anomalous case images were not included since the purpose of the test was to prove the feasibility of the classification of these low-resolution images using a neural network. Fig. 10 shows olive images 16 × 16 pixels ROI-scaled.

The structure of the autoencoder of the neural network has been

Conclusions

Training and testing of a neural network based on a physical chip to classify olives in the feed chain of a pitting machine have been successfully completed. The use of the physical chip NeuroMem CM1K, for its greater capacity and scalability, has been proven satisfactory and, therefore, it offers a great potential for sorting purposes. The CM1K chip detected between 97.6% and 94.76% of cases correctly. Only four neurons have been used for classification purposes: one neuron for empty pockets,

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (40)

  • M.F. Moller

    A scaled conjugate gradient algorithm for fast supervised learning

    Neural Networks

    (1993)
  • Q. Yang

    Classification of apple surface features using machine vision and neural networks

    Comput. Electron. Agric.

    (1993)
  • Behroozi Khazaei Nasser, Tavakoli Teymour, Ghassemian Hassan, Hadi Khoshtaghaza Mohammad, Banakar Ahmad, 2013. Applied...
  • Davies Mike, Srinivasa Narayan, Lin Tsung-Han, Chinya Gautham, Cao Yongqiang, Choday Sri Harsha, Dimou Georgios, Joshi...
  • Demuth Howard B., Beale Mark H., De Jess Orlando, Hagan Martin T., 2014. Neural Network...
  • Diaz, R., 2004. Classification and Quality Evaluation of Table...
  • Frenkel Charlotte, Lefebvre Martin, Legat Jean-Didier, Bol David, 2019. A 0.086-mm2 12.7-pJ/SOP 64k-Synapse 256-Neuron...
  • Fried Limor, 2019. Making machine learning arduino compatible: A gaming handheld that runs neural networks -...
  • Gatica Gabriel, Bestb Stanley, Ceronic José, Lefranca Gastón, 2013. Olive fruits recognition using neural...
  • General Visión, 2019. NM500 Chip...
  • General Visions, 2016. CM1K Chip...
  • General Visions, 2017. Braincard...
  • General Visions, 2017. TM_CM1K_Hardware_Manual....
  • General Visions, 2018. CogniPat SDK for Matlab....
  • General Visions, 2019. NeuroMem USB dongle....
  • General Visions, 2019. TM_NeuroMem_Technology_Reference_Guide....
  • General Visions. DS_CM1K....
  • General Visions. TM_TestNeurons_SimpleScript....
  • Goodfellow Ian, Bengio Yoshua, Courville Aaron, 2016. Deep...
  • Halgamuge Saman K., Poechmueller Werner, Glesner Manfred, 1994. An alternative approach for generation of membership...
  • Cited by (7)

    • Computer vision-based high-quality tea automatic plucking robot using Delta parallel manipulator

      2021, Computers and Electronics in Agriculture
      Citation Excerpt :

      The real world information is acquired with the cameras and then is converted to the digital form by digital image processing. Computer vision has been widely applied for the recognition and detection of a variety of fruits and plants (Lázaro et al., 2020; Nguyen et al., 2016; Sengar et al., 2018; Barnea et al., 2016). Specific for the tea, Zhang et al. (2019) proposed a method for real-time monitoring of the optimum harvesting time for picking fresh tea leaves based on machine vision.

    • A smart system for the automatic evaluation of green olives visual quality in the field

      2020, Computers and Electronics in Agriculture
      Citation Excerpt :

      This is due to both the complexity of spot shapes and their colour, which that can be confused with that of certain stages of olive ripeness. Machine learning techniques may be applied to extract some olive parameters or grading with good results (Diaz et al., 2004) and could also be used in industrial applications (de Jódar Lázaro et al., 2020). There are several studies and reviews that have researched the technologies available for assessing fruit and vegetable quality (Arendse et al., 2018; Bhargava & Bansal 2018; Ortaç et al., 2016; Srivastava and Sadistap, 2018; Tsouvaltzis et al., 2020; Zhang et al., 2018), but also for olive quality (Navarro Soto et al., 2018) and specifically for table olives (Campus et al., 2018).

    View all citing articles on Scopus
    View full text