Evolving Developmental Programs That Build Neural Networks for Solving Multiple Problems

Abstract

A developmental model of an artificial neuron is presented. In this model, a pair of neural developmental programs develop an entire artificial neural network of arbitrary size. The pair of neural chromosomes are evolved using Cartesian Genetic Programming. During development, neurons and their connections can move, change, die or be created. We show that this two-chromosome genotype can be evolved to develop into a single neural network from which multiple conventional artificial neural networks can be extracted. The extracted conventional ANNs share some neurons across tasks. We have evaluated the performance of this method on three standard classification problems: cancer, diabetes and the glass datasets. The evolved pair of neuron programs can generate artificial neural networks that perform reasonably well on all three benchmark problems simultaneously. It appears to be the first attempt to solve multiple standard classification problems using a developmental approach.

Appendices

Appendix: Detailed Algorithms

8.1.1 Developing the Brain and Evaluating the Fitness

The detailed algorithm for developing the brain and assessing its fitness is shown in Algorithm 1 There are two stages to development. The first (which we refer to as ‘pre’) occurs prior to a learning epoch loop (lines 3–6). While the second phase (referred to as ‘while’) occurs inside a learning epoch loop (lines 9–12).

Lines 13–22 are concerned with calculating fitness. For each computational problem an ANN is extracted from the underlying brain. This is carried by a function ExtractANN(problem, OutputAddress) which is detailed in Algorithm 9. This function extracts a feedforward ANN corresponding to each computational problem (this is stored in a phenotype which we do not detail here). The array OutputAddress stores the addresses of the output neurons associated with the computational problem. It is used together with the phenotype to extract the network of neurons that are required for the computational problem. Then the input data is supplied and the outputs of the ANN calculated. The class of a data instance is determined by the largest output. The learning loop (lines 8–29) develops the brain and exits if the fitness value (in this case classification accuracy) reduces (lines 23–27 in Algorithm 1). One can think of the ‘pre’ development phase as growing a neural network prior to training. The ‘while’ phase is a period of development within the learning phase. N _ep denotes the user-defined number of learning epochs. N _p represents the number of problems in the suite of problems being solved. N _ex(p) denotes the number of examples for each problem. A is the accuracy of prediction for a single training instance. F is the fitness over all examples. TF is the accumulated fitness over all problems. Fitness is normalised (lines 20 and 22).

Updating the Brain

Algorithm 2 shows the update brain process. This algorithm is run at each developmental step. It runs the soma and dendrite programs for each neuron and from the previously existing brain creates a new version (NewBrain) which eventually overwrites the previous brain at the last step (lines 52–53).

Algorithm 2 starts by analyzing the brain to determine the addresses and numbers of non-output and output neurons (lines 3–11). Then the non-output neurons are processed. The evolved soma program is executed and it returns a neuron with updated values for the neuron position, health and bias. These are stored in the variable UpdatedNeurV ars.

If the user-defined option to disallow non-output neuron movement is chosen then the updated neuron position is reset to that before the soma program is run (lines 16–17). Next the evolved dendrite programs are executed in all dendrites. The algorithmic details are given in Algorithm 6 (See Sect. 8.3.5).

Algorithm 1 Develop network and evaluate fitness

The neuron health is compared with the user-defined neuron death threshold NH _death and if the health exceeds the threshold the neuron survives (see lines 22–28). At this stage it is possible that the neuron has been given a position that is identical to one of the neurons in the developing brain (NewBrain) so one needs a mechanism for preventing this. This is accomplished by Algorithm 3 (Lines 19 and 46). It checks whether a collision has occurred and if so an increment MN _inc is added to the position and then it is bound to the interval [−1, 1]. In line 23 the updated neuron is written into NewBrain. A check is made in line 25 to see if the allowed number of neurons has been reached, if so the non output neuron update loop (lines 12–38) is exited and the output neuron section starts (lines 39–51). If the limit on numbers of neurons has not been reached, the updated neuron may replicate depending on whether its health is above the user-defined threshold, NH _health (line 29). The position of the new born neuron is immediately incremented by MN _inc so that it does not collide with its parent (line 30). However, its position needs to be checked also to see if it collides with any other neuron, in which case its position is incremented again until a position is found that causes no collision. This is done in the function IfCollision.

In CreateNewNeuron (see line 32) the bias, the incremented position and dendrites of the parent neuron are copied into the child neuron. However, the new neuron is given a health of 1.0 (the maximum value). The algorithm examines the non-output neurons (lines 39–51) and again is terminated if the allowed number of neurons is exceeded. The steps are similar to those carried out with non-output neurons, except that output neurons can not either die or replicate as their number is fixed by the number of outputs required by the computational problem being solved.

The details of the neuron collision avoidance mechanism is shown in Algorithm 3.

Algorithm 2 Update brain

Algorithm 3 Move neuron if it collides with another

Running the Soma

The UpdateBrain program calls the RunSoma program (Algorithm 4) to determine how the soma changes in each developmental step. The seven soma program inputs comprising the neuron health, position and bias, the averaged position, weight and health of the neuron’s dendrites and the problem type are supplied to the CGP encoded soma program (line 12). The array ProblemTypeInputs stores NumProblems+1 constants equally spaced between − 1 and 1. These are used to allow output neurons to know what computational problem they belong to.

The soma program has three outputs relating to the position, health and bias of the neuron. These are used to update the neuron (line 13).

Algorithm 4 RunSoma(Neuron)

Changing the Neuron Variables

The UpdateNeuron Algorithm 5 updates the neuron properties of health, position and bias according to three user-chosen options defined by a variable Incr_opt. If this is zero, then the soma program outputs determine directly the updated values of the soma’s health, position and bias. If Incr_opt is one or two, the updated values of the soma are changed from the parent neuron’s values in an incremental way. This is either a linear or nonlinear increment or decrement depending on whether the soma program’s outputs are greater than or less than or equal to zero (lines 8–16). The magnitudes of the increments is defined by the user-defined constants: δ _sh, δ _sp, δ _sb and sigmoid slope parameter, α (see Table 8.1).

Algorithm 5 Neuron update function

The increment methods described in Algorithm 5 change neural variables, so action needs to be taken to force the variables to strictly lie in the interval [−1, 1]. We call this ‘bounding’ (lines 34–36).This is accomplished using a hyperbolic tangent function.

Running All Dendrite Programs and Building a New Neuron

Algorithm 6 Run the evolved dendrite program in all dendrites

Algorithm 6 takes an existing neuron and creates a new neuron using the updated soma variables, position, health and bias which are stored in UpdateNeurV ars (from Algorithm 4) and the updated dendrites which result from running the dendrite program in all the dendrites. Initially (line 3–5), the updated soma variables are written into the updated neuron. The number of dendrites in the updated neuron is set to zero. In lines 8–11, the health of the non-updated neuron is examined and if it is above the dendrite health threshold for birth, a new dendrite is generated and the updated neuron gains a dendrite. If so, the neuron gains a dendrite created by a function GenerateDendrite(). This assigns a weight, health and position to the new dendrite. The weight and health is set to one and the position set to half the parent neuron position. These choices appeared to give good results.

Algorithm 7 Change dendrites according to the evolved dendrite program

Lines 12–33 are concerned with processing the dendrite program in all the dendrites of the non-updated neuron and updating the dendrites. If the updated dendrite has a health above its death threshold then it survives and gets written into the updated neuron (lines 22–28). Updated dendrites do not get written into the updated neuron if it already has the maximum allowed number of dendrites (line 25–27). In lines 30–33 a check is made as to whether the updated neuron has no dendrites. If this is so, it is given one of the dendrites of the non-updated neuron. Finally, the updated neuron is returned to the calling function.

Algorithm 6 calls the function RunDendrite (line 21). This function is detailed in Algorithm 7. It changes the dendrites of a neuron according to the evolved dendrite program. It begins by assigning the dendrites health, position and weight to the parent dendrite variables. It writes the dendrite program outputs to the internal variables health, weight and position. Then in lines 8–16 it defines the possible increments in health, weight and position that will be used to increment or decrement the parent variables according to the user defined incremental options (linear or non-linear). In lines 17–33 it respectively carries out the increments or decrements of the parent dendrite variables according whether the corresponding dendrite program outputs are greater than or less than or equal to zero. After this it bounds those variables. Finally, in lines 37–44 it updates the dendrites health, weight and position provided the adjusted health is above the dendrite death threshold (in other words it survives). Note that if Incr _opt = 0 then there is no incremental adjustment and the health, weight and position of the dendrites are just bounded (lines 34–36).

Algorithm 2 uses a function CreateNewNeuron to create a new neuron if the neuron health is above a threshold. This function is described in Algorithm 8. It makes the new born neuron the same as the parent (note, its position will be adjusted by the collision avoidance algorithm) except that it is given a health of one. Experiments suggested that this gave better results.

Algorithm 8 Create new neuron from parent neuron

Extracting Conventional ANNs from the Evolved Brain

In Algorithm 1, a conventional feed-forward ANN is extracted from the underlying collection of neurons (line 15). The algorithm for doing this is shown in Algorithm 9. Firstly, this algorithm determines the number of inputs to the ANN (line 5). Since inputs are shared across problems the number of inputs is set to be the maximum number of inputs that occur in the computational problem suite. If an individual problem has less inputs than this maximum, the extra inputs are set to 0.0. The brain array is sorted by position. The algorithm then examines all neurons (line 7) and calculates the number of non-output neurons and output neurons and stores the neuron data in arrays NonOutputNeurons and OutputNeurons. It also calculates their addresses in the brain array.

Algorithm 9 The extraction of neural networks from the underlying brain

Algorithm 10 Find which input or neuron a dendrite is closest to

The next phase is to go through all dendrites of the non-output neurons to determine which inputs or neurons they connect to (lines 19–33). The evolved neuron programs generate dendrites with end positions anywhere in the interval [−1, 1]. The end positions are converted to lengths (line 25). In this step the dendrite position is linearly mapped into the interval [0, 1]. To generate a valid neural network we assume that dendrites are automatically connected to the nearest neuron or input on the left. We refer to this as “snapping” (lines 28 and 44). The dendrites of non-output neurons are allowed to connect to either inputs or other non-output neurons on their left. However, output neurons are only allowed to connect to non-output neurons on their left. Algorithm 10 returns the address of the neuron or input that the dendrite snaps to. The dendrites of output neurons are not allowed to connect directly to inputs (see Line 4 of the GetClosest function), however, when neurons are allowed to move, there can occur a situation where an output neuron is positioned so that it is the first neuron on the right of the outputs. In that situation it can only connect to inputs. If this situation occurs then the initialisation of the variable AddressOfClosest to zero in the GetClosest function (line 2) means that all the dendrites of the output neuron will be connected to the first external input to the ANN network. Thus a valid network will still be extracted albeit with a rather useless output neuron. It is expected that evolution will avoid using programs that allow this to happen.

Algorithm 9 stores the information required to extract the ANN in an array called Phenotype. It contains the connection addresses of all neurons and their weights (lines 29–30 and 45–46). Finally it stores the addresses of the output neurons (OutputAddress) corresponding to the computational problem whose ANN is being extracted (lines 49–52). These define the outputs of the extracted ANNs when they are supplied with inputs (i.e. in the fitness function when the Accuracy is assessed (see Algorithm 1). The Phenotype is stored in the same format as Cartesian Genetic Programming (see Sect. 8.4) and decoded in a similar way to genotypes.