Keywords

1 Introduction

As it is known, forest fires are one of the most destructive natural hazards in the Mediterranean countries because of their significant impact on the natural environment, human beings and economy. For that reason, scientific community has invested lots of efforts in developing forest fire propagation models and computational tools that could help firefighter and civil protection to tackle those phenomena in a smart way in terms of resources allocation, extinguish actions and security. The quality results of these models depend not only on the propagation equations describing the behaviour of the fire, but also on the input data required to initialize the model. Typically, this data is subjected to a high degree of uncertainty and variability during the evolution of the event due to the dynamic nature of some of them such as meteorological conditions or moisture contents in the vegetation. Any strategy oriented to reduce this input data uncertainty requires a quality measure of the results to determine the goodness of the proposed strategy. Typically, on the forest fire spread prediction field, this assessment relies on a fitness/error function that must evaluate how well the prediction system reproduces the real behaviour of the fire. In order to speed up the process of finding good estimations of certain input parameters, the prediction systems tend to include a calibration stage prior to perform the forest fire spread prediction (prediction stage) [4]. In particular, we focus on a well know two-stage methodology, which is based on Genetic Algorithms (GA). In this scheme, the calibration stage runs a GA to reduce input data uncertainty. To do that, the GA generates an initial population (set of individuals) where each individual consists of a particular configuration of the input parameters from which reducing their uncertainty implies an improvement in terms of quality results. This initial population will evolve using the so called genetic operators (elitism, mutation, selection, crossover) to obtain an improved set of individuals that better reproduces the observed past behaviour of the fire. After several iterations of the GA, the best individual will be selected to predict the near future (prediction stage). A key point in all this process is the error function applied to determine the prediction quality of each individual. This function drives the GA’s evolution process, consequently, to establish an appropriate level of confidence in the way that the error function is assessed is crucial in the system, [2].

The goal of this work is to study and test eight different error functions to assess the simulation error in the case of forest fire spread prediction.

This paper is organized as follows. In Sect. 2 the data uncertainty problem when dealing with forest fires spread prediction and how it can be minimize with an appropriate error function selection is introduced. Section 3 details an analysis of the different proposed functions to compute the simulation error. Section 4 presents the experimental results and, finally, Sect. 5 summarizes the main conclusions and future work.

2 Reducing Input Data Uncertainty

In order to minimize the uncertainty in the input data when dealing with forest fire spread prediction, we focus on the Two-Stage prediction scheme [1]. The main goal of this approach is to introduce an adjustment stage previous to the prediction stage to better estimate certain input parameters according to the observed behaviour of the forest fire. The main idea behind this strategy is to extract relevant information from the recent past evolution of the fire that could be used to reduce data uncertainty in the near future forecast. As it has previously mentioned, this scheme relies on a GA. Each individual of the GA population consists of a particular input parameter setting that will be fed into the underlying forest fire spread simulator. In this work, the forest fire spread simulator used is FARSITE [7], however, the methodology described would be reproduced for any other simulator. Each simulation generates a forest fire propagation that must be compared to the real evolution of the fire. According to the similarity in terms of shape and area among the simulated propagation and the real fire behaviour, the corresponding input parameters set is scored with an error value. Typically, low errors indicates that the provided forecast is closer to the reality, meanwhile higher errors indicate certain degree of mismatching between the two propagations.

There are several metrics to compare real and simulated values and each one weighs the events involved differently, depending on the nature of the problem [2]. In the forest fire spread case, the map of the area where the fire is taking place is represented as a grid of cells. Each cell is labeled to know if it has been burnt or not in both cases: the real propagation map and the simulated fire spread map. Then, if one compare both maps cell by cell, we came up with different possibilities: cells that were burnt in both the actual and simulated spread (Hits), cells burnt in the simulated spread but not in the reality (False Alarms), cells burnt in the reality but not in the simulated fire (Misses) and cells that were not burnt in any case (Correct negatives) [6, 8]. These four possibilities are used to construct a \(2\times 2\) contingency table, as is shown in Fig. 1.

Fig. 1.
figure 1

Standard structure of a contingency table.

Full size image

Nevertheless, at the time to put a prediction system into practice, it is important to consider what a good forecast actually means, according to the underlying phenomena we are dealing with, since there are many factors that models do not take into account. In our case, when a fire is simulated, a free fire behavior is considered, without including human intervention (firefighters). Under this hypothesis, the simulations should overestimate the burnt area. For this reason, when a simulation produces an underestimated perimeter (front of the fire), it is considered a bad result. Because of this, we are interested in those error functions that minimize the impact of False Alarms above the impact of Misses. Let’s imagine two hypothetical forest fire spread predictions where the number of FA in one of them is the same value that the number of Misses in the other one. Under this assumption, we would expect that the error function provides a lower error in the first case than in the second case.

In addition, for the particular case of forest fire spread forecast, the cells labeled as correct negatives are ignored because the area of the map to be simulated may vary independently of the fire perimeter, so they may distort the measurement of the error. In the following section, a description of the different error functions analyzed in this work is done.

3 Error Function Description

We propose the analysis of eight different functions that are potential candidates to be used as error metric in the Calibration Stage of a forest fire spread forecast system. To study their strengths and lacks, we carried out a preliminary study to artificially compute the error of each function depending on its Hits, False Alarms and Misses. For this purpose, we consider a sample map with a number of real burnt cells equal to 55 (RealCell = 55). A simple and intuitive, way to test the behavior of a given error function consists of evaluating its value for all configurations of Hits, Misses and False Alarms when their values varies within the range [0 to 55]. With the computed errors we build a color map representation where the behavior of the Error function with regard to the False Alarms and the Misses is shown. On the right side of the color map, the legend will show a color scale for the error function values where blue color represents lowest errors, while red colour represents highest errors. In this section, we will use the described color map to understand the behaviour of all described error functions.

Our goal is to find an error function that grows faster as Misses increases rather than doing so as False Alarms increase, that is, solid error function candidates should present a faster variation along the Misses axis than along the False Alarms axis.

To facilitate the data processing, the elements of the contingency table are expressed in the context of difference between sets. The description of the conversion of these metrics is showed in the Table 1. Using this notation, in the subsequent sections, we shall describe different error functions that have been deployed in the above described Calibration Stage, and the advantages and drawbacks of each one are reported.

3.1 Bias Score or Frequency Bias (BIAS)

The BIAS score is the ratio of the number of correct forecasts to the number of correct observations [8]. Equation 1 describes this error function. This metric represents the normalized symmetric difference between the real map and the simulated map.

$$\begin{aligned} BIAS=\frac{Hits+FA}{Hits+Misses} \end{aligned}$$
(1)
Table 1. Elements of the contingency table expressed in the context of difference between sets.
Full size table

However, in the evolutionary process we focus on minimizing the ratio of the frequency of the erroneous simulated events, so we rely on a slight variation of the BIAS score. The same Error Function can be expressed in terms of the difference between sets where the initial fire is considered a point and, therefore, it can be removed from the equation. The obtained formula is shown in Eq. 2.

$$\begin{aligned} \in =\frac{Misses+FA}{Hits+Misses} = \frac{UCell-ICell}{RealCell} \end{aligned}$$
(2)
Fig. 2.
figure 2

Colour map representation of BIAS (Eq. 2).

Full size image

In Fig. 2 the behaviour of the function 2 depending on False Alarms and Misses is shown. As it can be seen, the computed error grows up slightly faster in the False Alarms direction than in the Misses direction, this effect means that Eq. 2 lightly penalizes False Alarms compared to Misses.

3.2 BIAS+False Alarm Rate (BIAS+FAR) (\(\in _{1}\))

This function is a combination of the previous formula (Eq. 2) and the False Alarm Rate (FAR). The FAR measures the proportion of the wrong events forecast (see Eq. 3).

$$\begin{aligned} FAR={FA / (Hits + FA)} \end{aligned}$$
(3)

Since we are interested in penalizing those cases that underestimate the forecast, \(\in _{1}\) combines BIAS and FAR. Equation 4 shows this new Fitness Function in terms of events and difference between cell sets.

$$\begin{aligned} \in _{1}&=\frac{1}{2} \cdot \left( \frac{Misses+FA}{Hits+Misses} + \frac{FA}{Hits + FA}\right) \\ \nonumber&=\frac{1}{2}\cdot \left( \frac{UCell-ICell}{RealCell} + \frac{SimCell-ICell}{SimCell} \right) \end{aligned}$$
(4)
Fig. 3.
figure 3

Colour Map representation of BIAS+FAR (Eq. 4)

Full size image

The behavior of this metric is shown in Fig. 3. As it can be seen, the variation of the error is not lineal. The color map shows that while the blue part forms a convex curve, the yellow zone of the map forms a concave curve. It means that, while the amount of cells is low, the Eq. 4 penalizes more the Misses than the False Alarms, whereas with a large number of cells it tends to provide higher error for underestimated areas than for overestimated ones.

3.3 BIAS-False Alarm Rate (BIAS-FAR) (\(\in _{2}\))

The next Error Function is quite similar to the previous equation (Eq. 4), but in this case the FAR is subtracted from the BIAS, as it is shown in the Eq. 5. As we said, FAR measures the rate of False Alarms, so the overestimated simulations have a high value of FAR. So, in this case, we subtract FAR from BIAS in order to provide a better position for those simulations that provide overestimated spread perimeters.

$$\begin{aligned}&\in _{2}\,=\frac{1}{2} \cdot \left( \frac{Misses+FA}{Hits+Misses} - \frac{FA}{Hits + FA}\right) \\ \nonumber&\qquad \quad \,=\frac{1}{2}\cdot \left( \frac{UCell-ICell}{RealCell} - \frac{SimCell-ICell}{SimCell} \right) \end{aligned}$$
(5)

The behavior of this metric is shown in Fig. 4. This figure shows that this metric presents a not linear behavior. We are able to see how the blue zone (low error) is bigger in the False Alarms axis than in the Misses one, it means that BIAS-FAR penalizes much more the Misses than the False Alarms. Moreover, a big green zone is present, where the computed error remains more or less constant. Above this green zone, we can see that the error grows faster in the False Alarms direction than in the Misses direction.

Fig. 4.
figure 4

Colour Map representation of BIAS-FAR (Eq. 5)

Full size image

The main problem of this function is that for values around \(Misses=55\) and \(False Alarms=10\) (see Fig. 4), the error is lower than some simulations with better fitness. So, we cannot be confident about this metric, since it could cause that individuals with a better adjustment are discarded before individuals with a worse adjustment.

3.4 FAR+Probability of Detection of Hit Rate (FAR+POD) (\(\in _{3}\))

The next Fitness Function used is a combination of Eq. 3 and the Probability of Detection of hits rate (POD). The POD formula relates the observed events and estimated positively with all ones, Eq. 6. It represents the probability of a phenomenon being detected.

$$\begin{aligned} POD=\frac{Hits}{Hits+Misses} \end{aligned}$$
(6)

However, as it was mentioned, we focus on minimizing the ratio of the frequency of the erroneous simulated events, so a slight variant of the POD is used, as expressed in formula 7.

$$\begin{aligned} \in \,=\frac{Misses}{Hits+Misses} \end{aligned}$$
(7)

The result of combining Eq. 7 with Eq. 3 is expressed in Eq. 8.

$$\begin{aligned}&\in _{3}\,=\frac{1}{2} \cdot \left( \frac{Misses}{Hits+Misses} + \frac{FA}{Hits + FA}\right) \\ \nonumber&\qquad \quad \,\,=\frac{1}{2} \cdot \left( \frac{RealCell-Icell}{RealCell} + \frac{SimCell-ICell}{SimCell}\right) \end{aligned}$$
(8)
Fig. 5.
figure 5

Colour Map representation of FAR+POD (Eq. 8)

Full size image

In Fig. 5, we can see that, while False Alarms are low, the error grows more slowly in the Misses direction than when the amount of FalseAlarms is high. In the same way, when the number of Misses is low, the error remains more or less constant in the False Alarms direction, but when the amount of Misses increases, the error grows up very fast in the False Alarms direction.

3.5 BIAS+Incorrectness Rate (BIAS+IR) (\(\in _{4}\))

This Error Function was proposed in [3]. Using this function, the individuals that provide overestimated prediction have a better error than those individuals that underestimate the fire evolution. This function is shown in Eq. 9.

$$\begin{aligned} \begin{array}{c} \in _{4}\,=\frac{1}{2} \cdot \left( \frac{Misses\,+\,FA}{Hits\,+\,Misses} + \frac{Misses\,+\,FA}{Hits\,+\,FA}\right) \\ \quad =\frac{1}{2} \cdot \left( \frac{UCell\,-\,Icell}{RealCell} + \frac{UCell\,-\,ICell}{SimCell}\right) \end{array} \end{aligned}$$
(9)
Fig. 6.
figure 6

Colour Map representation of BIAS+IR (Eq. 9)

Full size image

Figure 6 depicts the behaviour of this metric. As it can be seen, it shows a predominance of blue color, which means that the error growths very slowly. This might suggest that the errors obtained are very good in any case, but in the reality what happens is that this function practically treated every case, i.e., overestimated simulations and underestimated ones, in the same way. It is for a large number of cells where equation BIAS+IR provide lower errors for overestimated simulations than for underestimate simulations.

3.6 Adjustable Incorrectness Aggregation (AIA) (\(\in _{5}\))

As we said, we look for an equation that penalize more the underestimated simulated areas than the overestimated, this implies that Misses cells are worse than False Alarms cells. In order to penalize more Misses than False Alarms the Eq. 10 was proposed

$$\begin{aligned} \in _{5}\,=\alpha \cdot FA + \beta \cdot Misses=\alpha \cdot (SimCell-ICell) + \beta \cdot (RealCell-ICell) \end{aligned}$$
(10)

with \(\alpha =1\) and \(\beta =2\). This Fitness Function provides very high errors, but, in our case, this is not a problem, because we only want to compare the errors among the individuals but not the absolute value of the error.

Fig. 7.
figure 7

Colour Map representation of AIA (Eq. 10)

Full size image

Figure 7 represents the behaviour of this metric. As it can be observed, this metric clearly reproduces the behavior that we were expected. It is easy to see that Eq. 10 penalizes more the Misses than the False Alarms, which is what we are looking for.

3.7 AIA with Correctness BIAS (AIA+CB) (\(\in _{6}\))

This equation is close to Eq. 10 but, in this case, the Hits are removed (see Eq. 11). This implies that when using this error function, negatives values can be obtained but, as it was stated, we do not care about the value of the error but on the final ranking that is generated using it. In this case, the best individual will be the individual with a higher negative value.

$$\begin{aligned}&\in _{6}\,=\alpha \cdot FA + \beta \cdot Misses-\gamma \cdot Hits\\ \nonumber&=\alpha \cdot (SimCell-ICell) + \beta \cdot (RealCell-ICell)-\gamma \cdot ICell \end{aligned}$$
(11)
Fig. 8.
figure 8

Colour Map representation of AIA+CB (Eq. 11)

Full size image

As in the previous case, using this metric, the error of the overestimated predictions increase slower than in the case of underestimated simulations (see Fig. 8). The problem of Eqs. 10 and 11 is that the values of \(\alpha \), \(\beta \), and \(\gamma \) are fixed and they are not the best choice for all forest fires.

3.8 Ponderated FA-MISS Rate (PFA-MR) (\(\in _{7}\))

In order to overcome the restrictions of Eqs. 10, 12 has been proposed.

$$\begin{aligned} \in _{7}\,=\alpha \cdot FA + \beta \cdot Misses \end{aligned}$$
(12)

with

$$\begin{aligned} \alpha =\frac{Hits}{Hits+FA}=\frac{ICell}{SimCell} \end{aligned}$$
(13)

and

$$\begin{aligned} \beta =\alpha +\frac{Hits}{Hits+Misses}=\frac{ICell}{SimCell}+\frac{ICell}{RealCell} \end{aligned}$$
(14)

Then, the Error Function expressed in context of difference between sets corresponds to Eq. 15.

$$\begin{aligned} \in _{7}\,=\frac{ICell}{SimCell} \cdot (SimCell-ICell) + \left( \frac{ICell}{SimCell}+\frac{ICell}{RealCell}\right) \cdot (RealCell-ICell) \end{aligned}$$
(15)
Fig. 9.
figure 9

Color Map representation of PFA-MR (Eq. 11)

Full size image

In Fig. 9 the behavior of Eq. 15 is shown. As it was expected, it can be seen that this metric penalizes the Misses above the False Alarms. However, similarly to the case of BIAS-FAR, it behaves in a way that, in cases with high values of Misses (combined, in this case, with high values of False Alarms), it returns low error values. The big orange zone, between 20 and 45 Misses value, means that the maximum value of the error is achieved within this area. This implies that a large number of Misses can distort the selection of those individuals with a better fitting.

Based on the previous analyses and interpretations of the different proposed Error Functions, in the subsequent section we present an applied test using both synthetic and real results of a large fire that took place in Greece in 2011.

4 Experimental Study and Results

In order to analyze how the use of different error functions affects the prediction of the forest fire spread, we have selected as study case one event stored in the database of EFFIS (European Forest Fire Information System) [5]. In particular, we have retrieved the information of a past fire that took place in Greece during the summer season of 2011 in the region of Arkadia. The forest fire began on the 26th of August, and the total burnt area was 1.761 ha. In Fig. 10, it can be seen the fire perimeters at three different time instants: \(t_{0}\) (August 26th at 09:43am) called Perimeter 1, \(t_{1}\) (August 26th at 11:27am) corresponds to Perimeter 2 and \(t_{2}\) (August 27th at 08:49am) is Perimeter 3. The Two-Stage predictions strategy based on the Genetic Algorithm has been applied where the Perimeter 1 was used as initial perimeter (ignition Perimeter), Perimeter 2 was used in the Calibration Stage and the perimeter to be predicted is Perimeter 3.

Fig. 10.
figure 10

Fire perimeters corresponding to the Arkadia fire.

Full size image

Figure 11 shows the forest fire spread obtained using best individual at the end of the Calibration Stage and the forecast delivered using that individual for each error function defined in the previous section. As it can be observed, the fire evolution provided by the best individual at the end of the Calibration Stage is directly related to the Error Function applied. Using the POD+FAR and AIA+CB errors functions, the obtained best individuals fits well enough the bottom of the fire, see Fig. 11(d) and (g), but they overestimate the top very much. This could be a good result because it could happen that human intervention had stopped the real fire spread in that zone. However, the problem is that the corresponding predictions have a high overestimation of the burned area. The worst error formula is the seventh (PFA-MR), because the best individual obtained underestimates the real fire but the simulation in the Prediction stage has a very overestimate burned area.

Fig. 11.
figure 11

For each error function (a) to (h), the upper figure shows the forest fire spread obtained using the best individual provided by the Calibration stage and the bottom figure depicts the final forecast delivered using that individual. The error evaluated for each case is also included

Full size image

In this case, the prediction that better fits the burned area is the simulation obtained when BIAS is used in the Calibration Stage (see 11(a)). This function provides the forecast with the highest number of intersection cells, although in some regions the burned area is overestimated. As we can see, the Error Function with the lower number of intersections cells are the AIA equation, Fig. 11(f). The predictions obtained from BIAS+FAR, BIAS-FAR and BIAS+IR, Fig. 11(b), (c) and (e) respectively, have less intersection cells than the prediction using BIAS but, at the same time, it has less False Alarms.

5 Conclusions

In the last years, the simulation of complex systems have demonstrated to be a powerful tool to improve the fight against natural hazards. For this reason, an adequate error function to evaluate the fitness of these models is a key issue. In this work, we focus on the specific case of wild fires as one of the most worrisome natural disaster.

In this work, eight equations have been tested in order to evaluate the prediction quality for large forest fires taking into account the factor of overestimated/underestimated predictions compared to the real fire. The results show that different error functions imply different calibration of the parameters and, therefore, different forest fire spread predictions are obtained. After applying the proposed error functions in the Two-Stage methodology, we can conclude that FAR+POD and AIA+CB functions tend to select individuals that provide very overestimated predictions. These functions provide the fittest individual in the Calibration Stage. The problem is, that using the adjusted inputs in these cases, the obtained predictions present very overestimated burnt areas. The function that underestimated more the fire perimeter is the sixth Error function (AIA), using this equation, the obtained prediction is the one with fewer intersection cells. The PFA-MR function is dismissed due to its lack of reliability.

There are a set of three Error Functions which stand out above others. These equations are BIAS, BIAS+FAR and BIAS+IR. As a result of our experimental study, we can observe that we obtain more favorable results with this set of functions than with the rest. However, we are not able to determine undoubtedly which Equation is best for most of the cases. For this reason, our future work will be oriented to differentiate large fires of small fires, or fires with different meteorological conditions (wind prominence, mainly), in order to determine which Error function is more suitable, depending on these aspects.