Comparison of GAs in black-box scenarios

In black-box optimization scenarios, researchers have no control over the fitness functions, and hence genetic algorithms (GAs) are usually compared by the number of function evaluations. Commonly used statistics include arithmetic mean, median, and standard deviation. However, these statistics can be misleading. For example, when there exist unsolvable instances within limited time, median simply ignores those instances, and arithmetic mean is not applicable at all. In this paper, we propose comparison methods from a practical point of view. Specifically we propose three use cases which cover most of the situations that GA practitioners may encounter. Among these three use cases, two of them are matchups, which requires a pair of GAs to be compared with each other, while the other provides as a standalone performance indicator of GAs.