Systematic data generation and test design for solution algorithms on the example of SALBPGen for assembly line balancing

Abstract

Recently, the importance of correctly designed computational experiments for testing algorithms has been a subject of extended discussions. Whenever real-world data is lacking, generated data sets provide a substantive methodological tool for experiments. Focused research questions need to base on specialized, randomized and sufficiently large data sets, which are sampled from the population of interest. We integrate the generation of data sets into the process of scientific testing.

Until now, no appropriate generators or systematic data sets have been available for the assembly line balancing problem (ALBP). Computational experiments were mostly based on very limited data sets unsystematically collected from the literature and from some real-world cases. As a consequence, former performance analyses often come to contradictory conclusions and lack on statistical evidence.

We introduce SALBPGen, a new instance generator for the simple ALBP which can be applied and extended to any generalized ALBP, too. Unlike most generators, SALBPGen takes into account usual properties of precedence graphs in manufacturing. It is very flexible and able to create instances with very diverse structures under full control of the experiment’s designer. We also propose new challenging data sets, as shown with the new direct measure of instance’s hardness called trickiness. By two exemplary computational experiments, we illustrate how important insights can be gained with the help of the systematically generated data sets.

Introduction

Analysis of algorithms is performed according to three main approaches in the literature: theoretical worst-case analysis, theoretical average-case analysis and experimental analysis (Johnson, 2002). Recently, a growing interest has been observed in the research community towards the experimental analysis, where performance of algorithms is analyzed by running them on a set of real-world or generated instances of a certain optimization problem. It has been realized that theoretical analysis cannot tell a full story about performance of algorithms in real-world applications, whereas the experiments provide valuable insights and knowledge (Hall and Posner, 2001, Johnson, 2002). Experimental analysis is especially important in studies of NP-hard problems, where computational time needed by algorithms to find optimal solutions varies dramatically depending on the specifics of the problem instance.

Due to varying quality of conducted experiments in the literature, guidelines for performing computational experiments were developed by a number of research societies, such as by the Mathematical Optimization Society (Jackson et al., 1991), by the Mathematical Programming Society (Crowder et al., 1978) or by the Institute of Industrial Engineers (Lee et al., 2007). Without exception, one of the focuses in such guidelines takes the discussion on how to choose an adequate test data set. Since the number of real-world problem instances available in the literature is limited, one of the most valuable sources of test data is the generation of data sets by a random generator (Crowder et al., 1978, Hooker, 1995).

The above-mentioned guidelines point on the advantages and disadvantages of randomly generated experimental data sets. A major advantage is that a sufficient number of instances can be drawn from the population with desired characteristics, thus statistical analysis can be applied to make generalizations about the algorithms’ performance. A major possible disadvantage is that the generated instances may be unrealistic or poorly diversified, and thus may not correctly represent the population of instances of interest. However, as we will show in Section 4, it is possible to alleviate this disadvantage significantly.

In any case, whenever available, all the guidelines highly recommend using standard data generators and standard data sets. These are well-documented data generators, or data sets, that are a “standard reference” for researchers for conducting computational experiments. Such data sets and, especially, generators, are badly needed for a number of optimization problems, for example, the assembly line balancing problem (ALBP).

ALBP is a problem of an optimal allocation of tasks to working stations of an assembly line and describes the planning situation in manual assemblies. It can be found, for example, in automotive, white goods and electronics industries. The currently existing standard data set for simple ALBP (Scholl, 1993, Scholl, 1999) is not randomized and poorly diversified, i.e., it contains instances, differing from each other on just a few characteristics (see Section 4.4). Furthermore, no standard data generator for ALBP exists. The existing data set generators are restricted in their generation capability. They do not produce realistic problem instances and do not allow for a thorough comparison of solution algorithms, for example on instances with specified structures.

Therefore, we propose a new data set generator SALBPGen, which is developed according to the above-mentioned guidelines and generation principles. Moreover, SALBPGen is very flexible and, thus, it can serve as a standard data generator. First of all, SALBPGen is able to produce instances rather similar to real-world assembly line balancing situations. We collected information from our partners in German automobile industry and analyzed problem instances taken from practice, which are available in the literature. This knowledge enables us to formulate several structural characteristics, which can be controlled for in our generator to produce realistic and randomized problem instances. Secondly, with our generator, a (full) factorial design of computer experiments is possible. SALBPGen enables to fix and systematically vary important structure measures proposed in the literature. In other words, with our generator, it is not only possible to find out whether the algorithms perform better or worse for the test data set, but also, for what kind of instances its performance is especially weak or strong. Because of the ability of SALBGen to generate a large number of randomized instances, it is possible to generalize the findings from the data sets with help of statistical analysis. Put it all together, SALBPGen enables to generate easily new knowledge and important insights useful for the algorithm’s design and application, as we illustrate it in Section 6.

One of the important characteristics of generated problem instances is their hardness, i.e., difficulty to find an optimal solution. A measure of instance’s hardness, called trickiness measure, is one of the major contributions of this article. Hardness is one of the most important characteristics of the instances in a test data set, especially in standard sets (see, e.g., Hoffmann, 1990, Farhi et al., 1997). The developed trickiness measure is universal, in the sense that it can be applied to any optimization problem. It is also a direct and intuitive measure that does not rely on any specific solution algorithm, but describes a characteristic of the instance itself.

The paper is organized as follows. First, we describe the steps how to design correctly and in accordance with the existing guidelines a computational experiment on a generated data set (Section 2). These steps will form the logic of our further discussion. We outline the assembly line balancing problem in Section 3. In Section 4, we look at important characteristics of instances that are often (or are important to be) considered in computational experiments. Section 5 comments on the new generator SALBPGen and proposes new standard data sets. In Section 6 we show, how the new data sets generated by SALBPGen help to generate statistically supported knowledge. We conclude with a summary and an outlook in Section 7.