A massively parallel Grammatical Evolution technique with OpenCL

https://doi.org/10.1016/j.jpdc.2017.06.017Get rights and content

Highlights

  • A model of a massively parallel computation for Grammatical Evolution is proposed.

  • Evaluation of the proposed technique using OpenCL.

  • The candidate solutions and data are parallelized.

  • Performance gains of up to 528× were observed.

Abstract

Grammatical Evolution (GE) is a bio-inspired metaheuristic capable of evolving programs in an arbitrary language using a formal grammar. Among the major applications of the technique, the automatic inference of models from data can be highlighted. As with other genetic programming techniques, GE has a high computational cost. However, the algorithm has steps that can be computed independently, enabling the use of parallel computing to reduce the execution time and, consequently, making it possible its application to larger and more complex problems. Here, models of massively parallel computation for GE are studied and proposed using OpenCL, a framework for the creation of parallel algorithms in heterogeneous computing environments. Computational experiments were conducted to analyze the performance of an implementation using GPUs (Graphics Processing Units), when compared to a sequential implementation in CPUs (Central Processing Units). Finally, speedups of up to 528× were achieved, when all steps are performed in parallel in a GPU.

Introduction

The extraction of knowledge from data is a challenging task that finds application in many activities in different areas. In particular, the creation of mathematical models from data is one of the main tasks arising in the system identification field. Briefly, it consists in inferring the underlying functions that describe the behavior of a natural or artificial system from a set of samples, according to Ljung [25]. The inference of models in symbolic form is an important tool to assist specialists in extracting knowledge from data. One of the most common tasks in this area is known as parametric identification, where the structure of the model is established a priori, and then an optimization procedure is employed to find the numerical parameters of such model. Although the prior definition of the structure helps reducing the number of candidate solutions to be examined, the parametric modeling clearly contains a disadvantage: it requires previous domain knowledge – often unavailable – to define the model structure so that useful results can be obtained. A more general way of automating this knowledge discovery process is symbolic regression, that is, the inference of the model structure simultaneously with its numerical parameters. Summarizing Langley [24], some of the major challenges in assisting the process of automating scientific discovery include:

  • The methods should produce models which are easily understood by the user;

  • They should allow for the insertion of previous domain knowledge in the process;

  • They should be capable of inferring models also from small datasets;

  • They should allow for interaction with the specialists.

A branch of evolutionary computation known as Genetic Programming (GP) has been shown to be able to address those goals and has been successfully applied in several domains [14], [22].

Among its virtues, it can be highlighted that it requires less domain knowledge and allows for the creation of symbolic (and possibly intelligible) solutions. In short, GP is a powerful bioinspired metaheuristic capable of automatically evolving symbolic structures that can be thought of as computer programs [22]. As a result, a formal grammar can be adopted in order to (i) restrict the search space, (ii) introduce domain knowledge, and (iii) avoid the generation of invalid candidate programs. Here, Grammatical Evolution (GE) is used; one of the most popular GP variants [26], which is capable of evolving programs in an arbitrary language defined by a formal grammar [28], [29].

However, GE and other GP techniques have a high computational demand: their application to large real-world problems can be unfeasible.

The most time consuming step of the algorithm is the evaluation of the quality of the candidate solutions, which grows with the size of the dataset (number of instances and attributes) and the complexity of the models which are being evolved. However, GP techniques are naturally parallel, as there are steps that can be computed independently, suggesting the adoption of parallel or distributed computing in order to reduce the execution time and, thereby, enabling their application to large scale problems.

In the last decade, the use of GPGPU (General-purpose computing on graphics processing units) has been growing, simplifying the use of graphics processing units (GPUs) to implement general purpose algorithms. This increase is mainly due to GPU’s relatively low cost, while offering an impressive processing capacity. Thus, GPGPU represents an affordable alternative to accelerate scientific applications.

GE techniques have shown advantages when compared to other GP techniques: some works presented higher accuracy of the solutions [28], [29] or lower complexity of the generated models [4]. Also, grammar-based approaches allow to bias the search syntactically, as the models obtained must comply with the production rules of a formal grammar. Besides, existing knowledge can be included in the definition of the grammar. Despite the advantage of the formal grammar in the knowledge discovery process, the evaluation step in GE also includes a mapping procedure, where the programs are generated by decoding a binary string genotype. Thus, the parallelism of such type of method of modeling from data is important and necessary.

In this work, a parallel GE technique is proposed for GPGPU architecture, in which every algorithm step is performed by a GPU. To exploit this computing power, the OpenCL (Open Computing Language) framework was adopted. That was due to the fact that, unlike CUDA, it can be used with various types of devices from different manufacturers, with few or no changes in the source code [15].

Computational experiments from the symbolic regression domain were carried out to measure the achieved speedup, and the results obtained here show speedups of up to 528× when compared to the serial execution of the technique in a CPU.

Section snippets

Related work

As the candidate solutions in GP are computer programs, two strategies can be employed to evaluate their quality. They can be either executed step-by-step using an interpreter or be compiled into an executable machine code and executed directly. An advantage of the compilation approach is the faster execution of the candidate solutions, as their instructions are executed without an interpreter. On the other hand, the compilation process adds a large overhead, making the strategy less attractive.

Grammars

A grammar is a formalism that allows for the production of strings of symbols which belong to a language [12]. In this context, it can be used here as a constraint technique to limit the search space, only allowing for the generation of (correct) programs from a given language.

A context-free grammar G (the formal grammar adopted here) can be defined by a tuple (N,Σ,R,S), where N is a finite set of nonterminals; Σ is a finite set of terminals; R is a set of production rules; and S is the initial

Parallel computing

As Moore’s Law is no longer applicable [19], the industry has been adopting other strategies to improve the computing power, such as the creation of multi-core and many-core architectures. In the Single Program Multiple Data (SPMD) model, the same program is executed in parallel by all processing elements. Instead of running different programs in each unit, SPMD programs consist of a single executable program which is able to simulate the execution of various programs through the use of

Massively parallel grammatical Evolution

This section describes the proposed GE parallelization for the GPGPU architecture using OpenCL, in which the search process is completely executed in parallel. Section 5.1 presents the parallel strategy used for the evaluation step, while Section 5.2 describes the remaining phases of the search process.

Computational experiments

An experiment was carried out to analyze the performance of the proposed parallel technique. The running time was used to compare the techniques, and it was measured using the wall-clock time, which is the total elapsed time between the beginning and the end of the programs. Thus, not only the processing time is considered, but also the overhead, comprising the additional costs such as data transfers and compilation of the OpenCL kernels. In cases where a GPU is used, we also measured the

Conclusion and future works

An approach for massively parallel architectures was proposed to accelerate the execution of Grammatical Evolution with OpenCL, in order to allow the use of the technique for problems with large datasets. In the presented model all steps of GE are performed in parallel. Computational experiments were conducted to evaluate the computational performance of the proposed technique. The metrics adopted were the speedup regarding the total execution time, and the number of instructions executed by

Acknowledgments

The authors would like to thank the reviewers for their comments, which helped improve the paper, and the support provided by CNPq (grant 310778/2013-1), FAPEMIG (grant APQ-03414-15), FAPERJ (grant E-26/102.035/2009), CAPES, PPGMC/UFJF and LNCC .

Igor L.S. Russo He received a Computer Science degree (2015) and a M.Sc. degree in Computational Modeling (2017) from Universidade Federal de Juiz de Fora. He is currently pursuing his D.Sc. degree in Computational Modeling at Laboratório Nacional de Computação Científica and has experience in Computer Science, working in the following subjects: evolutionary computation, machine learning, and parallel computing.

References (44)

  • CanoA. et al.

    Speeding up multiple instance learning classification rules on GPUs

    Knowledge Inform. Syst.

    (2015)
  • ChittyD.

    A data parallel approach to genetic programming using programmable graphics hardware

  • ChomskyN.

    Syntactic structures

    (2002)
  • GasterB. et al.

    Heterogeneous Computing with OpenCL

    (2011)
  • GloverF.W. et al.

    Handbook of Metaheuristics

    (2003)
  • GothandaramanA. et al.

    Comparing hardware accelerators in scientific applications: A case study

    IEEE Trans. Parallel Distrib. Syst.

    (2011)
  • HardingS. et al.

    Fast genetic programming on GPUs

  • JuilleH. et al.

    Massively parallel genetic programming

  • KeijzerM.

    Improving symbolic regression with interval arithmetic and linear scaling

  • KnuthD.E.

    Backus normal form vs. Backus Naur form

    Commun. ACM

    (1964)
  • KornsM.F.

    Accuracy in symbolic regression

  • KozaJ.R.

    Genetic Programming: On the Programming of Computers by Means of Natural Selection

    The MIT Press

    (1992)
  • Igor L.S. Russo He received a Computer Science degree (2015) and a M.Sc. degree in Computational Modeling (2017) from Universidade Federal de Juiz de Fora. He is currently pursuing his D.Sc. degree in Computational Modeling at Laboratório Nacional de Computação Científica and has experience in Computer Science, working in the following subjects: evolutionary computation, machine learning, and parallel computing.

    Heder S. Bernardino He received a Computer Science degree from Universidade Federal de Juiz de Fora (2006), a M.Sc. degree in Computational Modeling from Universidade Federal de Juiz de Fora (2008) and a D.Sc. degree in Computational Modeling from Laboratório Nacional de Computação Científica (2012). He is currently Professor Adjunto at Universidade Federal de Juiz de Fora and has experience in Computer Science, working in the following subjects: evolutionary computation, machine learning, and parallel computing.

    Helio J.C. Barbosa is a Senior Technologist at the Laboratòrio Nacional de Computação Científica, Brazil. He received a Civil Engineering degree (1974) from the Federal University of Juiz de Fora, where he is an Associate Professor in the Computer Science Department, and M.Sc. (1978) and D.Sc. (1986) degrees in Civil Engineering from the Federal University of Rio de Janeiro, Brazil. During 1988–1990 he was a visiting scholar at the Division of Applied Mechanics, Stanford University, USA. His is currently mainly interested in the design and application of nature-inspired metaheuristics in engineering and biology.

    View full text