A massively parallel Grammatical Evolution technique with OpenCL
Introduction
The extraction of knowledge from data is a challenging task that finds application in many activities in different areas. In particular, the creation of mathematical models from data is one of the main tasks arising in the system identification field. Briefly, it consists in inferring the underlying functions that describe the behavior of a natural or artificial system from a set of samples, according to Ljung [25]. The inference of models in symbolic form is an important tool to assist specialists in extracting knowledge from data. One of the most common tasks in this area is known as parametric identification, where the structure of the model is established a priori, and then an optimization procedure is employed to find the numerical parameters of such model. Although the prior definition of the structure helps reducing the number of candidate solutions to be examined, the parametric modeling clearly contains a disadvantage: it requires previous domain knowledge – often unavailable – to define the model structure so that useful results can be obtained. A more general way of automating this knowledge discovery process is symbolic regression, that is, the inference of the model structure simultaneously with its numerical parameters. Summarizing Langley [24], some of the major challenges in assisting the process of automating scientific discovery include:
-
The methods should produce models which are easily understood by the user;
-
They should allow for the insertion of previous domain knowledge in the process;
-
They should be capable of inferring models also from small datasets;
-
They should allow for interaction with the specialists.
A branch of evolutionary computation known as Genetic Programming (GP) has been shown to be able to address those goals and has been successfully applied in several domains [14], [22].
Among its virtues, it can be highlighted that it requires less domain knowledge and allows for the creation of symbolic (and possibly intelligible) solutions. In short, GP is a powerful bioinspired metaheuristic capable of automatically evolving symbolic structures that can be thought of as computer programs [22]. As a result, a formal grammar can be adopted in order to (i) restrict the search space, (ii) introduce domain knowledge, and (iii) avoid the generation of invalid candidate programs. Here, Grammatical Evolution (GE) is used; one of the most popular GP variants [26], which is capable of evolving programs in an arbitrary language defined by a formal grammar [28], [29].
However, GE and other GP techniques have a high computational demand: their application to large real-world problems can be unfeasible.
The most time consuming step of the algorithm is the evaluation of the quality of the candidate solutions, which grows with the size of the dataset (number of instances and attributes) and the complexity of the models which are being evolved. However, GP techniques are naturally parallel, as there are steps that can be computed independently, suggesting the adoption of parallel or distributed computing in order to reduce the execution time and, thereby, enabling their application to large scale problems.
In the last decade, the use of GPGPU (General-purpose computing on graphics processing units) has been growing, simplifying the use of graphics processing units (GPUs) to implement general purpose algorithms. This increase is mainly due to GPU’s relatively low cost, while offering an impressive processing capacity. Thus, GPGPU represents an affordable alternative to accelerate scientific applications.
GE techniques have shown advantages when compared to other GP techniques: some works presented higher accuracy of the solutions [28], [29] or lower complexity of the generated models [4]. Also, grammar-based approaches allow to bias the search syntactically, as the models obtained must comply with the production rules of a formal grammar. Besides, existing knowledge can be included in the definition of the grammar. Despite the advantage of the formal grammar in the knowledge discovery process, the evaluation step in GE also includes a mapping procedure, where the programs are generated by decoding a binary string genotype. Thus, the parallelism of such type of method of modeling from data is important and necessary.
In this work, a parallel GE technique is proposed for GPGPU architecture, in which every algorithm step is performed by a GPU. To exploit this computing power, the OpenCL (Open Computing Language) framework was adopted. That was due to the fact that, unlike CUDA, it can be used with various types of devices from different manufacturers, with few or no changes in the source code [15].
Computational experiments from the symbolic regression domain were carried out to measure the achieved speedup, and the results obtained here show speedups of up to when compared to the serial execution of the technique in a CPU.
Section snippets
Related work
As the candidate solutions in GP are computer programs, two strategies can be employed to evaluate their quality. They can be either executed step-by-step using an interpreter or be compiled into an executable machine code and executed directly. An advantage of the compilation approach is the faster execution of the candidate solutions, as their instructions are executed without an interpreter. On the other hand, the compilation process adds a large overhead, making the strategy less attractive.
Grammars
A grammar is a formalism that allows for the production of strings of symbols which belong to a language [12]. In this context, it can be used here as a constraint technique to limit the search space, only allowing for the generation of (correct) programs from a given language.
A context-free grammar G (the formal grammar adopted here) can be defined by a tuple , where is a finite set of nonterminals; is a finite set of terminals; is a set of production rules; and is the initial
Parallel computing
As Moore’s Law is no longer applicable [19], the industry has been adopting other strategies to improve the computing power, such as the creation of multi-core and many-core architectures. In the Single Program Multiple Data (SPMD) model, the same program is executed in parallel by all processing elements. Instead of running different programs in each unit, SPMD programs consist of a single executable program which is able to simulate the execution of various programs through the use of
Massively parallel grammatical Evolution
This section describes the proposed GE parallelization for the GPGPU architecture using OpenCL, in which the search process is completely executed in parallel. Section 5.1 presents the parallel strategy used for the evaluation step, while Section 5.2 describes the remaining phases of the search process.
Computational experiments
An experiment was carried out to analyze the performance of the proposed parallel technique. The running time was used to compare the techniques, and it was measured using the wall-clock time, which is the total elapsed time between the beginning and the end of the programs. Thus, not only the processing time is considered, but also the overhead, comprising the additional costs such as data transfers and compilation of the OpenCL kernels. In cases where a GPU is used, we also measured the
Conclusion and future works
An approach for massively parallel architectures was proposed to accelerate the execution of Grammatical Evolution with OpenCL, in order to allow the use of the technique for problems with large datasets. In the presented model all steps of GE are performed in parallel. Computational experiments were conducted to evaluate the computational performance of the proposed technique. The metrics adopted were the speedup regarding the total execution time, and the number of instructions executed by
Acknowledgments
The authors would like to thank the reviewers for their comments, which helped improve the paper, and the support provided by CNPq (grant 310778/2013-1), FAPEMIG (grant APQ-03414-15), FAPERJ (grant E-26/102.035/2009), CAPES, PPGMC/UFJF and LNCC .
Igor L.S. Russo He received a Computer Science degree (2015) and a M.Sc. degree in Computational Modeling (2017) from Universidade Federal de Juiz de Fora. He is currently pursuing his D.Sc. degree in Computational Modeling at Laboratório Nacional de Computação Científica and has experience in Computer Science, working in the following subjects: evolutionary computation, machine learning, and parallel computing.
References (44)
- et al.
Accelerated parallel genetic programming tree evaluation with OpenCL
J. Parallel Distrib. Comput.
(2013) - et al.
Parallel multi-objective ant programming for classification using {GPUs}
J. Parallel Distrib. Comput.
(2013) - et al.
Parallel evaluation of Pittsburgh rule-based classifiers on GPUs
Neurocomputing
(2014) End of moore’s law: thermal (noise) death of integration in micro and nano electronics
Phys. Lett. A
(2002)- et al.
Fast evolutionary image processing using Multi-GPUs
- et al.
Parallelization of binary and real-coded genetic algorithms on GPU using CUDA
- et al.
Predicting the performance of job applicants by means of genetic programming
- et al.
Grammar-based immune programming
Nat. Comput.
(2011) - et al.
GPU-parallel subtree interpreter for genetic programming
- et al.
Speeding up the evaluation phase of GP classification algorithms on GPUs
Soft. Comput.
(2012)
Speeding up multiple instance learning classification rules on GPUs
Knowledge Inform. Syst.
A data parallel approach to genetic programming using programmable graphics hardware
Syntactic structures
Heterogeneous Computing with OpenCL
Handbook of Metaheuristics
Comparing hardware accelerators in scientific applications: A case study
IEEE Trans. Parallel Distrib. Syst.
Fast genetic programming on GPUs
Massively parallel genetic programming
Improving symbolic regression with interval arithmetic and linear scaling
Backus normal form vs. Backus Naur form
Commun. ACM
Accuracy in symbolic regression
Genetic Programming: On the Programming of Computers by Means of Natural Selection
The MIT Press
Cited by (5)
Human Activity Recognition Using Parallel Cartesian Genetic Programming
2021, 2021 IEEE Congress on Evolutionary Computation, CEC 2021 - ProceedingsMultifractal detrended fluctuation analysis parallel optimization strategy based on openMP for image processing
2020, Neural Computing and ApplicationsClassification and advantages parallel computing in process computation: A systematic literature review
2019, Proceedings - 2018 4th International Conference on Computing, Engineering, and Design, ICCED 2018Multi- and many-threaded heterogeneous parallel grammatical evolution
2018, Handbook of Grammatical Evolution
Igor L.S. Russo He received a Computer Science degree (2015) and a M.Sc. degree in Computational Modeling (2017) from Universidade Federal de Juiz de Fora. He is currently pursuing his D.Sc. degree in Computational Modeling at Laboratório Nacional de Computação Científica and has experience in Computer Science, working in the following subjects: evolutionary computation, machine learning, and parallel computing.
Heder S. Bernardino He received a Computer Science degree from Universidade Federal de Juiz de Fora (2006), a M.Sc. degree in Computational Modeling from Universidade Federal de Juiz de Fora (2008) and a D.Sc. degree in Computational Modeling from Laboratório Nacional de Computação Científica (2012). He is currently Professor Adjunto at Universidade Federal de Juiz de Fora and has experience in Computer Science, working in the following subjects: evolutionary computation, machine learning, and parallel computing.
Helio J.C. Barbosa is a Senior Technologist at the Laboratòrio Nacional de Computação Científica, Brazil. He received a Civil Engineering degree (1974) from the Federal University of Juiz de Fora, where he is an Associate Professor in the Computer Science Department, and M.Sc. (1978) and D.Sc. (1986) degrees in Civil Engineering from the Federal University of Rio de Janeiro, Brazil. During 1988–1990 he was a visiting scholar at the Division of Applied Mechanics, Stanford University, USA. His is currently mainly interested in the design and application of nature-inspired metaheuristics in engineering and biology.