A family of multi-concept program synthesisers in Alloy^⁎

Highlights

• Alloy^⁎ Program Synthesiser (APS) can be used for general purpose program synthesis.

• Implementing denotational semantics based synthesis for Turing complete problems.

• Exploring the notion of: state, control flow, while loops commands, and arrays.

• Also: deductive/inductive synthesis, code reuse, soft sketch and genetic algorithm.

• APS synthesized 17 well-known programs, including array based, such as Bubble sorting.

Abstract

Program synthesis aims to mechanise the task of programming from the user intent (using pre and post condition, examples and sketches). There are many approaches (or concepts) in program synthesis that are usually implemented in isolation: deductive, syntax-based, inductive, etc. In this paper, we present a characterisation of program synthesis as model finding, using Alloy^⁎. Such a characterisation unifies several of these concepts in a single model. Through model finding, we obtain a general framework for rapid development of a program synthesiser accommodating denotational semantics based synthesis, simultaneous deductive and inductive synthesis, software reuse, syntactic ingredients (the Alloy^⁎ scope of entities), and a new one: a soft sketch (a set of commands that must appear in the synthesised program but in no particular order of execution). Our family of synthesisers produce general purpose programs in the Java language. As the Alloy^⁎ synthesiser requires several rounds of user assistance to set scope, sketches, etc., particularly for complex problems, we integrated the model finder to a genetic algorithm module, where candidate solutions and user inputs are generated and mutated automatically. We carried out empirical evaluations on program synthesis successfully. As results, we verified that: (i) we can synthesise thirteen programs (Maj5, Maj8, IntSQRT, Max4, Modu, Fact, Fib, aMax, aDouble, aSum, eCount, aBubSort, aSelSort); (ii) inductive synthesis was faster than deductive synthesis; (iii) synthesis with reuse was faster; and (iv) Genetic Algorithm is better than user trial and error approach.

Introduction

Program synthesis typically performs a search over the space of programs to find a source code that is consistent with a variety of constraints (for instance, input-output examples, specifications, and incomplete programs — or sketches [1]). Program synthesis is considered the holy grail of Computer Science since the beginning of Artificial Intelligence in the 1950s [2]. Automatic program synthesis is gaining attention nowadays thanks to the advances in Artificial Intelligence and SAT/SMT theories (together with the development of efficient solvers). Such advances have led to recent developments in different trends in program synthesis like deductive synthesis and inductive synthesis.

In deductive synthesis, a program that satisfies a formal specification must be produced by deduction. One way to do that is to model the problem of program synthesis as a SAT/SMT problem so that a solver is employed to automatically find a program that satisfies a formal specification [2]. As a SAT/SMT solver can now check the satisfiability of very complex logical expressions in reasonable time, fully automatic deductive synthesisers have currently produced promising results [3], [4], [5], [6].

In the last decade, a new trend in program synthesis has appeared. Several applications of synthesis in the so called programming-by-examples field have been deployed in mass-market industrial products [7], [8], [9]. This kind of synthesis is known as inductive synthesis, where the formal specification is replaced by a set of input-output examples. The usage of examples instead of specifications comes from the Artificial Intelligence culture [10], [11], [12], [13].

In our previous study [14], we presented a deductive synthesiser which was built on top of the Alloy^⁎ [15] model finder. Here, the term deductive synthesiser means that the user intention is represented by a pre and post condition (like in a deductive approach). Alloy^⁎ subsequently tries to find a program that satisfies this contract. Note, however, that Alloy^⁎ is, at the end, always inductive. By inductive synthesiser we mean the usual definition: input and output examples are generalised into programs.

The syntax and the denotational semantics of Winskel's IMP (erative) language [16] were our starting point to build an Alloy^⁎-based Program Synthesiser (APS for short), whose syntax and semantics are based on Alloy language elements (such as signatures, relations, predicates, etc). We then applied the Alloy^⁎ model finder (the Alloy^⁎ Analyser) to search for a program that satisfies a contract written in terms of pre and post condition. Alloy^⁎ is a variant of the Alloy [17] model finder that implements a convenient Counter Example-Guided Inductive Synthesis (CEGIS) [18] algorithm. This facility, in combination with the Alloy^⁎ modelling language, made the construction of the Alloy^⁎ synthesiser interesting. By using the primitive implementation of CEGIS and the high-level modelling language of Alloy^⁎, we could encode the synthesis problem at a very abstract level (synthesis as a search over the states constrained by a denotational semantics). Our Alloy^⁎-based Program Synthesiser (APS) was able to synthesise high-level program constructs like sequential compositions, if-then-elses and while loops. Such results were similar to the synthesiser developed by Srivastava et al. [4], where a SAT/SMT solver was used directly. Alloy^⁎ is related to a large domain of applications such as program verification, software testing, fast prototyping, as well as teaching. Our goal is to exploit program synthesis in these domains in the future.

As Alloy^⁎ provides us with this platform for a rapid development of a synthesiser, the original synthesiser was easily extended and adapted to deal with examples instead of specifications (that is, instead of using a formal contract we can just use input-output examples). This allowed us to produce an inductive programming-by-example synthesiser with an almost effortless endeavour. The inductive version of the synthesiser was achieved by realising that Alloy^⁎ treats a set of input-output examples just as any regular constraint. The inductive version modified 4 lines of the deductive model and added 5 new lines. This new version takes as input not only examples but also sketches (programs with holes) [1]. It also possesses the ability to reuse functions by simply reusing the way the language was embedded into Alloy^⁎. For instance, a program that returns the maximum of 3 numbers can be implemented by calling the maximum of 2 numbers twice: max(x,max(y,z)). With the ability to call functions, it can (hopefully) avoid doing all programs from scratch.

In the attempt to synthesise more real-world related problems, we introduced arrays in the Alloy^⁎ specification. With this, we were able to synthesise well-known sorting algorithms like Bubble sort and Selection sort. In terms of arrays, there are synthesisers capable of: (i) mapping functions to search/sort over lists [4], [19], [20]; (ii) handling strings by extracting specific data [21], [22], [23]; or (iii) simulating finite size arrays [24]. APS outputs arrays written as Java's source code, such as:

In order to reduce the repetitive tasks of interacting with Alloy^⁎ to change the scope of the model entities and the task of verifying the correctness of the synthesised program, we enveloped APS around the concepts of Genetic Algorithms [25]. This extension gave rise to the Alloy^⁎ Program Synthesiser with Genetic Algorithm (APS-GA, for short), a variant of the APS that can generate user inputs, mutate candidate solutions, and automatically test the results, thus reducing both the user mental effort and amount of interactions. APS-GA was first introduced in our previous work [25]. In this article we describe it in more detail.

To explore the inductive aspect of APS, the user provides a small set of examples (to guide the synthesiser on producing candidate programs) and a larger set of test cases that is as comprehensive as possible to capture all expected input and output values (in order to enable APS-GA to check whether the solution is correct). Examples must be smaller than test cases in order to prevent state explosion. We detail this issue in the following sections.

By providing only examples and test cases as inputs, the synthesiser can be effective for simple problems. Although our sketches are optional, complex cases need them in general. Many synthesisers use sketches to guide and speed up the search for a solution. Our synthesiser is no different than the others but uses a more flexible solution. Our sketches are called soft sketches because they represent a set of commands that must appear in the synthesised program, but in no particular order (the order of execution of a traditional sketch is hard coded). It is worth noting that a soft sketch can also define (optionally) pieces of hard code as well. It is up to the user's needs. Also, pieces of soft sketches can occasionally occur more than once in the synthesised program if needed. Such a feature is impossible with a traditional sketch as it denotes a fixed template of the synthesised program.

Moreover, the Alloy^⁎ scope is produced automatically. This scope defines the amount of operations and commands that must appear in the synthesised program. For instance, a user can inform Alloy^⁎ to search for a program that contains exactly one while loop and at most 5 assignments. This information, which we call syntactic ingredients, was provided interactively by the user when using the APS, but is generated automatically with the aid of Genetic Algorithm (APS-GA). Finally, with Genetic Algorithm we also verify automatically (by producing Java programs and running tests on them) whether a synthesised program is correct or not with respect to the test cases provided. All these features reduce both the mental effort of the user and the interactions needed. With Genetic Algorithm, we were able to synthesise thirteen programs: integer square root (IntSQRT), majority of 5 (Maj5), majority of 8 (Maj8), maximum of 4 (Max4), modulo (Mod), factorial (Fact), Fibonacci (Fib), maximum element of an array (aMax), assignment a[i] = 2*i, for each i and array a (aDouble), sum of all elements of an array (aSUm), counting of occurrences of a given number in an array (eCount), Bubble Sort (aBubSort) and Selection Sort (aSelSort). They are found in the SyGuS competition [26], iJava and IntroClass, and Genetic programming communities.

In summary, the main contributions of this article are:

• The characterisation of synthesis of general-purpose programs as model finder that acts as a multi-concept platform for rapid development of a program synthesiser that features both inductive and deductive synthesis simultaneously;

• An empirical evaluation showing that an inductive synthesis is as efficient as (and, most of the times, better than) a deductive synthesis;

• The introduction of a synthesiser that, after a simple adjustment in the Alloy^⁎ specification, is able to: (i) reuse functions, where we have to define their syntax and well-formedness rules, and define their semantics in the Alloy^⁎ relational language; and (ii) deal with arrays, where we have to define the arrays themselves (their names and contents in terms of Alloy^⁎ sequences), the array well-formedness rules and an updated syntax and semantics for expressions and assignments as they were affected by the inclusion of arrays;

• An empirical evaluation about the performance of the synthesis of 7 well-known non-array programs (IntSQRT, Maj5, Maj8, Max4, Mod, Fact and Fib) and 6 well-known array programs (aMax, aDouble, aSUm, eCount, aBubSort, and aSelSort);

• An empirical analysis measuring the time taken: (i) to translate from an Alloy^⁎ model to a SAT solver language, (ii) to find a solution, and (iii) to synthesise with reuse. Results showed that deductive synthesis takes less time to translate to the SAT language than inductive synthesis but takes longer to find a solution, and that synthesising with reuse is faster.

This article is organised as follows. Section 2 presents an Alloy^⁎ specification (syntax, well-formedness rules, semantics and the synthesis predicate) corresponding to a deductive synthesiser. Section 3 shows how to extend this specification to implement an inductive synthesiser as well as the facility of reusing functions. Section 4 describes an extension to deal with arrays. Section 5 augments the Alloy^⁎-based Program Synthesiser (APS) with Genetic Algorithm aspects (APS-GA) and other facilities to ease the user experience. Section 6 performs an empirical evaluation discussing: deductive vs inductive approaches; reuse; the use of Alloy^⁎ model combined with Genetic Algorithm strategy; programs with arrays, and the effectiveness of the Genetic Algorithm. Section 7 compares our work with some related studies, and Section 8 concludes and addresses future work.