Valid Text-to-SQL Generation with Unification-Based DeepStochLog

Abstract

Large language models have been used to translate natural language questions to SQL queries. Without hard constraints on syntax and database schema, they occasionally produce invalid queries that are not executable. These failures limit the usage of these systems in real-life scenarios. We propose a neurosymbolic framework that imposes SQL syntax and schema constraints with unification-based definite clause grammars and thus guarantees the generation of valid queries. Our framework also builds a bi-directional interface to language models to leverage their natural language understanding abilities. The evaluation results on a subset of SQL grammars show that all our output queries are valid. This work is the first step towards extending language models with unification-based grammars. We demonstrate this extension enhances the validity, execution accuracy, and ground truth alignment of the underlying language model by a large margin. Our code is available at https://github.com/ML-KULeuven/deepstochlog-lm.

Appendices

A Task 1

1.1 A.1 Logic Program

Task 1 grammar is modeled as follows:

1.2 A.2 Training and Evaluation

We train settings 1 and 2 end-to-end for 7 epochs using the Adam optimizer [10] with a batch size of 8 and a learning rate of $1e^{-3}$. For evaluation, we employ DeepStochLog inference with the $(\max , \times )$ semiring, i.e. the exact inference. Examples of $column\_lm$ and $selection\_lm$ inputs are listed in Table 7. The inputs of $table\_lm$ have the same format as those of $column\_lm$.

Pre- and Post-processing. In pre-processing, we tokenize the ground truth SQL queries to sequences with list format. Table and column identifiers in the sequences are replaced by their semantic names [12]. The semantics names provided in Spider [34] are closer to natural expressions, which facilitate the understanding of language models. This replacement is also conducted for the facts in logic programs. After generation, we restore the identifiers in the output sequence to their original names and join the sequence to get the SQL query. This pre- and post-progressing are also performed in task 2 experiments.

B Task 2

1.1 B.1 Logic Program

Task 2 grammar covers two types of queries: 1) single selection and 2) two selections connected by the set operator “EXCEPT". The grammar for single-selection queries covers the “SELECT", “WHERE", “GROUP BY", and “ORDER BY" clauses including “DISTINCT" and aggregation functions in the “SELECT" clause, “HAVING" in the “GROUP BY" clause, and “ASC / DESC" and “LIMIT" in the “ORDER BY" clause. “WHERE" and “HAVING" allow one condition. For type 2), the two selection clauses have the format “SELECT [column] FROM [table]". The columns in the two selection clauses are currently restricted to foreign keys linking the two tables. Task 2 grammar is modeled as follows:

1.2 B.2 Underlying Language Models

We employ 11 fine-tuned T5-small models. All models used in this work are from Huggingface [28].

$table\_lm$ and $column\_lm$ provide the probability distribution over the given table and column domain respectively. Their output domains vary with the database. The output domain of other language models is fixed. $except\_lm$, $where\_lm$, $groupby\_lm$, $having\_lm$, $order\_lm$, $desc\_lm$, $limit\_lm$ are used to produce the probability distribution on the existence of the corresponding SQL clause. $selection\_lm$ provides the probability distribution over 10 possible selection branches. $operator\_lm$ outputs the probability distribution over possible operators in “WHERE" and “HAVING" conditions. For the queries with two selection clauses connected by “EXCEPT", we call $table\_lm$ twice to get probability distributions over the possible substitutions for the table in each selection clause. The pair of columns in each selection clause are decided deterministically based on the pair of tables. Our current framework does not predict any value in SQL conditions. We assume the gold values are given. When we predict wrong conditions that cannot be mapped to the gold values, we assign the values to 1.

$table\_lm$, $column\_lm$, and $opertor\_lm$ can be used at different positions. $table\_lm$ can be called twice for the selection clause before “EXCEPT" and the one after “EXCEPT". $column\_lm$ is used for the column in the “SELECT", “WHERE", “GROUP BY" and “ORDER BY" clauses. $operator\_lm$ is used for operators in “WHERE" and “HAVING" conditions. We add states in their inputs to help them distinguish different cases. Examples of $column\_lm$ inputs are shown in Table 7 to showcase the inputs with states. We also include examples of $where\_lm$ and $selection\_lm$ inputs in Table 7. $except\_lm$, $where\_lm$, $groupby\_lm$, $having\_lm$, $order\_lm$, $desc\_lm$, $limit\_lm$ shares a similar input format as $where\_lm$.

1.3 B.3 Training and Evaluation

In the text-to-SQL task, the grammar can always be written unambiguously, which leads to only one possible derivation for the ground-truth query Q. With the negative log-likelihood loss function and all positive samples in dataset $\mathcal {D}$ (target probability $t_i$=1.0),

$$\begin{aligned} \begin{aligned} (1) &= \min _{p} \sum \nolimits _{(G_i\theta _i, Q_i) \in D} -log(\prod \nolimits _{r_j \in d(G_i\theta _i)=Q_i}p_j^{k_j}) \\ &= \sum \nolimits _{(G_i\theta _i, Q_i) \in D} \sum \nolimits _{r_j \in d(G_i\theta _i)=Q_i} \min _{p_j} -k_jlogp_j \end{aligned} \end{aligned}$$

As all the intermediate goals are observable given Q, the learning problem collapses to supervised training.

We fine-tune the T5-small models using the Adam optimizer. Table 6 shows the number of fine-tuning epochs, the batch size, and the learning rate for each model. The hyper-parameters are determined with cross-validation.

In evaluation, we perform the greedy inference to speed up the inference process. Instead of considering the re-normalized distributions obtained from the language models, the greedy inference takes the substitution with the largest probability.

Table 6. Fine-tuning hyper-parameters of our T5-small models.

C Baselines

We fine-tune the vanilla T5-small baseline for 10 epochs using the Adam optimizer, a batch size of 32, and a learning rate $1e^{-3}$. T5-small + CFGs shares the same language models with ours T5-small + DCGs except the $column\_lm$. Without table unification, the domain of $column\_lm$ used in T5-small + CFGs covers all the columns in a given database. We fine-tune a T5-small model for $column\_lm$ in T5-small + CFGs using the Adam optimizer for 13 epochs with a batch size of 32, and a learning rate $5e^{-4}$. Table 8 shows the example inputs for T5-small and $column\_lm$ in T5-small + CFGs.

For the state-of-the-art models (DAIL-SQL [7], DIN-SQL [18], Graphix-T5 [13], and C3 [6]), we extract their predictions on the samples in our evaluation set from their official results for the full Spider development set [34].

Table 7. Examples of inputs of our T5-small models.

Table 8. Examples of baseline inputs.