Embed2Rule Scalable Neuro-Symbolic Learning via Latent Space Weak-Labelling

Abstract

Neuro-symbolic approaches have garnered much interest recently as a path toward endowing neural systems with robust reasoning capabilities. Most proposed end-to-end methods assume knowledge to be given in advance and do not scale up over many latent concepts. The recently proposed Embed2Sym tackles the scalability limitation by performing end-to-end neural training of a visual perception component from downstream labels to generate clusters in the latent space of symbolic concepts. These are later used to perform downstream symbolic reasoning but symbolic knowledge is still engineered. Taking inspiration from Embed2Sym, this paper introduces a novel method for scalable neuro-symbolic learning of first-order logic programs from raw data. The learned clusters are optimally labelled using sampled predictions of a pre-trained vision-language model. A SOTA symbolic learner, robust to noise, uses these labels to learn an answer set program that solves the reasoning task. Our approach, called Embed2Rule, is shown to achieve better accuracy than SOTA neuro-symbolic systems on existing benchmark tasks in most cases while scaling up to tasks that require far more complex reasoning and a large number of latent concepts.

Appendices

A Datasets

In this appendix, we describe the datasets used throughout the experiments described in the main paper.

1.1 A.1 Raw Datasets

For raw data (images), we use the standard MNIST [26] dataset as provided by the PyTorch module, and the Cards dataset by Cunnington et al. [5]. The Cards dataset is publicly available at: https://github.com/DanCunnington/FFNSL. For both datasets, we split the training images into a training and validation set, while using the test images as the test set.

Hitting Sets. For this task, we generated a dataset of 200K training samples, 1K validation samples, and 1K test samples for each case. The generation process was as follows: We randomly generated collections of sets of a given size (4-6) with a given number of elements (5 or 10) and checked if a hitting set of size 2 or less exists. We split the generated collections into training, validation and test sets with an equal number of positive and negative samples, ensuring there is no overlap between the sets. Then, we assigned each of the elements in a sample to a random image from MNIST with a label corresponding to that element, and from the appropriate split (train/validation/test). Note that MNIST images may repeat across different samples from the same split.

Visual Sudoku Classification. For the 4 $\times$ 4 cases (Fixed and Random), we generated 50K training samples, 5K validation samples and 5K test samples. For the 9 $\times$ 9 case, we generated 200K training samples, 20K validation and 20K test samples. When generating samples, we followed a process similar to Augustine et al. [2]. We began by generating valid boards and then corrupted half to generate negative samples. There are two types of corruption: 1. Replacement - a cell is randomly chosen and its content is replaced with a different digit. 2. Substitution - two cells are randomly chosen and their contents are swapped. We apply at least one corruption (at random) to each board and then flip a (biased) coin to decide if further corruptions are to be applied. If we get heads, we apply another random corruption and flip again until we get tails. We then checked to see if the multiple corruptions did not accidentally create a valid board. In the 4 $\times$ 4 cases, the coin has a bias of 0 (always landing on tails) so the board contains a single corruption. For the 9 $\times$ 9 Fixed case, we used a bias of 0.75, so a board contains, on average, 4 corruptions. The generated (valid and invalid) puzzles are then split into a training, validation and test set, with an equal number of positive and negative samples, and ensuring no overlap. Assigning MNIST images to cells followed the same procedure as Hitting Sets.

Follow Suit Winner. We generated 200K training samples, 1K validation and 1K test samples. The generation process is done by continuously simulating games and recording the winner. The process of assigning images to cards is similar to the other tasks.

B Training Details

In this section, we provide further details on the training of the Embed2Rule model and baselines in the various experiments described in the evaluation section of the paper.

1.1 B.1 Training Hardware

We trained on a system with an Intel i7-12700K @ 3600 MHz, 32 GB of RAM and a Nvidia GeForce RTX 3090.

1.2 B.2 Embed2Rule Hyperparameters

In all tasks, K-Means is trained with 1000 random samples from the training set, and another 1024 random samples are given to BLIP-2 for weak-labelling.

Hitting Sets. We employ LeNet-5 [25] as a perception network with the output layer changed to size 32 with ReLU activation. The output embeddings are concatenated and fed into the reasoning network, an MLP with residual connections. This MLP is comprised of a stack of residual blocks. Each block is a fully connected layer with an output size of 128, followed by a GELU activation, batch norm and finally adding the block input. We use 4 such residual blocks. The network is trained for 100 epochs (50 in the 5-4 case) with a learning rate of 0.0009 and a batch size of 256 using the Adam optimiser [18]. The prompt for BLIP-2 was simply the digit in numerical form. We set a uniform penalty of 100 to the ILASP examples and set a maximum rule length of 10 (to allow learning long choice rules). We provide ILASP with 50 examples from the training set.

Visual Sudoku Classification. We utilise a CNN similar to that used in SATNet [41] for the perception network containing 2 convolutional and max-pooling layers, followed by 2 feed-forward layers with ReLU activations functions. We use a transformer encoder [40] for the reasoning network employing multi-headed self-attention with learnable positional embeddings. We set the number of attention heads to 4, the number of encoder blocks to 1, the feed-forward layer embedding dimensions to 512, the activation functions to GELU, and a dropout value of 0.1. The inputs and outputs of the transformer are embeddings of dimensions $batch \times seq \times emb$, where batch is the batch size fixed to 512, seq is the sequence length set to 16 and 81 for the 4 $\times$ 4 and 9 $\times$ 9 Sudoku boards, respectively, and emb is the dimensions of the individual latent concept embeddings set to 128. We apply a single-layer classification head with a Sigmoid activation function to the layer-normalized outputs of the transformer to obtain the neural predictions. We train the network for 50 epochs using an AdamW optimizer with a linear warm-up phase from 0.0 to a base learning rate of 0.003 for 30% and 20% of the total number of iterations for the 4 $\times$ 4 and 9 $\times$ 9 boards, respectively, followed by a cosine annealing learning rate schedule between the base learning rate and 0.0001. We clip the gradient norm of each layer at 1.0. The prompt for BLIP-2 is again the digit in numerical form. We give ILASP 100 training examples with a uniform penalty of 1.

Follow Suit Winner. We use a ResNet-18 [13] as the perception network, with an output layer of size 128. As a reasoning network, we use a similar MLP architecture to the Hitting Sets task, with 4 residual blocks and a hidden size of 256. We train for 50 epochs with a learning rate of 0.0001 and a batch size of 128. For BLIP-2 we use a prompt of the format “The playing card rank of suit” where rank and suit are replaced with their appropriate values. We provide ILASP with 100 examples with a uniform penalty of 1.

1.3 B.3 Baseline Hyperparameters

We use the official implementations of NSIL and SLASH in our experiments, using the same perception networks as Embed2Rule. As both NSIL and SLASH are more data efficient than Embed2Rule, we used smaller training sets of 10K samples to avoid artificially increasing training time. We found no gain from using additional data.

NSIL. For Hitting Sets, we follow the same hyperparameters used in the official implementation. NSIL is trained for 20 iterations using the SGD optimiser with a learning rate of 0.0008, momentum of 0.7643 and a batch size of 64. The $\lambda$ value (that affects the weight of neural network confidence in symbolic learning) is set to 1. For Visual Sudoku Classification and Follow Suit Winner, no set of hyperparameters affects the result, as NSIL could not solve either. For Sudoku, as the number of iterations did not change the performance of NSIL, we reported training time results when using 5 iterations.

SLASH. For Visual Sudoku Classification, we trained for 20 epochs using the Adam optimiser with a learning rate of 0.001 and a batch size of 100 (for 4 $\times$ 4 random and 9 $\times$ 9 we attempted to use batch sizes as small as 8 to overcome the memory issues, but found it did not help. Due to a bug in the implementation, batch sizes smaller than 8 are not possible.). For the Hitting Sets task, we found no set of hyperparameters that would allow SLASH to learn from positive samples alone, and so we report results for 5 epochs of training.

C Symbolic Learning Tasks

We include here the background knowledge and mode declarations given to ILASP in each task.

D Induced Programs

We include here some of the answer set programs induced by Embed2Rule (using ILASP) to solve the reasoning tasks in the paper.