BiG2S: A dual task graph-to-sequence model for the end-to-end template-free reaction prediction

Abstract

Retrosynthesis and reaction outcome prediction are fundamental problems in organic chemistry and computer-aided synthesis planning (CASP), which are also crucial parts of computer-aided drug design. In recent years, deep learning has spawned a branch of methods which use machine translation frameworks with SMILES data representation to solve these problems. With the successive introduction of additional inverted reaction data as well as the molecular graph representation, the accuracy and validity of machine-transaction-based approaches have been further improved. In this work, we propose a bidirectional graph-to-sequence model (BiG2S) that combines the benefits of inverted reaction training and graph representation. The proposed approach has the ability to provide high-quality retrosynthesis and forward synthesis prediction simultaneously on various datasets, which achieves $5.5\%$ top-1 accuracy with only $0.1\%$ invalid results on USPTO-50k retrosynthesis task, and maintain $85.0\%$ top-1 accuracy for outcome prediction with the same model.

Graphical abstract

Appendices

Appendix A: Feature and parameter settings

Table 10 summarizes the atom and bond features used in BiG2S, the majority of them are adapted from Graph2SMILES [15]. To support the dual-task capability of the model, additional task labels and optional reaction type information have been added to the atom features.

Table 10 Atom and bond features used in model

Table 11 Initialization for BiG2S

BiG2S introduces a special initialization method with the normalization of each layer in Transformer which are based on DeepNet [30]. Since the output of GLU [28] is significantly smaller in variance than the vanilla FFN [17] when the input data has the same distribution (e.g., standard normal distribution), BiG2S performs different initializations for each of the two types of structures above. The initialization and normalization methods are shown in Tables 11 and 12, where $\textbf{W}_{query}$, $\textbf{W}_{key}$, $\textbf{W}_{value}$, and $\textbf{W}_{out}$ denote the learnable weights of query, key, value, and the final output in attention layer; $\textbf{W}_{FFN}$ and $\textbf{W}_{GLU}$ denote the learnable weights of all linear layers in these two structures; N and M correspond to the number of layers of encoder and decoder, respectively.

Table 12 Normalization for BiG2S

Additionally, the hyperparameter settings for each dataset are shown in Table 13, where the primary batch loading type for BiG2S is the number of chemical reactions, with additional reactants and product token counts to constrain the batch loading due to the extremely long reactions in USPTO-full.

Table 13 Hyperparameters setting for each dataset

Appendix B: Statistical results during training and visualization results in dual-task inference

The statistical results for the training accuracy of each molecular structure and the performance variance on the validation set throughout the training are shown in Figs. 4 and 5.

Fig. 4

The change of accuracy and SMILES invalid rate on the USPTO-50k validation set during model training. The vertical axis on the right side is represented by a logarithmic axis from $0\%$ to $10\%$ in order to display the invalid rate of the top-1 result

Fig. 5

The total number of occurrences of all SMILES tokens in the USPTO-50k training set and the predicting accuracy in the training periods. The blue bars inside indicate the number of token occurrences, and the red dashed line indicates the prediction accuracy of each token

Fig. 6

Top-5 results for additional dual-task prediction of reactions in USPTO-50k test set from BiG2S which is trained using USPTO-full. Ethane indicates that the prediction SMILES is invalid

Fig. 7

More results for the additional dual-task prediction. Ethane indicates that the prediction SMILES is invalid

We additionally perform reaction outcome prediction for the products and retrosynthesis for the reactants, as well as the results evaluation and SCScore [48] obtained from ASKCOS [47], the visualization results are shown in Figs. 6 and 7. Notice that when evaluating the retrosynthesis results of reactants, the result will be marked as “Highly ranked in ASKCOS” if its corresponding reactants (as the prediction results in ASKCOS forward synthesis prediction) rank in the top-5 when the results are input into ASKCOS for forward synthesis prediction, which is different from the evaluation of retrosynthesis results of the product that it requires the product to be the top-1 result in the forward synthesis results from ASKCOS (Fig. 2). Since the training of BiG2S only contains data related to the retrosynthesis of products and the reaction outcome prediction for reactants, the quality of model outputs is significantly reduced when these two additional tasks are performed, especially when multiple molecules of reactants are analyzed in retrosynthesis task at the same time, the model can therefore hardly obtain reasonable prediction results.

From the retrosynthesis results of the reactants, it is noticeable that the model attempts to generate the corresponding prediction results for each input molecule (such as the decomposition of 1-bromobut-2-yne in Fig. 7), but it is limited by the fact that this task is not involved in the training, which results in a considerable part of the molecules being the same as inputs or directly taking the product of the original reaction as one of the results. The retrosynthesis results are more reasonable when each molecule in reactants is input separately, some of which can further achieve a high ranking in the evaluation of ASKCOS. When performing forward synthesis prediction on each product, the results depend mainly on the bias of the model for various reaction types, reaction centers, and functional groups during training due to the lack of constraints and guidelines from other reactants.