Automating Genetic Algorithm Mutations for Molecules Using a Masked Language Model

Blanchard, Andrew; Chandra Shekar, Mayanka; Gao, Shang; Gounley, John; Lyngaas, Isaac; Glaser, Jens; Bhowmik, Debsindhu

doi:10.1109/TEVC.2022.3144045

Title: Automating Genetic Algorithm Mutations for Molecules Using a Masked Language Model

Journal Article · Thu Jan 20 00:00:00 EST 2022 · IEEE Transactions on Evolutionary Computation

DOI:https://doi.org/10.1109/TEVC.2022.3144045· OSTI ID:1845799

Blanchard, Andrew ^[1];

^[1]; Gao, Shang ^[1]; Gounley, John ^[1]; Lyngaas, Isaac ^[1];

^[1];

^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Inspired by the evolution of biological systems, genetic algorithms have been applied to generate solutions for optimization problems in a variety of scientific and engineering disciplines. For a given problem, a suitable genome representation must be defined along with a mutation operator to generate subsequent generations. Unlike natural systems which display a variety of complex rearrangements (e.g. mobile genetic elements), mutation for genetic algorithms commonly utilizes only random point-wise changes. Furthermore, generalizing beyond point-wise mutations poses a key difficulty as useful genome rearrangements depend on the representation and problem domain. To move beyond the limitations of manually defined point-wise changes, here we propose the use of techniques from masked language models to automatically generate mutations. As a first step, common subsequences within a given population are used to generate a vocabulary. The vocabulary is then used to tokenize each genome. A masked language model is trained on the tokenized data in order to generate possible rearrangements (i.e. mutations). In order to illustrate the proposed strategy, we use string representations of molecules and use a genetic algorithm to optimize for drug-likeness and synthesizability. Finally, our results show that moving beyond random point-wise mutations accelerates genetic algorithm optimization.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC05-00OR22725; AC02-06CH11357; AC52-07NA27344; AC5206NA25396

OSTI ID:: 1845799

Journal Information:: IEEE Transactions on Evolutionary Computation, Vol. 26, Issue 4; ISSN 1089-778X

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (30)

ZeRO: Memory optimizations Toward Training Trillion Parameter Models Rajbhandari, Samyam; Rasley, Jeff; Ruwase, Olatunji SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00024	conference	November 2020
Transformers: State-of-the-Art Natural Language Processing Wolf, Thomas; Debut, Lysandre; Sanh, Victor Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations https://doi.org/10.18653/v1/2020.emnlp-demos.6	conference	January 2020
Mapping the space of chemical reactions using attention-based neural networks Schwaller, Philippe; Probst, Daniel; Vaucher, Alain C. Nature Machine Intelligence, Vol. 3, Issue 2 https://doi.org/10.1038/s42256-020-00284-w	journal	January 2021
Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction Schwaller, Philippe; Laino, Teodoro; Gaudin, Théophile ACS Central Science, Vol. 5, Issue 9 https://doi.org/10.1021/acscentsci.9b00576	journal	August 2019
Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions Ertl, Peter; Schuffenhauer, Ansgar Journal of Cheminformatics, Vol. 1, Issue 1 https://doi.org/10.1186/1758-2946-1-8	journal	June 2009
Quantifying the chemical beauty of drugs Bickerton, G. Richard; Paolini, Gaia V.; Besnard, Jérémy Nature Chemistry, Vol. 4, Issue 2 https://doi.org/10.1038/nchem.1243	journal	January 2012
BERT-ATTACK: Adversarial Attack Against BERT Using BERT Li, Linyang; Ma, Ruotian; Guo, Qipeng Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) https://doi.org/10.18653/v1/2020.emnlp-main.500	conference	January 2020
The Molecule Evoluator. An Interactive Evolutionary Algorithm for the Design of Drug-Like Molecules Lameijer, Eric-Wubbo; Kok, Joost N.; Bäck, Thomas Journal of Chemical Information and Modeling, Vol. 46, Issue 2 https://doi.org/10.1021/ci050369d	journal	January 2006
High Performance I/O For Large Scale Deep Learning Aizman, Alex; Maltby, Gavin; Breuel, Thomas 2019 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData47090.2019.9005703	conference	December 2019
De Novo Drug Design Using Multiobjective Evolutionary Graphs Nicolaou, Christos A.; Apostolakis, Joannis; Pattichis, Costas S. Journal of Chemical Information and Modeling, Vol. 49, Issue 2 https://doi.org/10.1021/ci800308h	journal	January 2009
Mining a Chemical Database for Fragment Co-occurrence: Discovery of “Chemical Clichés” Lameijer, Eric-Wubbo; Kok, Joost N.; Bäck, Thomas Journal of Chemical Information and Modeling, Vol. 46, Issue 2 https://doi.org/10.1021/ci050370c	journal	January 2006
Japanese and Korean voice search Schuster, Mike; Nakajima, Kaisuke 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP.2012.6289079	conference	March 2012
SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules Weininger, David Journal of Chemical Information and Modeling, Vol. 28, Issue 1 https://doi.org/10.1021/ci00057a005	journal	February 1988
Simple Evolutionary Optimization Can Rival Stochastic Gradient Descent in Neural Networks Morse, Gregory; Stanley, Kenneth O. GECCO '16: Genetic and Evolutionary Computation Conference, Proceedings of the Genetic and Evolutionary Computation Conference 2016 https://doi.org/10.1145/2908812.2908916	conference	July 2016
Smiles-Bert Wang, Sheng; Guo, Yuzhi; Wang, Yuhong Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics https://doi.org/10.1145/3307339.3342186	conference	September 2019
Computer-Automated Evolution of an X-Band Antenna for NASA's Space Technology 5 Mission Hornby, Gregory. S.; Lohn, Jason D.; Linden, Derek S. Evolutionary Computation, Vol. 19, Issue 1 https://doi.org/10.1162/EVCO_a_00005	journal	March 2011
Population-based De Novo Molecule Generation, Using Grammatical Evolution Yoshikawa, Naruki; Terayama, Kei; Sumita, Masato Chemistry Letters, Vol. 47, Issue 11 https://doi.org/10.1246/cl.180665	journal	November 2018
Abandoning Objectives: Evolution Through the Search for Novelty Alone Lehman, Joel; Stanley, Kenneth O. Evolutionary Computation, Vol. 19, Issue 2 https://doi.org/10.1162/EVCO_a_00025	journal	June 2011
Stochastic Voyages into Uncharted Chemical Space Produce a Representative Library of All Possible Drug-Like Compounds Virshup, Aaron M.; Contreras-García, Julia; Wipf, Peter Journal of the American Chemical Society, Vol. 135, Issue 19 https://doi.org/10.1021/ja401184g	journal	May 2013
A Graph-Based Genetic Algorithm and Its Application to the Multiobjective Evolution of Median Molecules Brown, Nathan; McKay, Ben; Gilardoni, François Journal of Chemical Information and Computer Sciences, Vol. 44, Issue 3 https://doi.org/10.1021/ci034290p	journal	May 2004
GuacaMol: Benchmarking Models for de Novo Molecular Design Brown, Nathan; Fiscato, Marco; Segler, Marwin H. S. Journal of Chemical Information and Modeling, Vol. 59, Issue 3 https://doi.org/10.1021/acs.jcim.8b00839	journal	October 2018
Introduction to Evolutionary Computing Eiben, A. E.; Smith, J. E. Natural Computing Series https://doi.org/10.1007/978-3-662-44874-8	book	January 2015
Quality and Diversity Optimization: A Unifying Modular Framework Cully, Antoine; Demiris, Yiannis IEEE Transactions on Evolutionary Computation, Vol. 22, Issue 2 https://doi.org/10.1109/TEVC.2017.2704781	journal	April 2018
Mobility of Plasmids Smillie, C.; Garcillan-Barcia, M. P.; Francia, M. V. Microbiology and Molecular Biology Reviews, Vol. 74, Issue 3 https://doi.org/10.1128/MMBR.00020-10	journal	August 2010
Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks Segler, Marwin H. S.; Kogej, Thierry; Tyrchan, Christian ACS Central Science, Vol. 4, Issue 1 https://doi.org/10.1021/acscentsci.7b00512	journal	December 2017
“Found in Translation”: predicting outcomes of complex organic chemistry reactions using neural sequence-to-sequence models Schwaller, Philippe; Gaudin, Théophile; Lányi, Dávid Chemical Science, Vol. 9, Issue 28 https://doi.org/10.1039/C8SC02339E	journal	January 2018
Using GANs with adaptive training data to search for new molecules Blanchard, Andrew E.; Stanley, Christopher; Bhowmik, Debsindhu Journal of Cheminformatics, Vol. 13, Issue 1 https://doi.org/10.1186/s13321-021-00494-3	journal	February 2021
Bidirectional Molecule Generation with Recurrent Neural Networks Grisoni, Francesca; Moret, Michael; Lingwood, Robin Journal of Chemical Information and Modeling, Vol. 60, Issue 3 https://doi.org/10.1021/acs.jcim.9b00943	journal	January 2020
Extended-Connectivity Fingerprints Rogers, David; Hahn, Mathew Journal of Chemical Information and Modeling, Vol. 50, Issue 5 https://doi.org/10.1021/ci100050t	journal	April 2010
Randomized SMILES strings improve the quality of molecular generative models Arús-Pous, Josep; Johansson, Simon Viet; Prykhodko, Oleksii Journal of Cheminformatics, Vol. 11, Issue 1 https://doi.org/10.1186/s13321-019-0393-0	journal	November 2019

Similar Records

Adaptive language model training for molecular design

Journal Article · Thu Jun 08 00:00:00 EDT 2023 · Journal of Cheminformatics · OSTI ID:1845799

Blanchard, Andrew E.; Bhowmik, Debsindhu; Fox, Zachary; +4 more

Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports

Journal Article · Sun Apr 15 00:00:00 EDT 2018 · Journal of Digital Imaging (Online) · OSTI ID:1845799

Chen, Po-Hao; Zafar, Hanna; Galperin-Aizenberg, Maya; +1 more

Investigation of the mutagenic specificity of x-ray using a retroviral shuttle vector in CHO cells

Journal Article · Wed Jan 01 00:00:00 EST 1992 · Environmental and Molecular Mutagenesis; (United States) · OSTI ID:1845799

Skandalis, A; Grosovsky, A J; Drobetsky, E A; +1 more

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
genetic algorithms
vocabulary
training
predictive models
optimization
tokenization
bit error rate

Title: Automating Genetic Algorithm Mutations for Molecules Using a Masked Language Model

Citation Formats

References (30)

Similar Records

Related Subjects