Skip to main content
SpringerLink
Log in

MolBench: A Benchmark of AI Models for Molecular Property Prediction

  • Conference paper
  • First Online:
Benchmarking, Measuring, and Optimizing (Bench 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14521))

Included in the following conference series:

  • 128 Accesses

Abstract

In recent years, there has been a growing demand for the prediction of complex molecular properties in the fields of drug design, material science, and biotechnology. Compared to traditional laboratory methods, the deep learning method has many advantages such as saving enormously time and money. The deep learning method achieves revolutionary success in predicting molecular properties and many models based on the deep learning method has been developed in this field. However, there still lacks reliable and multidimensional benchmarks for evaluating these artificial intelligence (AI) models. In this paper, we develop a general method to evaluate AI models for predicting molecular properties. More precisely, we design multiple evaluation metrics based on the MoleculeNet datasets and introduce an extensible API interface to benchmark three types of AI models: molecular fingerprint based models, graph-based models, and pre-trained models. The purpose of the work is to establish a fair and reliable benchmark for future innovation in the field of molecular property prediction, emphasizing the importance of multidimensional perspectives.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://wiki.nci.nih.gov/display/NCIDTPdata/AIDS+Antiviral+Screen+Data.

  2. 2.

    https://tripod.nih.gov/tox21/challenge/.

References

  1. Abdi, H.: Coefficient of variation. Encycl. Res. Des. 1(5) (2010)

    Google Scholar 

  2. Artemov, A.V., Putin, E., Vanhaelen, Q., Aliper, A., Ozerov, I.V., Zhavoronkov, A.: Integrated deep learned transcriptomic and structure-based predictor of clinical trials outcomes. BioRxiv, p. 095653 (2016)

    Google Scholar 

  3. Blum, L.C., Reymond, J.L.: 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc. 131(25), 8732–8733 (2009)

    Article  Google Scholar 

  4. Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)

    Article  Google Scholar 

  5. Chen, T., Guestrin, C.: Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)

    Google Scholar 

  6. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)

    Article  Google Scholar 

  7. Delaney, J.S.: ESOL: estimating aqueous solubility directly from molecular structure. J. Chem. Inf. Comput. Sci. 44(3), 1000–1005 (2004)

    Article  Google Scholar 

  8. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database, pp. 248–255 (2009)

    Google Scholar 

  9. Dowden, H., Munro, J.: Trends in clinical success rates and therapeutic focus. Nat. Rev. Drug Discov. 18(7), 495–496 (2019)

    Article  Google Scholar 

  10. Dunn, A., Wang, Q., Ganose, A., Dopp, D., Jain, A.: Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. NPJ Comput. Mater. 6(1), 138 (2020)

    Google Scholar 

  11. Fang, X., et al.: Geometry-enhanced molecular representation learning for property prediction. Nat. Mach. Intell. 4(2), 127–134 (2022)

    Article  Google Scholar 

  12. Gardner, M.W., Dorling, S.: Artificial neural networks (the multilayer perceptron)-a review of applications in the atmospheric sciences. Atmos. Environ. 32(14–15), 2627–2636 (1998)

    Article  Google Scholar 

  13. Gasteiger, J., Groß, J., Günnemann, S.: Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123 (2020)

  14. Gaulton, A., et al.: ChEMBL: a large-scale bioactivity database for drug discovery. Nucleic Acids Res. 40(D1), D1100–D1107 (2012)

    Article  Google Scholar 

  15. Gaulton, A., et al.: A large-scale crop protection bioassay data set. Sci. Data 2(1), 1–7 (2015)

    Article  Google Scholar 

  16. Gayvert, K.M., Madhukar, N.S., Elemento, O.: A data-driven approach to predicting successes and failures of clinical trials. Cell Chem. Biol. 23(10), 1294–1301 (2016)

    Article  Google Scholar 

  17. Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: International Conference on Machine Learning, pp. 1263–1272. PMLR (2017)

    Google Scholar 

  18. Hay, M., Thomas, D.W., Craighead, J.L., Economides, C., Rosenthal, J.: Clinical development success rates for investigational drugs. Nat. Biotechnol. 32(1), 40–51 (2014)

    Article  Google Scholar 

  19. Hirohara, M., Saito, Y., Koda, Y., Sato, K., Sakakibara, Y.: Convolutional neural network based on smiles representation of compounds for detecting chemical motif. BMC Bioinform. 19, 83–94 (2018)

    Article  Google Scholar 

  20. Hu, W., et al.: Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 (2019)

  21. Jiao, R., Han, J., Huang, W., Rong, Y., Liu, Y.: Energy-motivated equivariant pretraining for 3D molecular graphs. arXiv preprint arXiv:2207.08824 (2022)

  22. Kearnes, S., McCloskey, K., Berndl, M., Pande, V., Riley, P.: Molecular graph convolutions: moving beyond fingerprints. J. Comput. Aided Mol. Des. 30, 595–608 (2016)

    Article  Google Scholar 

  23. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)

  24. Kuhn, M., Letunic, I., Jensen, L.J., Bork, P.: The sider database of drugs and side effects. Nucleic Acids Res. 44(D1), D1075–D1079 (2016)

    Article  Google Scholar 

  25. Li, Y., et al.: MUBen: benchmarking the uncertainty of pre-trained models for molecular property prediction. arXiv preprint arXiv:2306.10060 (2023)

  26. Liu, S., Guo, H., Tang, J.: Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint arXiv:2206.13602 (2022)

  27. Liu, Y., Wang, L., Liu, M., Zhang, X., Oztekin, B., Ji, S.: Spherical message passing for 3D graph networks. arXiv preprint arXiv:2102.05013 (2021)

  28. Martins, I.F., Teixeira, A.L., Pinheiro, L., Falcao, A.O.: A Bayesian approach to in silico blood-brain barrier penetration modeling. J. Chem. Inf. Model. 52(6), 1686–1697 (2012)

    Article  Google Scholar 

  29. Mobley, D.L., Guthrie, J.P.: Freesolv: a database of experimental and calculated hydration free energies, with input files. J. Comput. Aided Mol. Des. 28, 711–720 (2014)

    Article  Google Scholar 

  30. Mobley, D.L., Wymer, K.L., Lim, N.M., Guthrie, J.P.: Blind prediction of solvation free energies from the sampl4 challenge. J. Comput. Aided Mol. Des. 28, 135–150 (2014)

    Article  Google Scholar 

  31. Montavon, G., et al.: Machine learning of molecular electronic properties in chemical compound space. New J. Phys. 15(9), 095003 (2013)

    Article  Google Scholar 

  32. Ramakrishnan, R., Dral, P.O., Rupp, M., Von Lilienfeld, O.A.: Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1(1), 1–7 (2014)

    Article  Google Scholar 

  33. Ramakrishnan, R., Hartmann, M., Tapavicza, E., Von Lilienfeld, O.A.: Electronic spectra from TDDFT and machine learning in chemical space. J. Chem. Phys. 143(8) (2015)

    Google Scholar 

  34. Richard, A.M., et al.: Toxcast chemical landscape: paving the road to 21st century toxicology. Chem. Res. Toxicol. 29(8), 1225–1251 (2016)

    Article  Google Scholar 

  35. Rogers, D., Hahn, M.: Extended-connectivity fingerprints. J. Chem. Inf. Model. 50(5), 742–754 (2010)

    Article  Google Scholar 

  36. Rong, Y., et al.: Self-supervised graph transformer on large-scale molecular data. Adv. Neural Inf. Process. Syst. 33, 12559–12571 (2020)

    Google Scholar 

  37. Rupp, M., Tkatchenko, A., Müller, K.R., Von Lilienfeld, O.A.: Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett. 108(5), 058301 (2012)

    Article  Google Scholar 

  38. Schütt, K., Kindermans, P.J., Sauceda Felix, H.E., Chmiela, S., Tkatchenko, A., Müller, K.R.: Schnet: a continuous-filter convolutional neural network for modeling quantum interactions. Adv. Neural Inf. Process. Syst. 30 (2017)

    Google Scholar 

  39. Sterling, T., Irwin, J.J.: Zinc 15-ligand discovery for everyone. J. Chem. Inf. Model. 55(11), 2324–2337 (2015)

    Article  Google Scholar 

  40. Subramanian, G., Ramsundar, B., Pande, V., Denny, R.A.: Computational modeling of \(\beta \)-secretase 1 (BACE-1) inhibitors using ligand based approaches. J. Chem. Inf. Model. 56(10), 1936–1949 (2016)

    Article  Google Scholar 

  41. Suresh, S., Li, P., Hao, C., Neville, J.: Adversarial graph augmentation to improve graph contrastive learning. Adv. Neural Inf. Process. Syst. 34, 15920–15933 (2021)

    Google Scholar 

  42. Takamoto, M., et al.: Pdebench: an extensive benchmark for scientific machine learning. Adv. Neural Inf. Process. Syst. 35, 1596–1611 (2022)

    Google Scholar 

  43. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)

  44. Wang, Y., Wang, J., Cao, Z., Farimani, A.: MolCLR: molecular contrastive learning of representations via graph neural networks. arxiv 2021. arXiv preprint arXiv:2102.10056

  45. Wang, Y., Wang, J., Cao, Z., Barati Farimani, A.: Molecular contrastive learning of representations via graph neural networks. Nat. Mach. Intell. 4(3), 279–287 (2022)

    Article  Google Scholar 

  46. Watanabe, N., Ohnuki, Y., Sakakibara, Y.: Deep learning integration of molecular and interactome data for protein-compound interaction prediction. J. Cheminformatics 13(1), 36 (2021)

    Article  Google Scholar 

  47. Wu, Z., et al.: Moleculenet: a benchmark for molecular machine learning. Chem. Sci. 9(2), 513–530 (2018)

    Article  Google Scholar 

  48. Xia, J., et al.: Mole-BERT: rethinking pre-training graph neural networks for molecules. In: The Eleventh International Conference on Learning Representations (2022)

    Google Scholar 

  49. Xiong, Z., et al.: Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. J. Med. Chem. 63(16), 8749–8760 (2019)

    Article  Google Scholar 

  50. Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018)

  51. Yang, K., et al.: Analyzing learned molecular representations for property prediction. J. Chem. Inf. Model. 59(8), 3370–3388 (2019)

    Article  Google Scholar 

  52. Zaidi, S., et al.: Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133 (2022)

  53. Zhou, G., et al.: Uni-Mol: a universal 3D molecular representation learning framework (2023)

    Google Scholar 

Download references

Acknowledgements

The research was supported in part by the National Key Research and Program of China (2022ZD0117805), by the National Natural Science Foundation of China under grants 12071496 and 92370113, and by the Natural Science Foundation of the Guangdong Province under the grant 2023A1515012079.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China

    Xiuyu Jiang, Liqin Tan, Jianhuan Cen & Qingsong Zou

  2. Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Guangzhou, 510006, China

    Qingsong Zou

Authors
  1. Xiuyu Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liqin Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianhuan Cen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qingsong Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingsong Zou .

Editor information

Editors and Affiliations

  1. TU Wien, Vienna, Austria

    Sascha Hunold

  2. Chinese Academy of Sciences, Beijing, China

    Biwei Xie

  3. Illinois Institute of Technology, Chicago, IL, USA

    Kai Shu

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jiang, X., Tan, L., Cen, J., Zou, Q. (2024). MolBench: A Benchmark of AI Models for Molecular Property Prediction. In: Hunold, S., Xie, B., Shu, K. (eds) Benchmarking, Measuring, and Optimizing. Bench 2023. Lecture Notes in Computer Science, vol 14521. Springer, Singapore. https://doi.org/10.1007/978-981-97-0316-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-0316-6_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-0315-9

  • Online ISBN: 978-981-97-0316-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation