Skip to main content
SpringerLink
Log in
Book cover

International Conference on Service-Oriented Computing

ICSOC 2021: Service-Oriented Computing pp 464–478Cite as

ServiceBERT: A Pre-trained Model for Web Service Tagging and Recommendation

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 13121))

Abstract

Pre-trained models have shown their significant values on a number of natural language processing (NLP) tasks. However, there is still a lack of corresponding work in the field of service computing to effectively utilize the rich knowledge accumulated in the Web service ecosystem. In this paper, we propose ServiceBERT, which learns domain knowledge of Web service ecosystem aiming to support service intelligence tasks, such as Web API tagging and Mashup-oriented API recommendation. The ServiceBERT is developed with the Transformer-based neural architecture. In addition to using the objective of masked language modeling (MLM), we also introduce the replaced token detection (RTD) objective for efficiently learning pre-trained model. Finally, we also implement the contrastive learning to learn noise-invariant representations at the sentence level in pre-training stage. Comprehensive experiments on two service-related tasks successfully demonstrate the better performance of ServiceBERT through the comparison with a variety of representative methods.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://www.programmableweb.com/.

  2. 2.

    https://pytorch.org/.

  3. 3.

    http://www.nltk.org/.

  4. 4.

    https://nlp.stanford.edu/projects/glove/.

References

  1. Barros, A.P., Dumas, M.: The rise of web service ecosystems. Science and Engineering Faculty (2006)

    Google Scholar 

  2. Cao, Y., Liu, J., Cao, B., Shi, M., Wen, Y., Peng, Z.: Web services classification with topical attention based BI-LSTM. In: International Conference on Collaborative Computing: Networking, Applications and Worksharing, pp. 394–407 (2019)

    Google Scholar 

  3. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML 2020: 37th International Conference on Machine Learning, vol. 1, pp. 1597–1607 (2020)

    Google Scholar 

  4. Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: ELECTRA: pre-training text encoders as discriminators rather than generators. In: ICLR 2020 : Eighth International Conference on Learning Representations (2020)

    Google Scholar 

  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.N.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2018)

    Google Scholar 

  6. Devries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)

  7. Fang, H., Xie, P.: CERT: contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766 (2020)

  8. Fletcher, K.K.: An attention model for mashup tag recommendation. In: International Conference on Services Computing, pp. 50–64 (2020)

    Google Scholar 

  9. Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: International Conference on Learning Representations (2018)

    Google Scholar 

  10. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  11. Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), vol. 1, pp. 2227–2237 (2018)

    Google Scholar 

  12. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  13. Saied, M.A., Raelijohn, E., Batot, E., Famelis, M., Sahraoui, H.A.: Towards assisting developers in API usage by automated recovery of complex temporal patterns. Inf. Softw. Technol. 119(119), 106213 (2020)

    Google Scholar 

  14. Shi, M., Tang, Y., Liu, J.: Functional and contextual attention-based LSTM for service recommendation in mashup creation. IEEE Trans. Parallel Distrib. Syst. 30(5), 1077–1090 (2019)

    Article  Google Scholar 

  15. Shi, M., Tang, Y., Liu, J.: TA-BLSTM: tag attention-based bidirectional long short-term memory for service recommendation in mashup creation. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)

    Google Scholar 

  16. Szegedy, C., et al.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)

    Google Scholar 

  17. Uddin, G., Khomh, F., Roy, C.K.: Mining API usage scenarios from stack overflow. Inf. Softw. Technol. 122(122), 106277 (2020)

    Google Scholar 

  18. Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, vol. 30, pp. 5998–6008 (2017)

    Google Scholar 

  19. Wang, X., Liu, J., Liu, X., Cui, X., Wu, H.: A novel dual-graph convolutional network based web service classification framework. In: 2020 IEEE International Conference on Web Services (ICWS), pp. 281–288 (2020)

    Google Scholar 

  20. Wang, X., Liu, J., Liu, X., Cui, X., Wu, H.: A spatial and sequential combined method for web service classification. In: APWeb-WAIM 2020 : Proceedings of the 4th Asia-Pacific and Web-Age Information Management International Joint Conference on Web and Big Data, pp. 764–778 (2020)

    Google Scholar 

  21. Wang, X., Liu, X., Liu, J., Chen, X., Wu, H.: A novel knowledge graph embedding based API recommendation method for mashup development. World Wide Web 24(3), 869–894 (2021)

    Article  Google Scholar 

  22. Wang, X., et al.: SYNCOBERT: syntax-guided multi-modal contrastive pre-training for code representation. arXiv: Computation and Language (2021)

  23. Wei, J.W., Zou, K.: Eda: easy data augmentation techniques for boosting performance on text classification tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6381–6387 (2019)

    Google Scholar 

  24. Wu, Z., Wang, S., Gu, J., Khabsa, M., Sun, F., Ma, H.: CLEAR: contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466 (2020)

  25. Xiong, R., Wang, J., Zhang, N., Ma, Y.: Deep hybrid collaborative filtering for web service recommendation. Expert Syst. Appl. 110, 191–205 (2018)

    Article  Google Scholar 

  26. Yang, Y., et al.: ServeNet: a deep neural network for web services classification. In: 2020 IEEE International Conference on Web Services (ICWS), pp. 168–175 (2020)

    Google Scholar 

  27. Yao, L., Wang, X., Sheng, Q.Z., Benatallah, B., Huang, C.: Mashup recommendation by regularizing matrix factorization with API co-invocations. IEEE Trans. Serv. Comput. 14, 502–515 (2021)

    Article  Google Scholar 

  28. Ye, H., Cao, B., Chen, J., Liu, J., Wen, Y., Chen, J.: A web services classification method based on GCN. In: 2019 IEEE International Conference on Parallel and Distributed Processing with Applications, Big Data and Cloud Computing, Sustainable Computing and Communications, Social Computing and Networking (ISPA/BDCloud/SocialCom/SustainCom), pp. 1107–1114 (2019)

    Google Scholar 

  29. Yuan, W., Nguyen, H.H., Jiang, L., Chen, Y., Zhao, J., Yu, H.: API recommendation for event-driven android application development. Inf. Softw. Technol. 107, 30–47 (2019)

    Article  Google Scholar 

  30. Zhong, Y., Fan, Y., Tan, W., Zhang, J.: Web service recommendation with reconstructed profile from mashup descriptions. IEEE Trans. Autom. Sci. Eng. 15(2), 468–478 (2018)

    Article  Google Scholar 

Download references

Acknowledgment

This work is supported by the National Natural Science Foundation of China under Grants 61972290 and the National Key R&D Program of China under Grant 2018YFC1604000. This work is also supported by the National Natural Science Foundation of China (61962061) and partially supported by the Yunnan Provincial Foundation for Leaders of Disciplines in Science and Technology (202005AC160005).

Author information

Authors and Affiliations

  1. School of Computer Science, Wuhan University, Wuhan, China

    Xin Wang & Jin Liu

  2. Noah’s Ark Lab, Huawei, Shenzhen, China

    Pingyi Zhou & Yasheng Wang

  3. School of Information Technology, Deakin University, Geelong, Australia

    Xiao Liu

  4. School of Information Science and Engineering, Yunnan University, Kunming, China

    Hao Wu

Authors
  1. Xin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pingyi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yasheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Liu .

Editor information

Editors and Affiliations

  1. Zayed University, Dubai, United Arab Emirates

    Hakim Hacid

  2. Technical University of Berlin, Berlin, Germany

    Odej Kao

  3. Informatica Automatica Gestio, Sapienza University of Rome, Rome, Italy

    Massimo Mecella

  4. Departement d'Informatique, University of Quebec, Montreal, QC, Canada

    Naouel Moha

  5. UNSW Sydney, Sydney, NSW, Australia

    Hye-young Paik

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, X., Zhou, P., Wang, Y., Liu, X., Liu, J., Wu, H. (2021). ServiceBERT: A Pre-trained Model for Web Service Tagging and Recommendation. In: Hacid, H., Kao, O., Mecella, M., Moha, N., Paik, Hy. (eds) Service-Oriented Computing. ICSOC 2021. Lecture Notes in Computer Science(), vol 13121. Springer, Cham. https://doi.org/10.1007/978-3-030-91431-8_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-91431-8_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-91430-1

  • Online ISBN: 978-3-030-91431-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation