Skip to main content
SpringerLink
Log in

BERT for Sequence-to-Sequence Multi-label Text Classification

  • Conference paper
  • First Online:
Analysis of Images, Social Networks and Texts (AIST 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12602))

Abstract

We study the BERT language representation model and the sequence generation model with BERT encoder for the multi-label text classification task. We show that the Sequence Generating BERT model achieves decent results in significantly fewer training epochs compared to the standard BERT. We also introduce and experimentally examine a mixed model, an ensemble of BERT and Sequence Generating BERT models. Our experiments demonstrate that the proposed model outperforms current baselines in several metrics on three well-studied multi-label classification datasets with English texts and two private Yandex Taxi datasets with Russian texts.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/lancopku/SGM.

References

  1. Adhikari, A., Ram, A., Tang, R., Lin, J.: DocBERT: BERT for document classification. CoRR abs/1904.08398 (2019). http://arxiv.org/abs/1904.08398

  2. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2014)

    Google Scholar 

  3. Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: encoder-decoder approaches. In: Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111. Association for Computational Linguistics, Doha, Qatar, October 2014. https://doi.org/10.3115/v1/W14-4012

  4. Cohen, W.W., Singer, Y.: Context-sensitive learning methods for text categorization. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR 1996, pp. 307–315. ACM, New York, NY, USA (1996). https://doi.org/10.1145/243199.243278

  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2018)

    Google Scholar 

  6. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  7. Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1746–1751 (2014)

    Google Scholar 

  8. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2015). In: Conference Paper at the 3rd International Conference for Learning Representations, San Diego (2015)

    Google Scholar 

  9. Lewis, D.D., Yang, Y., Rose, T.G., Li, F.: RCV1: a new benchmark collection for text categorization research. J. Mach. Learn. Res. 5, 361–397 (2004)

    Google Scholar 

  10. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019)

    Google Scholar 

  11. Mao, Y., Tian, J., Han, J., Ren, X.: Hierarchical text classification with reinforced label assignment. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP (2019)

    Google Scholar 

  12. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26, pp. 3111–3119. Curran Associates, Inc. (2013)

    Google Scholar 

  13. Molino, P., Zheng, H., Wang, Y.: COTA: improving the speed and accuracy of customer support through ranking and deep networks. KDD (2018)

    Google Scholar 

  14. Nam, J., Menc’ia, E.L., Kim, H.J., Fürnkranz, J.: Maximizing subset accuracy with recurrent neural networks in multi-label classification. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA, pp. 5419–5429 (2017)

    Google Scholar 

  15. Peng, H., et al.: Large-scale hierarchical text classification with recursively regularized deep graph-CNN. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, 23–27 April 2018, pp. 1063–1072 (2018). https://doi.org/10.1145/3178876.3186005

  16. Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  17. Salakhutdinov, R., Torralba, A., Tenenbaum, J.B.: Learning to share visual appearance for multiclass object detection. In: CVPR, pp. 1481–1488. IEEE Computer Society (2011)

    Google Scholar 

  18. Salvador, A., Drozdzal, M., Giró i Nieto, X., Romero, A.: Inverse cooking: Recipe generation from food images. CoRR abs/1812.06164 (2018)

    Google Scholar 

  19. Schuster, M., Paliwal, K.: Bidirectional recurrent neural networks. Trans. Sig. Proc. 45(11), 2673–2681 (1997). https://doi.org/10.1109/78.650093

  20. Srivastava, N., Salakhutdinov, R.: Discriminative transfer learning with tree-based priors. In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a Meeting Held, 5–8 December 2013, Lake Tahoe, Nevada, United States, pp. 2094–2102 (2013)

    Google Scholar 

  21. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, 8–13 December 2014, Montreal, Quebec, Canada, pp. 3104–3112 (2014)

    Google Scholar 

  22. Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 2285–2294 (2016). https://doi.org/10.1109/CVPR.2016.251

  23. Wehrmann, J., Cerri, R., Barros, R.C.: Hierarchical multi-label classification networks. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, 10–15 July 2018, pp. 5225–5234 (2018)

    Google Scholar 

  24. Wiegreffe, S., Pinter, Y.: Attention is not not explanation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 11–20. Association for Computational Linguistics, Hong Kong, China, November 2019. https://doi.org/10.18653/v1/D19-1002, https://www.aclweb.org/anthology/D19-1002

  25. Wiseman, S., Rush, A.M.: Sequence-to-sequence learning as beam-search optimization. CoRR abs/1606.02960 (2016)

    Google Scholar 

  26. Wolf, T., et al.: Huggingface’s transformers: state-of-the-art natural language processing. arXiv abs/1910.03771 (2019)

    Google Scholar 

  27. Yan, Z., et al.: HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In: 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, 7–13 December 2015, pp. 2740–2748 (2015). https://doi.org/10.1109/ICCV.2015.314

  28. Yang, P., Ma, S., Zhang, Y., Lin, J., Su, Q., Sun, X.: A deep reinforced sequence-to-set model for multi-label text classification. CoRR abs/1809.03118 (2018)

    Google Scholar 

  29. Yang, P., Sun, X., Li, W., Ma, S., Wu, W., Wang, H.: SGM: sequence generation model for multi-label classification. In: COLING, pp. 3915–3926. Association for Computational Linguistics (2018)

    Google Scholar 

  30. Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. CoRR abs/1906.08237 (2019)

    Google Scholar 

  31. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489. Association for Computational Linguistics, San Diego, California, June 2016. https://doi.org/10.18653/v1/N16-1174

Download references

Acknowledgments

We want to express our gratitude to Yandex.Go Machine Learning team, and, in particular, to Tatiana Savelieva, Roman Khalkechev, Nikita Seleznev, and Alexander Parubchenko for their support and helpful discussions. We also want to thank all the people from Yandex.Taxi client support team who participated in collecting Y.Taxi Riders and Y.Taxi Drivers datasets.

Author information

Authors and Affiliations

  1. National Research University Higher School of Economics, Pokrovskii bulvar 11, Moscow, Russia

    Ramil Yarullin

  2. Yandex.Go, 1st Krasnogvardeysky Proezd 19, Moscow, Russia

    Ramil Yarullin

  3. Yandex, Ulitsa Lva Tolstogo 16, Moscow, Russia

    Pavel Serdyukov

Authors
  1. Ramil Yarullin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pavel Serdyukov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ramil Yarullin .

Editor information

Editors and Affiliations

  1. RWTH Aachen University, Aachen, Germany

    Wil M. P. van der Aalst

  2. University of Ljubljana, Ljubljana, Slovenia

    Vladimir Batagelj

  3. National Research University Higher School of Economics, Moscow, Russia

    Dmitry I. Ignatov

  4. Krasovskii Institute of Mathematics and Mechanics, Yekaterinburg, Russia

    Michael Khachay

  5. National Research University Higher School of Economics, St. Petersburg, Russia

    Olessia Koltsova

  6. University of Oslo, Oslo, Norway

    Andrey Kutuzov

  7. National Research University Higher School of Economics, Moscow, Russia

    Sergei O. Kuznetsov

  8. National Research University Higher School of Economics, Moscow, Russia

    Irina A. Lomazova

  9. Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  10. LORIA, Vandœuvre lès Nancy, France

    Amedeo Napoli

  11. Skolkovo Institute of Science and Technology, Moscow, Russia

    Alexander Panchenko

  12. University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

  13. Università Ca' Foscari Venezia, Venice, Italy

    Marcello Pelillo

  14. National Research University Higher School of Economics, Nizhny Novgorod, Russia

    Andrey V. Savchenko

  15. Kazan Federal University, Kazan, Russia

    Elena Tutubalina

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yarullin, R., Serdyukov, P. (2021). BERT for Sequence-to-Sequence Multi-label Text Classification. In: van der Aalst, W.M.P., et al. Analysis of Images, Social Networks and Texts. AIST 2020. Lecture Notes in Computer Science(), vol 12602. Springer, Cham. https://doi.org/10.1007/978-3-030-72610-2_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72610-2_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72609-6

  • Online ISBN: 978-3-030-72610-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation