Skip to main content
SpringerLink
Log in

DeepBT and NLP Data Augmentation Techniques: A New Proposal and a Comprehensive Study

  • Conference paper
  • First Online:
Intelligent Systems (BRACIS 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12319))

Included in the following conference series:

Abstract

Data Augmentation methods – a family of techniques designed for synthetic generation of training data – have shown remarkable results in various Deep Learning and Machine Learning tasks. Despite its widespread and successful adoption within the computer vision community, data augmentation techniques designed for natural language processing (NLP) tasks have exhibited much slower advances and limited success in achieving performance gains. As a consequence, with the exception of applications of back-translation to machine translation tasks, these techniques have not been as thoroughly explored by the wider NLP community. Recent research on the subject also still lacks a proper practical understanding of the relationship between data augmentation and several important aspects of model design, such as hyperparameters and regularization parameters. In this paper, we perform a comprehensive study of NLP data augmentation techniques, comparing their relative performance under different settings. We also propose Deep Back-Translation, a novel NLP data augmentation technique and apply it to benchmark datasets. We analyze the quality of the synthetic data generated, evaluate its performance gains and compare all of these aspects to previous existing data augmentation procedures.

Partially supported by Itaú-Unibanco, CNPq (grants 25860/2016-7 and 530307027/2017-1), and CAPES (Finance Code 001). Any opinions, findings, and conclusions expressed in this manuscript are those of the authors and do not necessarily reflect the views, official policy or position of Itaú-Unibanco.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The term Noised Back-Translation was not used in [7], but coined by [3].

  2. 2.

    https://cloud.google.com/translate/docs.

References

  1. Bao, F., Neumann, M., Vu, T.: CycleGAN-based emotion style transfer as data augmentation for speech emotion recognition. In: Proceedings of the Interspeech 2019, pp. 2828–2832, September 2019. https://doi.org/10.21437/Interspeech.2019-2293

  2. Basile, V., et al.: SemEval-2019 task 5: multilingual detection of hate speech against immigrants and women in twitter. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 54–63. Association for Computational Linguistics, Minneapolis, June 2019. https://doi.org/10.18653/v1/S19-2007

  3. Caswell, I., Chelba, C., Grangier, D.: Tagged back-translation. In: Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pp. 53–63. Association for Computational Linguistics, Florence, August 2019. https://doi.org/10.18653/v1/W19-5206

  4. Cortis, K., et al.: SemEval-2017 task 5: fine-grained sentiment analysis on financial microblogs and news. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 519–535. Association for Computational Linguistics, Stroudsburg (2017). https://doi.org/10.18653/v1/S17-2089

  5. Dao, T., Gu, A., Ratner, A., Smith, V., De Sa, C., Re, C.: A kernel theory of modern data augmentation. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 1528–1537. PMLR, Long Beach, 09–15 June 2019. http://proceedings.mlr.press/v97/dao19b.html

  6. Davis, B., Cortis, K., Vasiliu, L., Koumpis, A., Mcdermott, R., Handschuh, S.: Social sentiment indices powered by X-scores. In: ALLDATA 2016, The Second International Conference on Big Data, Small Data, Linked Data and Open Data, Lisbon, Portugal (2016)

    Google Scholar 

  7. Edunov, S., Ott, M., Auli, M., Grangier, D.: Understanding back-translation at scale. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489–500. Association for Computational Linguistics, Brussels, October–November 2018. https://doi.org/10.18653/v1/D18-1045

  8. Ferreira, T., Paiva, F., Silva, R., Paula, A., Costa, A., Cugnasca, C.: Assessing regression-based sentiment analysis techniques in financial texts. In: Anais do XVI Encontro Nacional de Inteligência Artificial e Computacional, pp. 729–740. SBC, Porto Alegre (2019). https://doi.org/10.5753/eniac.2019.9329, https://sol.sbc.org.br/index.php/eniac/article/view/9329

  9. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. The MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  10. Graça, M., Kim, Y., Schamper, J., Khadivi, S., Ney, H.: Generalizing back-translation in neural machine translation. In: Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pp. 45–52. Association for Computational Linguistics, Florence, August 2019. https://doi.org/10.18653/v1/W19-5205

  11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  12. Hernández-García, A., König, P.: Further advantages of data augmentation on convolutional neural networks. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11139, pp. 95–103. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01418-6_10

    Chapter  Google Scholar 

  13. Hoang, V.C.D., Koehn, P., Haffari, G., Cohn, T.: Iterative back-translation for neural machine translation. In: Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 18–24. Association for Computational Linguistics, Melbourne, July 2018. https://doi.org/10.18653/v1/W18-2703

  14. Imamura, K., Fujita, A., Sumita, E.: Enhancement of encoder and attention using target monolingual corpora in neural machine translation. In: Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 55–63. Association for Computational Linguistics, Melbourne, July 2018. https://doi.org/10.18653/v1/W18-2707

  15. Kobayashi, S.: Contextual augmentation: data augmentation by words with paradigmatic relations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 452–457. Association for Computational Linguistics, New Orleans, June 2018. https://doi.org/10.18653/v1/N18-2072

  16. Konda, K.R., Bouthillier, X., Memisevic, R., Vincent, P.: Dropout as data augmentation. arXiv abs/1506.08700 (2015)

    Google Scholar 

  17. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems. NIPS 2012, vol. 1, pp. 1097–1105. Curran Associates Inc., Red Hook (2012)

    Google Scholar 

  18. Mikołajczyk, A., Grochowski, M.: Data augmentation for improving deep learning in image classification problem. In: 2018 International Interdisciplinary PhD Workshop (IIPhDW), pp. 117–122 (2018)

    Google Scholar 

  19. Mohammad, S., Bravo-Marquez, F., Salameh, M., Kiritchenko, S.: SemEval-2018 task 1: affect in tweets. In: Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 1–17. Association for Computational Linguistics, New Orleans, June 2018. https://doi.org/10.18653/v1/S18-1001

  20. Pennington, J., Socher, R., Manning, C.D.: GloVe : global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  21. Salamon, J., Bello, J.P.: Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Process. Lett. 24(3), 279–283 (2017)

    Article  Google Scholar 

  22. Sennrich, R., Haddow, B., Birch, A.: Improving neural machine translation models with monolingual data. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86–96. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/P16-1009

  23. Springenberg, J., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. In: ICLR (workshop track) (2015). http://lmb.informatik.uni-freiburg.de/Publications/2015/DB15a

  24. Sugiyama, A., Yoshinaga, N.: Data augmentation using back-translation for context-aware neural machine translation. In: Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), pp. 35–44. Association for Computational Linguistics, Hong Kong, November 2019. https://doi.org/10.18653/v1/D19-6504

  25. Taylor, L., Nitschke, G.: Improving deep learning with generic data augmentation. In: 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1542–1547 (2018)

    Google Scholar 

  26. Wei, J., Zou, K.: EDA: easy data augmentation techniques for boosting performance on text classification tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6382–6388. Association for Computational Linguistics, Hong Kong, November 2019. https://doi.org/10.18653/v1/D19-1670

  27. Wen, Q., Sun, L., Song, X., Gao, J., Wang, X., Xu, H.: Time series data augmentation for deep learning: a survey (2020)

    Google Scholar 

  28. Yu, A.W., et al.: QANET: combining local convolution with global self-attention for reading comprehension. CoRR abs/1804.09541 (2018). https://arxiv.org/pdf/1804.09541

  29. Zhang, Y., Wallace, B.C.: A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. In: Proceedings of the 8th International Joint Conference on Natural Language Processing, pp. 253–263 (2017)

    Google Scholar 

  30. Zhao, D., Yu, G., Xu, P., Luo, M.: Equivalence between dropout and data augmentation: a mathematical check. Neural Netw. Off. J. Int. Neural Netw. Soc. 115, 82–89 (2019)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Escola Politécnica, Universidade de São Paulo, São Paulo, Brazil

    Taynan Maier Ferreira & Anna Helena Reali Costa

  2. Data Science Team, Itaú-Unibanco, São Paulo, Brazil

    Taynan Maier Ferreira

Authors
  1. Taynan Maier Ferreira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna Helena Reali Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Taynan Maier Ferreira .

Editor information

Editors and Affiliations

  1. Federal University of São Carlos, São Carlos, Brazil

    Ricardo Cerri

  2. Federal University of ABC, Santo Andre, Brazil

    Ronaldo C. Prati

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Maier Ferreira, T., Reali Costa, A.H. (2020). DeepBT and NLP Data Augmentation Techniques: A New Proposal and a Comprehensive Study. In: Cerri, R., Prati, R.C. (eds) Intelligent Systems. BRACIS 2020. Lecture Notes in Computer Science(), vol 12319. Springer, Cham. https://doi.org/10.1007/978-3-030-61377-8_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61377-8_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61376-1

  • Online ISBN: 978-3-030-61377-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation