Skip to main content
SpringerLink
Log in

Supervised Transfer Learning for Sequence Tagging of User-Generated-Content in Social Media

  • Conference paper
  • First Online:
Human Language Technology. Challenges for Computer Science and Linguistics (LTC 2017)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12598))

Included in the following conference series:

  • 284 Accesses

Abstract

Neural-networks based approaches for Natural Language Processing (NLP) are effective when dealing with learning from large amounts of annotated data. However, these are only available for a limited number of languages and domains due to the cost of the manual annotation. Particularly, despite the valuable importance of Social Media’s content for a variety of applications (e.g., public security, health monitoring, or trends highlight), this important domain is still poor in terms of annotated data. In this work, we analyse the impact of supervised sequential transfer learning to overcome the sparse data problem in the Tweets-domain by leveraging the huge annotated data available for the Newswire-domain. We experiment our approach on three NLP tasks: part-of-speech tagging, chunking and named entity recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    SeqEval package were used to calculate F1 metric.

  2. 2.

    https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html.

  3. 3.

    https://universaldependencies.org/u/pos/.

  4. 4.

    http://www.cs.cmu.edu/~ark/TweetNLP/annot_guidelines.pdf.

  5. 5.

    data splits portions are not mentioned in original papers.

  6. 6.

    https://www.clips.uantwerpen.be/conll2000/chunking/.

  7. 7.

    Note that, Sect. 5.1 experiments correspond to scheme D.

References

  1. Akbik, A., Bergmann, T., Vollgraf, R.: Pooled contextualized embeddings for named entity recognition. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Long and Short Papers, vol. 1, pp. 724–728 (2019)

    Google Scholar 

  2. Baumann, P., Pierrehumbert, J.B.: Using resource-rich languages to improve morphological analysis of under-resourced languages. In: LREC, pp. 3355–3359 (2014)

    Google Scholar 

  3. Changpinyo, S., Hu, H., Sha, F.: Multi-task learning for sequence tagging: an empirical study. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 2965–2977 (2018)

    Google Scholar 

  4. Chiu, J.P., Nichols, E.: Named entity recognition with bidirectional LSTM-CNNs. arXiv preprint arXiv:1511.08308 (2015)

  5. Derczynski, L., Nichols, E., van Erp, M., Limsopatham, N.: Results of the wnut2017 shared task on novel and emerging entity recognition. In: Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 140–147 (2017)

    Google Scholar 

  6. Derczynski, L., Ritter, A., Clark, S., Bontcheva, K.: Twitter part-of-speech tagging for all: overcoming sparse and noisy data. Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, 198–206 (2013)

    Google Scholar 

  7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Long and Short Papers, vol. 1, pp. 4171–4186 (2019)

    Google Scholar 

  8. Dirkson, A., Verberne, S.: Transfer learning for health-related twitter data. In: Proceedings of the Fourth Social Media Mining for Health Applications (# SMM4H) Workshop & Shared Task, pp. 89–92 (2019)

    Google Scholar 

  9. Duong, L.: Natural language processing for resource-poor languages. Ph.D. thesis, University of Melbourne (2017)

    Google Scholar 

  10. Gimpel, K., et al.: Part-of-speech tagging for twitter: annotation, features, and experiments. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. pp. 42–47. Association for Computational Linguistics (2011)

    Google Scholar 

  11. Gui, T., Zhang, Q., Gong, J., Peng, M., Liang, D., Ding, K., Huang, X.J.: Transferring from formal newswire domain with hypernet for twitter pos tagging. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2540–2549 (2018)

    Google Scholar 

  12. Gui, T., Zhang, Q., Huang, H., Peng, M., Huang, X.: Part-of-speech tagging for twitter with adversarial neural networks. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2411–2420 (2017)

    Google Scholar 

  13. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Long Papers, vol. 1, pp. 328–339 (2018)

    Google Scholar 

  14. Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., Wu, Y.: Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 (2016)

  15. Liu, Y., Zhu, Y., Che, W., Qin, B., Schneider, N., Smith, N.A.: Parsing tweets into universal dependencies. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Long Papers, vol. 1, pp. 965–975 (2018)

    Google Scholar 

  16. Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Long Papers, vol. 1, pp. 1064–1074 (2016)

    Google Scholar 

  17. Marcus, M., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of English: the penn treebank. Comput. Linguist. 19(2), 313–330 (1993)

    Google Scholar 

  18. März, L., Trautmann, D., Roth, B.: Domain adaptation for part-of-speech tagging of noisy user-generated text. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Long and Short Papers, vol. 1, pp. 3415–3420 (2019)

    Google Scholar 

  19. Meftah, S., Semmar, N.: A neural network model for part-of-speech tagging of social media texts. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)

    Google Scholar 

  20. Meftah, S., Semmar, N., Sadat, F., Raaijmakers, S.: Using neural transfer learning for morpho-syntactic tagging of South-Slavic Languages tweets. In: Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pp. 235–243 (2018)

    Google Scholar 

  21. Meftah, S., Tamaazousti, Y., Semmar, N., Essafi, H., Sadat, F.: Joint learning of pre-trained and random units for domain adaptation in part-of-speech tagging. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Long and Short Papers, vol. 1, pp. 4107–4112 (2019)

    Google Scholar 

  22. Mou, L., Meng, Z., Yan, R., Li, G., Xu, Y., Zhang, L., Jin, Z.: How transferable are neural networks in NLP applications? In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 479–489 (2016)

    Google Scholar 

  23. Nivre, J., et al.: Universal dependencies v1: a multilingual treebank collection. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pp. 1659–1666 (2016)

    Google Scholar 

  24. Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)

    Google Scholar 

  25. Owoputi, O., O’Connor, B., Dyer, C., Gimpel, K., Schneider, N., Smith, N.A.: Improved part-of-speech tagging for online conversational text with word clusters. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 380–390 (2013)

    Google Scholar 

  26. Pan, S.J., Yang, Q., et al.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    Article  Google Scholar 

  27. Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  28. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proceedings of NAACL-HLT, pp. 2227–2237 (2018)

    Google Scholar 

  29. Ritter, A., Clark, S., Etzioni, O., et al.: Named entity recognition in tweets: an experimental study. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1524–1534. Association for Computational Linguistics (2011)

    Google Scholar 

  30. Rizoiu, M.A., Wang, T., Ferraro, G., Suominen, H.: Transfer learning for hate speech detection in social media. arXiv preprint arXiv:1906.03829 (2019)

  31. Sang, E.F., De Meulder, F.: Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. arXiv:cs/0306050 (2003)

  32. Sang, T.K., Erik, F., Buchholz, S.: Introduction to the CoNLL-2000 shared task: chunking. In: Proceedings of CoNLL-2000, Lisbon, Portugal, pp. 127–132 (2000)

    Google Scholar 

  33. Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, vol. 1, pp. 173–180. Association for Computational Linguistics (2003)

    Google Scholar 

  34. Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Advances in Neural Information Processing Systems, pp. 3320–3328 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Département Intelligence Ambiante et Systèmes Interactifs, Université Paris-Saclay, CEA, 91191, Gif-sur-Yvette, France

    Sara Meftah, Nasredine Semmar & Othmane Zennaki

  2. UQAM, Montreal, QC, Canada

    Fatiha Sadat

Authors
  1. Sara Meftah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nasredine Semmar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Othmane Zennaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fatiha Sadat
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nasredine Semmar .

Editor information

Editors and Affiliations

  1. Adam Mickiewicz University, Poznań, Poland

    Zygmunt Vetulani

  2. Laboratoire d’Informatique pour la Méca, Orsay, France

    Patrick Paroubek

  3. Adam Mickiewicz University, Poznań, Poland

    Marek Kubis

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Meftah, S., Semmar, N., Zennaki, O., Sadat, F. (2020). Supervised Transfer Learning for Sequence Tagging of User-Generated-Content in Social Media. In: Vetulani, Z., Paroubek, P., Kubis, M. (eds) Human Language Technology. Challenges for Computer Science and Linguistics. LTC 2017. Lecture Notes in Computer Science(), vol 12598. Springer, Cham. https://doi.org/10.1007/978-3-030-66527-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-66527-2_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-66526-5

  • Online ISBN: 978-3-030-66527-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation