Abstract
As an essentially antecedent task of sentiment analysis, subjectivity detection refers to classifying sentences to be subjective ones containing opinions, or objective and neutral ones without bias. In the situations where impartial language is required, such as Wikipedia, subjectivity detection could play an important part. Recently, pretrained language models have proven to be effective in learning representations, profoundly boosting the performance among several NLP tasks. As a state-of-art pretrained model, BERT is trained on large unlabeled data with masked word prediction and next sentence prediction tasks. In this paper, we mainly explore utilizing BERT pretrained models with several combinations of fine-tuning methods, holding the intention to enhance performance in subjectivity detection task. Our experimental results reveal that optimum combinations of fine-tune and multi-task learning surplus the state-of-the-art on subjectivity detection and related tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aleksandrova, D., Lareau, F., Ménard, P.: Multilingual Sentence-Level Bias Detection in Wikipedia (2019)
Burges, C.J.C., Ragno, R., Le, Q.V.: Learning to rank with nonsmooth cost functions. In: NIPS (2006)
Caruana, R.: Multitask learning: A knowledge-based source of inductive bias. Mach. Learn. 28, 41–75 (1997)
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L.: SemEval2017 Task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In: Proceedings f the 11th International Workshop Semantic Evaluation, August 2017, pp. 1–14
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding (2018)
Hube, C., Fetahu, B.: Detecting biased statements in Wikipedia. In: The Web Conference, 1779–1786. International World Wide Web Conferences Steering Committee (2018)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting Association Computing Linguistics, vol. 1, pp. 328–339 (2018)
Hube, C., Fetahu, B.: Neural based statement classification for biased language. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 195–203. ACM (2019)
Karamibekr, M., Ghorbani, A.A.: Sentence subjectivity analysis in social domains. In: Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence (WI 2013), Atlanta, GA, USA (2013)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)
Lin, C., He, Y., Everson, R.: Sentence subjectivity detection with weakly-supervised learning. In: Proceedings of the 5th International Joint Conference on Natural Language Process, pp. 1153–1161, November 2011
Liu, X., He, P., Chen, W., Gao, J.: Multi-task deep neural networks for natural language understanding. In: Proceedings of the 57th Annual Meeting Association Computing Linguistics, pp. 4487–4496 (2019)
Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting Association Computing Linguistics, Human Language Technology vol. 1, pp. 142–150 (2011)
Montoyo, A., Martínez-Barco, P., Balahur, A.: Subjectivity and sentiment analysis: an overview of the current state of the area and envisaged developments. Decis. Supp. Syst. 53(4), 675–679 (2012)
Mäntylä, M.V., Graziotin, D., Kuutila, M.: The evolution of sentiment analysis—a review of research topics venues and top cited papers. Comput. Sci. Rev. 27, 16–32 (2018)
Pang, B., Lee, L.: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of ACL 2004 (2004)
Recasens, M., Danescu-Niculescu-Mizil, C., Jurafsky, D.: Linguistic models for analyzing and detecting biased language. In: Proceedings of ACL (2013)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000 + Questions for Machine Comprehension of Text. In: EMNLP (2016)
Ruder, S.: An overview of multi-task learning in deep neural networks (2017). http://arxiv.org/abs/1706.05098
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018). https://blog.openai.com/language-unsupervised/
Bowman, S.R., Angeli, G., Potts, C., Manning, C,D.: A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, 17–21 September 2015, pp. 632–642 (2015)
Smith, L.N.: Cyclical learning rates for training neural networks. In: WACV (2017)
Smith, L.N.: A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820 (2018)
Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune BERT for text classification? arXiv:1905.05583 (2019). https://arxiv.org/abs/1905.05583
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, “Attention Is All You Need”, CoRR, vol. abs/1706.03762, 2017
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding arXiv:1906.08237 (2019). https://arxiv.org/abs/1906.08237
Zhao, H., Lu, Z., Poupart, P.: Self-adaptive hierarchical sentence model (2015). https://arxiv.org/abs/1504.05070
Zhang, Y., Yang, Q.: A survey on multi-task learning. arXiv preprint arXiv:1707.08114 (2017)
Zhang, L., Wang, S., Liu, B.: Deep learning for sentiment analysis: a survey. In: Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, p. 25, March 2018
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Huo, H., Iwaihara, M. (2020). Utilizing BERT Pretrained Models with Various Fine-Tune Methods for Subjectivity Detection. In: Wang, X., Zhang, R., Lee, YK., Sun, L., Moon, YS. (eds) Web and Big Data. APWeb-WAIM 2020. Lecture Notes in Computer Science(), vol 12318. Springer, Cham. https://doi.org/10.1007/978-3-030-60290-1_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-60290-1_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60289-5
Online ISBN: 978-3-030-60290-1
eBook Packages: Computer ScienceComputer Science (R0)