Skip to main content
SpringerLink
Log in

Integrated BERT embeddings, BiLSTM-BiGRU and 1-D CNN model for binary sentiment classification analysis of movie reviews

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Now a days, understanding the review of the articles, movies are the major issue due to different sentiment present on them. Reviews are short texts which expressing the opinion of the writer on certain texts and express the sentiment related to them. In the recent past, many researchers pay attention to sentiment analysis. In this research work, a novel binary sentiment classification is proposed to classify either positive or negative sentiment. First, the Bidirectional Encoder Representations from Transformers (BERT) embeddings are introduced to tokenize and preprocess the input text. The Bidirectional Long Short-Term Memory (BiLSTM) – Bidirectional Gated Recurrent Unit (BiGRU) and 1-D Convolutional Neural Network (CNN) model is integrated and proposed for sentiment classification. The proposed integrated BERT Embedding and BiLSTM-BiGRU is applied to extract the specified target and self-attention layer is added for better understanding of context, further 1-D CNN along with few other deep learning layers, the sentiment is classified for the selected IMDB movie review dataset. The proposed BERT Embedding + BiLSTM-BiGRU + self-attention and 1-D CNN model is trained and validated with the IMDB movie review dataset. From the simulation, it is found that the testing accuracy and AUC (Area Under the Curve) values are 93.89% and 0.9828 respectively. The performance of the proposed integrated BERT Embedding + BiLSTM-BiGRU+ self-attention and 1-D CNN model is compared with existing models and it is observed that it outperforms better in binary sentiment classification analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and material

NA

Code availability

Custom Code.

References

  1. Alaparthi S, Mishra M (2020) Bidirectional Encoder Representations from Transformers (BERT): a sentiment analysis Odyssey, arXiv:2007.01127, 01-15

  2. Ali NM, Abd El Hamid MM, Youssif A (2019) Sentiment analysis for movies reviews dataset using deep learning models. Int J Data Min Knowl Manag Process 9(2/3):19–27. https://doi.org/10.5121/ijdkp.2019.9302

    Article  Google Scholar 

  3. Amit M, Adi S (2016) Word Embeddings and their use in sentence classification tasks. arXiv:1610.082229v1

  4. An H, Moon N (2019) Design of recommendation system for tourist spot using sentiment analysis based on CNN-LSTM. J Ambient Intell Humaniz Comput 13:1653–1663. https://doi.org/10.1007/s12652-019-01521-w

    Article  Google Scholar 

  5. Appel O, Chiclana F, Carter J, Fujita H (2016) A hybrid approach to the sentiment analysis problem at the sentence level. Knowl-Based Syst 108:110–124

    Article  Google Scholar 

  6. Bodapati JD, Veeranjaneyulu N, Shaik S (2019) Sentiment analysis from movie reviews using LSTMs. Ing Syst Inf 24(1):125–129. https://doi.org/10.18280/isi.240119

    Article  Google Scholar 

  7. Bojanowski P, Grave E, Joulin A, Mikolov T (2017) Enriching word vectors with subword information. Trans Assoc Comput Linguist 5:135–146

    Article  Google Scholar 

  8. Catal A, Nangir M (2017) A sentiment classification model based on multiple classifiers. Appl Soft Comput 50:135–141

    Article  Google Scholar 

  9. Chaturvedi I, Ong Y, Tsang IW, Welsch RE, Cambria E (2016) Learning word dependencies in text by means of a deep recurrent belief network. Knowl-Based Syst 108:144–154

    Article  Google Scholar 

  10. Chen Y, Zhang Z (2018) Research on text sentiment analysis based on CNNs and SVM, 2018 13th IEEE conference on industrial electronics and applications, pp 2731–2734

  11. Cheng Y, Yao L, Xiang G, Zhang G, Tang T, Zhong L (2020) Text sentiment orientation analysis based on Multi-Channel CNN and bidirectional GRU with attention mechanism. IEEE Access 8:134964–134975. https://doi.org/10.1109/ACCESS.2020.3005823

    Article  Google Scholar 

  12. Delvin J, Chang MW, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL- Human Language Technologies, pp 4171–4186

  13. Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target dependent twitter sentiment classification. 52nd Annual Meeting of the Association for Computational Linguistics, 2, pp 49–54

  14. Ganapathibhotla M, Liu B (2008) Mining opinions in comparative sentences. In: Proceedings of the 22nd international conference on computational linguistics, 1, pp 241–248

  15. Garg K, Lobiyal DK (2021) KL-NF technique for sentiment classification. Multimed Tools Appl 80:19885–19907. https://doi.org/10.1007/s11042-021-10559-y

    Article  Google Scholar 

  16. Gavilanes FM, López AT, Martínez JJ, Montenegro CE, Castaño GFJ (2016) Unsupervised method for sentiment analysis in online texts. Expert Syst Appl 58:57–75

    Article  Google Scholar 

  17. Graves A, Mohamed AR, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: IEEE international conference on acoustics, speech and signal processing, pp 6645–6649

    Google Scholar 

  18. Ivan T, Ryan MC (2008) Modeling online reviews with multi-grain topic models. Proceeding of the 17th international conference on world wide web 2008, pp 111–120, https://doi.org/10.1145/1367497.1367513

  19. Jindal N, Liu B (2006) Identifying comparative sentences in text documents. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp 244–251

    Chapter  Google Scholar 

  20. Khan AH, Qamar U, Bashir S (2019) SentiMI: introducing point-wise mutual information with SentiWordNet to improve sentiment polarity detection. Appl Soft Comput 39:140–153

    Article  Google Scholar 

  21. Kim Y (2014) Convolutional neural networks for sentence classification. In: Proceedings of the 2014 conference on empirical methods. EMNLP, pp 1746–1751

    Google Scholar 

  22. Lai H, Yan X (2021) Multimodal sentiment analysis with asymmetric window multi-attentions. Multimed Tools Appl. https://doi.org/10.1007/s11042-021-11234-y

  23. Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International conference on machine learning, pp 1188–1196

  24. Li D, Qian J (2016) Text sentiment analysis based on long short-term memory, first IEEE international conference on computer communication and the internet (ICCCI), pp 471–475. https://doi.org/10.1109/CCI.2016.7778967

    Book  Google Scholar 

  25. Li W, Gao S, Zhou H, Huang Z, Zhang K, Li W (2019) The automatic text classification method based on BERT and feature union. 2019 IEEE 25th international conference on parallel and distributed systems, pp 774–777. https://doi.org/10.1109/ICPADS47876.2019.00114.

  26. Liqiang N, Yi-Liang Z, Mohammad A, Jialie S, Tat-Seng C (2015) Bridging the vocabulary gap between health seekers and healthcare knowledge. IEEE Trans Knowl Data Eng 27:396–409

    Article  Google Scholar 

  27. Liu B (2015) Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge University Press

    Book  Google Scholar 

  28. Liu W, Cao G, Yin J (2019) Bi-level attention model for sentiment analysis of short texts. IEEE Access 7:13–22

    Google Scholar 

  29. Ma Y, Peng H, Cambria E (2018) Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://ojs.aaai.org/index.php/AAAI/article/view/12048

  30. Medhat W, Hassan A, Korashy H (2014) Sentiment analysis algorithms and applications: a survey. Ain Shams Eng J 5(4):1093–1113

    Article  Google Scholar 

  31. Minaee S, Azimi E, Abdolrashidi AA (2019) Deep-sentiment: sentiment analysis using ensemble of CNN and bi-LSTM models, arXiv:1904.04206v1

  32. Mitchell M, Aguilar J, Wilson T, Durme BV (2013) Open domain targeted sentiment. In: Proceedings of the 2013 conference on empirical methods in natural language processing, pp 1643–1654

    Google Scholar 

  33. Munikar M, Shakya S, Shreshtha A (2019) Fine-grained sentiment classification using BERT. Artificial Intelligence for Transforming Business and Society, 01–05

  34. Narayanan R, Liu B, Choudhary A (2009) Sentiment analysis of conditional sentences. In: Proceedings of the 2009 conference on empirical methods in natural language processing (EMNLP), vol 1, pp 180–189

    Google Scholar 

  35. Naz H, Ahuja S, Kumar D, Rishu (2021) DT-FNN based effective hybrid classification scheme for twitter sentiment analysis. Multimed Tools Appl 80:11443–11458. https://doi.org/10.1007/s11042-020-10190-3

    Article  Google Scholar 

  36. Ortigos A, Martin JM, Carro RM (2014) Sentiment analysis in Facebook and its application to e-learning. Comput Hum Behav 31:527–541

    Article  Google Scholar 

  37. Pennington J, Socher R, Manning C (2014) GloVe: global vectors for word representation. In: Proceedings of the 2014 conference on emperical methods in natural language processing, pp 1532–1543

  38. Ravi K, Ravi V (2015) A survey on opinion mining and sentiment analysis: tasks, approaches and applications. Knowl-Based Syst 89:14–46

    Article  Google Scholar 

  39. Rill S, Reinel D, Scheidt JZ, RV. (2014) PoliTwi: early detection of emerging political topics on twitter and the impact on concept level sentiment analysis. Knowl-Based Syst 69:24–33

    Article  Google Scholar 

  40. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  41. Shaukat Z, Zulfiqar AA, Xiao C, Azeem M, Mahmood T (2020) Sentiment analysis on IMDB using lexicon and neural networks. SN Appl Sci 2:148. https://doi.org/10.1007/s42452-019-1926-x

    Article  Google Scholar 

  42. Shen J, Liao X, Tao Z (2019) Sentence-level sentiment analysis via BERT and BiGRU. Proc. SPIE 11321, 2019 international conference on image and video processing, and artificial intelligence, 113212S. https://doi.org/10.1117/12.2550215

  43. Singh NK, Tomar DS, Sangaiah AK (2020) Sentiment analysis: a review and comparative analysis over social media. J Ambient Intell Humaniz Comput 11:97–117

    Article  Google Scholar 

  44. Tang D, Qin B, Liu T (2015) Deep learning for sentiment analysis: successful approaches and future challenges. Wiley Interdiscip Rev: Data Min Knowl Discov 5(6):292–303. https://doi.org/10.1002/widm.1171

    Article  Google Scholar 

  45. Tang D, Qin B, Wei F, Dong L, Liu T, Zhou M (2015) A joint segmentation and classification framework for sentence level sentiment classification. IEEE/ACM Trans Audio Speech Lang Process 23(11):1750–1761. https://doi.org/10.1109/TASLP.2015.2449071

    Article  Google Scholar 

  46. Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. Proceedings of the 2016 conference on empirical methods in natural language processing, pp 214–224

  47. Verma P, Khandelwal B (2019) Word embeddings and its application in deep learning. Int J Innov Technol Explor Eng 8(11):337–341

    Article  Google Scholar 

  48. Victor M, Bracha S, Lior R (2016) .Language models with pre-trained (GloVe) word Embeddings. arXiv:1610.03759

  49. Wadawadagi RS, Pagi VB (2019) An enterprise perspective of web content analysis research: a strategic road- map. Int J Knowl Web Intell 6(2):51–88

    Article  Google Scholar 

  50. Wiebe J, Wilson T, Cardie C (2005) Annotating expressions of opinions and emotions in language. Lang Resour Eval 39(2–3):165–210

    Article  Google Scholar 

  51. Xu K, Liao SS, Li J, Song Y (2011) Mining comparative opinions from customer reviews for competitive intelligence. Decis Support Syst 50(4):743–754

    Article  Google Scholar 

  52. Yang S, Ko Y (2011) Extracting comparative entities and predicates from texts using comparative type classification. In: Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, 1, pp 1636–1644

  53. Yu Q, Zhao H, Wang Z (2019) Attention-based bidirectional gated recurrent unit neural networks for sentiment analysis. In: Proceedings of the 2nd international conference on artificial intelligence and pattern recognition, pp 116–119. https://doi.org/10.1145/3357254.3357262

    Chapter  Google Scholar 

  54. Zhang L, Wang S, Liu B (2017) Deep learning for sentiment analysis: a survey. Wires Data Min Knowl Discov 2018:E1253

    Google Scholar 

  55. Zhou K, Long F (2018) Sentiment analysis of text based on CNN and bi-directional LSTM model. Proceedings of the 24th international conference on automation and computing, pp 1–5. https://doi.org/10.23919/IConAC.2018.8749069

Download references

Funding

NA

Author information

Authors and Affiliations

  1. School of Electronics Engineering, Vellore Institute of Technology, Vellore, India

    Bhart Gupta, P. Prakasam & T. Velmurugan

Authors
  1. Bhart Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Prakasam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. T. Velmurugan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to P. Prakasam.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper and that the work presented in this article is not supported by any funding agency.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gupta, B., Prakasam, P. & Velmurugan, T. Integrated BERT embeddings, BiLSTM-BiGRU and 1-D CNN model for binary sentiment classification analysis of movie reviews. Multimed Tools Appl 81, 33067–33086 (2022). https://doi.org/10.1007/s11042-022-13155-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13155-w

Keywords

Search

Navigation