Skip to main content
SpringerLink
Log in

Low resource language specific pre-processing and features for sentiment analysis task

  • Original Paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

Sentiment analysis is a classification task where polarity of textual data is identified, i.e. to analyze whether a sentence or document expresses a negative, positive or neutral sentiment. Manipuri is a less privileged, highly agglutinative and tonal language. Despite being a scheduled language of Indian Constitution, it is also a resource constrained language. In this work, we report the sentiment analysis for Manipuri using different types of machine learning based approaches. The dataset used in our work is collected from local daily newspaper. The novelty of this work is that we carry out language specific pre-processing tasks such as transliteration, building negative morpheme based lexicon and filtering of noisy words. Using them as additional linguistic features in our models improves the classification result in terms of precision, recall and F-score. The ensemble voting of best three classifiers based on TF-IDF perform better than BM25 based classifiers and other stand-alone classifiers. Based on this result, we attempt to classify the sentiment of news articles during a certain period of time. Further, we report the finding of deep learning based approaches on the same dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://hueiyenlanpao.com/.

  2. http://thesangaiexpress.com/manipuri/.

  3. https://github.com/LSMeetei/sentiment.

  4. http://tabish.freeshell.org/eeyek/script/.

  5. http://www.e-pao.net/.

  6. https://scikit-learn.org/stable/.

  7. https://github.com/arosh/BM25Transformer.

  8. http://www.numpy.org/.

  9. https://scikit-learn.org/stable/.

References

  • Albayati, A. Q., Al-Araji, A. S., & Ameen, S. H. A Method of Deep Learning Tackles Sentiment Analysis Problem in Arabic Texts.

  • Cambria, E., & Hussain, A. (2015). SenticNet. In Sentic Computing (pp. 23–71). Springer, Cham.

  • Cambria, E., Das, D., Bandyopadhyay, S., & Feraco, A. (2017). Affective computing and sentiment analysis. In A practical guide to sentiment analysis (pp. 1–10). Springer, Cham.

  • Das, A., & Bandyopadhyay, S. (2010, August). SentiWordNet for Indian languages. In Proceedings of the Eighth Workshop on Asian Language Resources (pp. 56–63).

  • Dashtipour, K., Gogate, M., Li, J., Jiang, F., Kong, B., & Hussain, A. (2020). A hybrid Persian sentiment analysis framework: Integrating dependency grammar based rules and deep neural networks. Neurocomputing, 380, 1–10.

    Article  Google Scholar 

  • Denecke, K. (2008, April). Using sentiwordnet for multilingual sentiment analysis. In 2008 IEEE 24th International Conference on Data Engineering Workshop (pp. 507–512). IEEE.

  • El-Haj, M., Kruschwitz, U., & Fox, C. (2015). Creating language resources for under-resourced languages: methodologies, and experiments with Arabic. Language Resources and Evaluation, 49(3), 549–580.

    Article  Google Scholar 

  • Gangula, R. R. R., & Mamidi, R. (2018, May). Resource creation towards automated sentiment analysis in telugu (a low resource language) and integrating multiple domain sources to enhance sentiment prediction. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

  • Goldberg, A. B., & Zhu, X. (2006, June). Seeing stars when there aren’t many stars: graph-based semi-supervised learning for sentiment categorization. In Proceedings of the first workshop on graph based methods for natural language processing (pp. 45-52). Association for Computational Linguistics.

  • Haddi, E., Liu, X., & Shi, Y. (2013). The role of text pre-processing in sentiment analysis. Procedia Computer Science, 17, 26–32.

    Article  Google Scholar 

  • Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.

    Article  Google Scholar 

  • Hu, X., Tang, J., Gao, H., & Liu, H. (2013, May). Unsupervised sentiment analysis with emotional signals. In Proceedings of the 22nd international conference on World Wide Web (pp. 607–618). ACM.

  • Jang, H., & Shin, H. (2010, August). Language-specific sentiment analysis in morphologically rich languages. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters (pp. 498–506). Association for Computational Linguistics.

  • Jianqiang, Z., & Xiaolin, G. (2017). Comparison research on text pre-processing methods on twitter sentiment analysis. IEEE Access, 5, 2870–2879.

    Article  Google Scholar 

  • Johansson, F., Brynielsson, J., & Quijano, M. N. (2012, August). Estimating citizen alertness in crises using social media monitoring and analysis. In 2012 European Intelligence and Security Informatics Conference (pp. 189–196). IEEE.

  • Khan, A., & Baharudin, B. (2011, September). Sentiment classification using sentence-level semantic orientation of opinion terms from blogs. In 2011 National Postgraduate Conference (pp. 1–7). IEEE.

  • Kim, S. M., & Hovy, E. (2004, August). Determining the sentiment of opinions. In Proceedings of the 20th international conference on Computational Linguistics (p. 1367). Association for Computational Linguistics.

  • Le, T. A., Moeljadi, D., Miura, Y., & Ohkuma, T. (2016, December). Sentiment analysis for low resource languages: A study on informal Indonesian tweets. In Proceedings of the 12th Workshop on Asian Language Resources (ALR12) (pp. 123–131).

  • Lo, S. L., Cambria, E., Chiong, R., & Cornforth, D. (2017). Multilingual sentiment analysis: From formal to informal and scarce resource languages. Artificial Intelligence Review, 48(4), 499–527.

    Article  Google Scholar 

  • Mishne, G. (2005, August). Experiments with mood classification in blog posts. In Proceedings of ACM SIGIR 2005 workshop on stylistic analysis of text for information access (Vol. 19, pp. 321–327).

  • Na, J. C., Sui, H., Khoo, C. S., Chan, S., & Zhou, Y. (2004). Effectiveness of simple linguistic processing in automatic sentiment classification of product reviews. International ISKO Conference.

  • Nasukawa, T., & Yi, J. (2003, October). Sentiment analysis: Capturing favorability using natural language processing. In Proceedings of the 2nd international conference on Knowledge capture (pp. 70–77). ACM.

  • Niu, Y., Zhu, X., Li, J., & Hirst, G. (2005). Analysis of polarity information in medical text. In AMIA annual symposium proceedings (Vol. 2005, p. 570). American Medical Informatics Association.

  • Nongmeikapam, K., Khangembam, D., Hemkumar, W., Khuraijam, S., & Bandyopadhyay, S. (2014). Verb based manipuri sentiment analysis. International Journal on Natural Language Computing (IJNLC), 3, 12–13.

    Google Scholar 

  • Pak, A., & Paroubek, P. (2010, May). Twitter as a corpus for sentiment analysis and opinion mining. In LREc (Vol. 10, No. 2010, pp. 1320–1326).

  • Pang, B., Lee, L., & Vaithyanathan, S. (2002, July). Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing, Vol. 10 (pp. 79–86). Association for Computational Linguistics.

  • Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., & Vanderplas, J. (2011). Scikit-learn: Machine learning in Python. Journal of machine learning research, 12(Oct), 2825–2830.

  • Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M. M., & Gatford, M. (1995). Okapi at TREC-3. Nist Special Publication Sp, 109, 109.

    Google Scholar 

  • Singh T.D., Singh T.J., Shadang M., & Thokchom S. (2021) Review Comments of Manipuri Online Video: Good, Bad or Ugly. In: Maji A.K., Saha G., Das S., Basu S., Tavares J.M.R.S. (eds) Proceedings of the International Conference on Computing and Communication Systems. Lecture Notes in Networks and Systems, vol 170. Springer, Singapore.

  • Singh, T. D. (2012, December). Bidirectional bengali script and meetei mayek transliteration of web based manipuri news corpus. In Proceedings of the 3rd Workshop on South and Southeast Asian Natural Language Processing (pp. 181–190)

  • Singh, T. D., & Bandyopadhyay, S. (2006). Word class and sentence type identification in manipuri morphological analyzer,” In Proceedings of MSPIL, Mumbai, India, 11-–17.

  • Singh, T. D., & Bandyopadhyay, S. (2010, August). Web Based Manipuri Corpus for Multiword NER and Reduplicated MWEs Identification using SVM. In Proceedings of the 1st Workshop on South and Southeast Asian Natural Language Processing (pp. 35–42).

  • Sixto, J., Almeida, A., & López-de-Ipiña, D. (2016, June). Improving the sentiment analysis process of Spanish Tweets with BM25. In International Conference on Applications of Natural Language to Information Systems (pp. 285–291). Springer, Cham.

  • Sixto, J., Almeida, A., & López-de-Ipiña, D. (2016, September). An approach to subjectivity detection on Twitter using the structured information. In International Conference on Computational Collective Intelligence (pp. 121–130). Springer, Cham.

  • Sixto, J., Almeida, A., & Löpez-de-Ipiña, D. (2018). Analysis of the Structured Information for Subjectivity Detection in Twitter. In Transactions on Computational Collective Intelligence XXIX (pp. 163–181). Springer, Cham.

  • Vilares, D., Peng, H., Satapathy, R., & Cambria, E. (2018, November). BabelSenticNet: a commonsense reasoning framework for multilingual sentiment analysis. In 2018 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 1292–1298). IEEE.

  • Wilson, T., Wiebe, J., & Hoffmann, P. (2005). Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing.

  • Yu, H., & Hatzivassiloglou, V. (2003, July). Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Proceedings of the 2003 conference on Empirical methods in natural language processing (pp. 129–136). Association for Computational Linguistics.

  • Zhang, Y., & Wallace, B. (2015). A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. arXiv:1510.03820.

  • Zhang, W., Xu, H., & Wan, W. (2012). Weakness finder: Find product weakness from Chinese reviews by using aspects based sentiment analysis. Expert Systems with Applications, 39(11), 10283–10291.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to express our appreciation to L. Arunkumar's team and Preety Q. Sinam for their assistance without whom the compilation of this corpus would not have been possible. We also thank the anonymous reviewers for their careful reading and their many insightful comments, which helped us to improve our manuscript.

Author information

Authors and Affiliations

  1. Center for Natural Language Processing (CNLP), National Institute of Technology Silchar, Silchar, Assam, India

    Loitongbam Sanayai Meetei, Thoudam Doren Singh & Sivaji Bandyopadhyay

  2. Department of Computer Science and Engineering, National Institute of Technology Silchar, Silchar, Assam, India

    Loitongbam Sanayai Meetei, Thoudam Doren Singh, Samir Kumar Borgohain & Sivaji Bandyopadhyay

Authors
  1. Loitongbam Sanayai Meetei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thoudam Doren Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Samir Kumar Borgohain
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sivaji Bandyopadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Loitongbam Sanayai Meetei.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Meetei, L.S., Singh, T.D., Borgohain, S.K. et al. Low resource language specific pre-processing and features for sentiment analysis task. Lang Resources & Evaluation 55, 947–969 (2021). https://doi.org/10.1007/s10579-021-09541-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-021-09541-9

Keywords

Search

Navigation