Skip to main content
SpringerLink
Log in

How Reliable Is Sentiment Analysis? A Multi-domain Empirical Investigation

  • Conference paper
  • First Online:
Book cover Web Information Systems and Technologies (WEBIST 2016)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 292))

Included in the following conference series:

  • 503 Accesses

Abstract

Sentiment analysis (also known as opinion mining) is frequently used in monitoring public opinions on the internet. For example, it can help marketers evaluate the success of an ad campaign. It can also be used to assess public opinions during a political campaign. As a result, many businesses and organizations are exploring the potential value of employing sentiment analysis as a part of their business and social intelligence strategies. However, the technology isn’t fully mature yet. As a result, if not used carefully, the results from sentiment analysis can be misleading. In this paper, we present an empirical investigation of the effectiveness of using current sentiment analysis tools to assess people’s opinions in five different domains. The results were very uneven, from decent (e.g., hotel reviews) to poor (e.g., comments on public policies). We also proposed several effectiveness indicators that can be used to signal the appropriateness of using these tools in specific domains.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://text-processing.com/docs/sentiment.html.

  2. 2.

    https://cloud.google.com/prediction/docs.

  3. 3.

    http://www.nltk.org/api/nltk.classify.html.

  4. 4.

    https://semantria.com/.

  5. 5.

    http://sentimentanalyzer.appspot.com/.

  6. 6.

    http://sentistrength.wlv.ac.uk/.

  7. 7.

    https://www.publicapis.com/mlanalyzer.

  8. 8.

    http://text-processing.com/demo/sentiment/.

  9. 9.

    https://github.com/nik0spapp/unsupervisedsentiment.

  10. 10.

    https://www.fcc.gov/rulemaking/most-active-proceedings.

References

  1. Andreevskaia, A., Bergler, S.: Unsupervised sentiment analysis with emotional signals. In: 11th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2006 (2006)

    Google Scholar 

  2. Belkin, D.: Harvard accused of bias against Asian-Americans. Wall Street J. (2015). http://www.wsj.com/articles/asian-american-organizations-seek-federal-probe-ofharvard-admission-policies-1431719348

  3. Bob Lannon, A.P.: What can we learn from 800,000 public comments on the FCC’s net neutrality plan? @ONLINE (2014)

    Google Scholar 

  4. Chung, W., Zeng, D.: Social-media-based public policy informatics: sentiment and network analyses of U.S. immigration and border security. J. Assoc. Inf. Sci. Technol. 67(7), 1588–1606 (2015)

    Article  Google Scholar 

  5. Diakopoulos, N.A., Shamma, D.A.: Characterizing debate performance via aggregated twitter sentiment. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2010, pp. 1195–1198. ACM, New York (2010)

    Google Scholar 

  6. Ghose, A., Ipeirotis, P.G., Sundararajan, A.: Opinion mining using econometrics: a case study on reputation systems. In: Proceedings of the 44th Annual Meeting of the Association for Computational Linguistics (2007)

    Google Scholar 

  7. Hu, X., Tang, J., Gao, H., Liu, H.: Unsupervised sentiment analysis with emotional signals. In Proceedings of the 22nd International Conference on World Wide Web, WWW 2013, Republic and Canton of Geneva, Switzerland, pp. 607–618. International World Wide Web Conferences Steering Committee (2013)

    Google Scholar 

  8. Kilgarriff, A., Rose, T.: Measures for corpus similarity and homogeneity. In: 3rd Conference on Empirical Methods in Natural Language Processing (1998)

    Google Scholar 

  9. Liu, B.: Sentiment Analysis and Opinion Mining. Morgan and Claypool, New York (2012)

    Google Scholar 

  10. Maynard, D., Funk, A.: Automatic detection of political opinions in tweets. In: García-Castro, R., Fensel, D., Antoniou, G. (eds.) ESWC 2011. LNCS, vol. 7117, pp. 88–99. Springer, Heidelberg (2012). doi:10.1007/978-3-642-25953-1_8

    Chapter  Google Scholar 

  11. Pang, B., Lee, L.: A sentimental education: sentiment analysis using subjectivity. In: Proceedings of ACL, pp. 271–278 (2004)

    Google Scholar 

  12. Pang, B., Lee, L.: Opinion mining and sentiment analysis. Found. Trends Inf. Retr. 2(1–2), 1–135 (2008)

    Article  Google Scholar 

  13. Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing, EMNLP 2002, Stroudsburg, PA, USA, vol. 10, pp. 79–86. Association for Computational Linguistics (2002)

    Google Scholar 

  14. Pappas, N., Katsimpras, G., Stamatatos, E.: Distinguishing the popularity between topics: a system for up-to-date opinion retrieval and mining in the web. In: 14th International Conference on Intelligent Text Processing and Computational Linguistics (2013)

    Google Scholar 

  15. Plackett, R.L.: Karl pearson and the chi-squared test. Int. Stat. Rev. 51, 59–72 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  16. Read, J., Carroll, J.: Weakly supervised techniques for domain-independent sentiment classification. In: Proceedings of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion, TSA 2009, New York, NY, USA, pp. 45–52 (2009)

    Google Scholar 

  17. Shi, H.-X. and Li, X.-J.: A sentiment analysis model for hotel reviews based on supervised learning. In: 2011 International Conference on Machine Learning and Cybernetics (ICMLC), vol. 3, pp. 950–954 (2011)

    Google Scholar 

  18. Singhal, A.: Modern information retrieval: a brief overview. IEEE Data Eng. Bull. 24(4), 35–43 (2001)

    Google Scholar 

  19. Turney, P.D.: Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL 2002, Stroudsburg, PA, USA, pp. 417–424. Association for Computational Linguistics (2002)

    Google Scholar 

  20. Wang, H., Can, D., Kazemzadeh, A., Bar, F., Narayanan, S.: A system for real-time twitter sentiment analysis of 2012 U.S. presidential election cycle. In: Proceedings of the ACL 2012 System Demonstrations, ACL 2012, Stroudsburg, PA, USA, pp. 115–120. Association for Computational Linguistics (2012)

    Google Scholar 

  21. Wang, H., Lu, Y., Zhai, C.: Latent aspect rating analysis without aspect keyword supervision. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2011, New York, NY, USA, pp. 618–626 (2011)

    Google Scholar 

  22. Peng, W., Park, D.H.: Generate adjective sentiment dictionary for social media sentiment analysis using constrained nonnegative matrix factorization. In: The International AAAI Conference on Web and Social Media, ICWSM (2011)

    Google Scholar 

  23. Zhou, X., Tao, X., Yong, J., Yang, Z.: Sentiment analysis on tweets for social events. In: 2013 IEEE 17th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 557–562 (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Maryland, Baltimore County, Baltimore, MD, 21250, USA

    Tao Ding & Shimei Pan

Authors
  1. Tao Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shimei Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Tao Ding or Shimei Pan .

Editor information

Editors and Affiliations

  1. University of Paris, Paris, France

    Valérie Monfort

  2. Department of Information Systems and Databases, RWTH Aachen University, Aachen, Germany

    Karl-Heinz Krempels

  3. Faculty of Social Sciences, University of Agder, Kristiansand, Norway

    Tim A. Majchrzak

  4. Center for Information Technology, FBK-ICT IRST, Trento, Italy

    Paolo Traverso

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Ding, T., Pan, S. (2017). How Reliable Is Sentiment Analysis? A Multi-domain Empirical Investigation. In: Monfort, V., Krempels, KH., Majchrzak, T., Traverso, P. (eds) Web Information Systems and Technologies. WEBIST 2016. Lecture Notes in Business Information Processing, vol 292. Springer, Cham. https://doi.org/10.1007/978-3-319-66468-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-66468-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-66467-5

  • Online ISBN: 978-3-319-66468-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation