Skip to main content
Springer Nature Link
Log in

Tweet Classification Using Sentiment Analysis Features and TF-IDF Weighting for Improved Flu Trend Detection

  • Conference paper
  • First Online:
Machine Learning and Data Mining in Pattern Recognition (MLDM 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10934))

Abstract

Social Networking Sites (SNS) such as Twitter are widely used by users of diverse ages. The rate of the data in SNS has made it become an efficient resource for real-time analysis. Thus, SNS data can effectively be used to track disease outbreaks and provide necessary warnings earlier than official agencies such as the American Center of Disease Control and Prevention. In this study, we show that sentiment analysis features and weighting techniques such as Term Frequency-Inverse Document Frequency (TF-IDF) can improve the accuracy of flu tweet classification. Various machine learning algorithms were evaluated to classify tweets to either flu-related or unrelated and then adopt the one with better accuracy. The results show that the proposed approach is useful for flu disease surveillance models/systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Moorhead, S.A., Hazlett, D.E., Harrison, L., Carroll, J.K., Irwin, A., Hoving, C.: A new dimension of health care: systematic review of the uses, benefits, and limitations of social media for health communication. J. Med. Internet Res. 15(4), e85 (2013)

    Article  Google Scholar 

  2. Nurwidyantoro, A., Winarko, E.: Event detection in social media: a survey. In: 2013 International Conference on ICT for Smart Society (ICISS), pp. 1–5. IEEE (2013)

    Google Scholar 

  3. Itoh, M., Yokoyama, D., Toyoda, M., Tomita, Y., Kawamura, S., Kitsuregawa, M.: Visual fusion of mega-city big data: an application to traffic and tweets data analysis of metro passengers. In: 2014 IEEE International Conference on Big Data (Big Data), pp. 431–440. IEEE (2014)

    Google Scholar 

  4. Wang, X., Zeng, K., Zhao, X.-L., Wang, F.-Y.: Using web data to enhance traffic situation awareness. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 195–199. IEEE (2014)

    Google Scholar 

  5. Zhang, S.: Using Twitter to enhance traffic incident awareness. In: 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pp. 2941–2946. IEEE (2015)

    Google Scholar 

  6. Kosala, R., Adi, E., et al.: Harvesting real time traffic information from Twitter. Procedia Eng. 50, 1–11 (2012)

    Article  Google Scholar 

  7. Abel, F., Hauff, C., Houben, G.-J., Stronkman, R., Tao, K.: Twitcident: fighting fire with information from social web streams. In: Proceedings of the 21st International Conference on World Wide Web, pp. 305–308. ACM (2012)

    Google Scholar 

  8. Terpstra, T., de Vries, A., Stronkman, R., Paradies, G.L.: Towards a realtime Twitter analysis during crises for operational crisis management. Simon Fraser University, Burnaby, BC, Canada (2012)

    Google Scholar 

  9. Adam, N., Eledath, J., Mehrotra, S., Venkatasubramanian, N.: Social media alert and response to threats to citizens (SMART-C). In: 2012 8th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom), pp. 181–189. IEEE (2012)

    Google Scholar 

  10. Abel, F., Hauff, C., Houben, G.-J., Stronkman, R., Tao, K.: Semantics + filtering + search = twitcident. Exploring information in social web streams. In: Proceedings of the 23rd ACM Conference on Hypertext and Social Media, pp. 285–294. ACM (2012)

    Google Scholar 

  11. Sakaki, T., Okazaki, M., Matsuo, Y.: Earthquake shakes Twitter users: real-time event detection by social sensors. In: Proceedings of the 19th International Conference on World Wide Web, pp. 851–860. ACM (2010)

    Google Scholar 

  12. Qusef, A., Ismail, K.: Social media in project communications management. In: 2016 7th International Conference on Computer Science and Information Technology (CSIT), pp. 1–5, July 2016

    Google Scholar 

  13. Treboux, J., Cretton, F., Evéquoz, F., Calvé, A.L., Genoud, D.: Mining and visualizing social data to inform marketing decisions. In: 2016 IEEE 30th International Conference on Advanced Information Networking and Applications (AINA), pp. 66–73, March 2016

    Google Scholar 

  14. Wan, S., Paris, C., Georgakopoulos, D.: Social media data aggregation and mining for internet-scale customer relationship management. In: 2015 IEEE International Conference on Information Reuse and Integration (IRI), pp. 39–48, August 2015

    Google Scholar 

  15. Burgess, J., Bruns, A.: Twitter archives and the challenges of “big social data” for media and communication research. M/C J. 15(5) (2012)

    Google Scholar 

  16. Yang, B., Guo, W., Chen, B., Yang, G., Zhang, J.: Estimating mobile traffic demand using Twitter. IEEE Wirel. Commun. Lett. 5(4), 380–383 (2016)

    Article  Google Scholar 

  17. Jackoway, A., Samet, H., Sankaranarayanan, J.: Identification of live news events using Twitter. In: Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Location-Based Social Networks, pp. 25–32. ACM (2011)

    Google Scholar 

  18. Ishikawa, S., Arakawa, Y., Tagashira, S., Fukuda, A.: Hot topic detection in local areas using Twitter and Wikipedia. In: ARCS Workshops (ARCS), pp. 1–5. IEEE (2012)

    Google Scholar 

  19. Petrovic, S., Osborne, M., Lavrenko, V.: The Edinburgh Twitter corpus. In: Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics in a World of Social Media, pp. 25–26 (2010)

    Google Scholar 

  20. Osborne, M., Petrovic, S., McCreadie, R., Macdonald, C., Ounis, I.: Bieber no more: first story detection using Twitter and Wikipedia. In: SIGIR 2012 Workshop on Time-Aware Information Access (2012)

    Google Scholar 

  21. Naveed, N., Gottron, T., Kunegis, J., Alhadi, A.C.: Bad news travel fast: a content-based analysis of interestingness on Twitter. In: Proceedings of the 3rd International Web Science Conference, WebSci 2011, pp. 8:1–8:7. ACM, New York (2011)

    Google Scholar 

  22. Corley, C.D., Cook, D.J., Mikler, A.R., Singh, K.P.: Text and structural data mining of influenza mentions in web and social media. Int. J. Environ. Res. Public Health 7(2), 596–615 (2010)

    Article  Google Scholar 

  23. Hwang, M.-H., Wang, S., Cao, G., Padmanabhan, A., Zhang, Z.: Spatiotemporal transformation of social media geostreams: a case study of Twitter for flu risk analysis. In: Proceedings of the 4th ACM SIGSPATIAL International Workshop on GeoStreaming, pp. 12–21. ACM (2013)

    Google Scholar 

  24. Polgreen, P.M., Chen, Y., Pennock, D.M., Nelson, F.D., Weinstein, R.A.: Using internet searches for influenza surveillance. Clin. Infect. Dis. 47(11), 1443–1448 (2008)

    Article  Google Scholar 

  25. Goel, S., Hofman, J.M., Lahaie, S., Pennock, D.M., Watts, D.J.: Predicting consumer behavior with web search. Proc. Natl. Acad. Sci. 107(41), 17486–17490 (2010)

    Article  Google Scholar 

  26. Scharkow, M., Vogelgesang, J.: Measuring the public agenda using search engine queries. Int. J. Public Opin. Res. 23(1), 104–113 (2011)

    Article  Google Scholar 

  27. Dugas, A.F., Hsieh, Y.-H., Levin, S.R., Pines, J.M., Mareiniss, D.P., Mohareb, A., Gaydos, C.A., Perl, T.M., Rothman, R.E.: Google flu trends: correlation with emergency department influenza rates and crowding metrics. Clin. Infect. Dis. 54(4), 463–469 (2012)

    Article  Google Scholar 

  28. Morrison, J.L., Breitling, R., Higham, D.J., Gilbert, D.R.: GeneRank: using search engine technology for the analysis of microarray experiments. BMC Bioinform. 6(1), 1 (2005)

    Article  Google Scholar 

  29. Ginsberg, J., Mohebbi, M.H., Patel, R.S., Brammer, L., Smolinski, M.S., Brilliant, L.: Detecting influenza epidemics using search engine query data. Nature 457(7232), 1012–1014 (2009)

    Article  Google Scholar 

  30. Lee, K., Agrawal, A., Choudhary, A.: Real-time disease surveillance using Twitter data: demonstration on flu and cancer. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1474–1477. ACM (2013)

    Google Scholar 

  31. Suh, B., Hong, L., Pirolli, P., Chi, Ed H.: Want to be retweeted? Large scale analytics on factors impacting retweet in Twitter network. In: 2010 IEEE Second International Conference on Social Computing (socialcom), pp. 177–184. IEEE (2010)

    Google Scholar 

  32. Broniatowski, D.A., Paul, M.J., Dredze, M.: National and local influenza surveillance through Twitter: an analysis of the 2012–2013 influenza epidemic. PLoS One 8(12), e83672 (2013)

    Article  Google Scholar 

  33. Lamb, A., Paul, M.J., Dredze, M.: Separating fact from fear: tracking flu infections on Twitter. In: North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013), June 2013

    Google Scholar 

  34. Aramaki, E., Maskawa, S., Morita, M.: Twitter catches the flu: detecting influenza epidemics using Twitter. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1568–1576. Association for Computational Linguistics (2011)

    Google Scholar 

  35. Santos, J.C., Matos, S.: Analysing Twitter and web queries for flu trend prediction. Theor. Biol. Med. Model. 11(1), S6 (2014)

    Article  Google Scholar 

  36. Cui, X., Yang, N., Wang, Z., Cheng, H., Zhu, W., Li, H., Ji, Y., Liu, C.: Chinese social media analysis for disease surveillance. Pers. Ubiquit. Comput. 19(7), 1125–1132 (2015)

    Article  Google Scholar 

  37. Byrd, K., Mansurov, A., Baysal, O.: Mining Twitter data for influenza detection and surveillance. In: IEEE/ACM International Workshop on Software Engineering in Healthcare Systems (SEHS), pp. 43–49. IEEE (2016)

    Google Scholar 

  38. Sanders, N.J.: Sanders-Twitter Sentiment Corpus (2011). http://www.sananalytics.com/lab/twitter-sentiment/. Accessed 20 Oct 2017

  39. Amazon MTurk: Amazon Mechanical Turk (MTurk)

    Google Scholar 

  40. Bird, S.: NLTK: the natural language toolkit. In: Proceedings of the COLING/ACL on Interactive Presentation Sessions, pp. 69–72. Association for Computational Linguistics (2006)

    Google Scholar 

  41. Singh, J., Gupta, V.: A systematic review of text stemming techniques. Artif. Intell. Rev. 48(2), 157–217 (2017)

    Article  Google Scholar 

  42. Joachims, T.: A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. Technical report, Carnegie-mellon univ pittsburgh pa dept of computer science (1996)

    Google Scholar 

  43. Loria, S., Keen, P., Honnibal, M., Yankovsky, R., Karesh, D., Dempsey, E., et al.: TextBlob: simplified text processing. Secondary TextBlob: Simplified Text Processing (2014)

    Google Scholar 

  44. Zweig, M.H., Campbell, G.: Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin. Chem. 39(4), 561–577 (1993)

    Google Scholar 

  45. Alessa, A., Faezipour, M.: A review of influenza detection and prediction through social networking sites. Theor. Biol. Med. Model. 15(2), 1–27 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering, University of Bridgeport, Bridgeport, CT, 06604, USA

    Ali Alessa & Miad Faezipour

Authors
  1. Ali Alessa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miad Faezipour
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miad Faezipour .

Editor information

Editors and Affiliations

  1. Institute of Computer Vision and Applied Computer Sciences, Leipzig, Germany

    Petra Perner

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alessa, A., Faezipour, M. (2018). Tweet Classification Using Sentiment Analysis Features and TF-IDF Weighting for Improved Flu Trend Detection. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2018. Lecture Notes in Computer Science(), vol 10934. Springer, Cham. https://doi.org/10.1007/978-3-319-96136-1_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-96136-1_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-96135-4

  • Online ISBN: 978-3-319-96136-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics