Skip to main content
SpringerLink
Log in

Mining streams of short text for analysis of world-wide event evolutions

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Streams of short text, such as news titles, enable us to effectively and efficiently learn the real world events that occur anywhere and anytime. Short text messages that are companied by timestamps and generally brief events using only a few words differ from other longer text documents, such as web pages, news stories, blogs, technical papers and books. For example, few words repeat in the same news titles, thus frequency of the term (i.e., TF) is not as important in short text corpus as in longer text corpus. Therefore, analysis of short text faces new challenges. Also, detecting and tracking events through short text analysis need to reliably identify events from constant topic clusters; however, existing methods, such as Latent Dirichlet Allocation (LDA), generates different topic results for a corpus at different executions. In this paper, we provide a Finding Topic Clusters using Co-occurring Terms (FTCCT) algorithm to automatically generate topics from a short text corpus, and develop an Event Evolution Mining (EEM) algorithm to discover hot events and their evolutions (i.e., the popularity degrees of events changing over time). In FTCCT, a term (i.e., a single word or a multiple-words phrase) belongs to only one topic in a corpus. Experiments on news titles of 157 countries within 4 months (from July to October, 2013) demonstrate that our FTCCT-based method (combining FTCCT and EEM) achieves far higher quality of the event’s content and description words than LDA-based method (combining LDA and EEM) for analysis of streams of short text. Our method also visualizes the evolutions of the hot events. The discovered world-wide event evolutions have explored some interesting correlations of the world-wide events; for example, successive extreme weather phenomenon occur in different locations - typhoon in Hong Kong and Philippines followed hurricane and storm flood in Mexico in September 2013.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Allan, J., Papka, R., and Lavrenko, V.: On-line new event detection and traking, Proc. of the 21st ACM International Conference on Research and Development in Information Retrieval, 1998.

  2. Becker, H., Naaman, M., and Gravano, L.: Beyond trending topics: real-world event identification on Twitter, ICWSM′11, 2011.

  3. Blei, D. M., and Lafferty, J. D.: Dynamic topic models, Proc. of ICML′06, pp. 113–120, 2006.

  4. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J Mach Learn Res 3, 993–1022 (2003)

    MATH  Google Scholar 

  5. Cui, W., Liu, S., Tan, L., Shi, C., Song, Y., Gao, Z.J., Tong, X., Qu, H.: Textflow: towards better understanding of evolving topics in text. IEEE Trans Vis Comput Graph 17(12), 2412–2421 (2011)

    Article  Google Scholar 

  6. Dou, W., Wang, X., Skau, D., Ribarsky, W., and Zhou, M. X.: LeadLine: interactive visual analysis of text data through event identification and exploration, Proc. of IEEE Conference on Visual Analytics Science and Technology, pp. 93–102, 2012.

  7. Ester, M., Kregel, H. P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise, SIGKDD′96, pp. 226–231, 1996.

  8. Ha-Thuc, V., Mejova, Y., Harris C., and Srinivasan, P.: A relevance-based topic model for news event tracking, SIGIR′09, pp. 764–765, 2009.

  9. Havre, S., Hetzler, B., Nowell, L.: ThemeRiver: visualizing theme changes over time, Proc. of IEEE Symposium on Information Visualization, 2000.

  10. Hu, Y., John, A., Wang, F., Kambhampati, S.: Et-lda: Joint topic modeling for aligning events and their twitter feedback, AAAI′12, pp. 59–65, 2012.

  11. Jaccard, P.: Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc vaud sci nat 37, 547–579 (1901)

    Google Scholar 

  12. Kleinberg, J.: Bursty and hierarchical structure in streams, Proc. of ACM SIGKDD′02, 2002.

  13. Kleinberg, J.: Temporal dynamics of on-line information streams, In Data Stream Management: Processing High-Speed Data, 2006.

  14. Lau, J., Collier, N., Baldwin, T.: On-line trend analysis with topic models: twitter trends detection topic model online, Proc. of COLING′12, pp. 1519–1534, 2012.

  15. Michelson, M., Macskassy, S. A.: Discovering users topics of interest on twitter: a first look. Proc. of 4th Workshop on Analytics for Noisy Unstructured Text Data, 2010.

  16. Reed, C.: Latent Dirichlet allocation: towards a deeper understanding, mlg.eng.cam.ac.uk/creed/Notes/lda-tutorial-reed.pdf, 2012.

  17. Robertson, S.: Understanding inverse document frequency: on theoretical arguments for IDF. J Doc 60(5), 503–520 (2004)

    Article  Google Scholar 

  18. Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Inf Process Manag 24(5), 513–523 (1988)

    Article  Google Scholar 

  19. Shin, Y., Ryo, C., Park, J.: Automatic extraction of persistent topics from social text streams. World Wide Web J (2013). doi:10.1007/s11280-013-0251-3

    Google Scholar 

  20. Van Rijsbergen, C.J.: A theoretical basis for the use of co-occurrence data in information retrieval. J Doc 33(2), 106–119 (1977)

    Article  Google Scholar 

  21. Wang, C., Blei, D. M., and Heckerman, D.: Continuous time dynamic topic models, Proc. of the 24th Conference on Uncertainty in Artificial Intelligence (UAI′08), pp. 579–586, 2008.

  22. Wang, X. and McCallum, A.: Topics over time: a non-Markov continuous time model of topical trends, Proc. of ACM SIGKDD′06, pp. 424–433, 2006.

  23. Wang, X., Mohanty, N., and McCallum, A.: Group and topic discovery from relations and text, Proc. of SIGKDD Workshop on Link Discovery: Issues, Approaches and Applications (LinkKDD′05), pp. 28–35, 2005.

  24. You, Y., Huang, G., Cao, J., Chen, E., He, J., Zhang, Y., and Hu, L.: GEAM: A general and event-related aspects model for twitter event detection, WISE 2013, Part II, LNCS 8181, pp. 319–332, 2013.

  25. Zhang, J., Song, Y., Zhang, C., and Liu, S.: Evolutionary hierarchical dirichlet processes for multiple correlated time-varying corpora, Proc. of ACM SIGKDD′10, pp. 1079–1088, 2010.

Download references

Author information

Authors and Affiliations

  1. School of Information Technology, Deakin University, Victoria, Australia

    Guangyan Huang & Wanlei Zhou

  2. Centre for Applied Informatics, College of Engineering and Science, Victoria University, Victoria, Australia

    Jing He, Yanchun Zhang & Peng Zhang

  3. Computer Science Department, South China Normal University, Guangzhou, China

    Hai Liu

  4. State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, China

    Zhiming Ding

  5. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Yue You & Jian Cao

Authors
  1. Guangyan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanchun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wanlei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhiming Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yue You
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jian Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangyan Huang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, G., He, J., Zhang, Y. et al. Mining streams of short text for analysis of world-wide event evolutions. World Wide Web 18, 1201–1217 (2015). https://doi.org/10.1007/s11280-014-0293-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-014-0293-1

Keywords

Search

Navigation