Abstract
We here present an operational deep learning pipeline for classifying life events from individual tweets, using job loss as a use case and Twitter data collected between 2010 and 2013 (historic sample from the public stream). The pipeline includes identification of keywords through snowball sampling, multiple rater manual annotation, supervised deep learning, text processing (word embedding, bag of words) and architecture selection (convolutional, shallow-and-wide convolutional, and long-short-term memory) with parameter optimization, external validation and feedback learning. After model optimization, a shallow-and-wide network with a pre-trained 200-dimensional word2vec achieved a precision of 78% (over an average single keyword precision of 50%) and an area under receiver operating characteristic of 86%. Precision and recall also increased by 5% using bag of words. When tested on tweets with ambiguous annotations (i.e. tweets that were hard for human annotators to classify), the network achieved 65% precision. Finally, on a random set of tweets that did not contain any of the snowballed keywords, 30% were classified as job loss events; this putatively false positive set can be used to reinforce the learner’s training. In conclusion, the pipeline streamlines both the manual and automated process, providing feedback reinforcement (snowballing and external tweets), and shows good performance on classifying individual tweets on the use case, potentially saving human resources needed to collate such data for research studies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Reece, A.G., Reagan, A.J., Lix, K.L.M., Dodds, P.S., Danforth, C.M., Langer, E.J.: Forecasting the onset and course of mental illness with Twitter data. Sci. Rep. 7(1), 13006 (2017). https://doi.org/10.1038/s41598-017-12961-9
Tausczik, Y.R., Pennebaker, J.W.: The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29(1), 24–54 (2010). https://doi.org/10.1177/0261927X09351676
Alsaedi, N., Burnap, P., Rana, O.: Can we predict a riot? Disruptive event detection using Twitter. ACM Trans. Internet Technol. 17(2), 18:1–18:26 (2017). https://doi.org/10.1145/2996183
Sumner, C., Byers, A., Boochever, R., Park, G.J.: Predicting dark triad personality traits from Twitter usage and a linguistic analysis of tweets. In: 2012 11th International Conference on Machine Learning and Applications, vol. 2, pp. 386–393 (2012). https://doi.org/10.1109/ICMLA.2012.218
Makazhanov, A., Rafiei, D.: Predicting political preference of Twitter users. In: 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013), pp. 298–305 (2013). https://doi.org/10.1145/2492517.2492527
Conover, M., Gonçalves, B., Ratkiewicz, J., Flammini, A., Menczer, F.: Predicting the political alignment of Twitter users. In: 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, pp. 192–199 (2011). https://doi.org/10.1109/PASSAT/SocialCom.2011.34
Ramos, J.P.: Using TF-IDF to determine word relevance in document queries. Presented at the First International Conference on Machine Learning, New Brunswick, NJ, USA (2003)
Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42(1–2), 177–196 (2001). https://doi.org/10.1023/A:1007617005950
Won, D., Steinert-Threlkeld, Z.C., Joo, J.: Protest activity detection and perceived violence estimation from social media images. ArXiv:1709.06204 [Cs] (2017). http://arxiv.org/abs/1709.06204
Zhang, Z., He, Q., Gao, J., Ni, M.: A deep learning approach for detecting traffic accidents from social media data. Transp. Res. Part C: Emerg. Technol. 86, 580–596 (2018). https://doi.org/10.1016/j.trc.2017.11
Founta, A.-M., Chatzakou, D., Kourtellis, N., Blackburn, J., Vakali, A., Leontiadis, I.: A unified deep learning architecture for abuse detection. ArXiv:1802.00385 [Cs] (2018). http://arxiv.org/abs/1802.00385
Badjatiya, P., Gupta, S., Gupta, M., Varma, V.: Deep learning for hate speech detection in tweets. In: Proceedings of the 26th International Conference on World Wide Web Companion, pp. 759–760. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2017). https://doi.org/10.1145/3041021.3054223
Pitsilis, G.K., Ramampiaro, H., Langseth, H.: Detecting offensive language in tweets using deep learning. ArXiv:1801.04433 [Cs] (2018). http://arxiv.org/abs/1801.04433
Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation, pp. 1532–1543. Association for Computational Linguistics (2014). https://doi.org/10.3115/v1/D14-1162
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality, p. 9 (n.d.)
Bian, J., et al.: Mining Twitter to assess the public perception of the “internet of things”. PLoS One 11(7), e0158450 (2016). https://doi.org/10.1371/journal.pone.0158450
McHugh, M.L.: Interrater reliability: the kappa statistic. Biochem. Med. 22(3), 276–282 (2012)
Goodman, L.A.: Snowball sampling. Ann. Math. Stat. 32(1), 148–170 (1961). https://doi.org/10.1214/aoms/1177705148
Nadeau, C., Bengio, Y.: Inference for the generalization error. In: Proceedings of the 12th International Conference on Neural Information Processing Systems, pp. 307–313. MIT Press, Cambridge (1999). http://dl.acm.org/citation.cfm?id=3009657.3009701
LeCun, Y., Bengio, Y.: Convolutional networks for images, speech, and time-series, p. 15 (n.d.)
Le, H.T., Cerisara, C., Denis, A.: Do convolutional networks need to be deep for text classification? ArXiv:1707.04108 [Cs] (2017). http://arxiv.org/abs/1707.04108
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Das, S., Wong, W.K., Dietterich, T., Fern, A., Emmott, A.: Incorporating expert feedback into active anomaly discovery. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 853–858 (2016). https://doi.org/10.1109/ICDM.2016.0102
Yin, W., Kann, K., Yu, M., Schütze, H.: Comparative study of CNN and RNN for natural language processing (2017). https://arxiv.org/abs/1702.01923
Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. ArXiv:1509.01626 [Cs] (2015). http://arxiv.org/abs/1509.01626
The Xapian Project (n.d.). https://xapian.org/. Assessed 28 June 2018
Welcome to Python.org (n.d.). https://www.python.org/. Assessed 28 June 2018
TensorFlow (n.d.). https://www.tensorflow.org/. Assessed 28 June 2018
Palinkas, L.A., Horwitz, S.M., Green, C.A., Wisdom, J.P., Duan, N., Hoagwood, K.: Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm. Policy Ment. Health 42(5), 533–544 (2015)
Acknowledgements
MP, JB, and XD are in part supported by US NSF grant SES 1734134.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Du, X., Bian, J., Prosperi, M. (2019). An Operational Deep Learning Pipeline for Classifying Life Events from Individual Tweets. In: Lossio-Ventura, J., Muñante, D., Alatrista-Salas, H. (eds) Information Management and Big Data. SIMBig 2018. Communications in Computer and Information Science, vol 898. Springer, Cham. https://doi.org/10.1007/978-3-030-11680-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-11680-4_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-11679-8
Online ISBN: 978-3-030-11680-4
eBook Packages: Computer ScienceComputer Science (R0)