Skip to main content
SpringerLink
Log in

Deep dynamic neural networks for temporal language modeling in author communities

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Language models are at the heart of numerous works, notably in the text mining and information retrieval communities. These statistical models aim at extracting word distributions, from simple unigram models to recurrent approaches with latent variables that capture subtle dependencies in texts. However, those models are learned from word sequences only, and authors’ identities, as well as publication dates, are seldom considered. We propose a neural model, based on recurrent language modeling (e.g., LSTM), which aims at capturing language diffusion tendencies in author communities through time. By conditioning language models with author and dynamic temporal vector states, we are able to leverage the latent dependencies between the text contexts. The model captures language evolution of authors via a shared temporal prediction function in a latent space, which allows to handle a variety of modeling tasks, including completion and prediction of language models through time. Experiments show the performances of the approach, compared to several temporal and non-temporal language baselines on two real-world corpora.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Code available at https://github.com/edouardelasalles/dar.

  2. Please note that this dataset contains papers with multi-authors. In the experiments reported below, we consider each text as written individually by each of its authors (i.e., it is contained in the individual text set of each of its authors). Combinations of author representations to model multi-authorship could correspond to an interesting perspective as future work.

  3. We did not use WordPiece in the NYT corpus since we noticed in experiments that it led to dramatic overfitting.

References

  1. Ammar W, Groeneveld D, Bhagavatula C, Beltagy I, Crawford M, Downey D, Dunkelberger J, Elgohary A, Feldman S, Ha V, Kinney R, Kohlmeier S, Lo K, Murray T, Ooi H-H, Peters M, Power J, Skjonsberg S, Wang LL, Wilhelm C, Yuan Z, van Zuylen M, Etzioni O (2018) Construction of the literature graph in semantic scholar. In: NAACL

  2. Bai S, Kolter JZ, Koltun V (2018) An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. In: CoRR

  3. Bamler R, Mandt S (2017) Dynamic word embeddings. In: ICML

  4. Bengio Y, Ducharme R, Vincent P, Jauvin C (2003) A neural probabilistic language model. JMLR 3:1137–1155

    MATH  Google Scholar 

  5. Bishop CM, Hinton GE, Strachan IG (1997) GTM through time. In: ICANN

  6. Blei DM, Lafferty JD (2006) Dynamic topic models. In: ICML

  7. Blei DM, Ng AY, Jordan MI (2002) Latent dirichlet allocation. JMLR 3:993–1022

    MATH  Google Scholar 

  8. Chiu C-C, Sainath TN, Wu Y, Prabhavalkar R, Nguyen P, Chen Z, Kannan A, Weiss RJ, Rao K, Gonina K et al (2018) State-of-the-art speech recognition with sequence-to-sequence models. In: ICASSP

  9. Chung J, Kastner K, Dinh L, Goel K, Courville AC, Bengio Y (2015) A recurrent latent variable model for sequential data. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol 28. Curran Associates Inc, Red Hook, pp 2980–2988

    Google Scholar 

  10. Devlin J, Chang M-W, Lee K, Toutanova K (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: ACL

  11. Ding SH, Fung BC, Iqbal F, Cheung WK (2017) Learning stylometric representations for authorship analysis. IEEE Trans Cybern 49:107–121

    Article  Google Scholar 

  12. Eger S, Mehler A (2017) On the linearity of semantic change: Investigating meaning variation via dynamic graph models. In: ACL

  13. Fedus W, Goodfellow I, Dai AM (2018) Maskgan: better text generation via filling in the\_. In: ICLR

  14. Fraccaro M, Sønderby SK, Paquet U, Winther O (2016) Sequential neural models with stochastic layers. In: NeurIPS

  15. Gerrish SM, Blei DM (2010) A language-based approach to measuring scholarly impact. In: ICML

  16. Hall D, Jurafsky D, Manning CD (2008) Studying the history of ideas using topic models. In: EMNLP

  17. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  18. Iwata T, Yamada T, Sakurai Y, Ueda N (2012) Sequential modeling of topic dynamics with multiple timescales. In: TKDD

  19. Kabán A, Girolami MA (2002) A dynamic probabilistic model to visualise topic evolution in text streams. JIIS 18:107–125

    Google Scholar 

  20. Kenter T, Wevers M, Huijnen P, de Rijke M (2015) Ad hoc monitoring of vocabulary shifts over time. In: ICIKM

  21. Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: ICLR

  22. Krishnan RG, Shalit U, Sontag D (2017) Structured inference networks for nonlinear state space models. AAAI, Menlo Park

    Google Scholar 

  23. Lample G, Ott M, Conneau A, Denoyer L, Ranzato M (2018) Phrase-based and neural unsupervised machine translation. In: EMNLP

  24. Le QV, Mikolov T (2014) Distributed representations of sentences and documents. In: ICML

  25. Loper E, Bird S (2002) Nltk: the natural language toolkit. In: ACL workshop on effective tools and methodologies for teaching natural language processing and computational linguistics

  26. Melis G, Dyer C, Blunsom P (2018) On the state of the art of evaluation in neural language models. In: ICLR

  27. Merity S, Keskar NS, Socher R (2018a) An analysis of neural language modeling at multiple scales. In: CoRR

  28. Merity S, Keskar NS, Socher R (2018b) Regularizing and optimizing LSTM language models. In: ICLR

  29. Mikolov T, Karafiát M, Burget L, Černockỳ J, Khudanpur S (2010) Recurrent neural network based language model. In: ISCA

  30. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2014) Distributed representations of words and phrases and their compositionality. In: NeurIPS

  31. Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: NAACL

  32. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. In: OpenAI Blog

  33. Rezende DJ, Mohamed S, Wierstra D (2014) Stochastic backpropagation and approximate inference in deep generative models. In: ICML

  34. Rudolph MR, Blei DM (2017) Dynamic bernoulli embeddings for language evolution. In: CoRR

  35. Rudolph MR, Ruiz FJR, Athey S, Blei DM (2017) Structured embedding models for grouped data. In: NeurIPS

  36. Saito K, Kimura M, Ohara K, Motoda H (2009) Learning continuous-time information diffusion model for social behavioral data analysis. ACML, Ahmedabad

    Book  Google Scholar 

  37. Song F, Croft WB (1999) A general language model for information retrieval. In: ICIKM

  38. Subramanian S, Lample G, Smith EM, Denoyer L, Ranzato M, Boureau Y-L (2018) Multiple-attribute text style transfer. In: ICLR

  39. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: NeurIPS

  40. Vinyals O, Toshev A, Bengio S, Erhan D (2017) Show and tell: lessons learned from the 2015 mscoco image captioning challenge. In: PAMI

  41. Wang C, Blei D, Heckerman D (2008) Continuous time dynamic topic models. In: UAI

  42. Wang E, Silva J, Willett R, Carin L (2011) Dynamic relational topic model for social network analysis with noisy links. In: SSP

  43. Wang W, Gan Z, Xu H, Zhang R, Wang G, Shen D, Chen C, Carin L (2019) Topic-guided variational auto-encoder for text generation. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). Association for Computational Linguistics, Minneapolis, pp 166–177

  44. Wang X, Mccallum A (2006) Topics over time: A non-Markov continuous-time model of topical trends. In: SIGKDD

  45. Yang Z, Hu Z, Dyer C, Xing EP, Berg-Kirkpatrick T (2018) Unsupervised text style transfer using language models as discriminators. In: NeurIPS

  46. Yao Z, Sun Y, Ding W, Rao N, Xiong H (2017) Discovery of evolving semantics through dynamic word embedding learning. In: CoRR

  47. Yao Z, Sun Y, Ding W, Rao N, Xiong H (2018) Dynamic word embeddings for evolving semantic discovery. In: WSDM

Download references

Author information

Authors and Affiliations

  1. LIP6, Sorbonne Université, 75005, Paris, France

    Edouard Delasalles, Sylvain Lamprier & Ludovic Denoyer

Authors
  1. Edouard Delasalles
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sylvain Lamprier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ludovic Denoyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sylvain Lamprier.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Delasalles, E., Lamprier, S. & Denoyer, L. Deep dynamic neural networks for temporal language modeling in author communities. Knowl Inf Syst 63, 733–757 (2021). https://doi.org/10.1007/s10115-020-01539-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-020-01539-z

Keywords

Search

Navigation