Skip to main content
SpringerLink
Log in

Learning general temporal point processes based on dynamic weight generation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Most real-world events relate to historical cases scholastically and chronologically. Some increase the probability of the next event in the future, while some do not. Many approaches leverage temporal point processes with explicitly defined probability intensity functions concerning time t and history \({\mathscr{H}}\), such as Poisson Process and Hawkes Process, to measure the time-varying probability. However, fixed-form intensity functions can limit the performance of temporal point process models, owing to the lack of prior knowledge of required intensity functions and the complexity of the real world. Neural networks’ ability to approximate functions makes the neural temporal point process a promising approach compared with the traditional temporal point process. In the paper, we identify several drawbacks the previous works have in meeting the mathematical constraints before designing a matrix-multiplication-based model to tackle these drawbacks. The experimental results reveal that our model is superior to other neural temporal point processes with better mathematical interpretability and extrapolation capability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://www.kaggle.com/colemaclean/subreddit-interactions

  2. https://www.kaggle.com/caganseval/earthquake

References

  1. Du N, Dai H, Trivedi R, Upadhyay U, Gomez-Rodriguez M, Song L (2016) Recurrent marked temporal point processes. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, USA, pp 1555–1564

  2. Mei H, Eisner J (2017) The neural hawkes process: A neurally self-modulating multivariate point process. In: Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp 6754–6764

  3. Zuo S, Jiang H, Li Z, Zhao T, Zha H (2020) Transformer Hawkes process. In: Proceedings of the 37th international conference on machine learning, Proceedings of machine learning research. PMLR, vol 119, pp 11692–11702

  4. Li S, Xiao S, Zhu S, Du N, Xie Y, Song L (2018) Learning temporal point processes via reinforcement learning. In: Advances in neural information processing systems 31: Annual conference on neural information processing systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal Canada, pp 10804–10814

  5. Guo R, Li J, Liu H (2018) INITIATOR: noise-contrastive estimation for marked temporal point process. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, IJCAI 2018, July 13-19, 2018. ijcai.org, Stockholm, Sweden, pp 2191–2197

  6. Okawa M, Iwata T, Kurashima T, Tanaka Y, Toda H, Ueda N (2019) Deep mixture point processes: Spatio-temporal event prediction with rich contextual information. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, KDD 2019, August 4-8, 2019. ACM, Anchorage, AK, USA, pp 373–383

  7. Xiao S, Yan J, Farajtabar M, Song L, Yang X, Zha H (2019) Learning time series associated event sequences with recurrent point process networks. IEEE Trans Neural Netw Learn Syst 30(10):3124–3136

    Article  Google Scholar 

  8. Omi T, Ueda N, Aihara K (2019) Fully neural network based model for general temporal point processes. In: Advances in neural information processing systems 32: Annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp 2120–2129

  9. Shchur O, Bilos M, Günnemann S (2020) Intensity-free learning of temporal point processes. In: 8th international conference on learning representations, ICLR 2020, April 26-30, 2020. OpenReview.net, Addis Ababa, Ethiopia

  10. Enguehard J, Busbridge D, Bozson A, Woodcock C, Hammerla N (2020) Neural temporal point processes for modelling electronic health records. In: Proceedings of the machine learning for health NeurIPS workshop, Proceedings of machine learning research. PMLR, vol 136, pp 85–113

  11. Mei H, Wan T, Eisner J (2020) Noise-contrastive estimation for multivariate point processes. In: Advances in neural information processing systems, vol 33. Curran Associates Inc, pp 5204– 5214

  12. Sütfeld LR, Brieger F, Finger H, Füllhase S, Pipa G (2020) Adaptive blending units: trainable activation functions for deep neural networks. In: Intelligent computing - proceedings of the 2020 computing conference, volume 3, advances in intelligent systems and computing, vol 1230. Springer, pp 37–50

  13. Rubanova Y, Chen TQ, Duvenaud D (2019) Latent ordinary differential equations for irregularly-sampled time series. In: Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC Canada, pp 5321–5331

  14. Jia J, Benson AR (2019) Neural jump stochastic differential equations. In: Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC Canada, pp 9843–9854

  15. Chen RTQ, Amos B, Nickel M (2021) Learning neural event functions for ordinary differential equations. In: 9th international conference on learning representations, ICLR 2021, virtual only, May 3–7, 2021

  16. Chen RTQ, Amos B, Nickel M (2021) Neural spatio-temporal point processes. In: 9th international conference on learning representations, ICLR 2021, virtual only, May 3–7, 2021

  17. Montavon G, Samek W, Müller KR (2018) Methods for interpreting and understanding deep neural networks. Digit Sig Process 73:1–15

    Article  MathSciNet  Google Scholar 

  18. Jacot A, Hongler C, Gabriel F (2018) Neural tangent kernel: Convergence and generalization in neural networks. In: Advances in neural information processing systems 31: annual conference on neural information processing systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp 8580–8589

  19. Chen M, Jiang H, Liao W, Zhao T (2019) Efficient approximation of deep relu networks for functions on low dimensional manifolds. In: Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp 8172–8182

  20. Xu K, Zhang M, Li J, Du SS, Kawarabayashi KI, Jegelka S (2021) How neural networks extrapolate: From feedforward to graph neural networks. In: 9th international conference on learning representations, ICLR 2021, virtual only, May 3–7, 2021

  21. Nguyen T, Raghu M, Kornblith S (2021) Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. In: 9th international conference on learning representations, ICLR 2021, virtual only, May, 3–7, 2021

  22. Shokri R, Stronati M, Song C, Shmatikov V (2017) Membership inference attacks against machine learning models. In: 2017 IEEE symposium on security and privacy (SP), pp 3–18

  23. Yeom S, Giacomelli I, Fredrikson M, Jha S (2018) Privacy risk in machine learning: Analyzing the connection to overfitting. In: 2018 IEEE 31st computer security foundations symposium (CSF), pp 268–282

  24. Ramsauer H, Schäfl B, Lehner J, Seidl P, Widrich M, Gruber L, Holzleitner M, Adler T, Kreil D, Kopp MK, Klambauer G, Brandstetter J, Hochreiter S (2021) Hopfield networks is all you need. In: 9th international conference on learning representations, ICLR 2021, virtual only May 3–7, 2021

  25. Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R (2020) NeRF: Representing scenes as neural radiance fields for view synthesis. In: Computer vision - ECCV 2020 - 16th european conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, vol 12346, pp 405–421

  26. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA USA, pp 5998–6008

  27. Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, vol 1 (Long and Short Papers), pp 4171–4186

  28. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2021) An image is worth 16x16 words: transformers for image recognition at scale. In: 9th international conference on learning representations, ICLR 2021, Virtual Only May 3–7, 2021

  29. Sill J (1998) Monotonic networks. In: Advances in neural information processing systems, vol 10, pp 661–667

  30. You S, Ding D, Canini KR, Pfeifer J, Gupta MR (2017) Deep lattice networks and partial monotonic functions. In: Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp 2981–2989

  31. Wehenkel A, Louppe G (2019) Unconstrained monotonic neural networks. In: Advances in neural information processing systems 32: Annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp 1543–1553

  32. Zhuang T, Zhang Z, Huang Y, Zeng X, Shuang K, Li X (2020) Neuron-level structured pruning using polarization regularizer. Adv Neural Inf Process Syst, vol 33

  33. Mescheder LM, Geiger A, Nowozin S (2018) Which training methods for gans do actually converge?. In: Proceedings of the 35th international conference on machine learning, ICML 2018, Stockholmsmässan, July 10-15, 2018, Proceedings of machine learning research. PMLR, vol 80, Stockholm, Sweden, pp 3478–3487

  34. Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: 7th international conference on learning representations, ICLR 2019, May 6–9, 2019, New Orleans, LA, USA

  35. Lobov SA, Mikhaylov AN, Shamshin M, Makarov VA, Kazantsev VB (2020) Spatial properties of stdp in a self-learning spiking neural network enable controlling a mobile robot. Front. Neurosci. 14:88

    Article  Google Scholar 

  36. Naveros F, Luque NR, Ros E, Arleo A (2020) Vor adaptation on a humanoid icub robot using a spiking cerebellar model. IEEE Trans Cybern 50(11):4744–4757

    Article  Google Scholar 

  37. Tang G, Shah A, Michmizos KP (2019) Spiking neural network on neuromorphic hardware for energy-efficient unidimensional slam. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 4176–4181

  38. Mozafari M, Kheradpisheh SR, Masquelier T, Nowzari-Dalini A, Ganjtabesh M (2018) First-spike-based visual categorization using reward-modulated stdp. IEEE Trans Neural Netw Learn Syst 29(12):6178–6190

    Article  Google Scholar 

  39. Zhu L, Dong S, Li J, Huang T, Tian Y (2020) Retina-like visual image reconstruction via spiking neural model. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1435–1443

  40. Yu Q, Yan R, Tang H, Tan KC, Li H (2016) A spiking neural network system for robust sequence recognition. IEEE Trans Neural Netw Learn Syst 27:621–635

    Article  MathSciNet  Google Scholar 

  41. Song Z, Xiang S, Ren Z, Han G, Hao Y (2020) Spike sequence learning in a photonic spiking neural network consisting of vcsels-sa with supervised training. IEEE J Sel Top Quant Electron 26(5):1–9

    Article  Google Scholar 

  42. Maciag PS, Kryszkiewicz M, Bembenik RL, Lobo J, Del Ser J (2021) Unsupervised anomaly detection in stream data with online evolving spiking neural networks. Neural Netw 139:118–139

    Article  Google Scholar 

  43. Zhang M, Qu H, Belatreche A, Chen Y, Yi Z (2019) A highly effective and robust membrane potential-driven supervised learning method for spiking neurons. IEEE Trans Neural Netw Learn Syst 30(1):123–137

    Article  Google Scholar 

  44. Wang X, Lin X, Dang X (2020) Supervised learning in spiking neural networks: A review of algorithms and evaluations. Neural Netw 125:258–280

    Article  Google Scholar 

  45. Yang S, Gao T, Wang J, Deng B, Lansdell B, Linares-Barranco B (2021) Efficient spikedriven learning with dendritic event-based processing. Front Neurosci, vol 15. Publisher: Frontiers

  46. Huh D, Sejnowski TJ (2018) Gradient descent for spiking neural networks. In: Advances in neural information processing systems 31: Annual conference on neural information processing systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp 1440–1450

  47. Yang S, Wang J, Zhang N, Deng B, Pang Y, Azghadi MR (2021) CerebelluMorphic: large-scale neuromorphic model and architecture for supervised motor learning. IEEE Trans Neural Netw Learn Syst, pp 1–15

  48. Yang S, Deng B, Wang J, Liu C, Li H, Lin Q, Fietkiewicz C, Loparo KA (2019) Design of hidden-property-based variable universe fuzzy control for movement disorders and its efficient reconfigurable implementation. IEEE Trans Fuzzy Syst 27(2):304–318

    Article  Google Scholar 

  49. Yang S, Deng B, Wang J, Li H, Lu M, Che Y, Wei X, Loparo KA (2020) Scalable digital neuromorphic architecture for large-scale biophysically meaningful neural network with multi-compartment neurons. IEEE Trans Neural Netw Learn Syst 31(1):148–162

    Article  Google Scholar 

  50. Zhao Q, Erdogdu MA, He HY, Rajaraman A, Leskovec J (2015) SEISMIC: A self-exciting point process model for predicting tweet popularity. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, August 10-13, 2015. ACM, Sydney, NSW, Australia, pp 1513–1522

Download references

Acknowledgements

This work is funded by the National Natural Science Foundation of China (ID:61877051) and the National Science Foundation Project of CQ (ID: cstc2018jscx-msyb1042 and cstc2018jscx-msybX0273). We thank Senlin Shu and Xiaofei Xu for all the inspiring discussions about the design of the dynamic weight layer. Additionally, we acknowledge all the developers and researchers for developing useful machine learning tools that enable our experiments.

Author information

Authors and Affiliations

  1. School of Computer and Information Science, Southwest University, Chongqing, China

    Sishun Liu & Li Li

Authors
  1. Sishun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Li, L. Learning general temporal point processes based on dynamic weight generation. Appl Intell 52, 3678–3690 (2022). https://doi.org/10.1007/s10489-021-02590-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02590-1

Keywords

Search

Navigation