To model sequentially observed multivariate nonstationary count data, we propose a switching Poisson-gamma dynamical systems (SPGDS), a dynamic probabilistic network with switching mechanism. Different from previous models, SPGDS assigns its latent variables into mixture of gamma distributed parameters to model complex sequences and describe the nonlinear dynamics, meanwhile, capture various temporal dependencies. Moreover, SPGDS can model all discrete and nonnegative real data by linking them to latent counts. To take advantage of Bayesian nonparametrics in handling the unknown number of mixture components, we integrate Dirichlet process (DP) mixture into SPGDS and develop an infinite switching Poisson-gamma dynamical systems (iSPGDS). For efficient and nonparametric inference, we develop a infinite switching recurrent variational inference network, combined with a scalable hybrid stochastic gradient-MCMC and variational inference method, which is scalable to large scale sequences and fast in out-of-sample prediction. Besides, to handle the time-series categorization task, we further propose an supervised attention iSPGDS (attn-iSPGDS), which combines the representation power of iSPGDS, discriminative power of deep neural networks, and selection power of the attention mechanism under a principled probabilistic framework. Experiments on both unsupervised and supervised tasks demonstrate that the proposed model not only has excellent fitting and prediction performance on complex sequences, but also separates different dynamical patterns within them.