Abstract
Massive spatio-temporal data are challenging for statistical analysis due to their low signal-to-noise ratios and high-dimensional spatio-temporal structure. To resolve these issues, we propose a novel Dirichlet process particle filter (DPPF) model. The Dirichlet process models a set of stochastic functions as probability distributions for dimension reduction, and the particle filter is used to solve the nonlinear filtering problem with sequential Monte Carlo steps where the data has a low signal-to-noise ratio. Our data set is derived from surveillance data on emergency visits for influenza-like and respiratory illness (from 2008 to 2010) from the Indiana Public Health Emergency Surveillance System. The DPPF develops a dynamic data-driven applications system (DDDAS) methodology for disease outbreak detection. Numerical results show that our model significantly improves the outbreak detection performance in real data analysis.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
C.E. Antoniak, Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Stat. 2, 1152–1174 (1974)
C.H. Bishop, B.J. Etherton, S.J. Majumdar, Adaptive sampling with the ensemble transform Kalman filter. Part I: theoretical aspects. Mon. Weather Rev. 129(3), 420–436 (2001)
D. Blackwell, J.B. MacQueen, Ferguson distributions via pólya urn schemes. Ann. Stat. 1, 353–355 (1973)
R. Brookmeyer, D.F. Stroup, Monitoring the Health of Populations: Statistical Principles and Methods for Public Health Surveillance (Oxford University Press, New York, 2003)
G. Burgers, P. Jan van Leeuwen, G. Evensen, Analysis scheme in the ensemble Kalman filter. Mon. Weather Rev. 126(6), 1719–1724 (1998)
K. Burghardt et al., Testing modeling assumptions in the West Africa Ebola outbreak. Sci. Rep. 6, 34598 (2016). https://doi.org/10.1038/srep34598
B. Cai, A.B. Lawson, M. Hossain, J. Choi, R.S. Kirby, J. Liu et al., Bayesian semiparametric model with spatially–temporally varying coefficients selection. Stat. Med. 32(21), 3670–3685 (2013)
CDC, Weekly u.s. influenza surveillance report, 2007–2008, 2008–2009, 2009–2010 (2016)
A.J. Chorin, M. Morzfeld, X. Tu, A survey of implicit particle filters for data assimilation, in State-Space Models, ed. by Y. Zeng, S. Wu (Springer, New York, 2013), pp. 63–88
Y. Chung, D.B. Dunson, The local Dirichlet process. Ann. Inst. Stat. Math. 63(1), 59–80 (2011)
J.A. Duan, M. Guindani, A.E. Gelfand, Generalized spatial Dirichlet process models. Biometrika 94(4), 809–825 (2007)
D.B. Dunson, J.-H. Park, Kernel stick-breaking processes. Biometrika 95(2), 307–323 (2008)
M.D. Escobar, M. West, Bayesian density estimation and inference using mixtures. J. Am. Stat. Assoc. 90(430), 577–588 (1995)
G. Evensen, Sequential data assimilation with a nonlinear quasi-geostrophic model using monte carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99(C5), 10143–10162 (1994)
T.S. Ferguson, A Bayesian analysis of some nonparametric problems. Ann. Stat. 1, 209–230 (1973)
R.D. Fricker, B.L. Hegler, D.A. Dunfee, Comparing syndromic surveillance detection methods: ears versus a cusum-based methology. Stat. Med. 27, 3407–3429 (2008)
M. Fuentes, B. Reich, Multivariate spatial nonparametric modelling via kernel processes mixing. Stat. Sin. 23(1), 75–97 (2013)
A.E. Gelfand, A. Kottas, S.N. MacEachern, Bayesian nonparametric spatial modeling with Dirichlet process mixing. J. Am. Stat. Assoc. 100(471), 1021–1035 (2005)
P.J. Green, S. Richardson, Hidden Markov models and disease mapping. J. Am. Stat. Assoc. 97(460), 1055–1070 (2002)
M.S. Grewal, A.P. Andrews, A.K. Filtering, Theory and practice using matlab, 3rd edn. (Wiley, Hoboken, 2001)
R.E. Kalman, A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960)
K. Kleinman, Generalized linear models and generalized linear mixed models for small-area surveillance, in Spatial and Syndromic Surveillance for Public Health, ed. by A.B. Lawson, K. Kleinman (Wiley, West Sussex, 2005), pp. 77–94
L. Knorr-Held, S. Richardson, A hierarchical model for space–time surveillance data on meningococcal disease incidence. J. R. Stat. Soc. Ser. C Appl. Stat. 52(2), 169–183 (2003)
A. Kottas, J.A. Duan, A.E. Gelfand, Modeling disease incidence data with spatial and spatio temporal Dirichlet process mixtures. Biom. J. 50(1), 29–42 (2008)
A.B. Lawson, K. Kleinman et al., Spatial and Syndromic Surveillance for Public Health (Wiley, New York, 2005)
Y. Le Strat, F. Carrat, Monitoring epidemiologic surveillance data using hidden Markov models. Stat. Med. 18(24), 3463–3478 (1999)
J. Mandel, J.D. Beezley, An Ensemble Kalman-Particle Predictor-Corrector Filter for Non-Gaussian Data Assimilation (Springer, Berlin/Heidelberg, 2009), pp. 470–478
J. Mandel, J.D. Beezley, A.K. Kochanski, V.Y. Kondratenko, M. Kim, Assimilation of perimeter data and coupling with fuel moisture in a wildland fire–atmosphere DDDAS. Proc. Comput. Sci. 9, 1100–1109 (2012)
J. Mandel, L.S. Bennethum, M. Chen, J.L. Coen, C.C. Douglas, L.P. Franca, C.J. Johns, M. Kim, A.V. Knyazev, R. Kremens, V. Kulkarni, G. Qin, A. Vodacek, J. Wu, W. Zhao, A. Zornes, Towards a Dynamic Data Driven Application System for Wildfire Simulation (Springer, Berlin/Heidelberg, 2005), pp. 632–639
A. Patra, M. Bursik, J. Dehn, M. Jones, M. Pavolonis, E.B. Pitman, T. Singh, P. Singla, P. Webley, A DDDAS framework for volcanic ash propagation and hazard analysis. Proc. Comput. Sci. 9, 1090–1099 (2012)
A.K. Patra, M. Bursik, J. Dehn, M. Jones, R. Madankan, D. Morton, M. Pavolonis, E.B. Pitman, S. Pouget, T. Singh et al., Challenges in developing DDDAS based methodology for volcanic ash hazard analysis–effect of numerical weather prediction variability and parameter estimation. Proc. Comput. Sci. 18, 1871–1880 (2013)
A. Rodriguez, D.B. Dunson, A.E. Gelfand, The nested Dirichlet process. J. Am. Stat. Assoc. 103(483), 1131–1154 (2008)
H. Seybold, S. Ravela, P. Tagade, Ensemble Learning in Non-Gaussian Data Assimilation (Springer, Cham, 2015), pp. 227–238
Y.W. Teh, M.I. Jordan, M.J. Beal, D.M. Blei, Hierarchical Dirichlet processes. J. Am. Stat. Assoc. 101, 1566–1581 (2006)
A. Vodacek, J.P. Kerekes, M.J. Hoffman, Adaptive optical sensing in an object tracking DDDAS. Proc. Comput. Sci. 9, 1159–1166 (2012)
L.A. Waller, B.P. Carlin, H. Xia, A. Gelfand, Hierarchical spatio-temporal mapping of disease rates. J. Am. Stat. Assoc. 92, 607–617 (1997)
R.E. Watkins, S. Eagleson, B. Veenendaal, G. Wright, A.J. Plant, Disease surveillance using a hidden Markov model. BMC Med. Inform. Decis. Mak. 9(1), 1 (2009)
J. Zou, A.F. Karr, D. Banks, M.J. Heaton, G. Datta, J. Lynch, F. Vera, Bayesian methodology for the analysis of spatial–temporal surveillance data. Stat. Anal. Data Min. 5(3), 194–204 (2012)
J. Zou, A.F. Karr, G. Datta, J. Lynch, S.J. Grannis, A Bayesian spatio-temporal approach for real-time detection of disease outbreaks: a case study. BMC Med. Inform. Decis. Mak. 14(108), 1–18 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Yan, H., Zhang, Z., Zou, J. (2018). Dynamic Space-Time Model for Syndromic Surveillance with Particle Filters and Dirichlet Process. In: Blasch, E., Ravela, S., Aved, A. (eds) Handbook of Dynamic Data Driven Applications Systems. Springer, Cham. https://doi.org/10.1007/978-3-319-95504-9_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-95504-9_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-95503-2
Online ISBN: 978-3-319-95504-9
eBook Packages: Computer ScienceComputer Science (R0)