Detecting outliers in streaming time series data from ARM distributed sensors

Lu, Yuping; Kumar, Jitendra; Collier, Nathaniel; Krishna, Bhargavi; Langston, Michael A.

doi:10.1109/ICDMW.2018.00117

Title: Detecting outliers in streaming time series data from ARM distributed sensors

Conference · Thu Nov 01 00:00:00 EDT 2018

DOI:https://doi.org/10.1109/ICDMW.2018.00117· OSTI ID:1491320

Lu, Yuping ^[1];

^[1];

^[1]

ORNL

The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliersthrough a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

View Conference

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1491320

Resource Relation:: Conference: IEEE International Conference on Data Mining Workshops - Singapore, , Singapore - 11/17/2018 10:00:00 AM-11/20/2018 10:00:00 AM

Country of Publication:: United States

Language:: English

References (21)

The ARM Data Quality Program Peppler, Randy A.; Kehoe, Kenneth E.; Monroe, Justin W. Meteorological Monographs, Vol. 57 https://doi.org/10.1175/AMSMONOGRAPHS-D-15-0039.1	journal	April 2016
The ARM Data System and Archive McCord, Raymond; Voyles, Jimmy Meteorological Monographs, Vol. 57 https://doi.org/10.1175/AMSMONOGRAPHS-D-15-0043.1	journal	April 2016
Potential of Multivariate Quantitative Methods for Delineation and Visualization of Ecoregions Hargrove, William W.; Hoffman, Forrest M. Environmental Management, Vol. 34, Issue S1 https://doi.org/10.1007/s00267-003-1084-0	journal	April 2004
The benefits of multivariate singular spectrum analysis over the univariate version Rodrigues, Paulo Canas; Mahmoudvand, Rahim Journal of the Franklin Institute, Vol. 355, Issue 1 https://doi.org/10.1016/j.jfranklin.2017.09.008	journal	January 2018
Deploying the ARM Sites and Supporting Infrastructure Cress, Ted S.; Sisterson, Douglas L. Meteorological Monographs, Vol. 57 https://doi.org/10.1175/AMSMONOGRAPHS-D-15-0049.1	journal	April 2016
Spatio-Temporal Outlier Detection in Large Databases Birant, Derya; Kut, Alp Journal of Computing and Information Technology, Vol. 14, Issue 4 https://doi.org/10.2498/cit.2006.04.04	journal	January 2006
Anomaly detection in streaming environmental sensor data: A data-driven modeling approach Hill, David J.; Minsker, Barbara S. Environmental Modelling & Software, Vol. 25, Issue 9 https://doi.org/10.1016/j.envsoft.2009.08.010	journal	September 2010
An analysis of Australian seasonal rainfall anomalies: 1950–1987. II: Temporal variability and teleconnection patterns Drosdowsky, Wasyl International Journal of Climatology, Vol. 13, Issue 2 https://doi.org/10.1002/joc.3370130202	journal	March 1993
Outlier Detection for Temporal Data: A Survey Gupta, Manish; Gao, Jing; Aggarwal, Charu C. IEEE Transactions on Knowledge and Data Engineering, Vol. 26, Issue 9 https://doi.org/10.1109/TKDE.2013.184	journal	September 2014
An algorithm for the machine calculation of complex Fourier series Cooley, James W.; Tukey, John W. Mathematics of Computation, Vol. 19, Issue 90 https://doi.org/10.1090/S0025-5718-1965-0178586-1	journal	May 1965
A scientific data processing framework for time series NetCDF data Gaustad, Krista; Shippert, Tim; Ermold, Brian Environmental Modelling & Software, Vol. 60 https://doi.org/10.1016/j.envsoft.2014.06.005	journal	October 2014
Anomaly Detection and Diagnosis Algorithms for Discrete Symbol Sequences with Applications to Airline Safety Budalakoti, S.; Srivastava, A. N.; Otey, M. E. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), Vol. 39, Issue 1 https://doi.org/10.1109/TSMCC.2008.2007248	journal	January 2009
Using Clustered Climate Regimes to Analyze and Compare Predictions from Fully Coupled General Circulation Models Hoffman, Forrest M.; Hargrove, William W.; Erickson, David J. Earth Interactions, Vol. 9, Issue 10 https://doi.org/10.1175/EI110.1	journal	July 2005
Outliers in Time Series Fox, A. J. Journal of the Royal Statistical Society: Series B (Methodological), Vol. 34, Issue 3 https://doi.org/10.1111/j.2517-6161.1972.tb00912.x	journal	July 1972
Admit Sequeira, Karlton; Zaki, Mohammed Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining https://doi.org/10.1145/775047.775103	conference	July 2002
Comparative Evaluation of Anomaly Detection Techniques for Sequence Data Chandola, Varun; Mithal, Varun; Kumar, Vipin 2008 Eighth IEEE International Conference on Data Mining https://doi.org/10.1109/ICDM.2008.151	conference	December 2008
The Atmospheric Radiation Measurement (ARM) Program: Programmatic Background and Design of the Cloud and Radiation Test Bed Stokes, Gerald M.; Schwartz, Stephen E. Bulletin of the American Meteorological Society, Vol. 75, Issue 7 https://doi.org/10.1175/1520-0477(1994)075<1201:TARMPP>2.0.CO;2	journal	July 1994
NetCDF: an interface for scientific data access Rew, R.; Davis, G. IEEE Computer Graphics and Applications, Vol. 10, Issue 4 https://doi.org/10.1109/38.56302	journal	July 1990
Basic Singular Spectrum Analysis and forecasting with R Golyandina, Nina; Korobeynikov, Anton Computational Statistics & Data Analysis, Vol. 71 https://doi.org/10.1016/j.csda.2013.04.009	journal	March 2014
VII. Note on regression and inheritance in the case of two parents Pearson, Karl Proceedings of the Royal Society of London, Vol. 58, Issue 347-352, p. 240-242 https://doi.org/10.1098/rspl.1895.0041	journal	January 1895
Relationship between Singular Spectrum Analysis and Fourier analysis: Theory and application to the monitoring of volcanic activity Bozzo, Enrico; Carniel, Roberto; Fasino, Dario Computers & Mathematics with Applications, Vol. 60, Issue 3 https://doi.org/10.1016/j.camwa.2010.05.028	journal	August 2010

Similar Records

Real-time statistical quality control and ARM

Conference · Fri May 01 00:00:00 EDT 1992 · OSTI ID:1491320

Blough, D K

Real-time statistical quality control and ARM

Conference · Fri May 01 00:00:00 EDT 1992 · OSTI ID:1491320

Blough, D K

Analysis and Calibration of CRF Raman Lidar Cloud Liquid Water Measurements

Technical Report · Wed Oct 31 00:00:00 EDT 2007 · OSTI ID:1491320

Turner, D D; Whiteman, D N; Russo, F

Title: Detecting outliers in streaming time series data from ARM distributed sensors

Citation Formats

References (21)

Similar Records

Related Subjects