skip to main content
10.1145/3488560.3498461acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
research-article
Open access

ESC-GAN: Extending Spatial Coverage of Physical Sensors

Published: 15 February 2022 Publication History

Abstract

Scientific discoveries and studies about our physical world have long benefited from large-scale and planetary sensing, from weather forecasting to wildfire monitoring. However, the limited deployment of sensors in the environment due to cost or physical access constraints has lagged behind our ever-growing need for increased data coverage and higher resolution, impeding timely and precise monitoring and understanding of the environment. Therefore, we seek to extend the spatial coverage of analysis based on existing sensory data, that is, to "generate" data for locations where no historical data exists. This problem is fundamentally different and more challenging than the traditional spatio-temporal imputation that assumes data for any particular location are only partially missing across time. Inspired by the success of Generative Adversarial Network (GAN) in imputation, we propose a novel ESC-GAN. We observe that there are local patterns in nearby locations, as well as trends in a global manner (e.g., temperature drops as altitude increases regardless of the location). As local patterns may exhibit at different scales (from meters to kilometers), we employ a multi-branch generator to aggregate information of different granularity. More specifically, each branch in the generator contains 1) randomly masked 3D partial convolutions at different resolutions to capture the local patterns and 2) global attention modules for global similarity. Next, we adversarially train a 3D convolution-based discriminator to distinguish the generator's output from the ground truth. Extensive experiments on three geo-sensor datasets demonstrate that ESC-GAN outperforms state-of-the-art methods on extending spatial coverage and also achieves the best results on a traditional spatio-temporal imputation task.

Supplementary Material

MP4 File (WSDM22-fp468.mp4)
Geo-sensors are critical for monitoring our ecosystem, but they are sparsely deployed due to e.g., physical constraints. We seek to extend the spatial coverage of analysis based on existing sensory data, that is, to generate data for locations where no historical data exists. This problem is more challenging than the traditional spatio-temporal imputation that assumes data for any particular location are only partially missing across time. In view of the non-linear and stochastic nature of spatio-temporal data, we design a Generative Adversarial Network-based model ESC-GAN to address the challenges. We observe that there are local patterns in nearby locations, as well as trends in a global manner (e.g., temperature drops as altitude increases). As local patterns may exhibit at different scales, we employ a multi-branch generator to aggregate information of different granularity. Extensive experiments on geo-sensor datasets demonstrate state-of-the-art performance under different missing scenarios.

References

[1]
Lorenzo Alfieri, Peter Burek, Emanuel Dutra, Blazej Krzeminski, David Muraro, Jutta Thielen, and Florian Pappenberger. 2013. GloFAS--global ensemble streamflow forecasting and flood early warning. Hydrology and Earth System Sciences, Vol. 17, 3 (2013), 1161--1175.
[2]
Craig F Ansley and Robert Kohn. 1984. On the estimation of ARIMA models with missing values. In Time series analysis of irregularly observed data. Springer.
[3]
Gabriel Appleby, Linfeng Liu, and Li-Ping Liu. 2020. Kriging Convolutional Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 3187--3194.
[4]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
[5]
S van Buuren and Karin Groothuis-Oudshoorn. 2010. mice: Multivariate imputation by chained equations in R. Journal of statistical software (2010), 1--68.
[6]
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. 2018. Brits: Bidirectional recurrent imputation for time series. Advances in Neural Information Processing Systems, Vol. 31 (2018), 6775--6785.
[7]
Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. 2019. Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops . 0--0.
[8]
Ya-Liang Chang, Zhe Yu Liu, Kuan-Ying Lee, and Winston Hsu. 2019. Free-form video inpainting with 3d gated convolution and temporal patchgan. In Proceedings of the IEEE International Conference on Computer Vision . 9066--9075.
[9]
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent neural networks for multivariate time series with missing values. Scientific reports, Vol. 8, 1 (2018), 1--12.
[10]
Feng-Wen Chen and Chen-Wuing Liu. 2012. Estimation of the spatial rainfall distribution using inverse distance weighting (IDW) in the middle of Taiwan. Paddy and Water Environment, Vol. 10, 3 (2012), 209--222.
[11]
Xinyu Chen and Lijun Sun. 2021. Bayesian temporal factorization for multidimensional time series prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[12]
Kevin Cowtan and Robert G Way. 2014. Coverage bias in the HadCRUT4 temperature series and its impact on recent temperature trends. Quarterly Journal of the Royal Meteorological Society, Vol. 140, 683 (2014), 1935--1944.
[13]
JW Gjerloev. 2009. A global ground-based magnetometer initiative. Eos, Transactions American Geophysical Union, Vol. 90, 27 (2009), 230--231.
[14]
Yongshun Gong, Zhibin Li, Jian Zhang, Wei Liu, Bei Chen, and Xiangjun Dong. 2020. A Spatial Missing Value Imputation Method for Multi-view Urban Statistical Data. In IJCAI . 1310--1316.
[15]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, Vol. 27 (2014), 2672--2680.
[16]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[17]
Andrew T Hudak, Nicholas L Crookston, Jeffrey S Evans, David E Hall, and Michael J Falkowski. 2008. Nearest neighbor imputation of species-level, plot-scale forest structure attributes from LiDAR data. Remote Sensing of Environment, Vol. 112, 5 (2008), 2232--2245.
[18]
Christopher Kadow, David Matthew Hall, and Uwe Ulbrich. 2020. Artificial intelligence reconstructs missing climate information. Nature Geoscience (2020).
[19]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[20]
Lixin Li and Peter Revesz. 2004. Interpolation methods for spatio-temporal geographic data. Computers, Environment and Urban Systems, Vol. 28, 3 (2004), 201--227.
[21]
Lixin Li, Xingyou Zhang, James B Holt, Jie Tian, and Reinhard Piltner. 2011. Spatiotemporal interpolation methods for air pollution exposure. In Ninth Symposium of Abstraction, Reformulation, and Approximation .
[22]
Steven Cheng-Xian Li and Benjamin Marlin. 2020. Learning from irregularly-sampled time series: A missing data perspective. In International Conference on Machine Learning. PMLR, 5937--5946.
[23]
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2017. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926 (2017).
[24]
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV). 85--100.
[25]
Yukai Liu, Rose Yu, Stephan Zheng, Eric Zhan, and Yisong Yue. 2019. Naomi: Non-autoregressive multiresolution sequence imputation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems .
[26]
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. 2018. Multivariate time series imputation with generative adversarial networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems .
[27]
Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. 2019. E2gan: End-to-end generative adversarial network for multivariate time series imputation. In AAAI Press. 3094--3100.
[28]
Jiawei Ma, Zheng Shou, Alireza Zareian, Hassan Mansour, Anthony Vetro, and Shih-Fu Chang. 2019. Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation. (2019).
[29]
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. 2010. Spectral regularization algorithms for learning large incomplete matrices. The Journal of Machine Learning Research, Vol. 11 (2010), 2287--2322.
[30]
Congcong Miao, Jiajun Fu, Jilong Wang, Heng Yu, Botao Yao, Anqi Zhong, Jie Chen, and Zekun He. 2021. Predicting Crowd Flows via Pyramid Dilated Deeper Spatial-temporal Network. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining . 806--814.
[31]
Colin P Morice, John J Kennedy, Nick A Rayner, and Phil D Jones. 2012. Quantifying uncertainties in global and regional temperature change using an ensemble of observational estimates: The HadCRUT4 data set. Journal of Geophysical Research: Atmospheres, Vol. 117, D8 (2012).
[32]
Colin P Morice, John J Kennedy, Nick A Rayner, JP Winn, Emma Hogan, RE Killick, RJH Dunn, TJ Osborn, PD Jones, and IR Simpson. 2020. An updated assessment of near-surface temperature change from 1850: the HadCRUT5 dataset. Journal of Geophysical Research: Atmospheres (2020), e2019JD032361.
[33]
Morten Morup, Daniel M Dunlavy, Evrim Acar, and Tamara Gibson Kolda. 2010. Scalable tensor factorizations with missing data. Technical Report. Sandia National Laboratories.
[34]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
[35]
Alex Rubinsteyn and Sergey Feldman. [n.d.]. fancyimpute: An Imputation Library for Python . https://github.com/iskandr/fancyimpute
[36]
Divya Saxena and Jiannong Cao. 2019. D-GAN: Deep Generative Adversarial Nets for Spatio-Temporal Prediction. arXiv preprint arXiv:1907.08556 (2019).
[37]
Michael L Stein. 2012. Interpolation of spatial data: some theory for kriging .Springer Science & Business Media.
[38]
Koh Takeuchi, Hisashi Kashima, and Naonori Ueda. 2017. Autoregressive tensor factorization for spatio-temporal predictions. In 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 1105--1110.
[39]
Xianfeng Tang, Boqing Gong, Yanwei Yu, Huaxiu Yao, Yandong Li, Haiyong Xie, and Xiaoyu Wang. 2019. Joint modeling of dense and incomplete trajectories for citywide traffic volume inference. In The World Wide Web Conference. 1806--1817.
[40]
Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. 2020. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In Proceedings of the AAAI Conference on Artificial Intelligence .
[41]
Waldo R Tobler. 1970. A computer movie simulating urban growth in the Detroit region. Economic geography, Vol. 46, sup1 (1970), 234--240.
[42]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
[43]
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018a. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7794--7803.
[44]
Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. 2018b. Image inpainting via generative multi-column convolutional neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems .
[45]
Yuankai Wu, Dingyi Zhuang, Aurelie Labbe, and Lijun Sun. 2020. Inductive graph neural networks for spatiotemporal kriging. arXiv preprint arXiv:2006.07527 (2020).
[46]
Pingping Xie and Phillip A Arkin. 1997. Global precipitation: A 17-year monthly analysis based on gauge observations, satellite estimates, and numerical model outputs. Bulletin of the American Meteorological Society, Vol. 78, 11 (1997), 2539--2558.
[47]
Xiuwen Yi, Yu Zheng, Junbo Zhang, and Tianrui Li. 2016. ST-MVL: filling missing values in geo-sensory time series data. (2016).
[48]
Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. 2020. Disentangled non-local neural networks. In European Conference on Computer Vision. Springer, 191--207.
[49]
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2019. Free-form image inpainting with gated convolution. In ICCV .
[50]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Self-attention generative adversarial networks. In International Conference on Machine Learning . 7354--7363.
[51]
Lin Zhang, Alexander Gorovits, Wenyu Zhang, and Petko Bogdanov. 2020 a. Learning Periods from Incomplete Multivariate Time Series. In ICDM .
[52]
Yingxue Zhang, Yanhua Li, Xun Zhou, Xiangnan Kong, and Jun Luo. 2020 b. Curb-GAN: Conditional Urban Traffic Estimation through Spatio-Temporal Generative Adversarial Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 842--852.

Cited By

View all
  • (2024)Inductive and adaptive graph convolution networks equipped with constraint task for spatial–temporal traffic data krigingKnowledge-Based Systems10.1016/j.knosys.2023.111325284:COnline publication date: 17-Apr-2024
  • (2023)Unleashing the Power of Shared Label Structures for Human Activity RecognitionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615101(3340-3350)Online publication date: 21-Oct-2023
  • (2022)Semi-GAN: An Improved GAN-Based Missing Data Imputation Method for the Semiconductor IndustryIEEE Access10.1109/ACCESS.2022.318887110(72328-72338)Online publication date: 2022

Index Terms

  1. ESC-GAN: Extending Spatial Coverage of Physical Sensors

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining
        February 2022
        1690 pages
        ISBN:9781450391320
        DOI:10.1145/3488560
        This work is licensed under a Creative Commons Attribution International 4.0 License.

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 15 February 2022

        Check for updates

        Author Tags

        1. generative adversarial network
        2. imputation
        3. self-attention
        4. spatio-temporal data
        5. super resolution

        Qualifiers

        • Research-article

        Funding Sources

        Conference

        WSDM '22

        Acceptance Rates

        Overall Acceptance Rate 498 of 2,863 submissions, 17%

        Upcoming Conference

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)189
        • Downloads (Last 6 weeks)13
        Reflects downloads up to 13 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Inductive and adaptive graph convolution networks equipped with constraint task for spatial–temporal traffic data krigingKnowledge-Based Systems10.1016/j.knosys.2023.111325284:COnline publication date: 17-Apr-2024
        • (2023)Unleashing the Power of Shared Label Structures for Human Activity RecognitionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615101(3340-3350)Online publication date: 21-Oct-2023
        • (2022)Semi-GAN: An Improved GAN-Based Missing Data Imputation Method for the Semiconductor IndustryIEEE Access10.1109/ACCESS.2022.318887110(72328-72338)Online publication date: 2022

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Login options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media