Skip to main content
SpringerLink
Log in

Spatio-temporal fusion for remote sensing data: an overview and new benchmark

  • Review
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Spatio-temporal fusion (STF) aims at fusing (temporally dense) coarse resolution images and (temporally sparse) fine resolution images to generate image series with adequate temporal and spatial resolution. In the last decade, STF has drawn a lot of attention and many STF methods have been developed. However, to date the STF domain still lacks benchmark datasets, which is a pressing issue that needs to be addressed in order to foster the development of this field. In this review, we provide (for the first time in the literature) a robust benchmark STF dataset that includes three important characteristics: (1) diversity of regions, (2) long timespan, and (3) challenging scenarios. We also provide a survey of highly representative STF techniques, along with a detailed quantitative and qualitative comparison of their performance with our newly presented benchmark dataset. The proposed dataset is public and available online.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Shen M, Tang Y, Chen J, et al. Influences of temperature and precipitation before the growing season on spring phenology in grasslands of the central and eastern Qinghai-Tibetan Plateau. Agric For Meteor, 2011, 151: 1711–1722

    Article  Google Scholar 

  2. Amorós-López J, Gómez-Chova L, Alonso L, et al. Multitemporal fusion of Landsat/TM and ENVISAT/MERIS for crop monitoring. Int J Appl Earth Observation GeoInf, 2013, 23: 132–141

    Article  Google Scholar 

  3. Johnson M D, Hsieh W W, Cannon A J, et al. Crop yield forecasting on the Canadian Prairies by remotely sensed vegetation indices and machine learning methods. Agric For Meteor, 2016, 218–219: 74–84

    Article  Google Scholar 

  4. Liao C, Wang J, Dong T, et al. Using spatio-temporal fusion of Landsat-8 and MODIS data to derive phenology, biomass and yield estimates for corn and soybean. Sci Total Environ, 2019, 650: 1707–1721

    Article  Google Scholar 

  5. Nduati E, Sofue Y, Matniyaz A, et al. Cropland mapping using fusion of multi-sensor data in a complex urban/periurban area. Remote Sens, 2019, 11: 207

    Article  Google Scholar 

  6. Zhang M, Lin H, Wang G X, et al. Estimation of vegetation productivity using a Landsat 8 time series in a heavily urbanized area, central China. Remote Sens, 2019, 11: 133

    Article  Google Scholar 

  7. Hwang T, Song C, Bolstad P V, et al. Downscaling real-time vegetation dynamics by fusing multi-temporal MODIS and Landsat NDVI in topographically complex terrain. Remote Sens Environ, 2011, 115: 2499–2512

    Article  Google Scholar 

  8. Knipper K R, Kustas W P, Anderson M C, et al. Evapotranspiration estimates derived using thermal-based satellite remote sensing and data fusion for irrigation management in California vineyards. Irrig Sci, 2019, 37: 431–449

    Article  Google Scholar 

  9. Pan Y Q, Shen F, Wei X D. Fusion of Landsat-8/OLI and GOCI data for hourly mapping of suspended particulate matter at high spatial resolution: a case study in the Yangtze (Changjiang) estuary. Remote Sens, 2018, 10: 158

    Article  Google Scholar 

  10. Yang X, Lo C P. Using a time series of satellite imagery to detect land use and land cover changes in the Atlanta, Georgia metropolitan area. Int J Remote Sens, 2002, 23: 1775–1798

    Article  Google Scholar 

  11. Heimhuber V, Tulbure M G, Broich M. Addressing spatio-temporal resolution constraints in Landsat and MODIS-based mapping of large-scale floodplain inundation dynamics. Remote Sens Environ, 2018, 211: 307–320

    Article  Google Scholar 

  12. Pastick N J, Wylie B K, Wu Z T. Spatiotemporal analysis of Landsat-8 and Sentinel-2 data to support monitoring of dryland ecosystems. Remote Sens, 2018, 10: 791–806

    Article  Google Scholar 

  13. Chiesi M, Battista P, Fibbi L, et al. Spatio-temporal fusion of NDVI data for simulating soil water content in heterogeneous Mediterranean areas. Eur J Remote Sens, 2019, 52: 88–95

    Article  Google Scholar 

  14. Li X C, Zhou Y Y, Asrar G R, et al. Response of vegetation phenology to urbanization in the conterminous United States. Glob Change Biol, 2017, 23: 2818–2830

    Article  Google Scholar 

  15. Zhu X L, Cai F Y, Tian J Q, et al. Spatiotemporal fusion of multisource remote sensing data: literature survey, taxonomy, principles, applications, and future directions. Remote Sens, 2018, 10: 527

    Article  Google Scholar 

  16. Gao F, Masek J G, Schwaller M R, et al. On the blending of the Landsat and MODIS surface reflectance: predicting daily Landsat surface reflectance. IEEE Trans Geosci Remote Sens, 2006, 44: 2207–2218

    Article  Google Scholar 

  17. Hilker T, Wulder M A, Coops N C, et al. A new data fusion model for high spatial- and temporal-resolution mapping of forest disturbance based on Landsat and MODIS. Remote Sens Environ, 2009, 113: 1613–1627

    Article  Google Scholar 

  18. Zhu X L, Chen J, Gao F, et al. An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions. Remote Sens Environ, 2010, 114: 2610–2623

    Article  Google Scholar 

  19. Shen H, Wu P, Liu Y, et al. A spatial and temporal reflectance fusion model considering sensor observation differences. Int J Remote Sens, 2013, 34: 4367–4383

    Article  Google Scholar 

  20. Wang P J, Gao F, Masek J G. Operational data fusion framework for building frequent Landsat-like imagery. IEEE Trans Geosci Remote Sens, 2014, 52: 7353–7365

    Article  Google Scholar 

  21. Wang Q, Zhang Y, Onojeghuo A O, et al. Enhancing spatio-temporal fusion of MODIS and Landsat data by incorporating 250 m MODIS data. IEEE J Sel Top Appl Earth Observ Remote Sens, 2017, 10: 4116–4123

    Article  Google Scholar 

  22. Zhao Y, Huang B, Song H. A robust adaptive spatial and temporal image fusion model for complex land surface changes. Remote Sens Environ, 2018, 208: 42–62

    Article  Google Scholar 

  23. Huang B, Wang J, Song H H, et al. Generating high spatiotemporal resolution land surface temperature for urban heat island monitoring. IEEE Geosci Remote Sens Lett, 2013, 10: 1011–1015

    Article  Google Scholar 

  24. Xia H, Chen Y, Li Y, et al. Combining kernel-driven and fusion-based methods to generate daily high-spatial-resolution land surface temperatures. Remote Sens Environ, 2019, 224: 259–274

    Article  Google Scholar 

  25. Weng Q, Fu P, Gao F. Generating daily land surface temperature at Landsat resolution by fusing Landsat and MODIS data. Remote Sens Environ, 2014, 145: 55–67

    Article  Google Scholar 

  26. Xu C, Qu J J, Hao X, et al. Downscaling of surface soil moisture retrieval by combining MODIS/Landsat and in situ measurements. Remote Sens, 2018, 10: 210

    Article  Google Scholar 

  27. Houborg R, McCabe M F, Gao F. A spatio-temporal enhancement method for medium resolution LAI (STEM-LAI). Int J Appl Earth Observ GeoInf, 2016, 47: 15–29

    Google Scholar 

  28. Wu P, Shen H, Zhang L, et al. Integrated fusion of multi-scale polar-orbiting and geostationary satellite observations for the mapping of high spatial and temporal resolution land surface temperature. Remote Sens Environ, 2015, 156: 169–181

    Article  Google Scholar 

  29. Fu D, Chen B, Wang J, et al. An improved image fusion approach based on enhanced spatial and temporal the adaptive reflectance fusion model. Remote Sens, 2013, 5: 6346–6360

    Article  Google Scholar 

  30. Liu M, Liu X, Wu L, et al. A modified spatiotemporal fusion algorithm using phenological information for predicting reflectance of paddy rice in southern China. Remote Sens, 2018, 10: 772

    Article  Google Scholar 

  31. Cheng Q, Liu H, Shen H, et al. A spatial and temporal nonlocal filter-based data fusion method. IEEE Trans Geosci Remote Sens, 2017, 55: 4476–4488

    Article  Google Scholar 

  32. Hazaymeh K, Hassan Q K. Spatiotemporal image-fusion model for enhancing the temporal resolution of Landsat-8 surface reflectance images using MODIS images. J Appl Remote Sens, 2015, 9: 096095

    Article  Google Scholar 

  33. Wang J, Huang B. A rigorously-weighted spatiotemporal fusion model with uncertainty analysis. Remote Sens, 2017, 9: 990

    Article  Google Scholar 

  34. Roy D P, Ju J, Lewis P, et al. Multi-temporal MODIS-Landsat data fusion for relative radiometric normalization, gap filling, and prediction of Landsat data. Remote Sens Environ, 2008, 112: 3112–3130

    Article  Google Scholar 

  35. Wu B, Huang B, Cao K, et al. Improving spatiotemporal reflectance fusion using image inpainting and steering kernel regression techniques. Int J Remote Sens, 2017, 38: 706–727

    Article  Google Scholar 

  36. Wang J, Huang B. A spatiotemporal satellite image fusion model with autoregressive error correction (AREC). Int J Remote Sens, 2018, 39: 6731–6756

    Article  Google Scholar 

  37. Wang Q, Blackburn G A, Onojeghuo A O, et al. Fusion of Landsat 8 OLI and Sentinel-2 MSI data. IEEE Trans Geosci Remote Sens, 2017, 55: 3885–3899

    Article  Google Scholar 

  38. Malleswara Rao J, Rao C V, Senthil Kumar A, et al. Spatiotemporal data fusion using temporal high-pass modulation and edge primitives. IEEE Trans Geosci Remote Sens, 2015, 53: 5853–5860

    Article  Google Scholar 

  39. Sun Y, Zhang H, Shi W. A spatio-temporal fusion method for remote sensing data using a linear injection model and local neighbourhood information. Int J Remote Sens, 2019, 40: 2965–2985

    Article  Google Scholar 

  40. Yin G, Li A, Jin H, et al. Spatiotemporal fusion through the best linear unbiased estimator to generate fine spatial resolution NDVI time series. Int J Remote Sens, 2018, 39: 3287–3305

    Article  Google Scholar 

  41. Liao C, Wang J, Pritchard I, et al. A spatio-temporal data fusion model for generating NDVI time series in heterogeneous regions. Remote Sens, 2017, 9: 1125

    Article  Google Scholar 

  42. Bhattarai N, Quackenbush L J, Dougherty M, et al. A simple Landsat-MODIS fusion approach for monitoring seasonal evapotranspiration at 30 m spatial resolution. Int J Remote Sens, 2015, 36: 115–143

    Article  Google Scholar 

  43. Kwan C, Budavari B, Gao F, et al. A hybrid color mapping approach to fusing MODIS and Landsat images for forward prediction. Remote Sens, 2018, 10: 520

    Article  Google Scholar 

  44. Zhang Y, Foody G M, Ling F, et al. Spatial-temporal fraction map fusion with multi-scale remotely sensed images. Remote Sens Environ, 2018, 213: 162–181

    Article  Google Scholar 

  45. Wang Q, Atkinson P M. Spatio-temporal fusion for daily Sentinel-2 images. Remote Sens Environment, 2017, 204: S0034425717305096

    Google Scholar 

  46. Zhukov B, Oertel D, Lanzl F, et al. Unmixing-based multisensor multiresolution image fusion. IEEE Trans Geosci Remote Sens, 1999, 37: 1212–1226

    Article  Google Scholar 

  47. Zurita-Milla R, Clevers J, Schaepman M E. Unmixing-based Landsat TM and MERIS FR data fusion. IEEE Geosci Remote Sens Lett, 2008, 5: 453–457

    Article  Google Scholar 

  48. Maselli F, Rembold F. Integration of LAC and GAC NDVI data to improve vegetation monitoring in semi-arid environments. Int J Remote Sens, 2002, 23: 2475–2488

    Article  Google Scholar 

  49. Niu Z. Use of MODIS and Landsat time series data to generate high-resolution temporal synthetic Landsat data using a spatial and temporal reflectance fusion model. J Appl Remote Sens, 2012, 6: 063507

    Article  Google Scholar 

  50. Wu M Q, Huang W, Niu Z, et al. Generating daily synthetic Landsat imagery by combining Landsat and MODIS Data. Sensors, 2015, 15: 24002–24025

    Article  Google Scholar 

  51. Zhang W, Li A, Jin H, et al. An enhanced spatial and temporal data fusion model for fusing Landsat and MODIS surface reflectance to generate high temporal Landsat-like data. Remote Sens, 2013, 5: 5346–5368

    Article  Google Scholar 

  52. Huang B, Zhang H. Spatio-temporal reflectance fusion via unmixing: accounting for both phenological and land-cover changes. Int J Remote Sens, 2014, 35: 6213–6233

    Article  Google Scholar 

  53. Xu Y, Huang B, Xu Y Y, et al. Spatial and temporal image fusion via regularized spatial unmixing. IEEE Geosci Remote Sens Lett, 2015, 12: 1362–1366

    Article  Google Scholar 

  54. Mizuochi H, Hiyama T, Ohta T, et al. Development and evaluation of a lookup-table-based approach to data fusion for seasonal wetlands monitoring: an integrated use of AMSR series, MODIS, and Landsat. Remote Sens Environ, 2017, 199: 370–388

    Article  Google Scholar 

  55. Lu M, Chen J, Tang H, et al. Land cover change detection by integrating object-based data blending model of Landsat and MODIS. Remote Sens Environ, 2016, 184: 374–386

    Article  Google Scholar 

  56. Rao Y, Zhu X, Chen J, et al. An improved method for producing high spatial-resolution NDVI time series datasets with multi-temporal MODIS NDVI data and Landsat TM/ETM+ images. Remote Sens, 2015, 7: 7865–7891

    Article  Google Scholar 

  57. Huang B, Song H H. Spatiotemporal reflectance fusion via sparse representation. IEEE Trans Geosci Remote Sens, 2012, 50: 3707–3716

    Article  Google Scholar 

  58. Song H H, Huang B. Spatiotemporal satellite image fusion through one-pair image learning. IEEE Trans Geosci Remote Sens, 2013, 51: 1883–1896

    Article  Google Scholar 

  59. Li D, Li Y, Yang W, et al. An enhanced single-pair learning-based reflectance fusion algorithm with spatiotemporally extended training samples. Remote Sens, 2018, 10: 1207

    Article  Google Scholar 

  60. Zhao C, Gao X, Emery W J, et al. An integrated spatio-spectral-temporal sparse representation method for fusing remote-sensing images with different resolutions. IEEE Trans Geosci Remote Sens, 2018, 56: 3358–3370

    Article  Google Scholar 

  61. Wu B, Huang B, Zhang L. An error-bound-regularized sparse coding for spatiotemporal reflectance fusion. IEEE Trans Geosci Remote Sens, 2015, 53: 6791–6803

    Article  Google Scholar 

  62. Wei J, Wang L, Liu P, et al. Spatiotemporal fusion of remote sensing images with structural sparsity and semi-coupled dictionary learning. Remote Sens, 2017, 9: 21

    Article  Google Scholar 

  63. Wei J, Wang L, Liu P, et al. Spatiotemporal fusion of MODIS and Landsat-7 reflectance images via compressed sensing. IEEE Trans Geosci Remote Sens, 2017, 55: 7126–7139

    Article  Google Scholar 

  64. Liu X, Deng C, Wang S, et al. Fast and accurate spatiotemporal fusion based upon extreme learning machine. IEEE Geosci Remote Sens Lett, 2016, 13: 2039–2043

    Article  Google Scholar 

  65. Song H H, Liu Q, Wang G, et al. Spatiotemporal satellite image fusion using deep convolutional neural networks. IEEE J Sel Top Appl Earth Observations Remote Sens, 2018, 11: 821–829

    Article  Google Scholar 

  66. Liu X, Deng C, Chanussot J, et al. StfNet: a two-stream convolutional neural network for spatiotemporal image fusion. IEEE Trans Geosci Remote Sens, 2019, 57: 6552–6564

    Article  Google Scholar 

  67. Tan Z, Yue P, Di L, et al. Deriving high spatiotemporal remote sensing images using deep convolutional network. Remote Sens, 2018, 10: 1066

    Article  Google Scholar 

  68. Tao X, Liang S, Wang D, et al. Improving satellite estimates of the fraction of absorbed photosynthetically active radiation through data integration: methodology and validation. IEEE Trans Geosci Remote Sens, 2018, 56: 2107–2118

    Article  Google Scholar 

  69. Ke Y, Im J, Park S, et al. Downscaling of MODIS one kilometer evapotranspiration using Landsat-8 data and machine learning approaches. Remote Sens, 2016, 8: 215

    Article  Google Scholar 

  70. Boyte S P, Wylie B K, Rigge M B, et al. Fusing MODIS with Landsat 8 data to downscale weekly normalized difference vegetation index estimates for central Great Basin rangelands, USA. GISci Remote Sens, 2018, 55: 376–399

    Article  Google Scholar 

  71. Moosavi V, Talebi A, Mokhtari M H, et al. A wavelet-artificial intelligence fusion approach (WAIFA) for blending Landsat and MODIS surface temperature. Remote Sens Environ, 2015, 169: 243–254

    Article  Google Scholar 

  72. Li Y F, Li J, He L, et al. A sensor-bias driven spatio-temporal fusion model based on convolutional neural networks. Sci China Inf Sci, 2020, 63: 140302

    Article  Google Scholar 

  73. Xue J, Leung Y, Fung T. A Bayesian data fusion approach to spatio-temporal fusion of remotely sensed images. Remote Sens, 2017, 9: 1310

    Article  Google Scholar 

  74. Li A, Bo Y, Zhu Y, et al. Blending multi-resolution satellite sea surface temperature (SST) products using Bayesian maximum entropy method. Remote Sens Environ, 2013, 135: 52–63

    Article  Google Scholar 

  75. Huang B, Zhang H, Song H, et al. Unified fusion of remote-sensing imagery: generating simultaneously high-resolution synthetic spatial-temporal-spectral earth observations. Remote Sens Lett, 2013, 4: 561–569

    Article  Google Scholar 

  76. Shen H, Meng X, Zhang L. An integrated framework for the spatio-temporal-spectral fusion of remote sensing images. IEEE Trans Geosci Remote Sens, 2016, 54: 7135–7148

    Article  Google Scholar 

  77. Gevaert C M, García-Haro F J. A comparison of STARFM and an unmixing-based algorithm for Landsat and MODIS data fusion. Remote Sens Environ, 2015, 156: 34–44

    Article  Google Scholar 

  78. Ma J, Zhang W, Marinoni A, et al. An improved spatial and temporal reflectance unmixing model to synthesize time series of Landsat-like images. Remote Sens, 2018, 10: 1388

    Article  Google Scholar 

  79. Xie D, Zhang J, Zhu X, et al. An improved STARFM with help of an unmixing-based method to generate high spatial and temporal resolution remote sensing data in complex heterogeneous regions. Sensors, 2016, 16: 207

    Article  Google Scholar 

  80. Zhu X L, Helmer E H, Gao F, et al. A flexible spatiotemporal method for fusing satellite images with different resolutions. Remote Sens Environ, 2016, 172: 165–177

    Article  Google Scholar 

  81. Liu M, Yang W, Zhu X, et al. An improved flexible spatiotemporal data fusion (IFSDAF) method for producing high spatiotemporal resolution normalized difference vegetation index time series. Remote Sens Environ, 2019, 227: 74–89

    Article  Google Scholar 

  82. Cui J, Zhang X, Luo M. Combining linear pixel unmixing and STARFM for spatiotemporal fusion of Gaofen-1 wide field of view imagery and MODIS imagery. Remote Sens, 2018, 10: 1047

    Article  Google Scholar 

  83. Li X, Ling F, Foody G M, et al. Generating a series of fine spatial and temporal resolution land cover maps by fusing coarse spatial resolution remotely sensed images and fine spatial resolution land cover maps. Remote Sens Environ, 2017, 196: 293–311

    Article  Google Scholar 

  84. Ping B, Meng Y S, Su F Z. An enhanced linear spatio-temporal fusion method for blending Landsat and MODIS data to synthesize Landsat-Like imagery. Remote Sens, 2018, 10: 881

    Article  Google Scholar 

  85. Quan J, Zhan W, Ma T, et al. An integrated model for generating hourly Landsat-like land surface temperatures over heterogeneous landscapes. Remote Sens Environ, 2018, 206: 403–423

    Article  Google Scholar 

  86. Liu W, Zeng Y, Li S, et al. An improved spatiotemporal fusion approach based on multiple endmember spectral mixture analysis. Sensors, 2019, 19: 2443

    Article  Google Scholar 

  87. Liao L, Song J, Wang J, et al. Bayesian method for building frequent Landsat-Like NDVI datasets by integrating MODIS and Landsat NDVI. Remote Sens, 2016, 8: 452

    Article  Google Scholar 

  88. Xue J, Leung Y, Fung T. An unmixing-based Bayesian model for spatio-temporal satellite image fusion in heterogeneous landscapes. Remote Sens, 2019, 11: 324

    Article  Google Scholar 

  89. Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process, 2004, 13: 600–612

    Article  Google Scholar 

  90. Renza D, Martinez E, Arquero A. A new approach to change detection in multispectral images by means of ERGAS index. IEEE Geosci Remote Sens Lett, 2013, 10: 76–80

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (Grant Nos. 61771496, 61571195), National Key Research and Development Program of China (Grant No. 2017YFB0502900), and Guangdong Provincial Natural Science Foundation (Grant No. 2017A030313382). The authors would like to thank the developers of STARFM, ESTARFM, FSDAF and STFDCNN algorithms for sharing their codes.

Author information

Authors and Affiliations

  1. Guangdong Provincial Key Laboratory of Urbanization and Geo-simulation, School of Geography and Planning, Sun Yat-sen University, Guangzhou, 510275, China

    Jun Li & Yunfei Li

  2. School of Automation Science and Engineering, South China University of Technology, Guangzhou, 510640, China

    Lin He

  3. State Key Laboratory of Earth Surface Processes and Resource Ecology, Institute of Remote Sensing Science and Engineering, Faculty of Geographical Science, Beijing Normal University, Beijing, 100875, China

    Jin Chen

  4. Hyperspectral Computing Laboratory, Department of Technology of Computers and Communications, Escuela Politécnica, University of Extremadura, Cáceres, E-10071, Spain

    Antonio Plaza

Authors
  1. Jun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunfei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lin He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Antonio Plaza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jun Li or Lin He.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Li, Y., He, L. et al. Spatio-temporal fusion for remote sensing data: an overview and new benchmark. Sci. China Inf. Sci. 63, 140301 (2020). https://doi.org/10.1007/s11432-019-2785-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-019-2785-y

Keywords

Search

Navigation