Skip to main content
Log in

Abstract

Dynamic weather such as rain and snow causes complex spatio-temporal intensity fluctuations in videos. Such fluctuations can adversely impact vision systems that rely on small image features for tracking, object detection and recognition. While these effects appear to be chaotic in space and time, we show that dynamic weather has a predictable global effect in frequency space. For this, we first develop a model of the shape and appearance of a single rain or snow streak in image space. Detecting individual streaks is difficult even with an accurate appearance model, so we combine the streak model with the statistical characteristics of rain and snow to create a model of the overall effect of dynamic weather in frequency space. Our model is then fit to a video and is used to detect rain or snow streaks first in frequency space, and the detection result is then transferred to image space. Once detected, the amount of rain or snow can be reduced or increased. We demonstrate that our frequency analysis allows for greater accuracy in the removal of dynamic weather and in the performance of feature extraction than previous pixel-based or patch-based methods. We also show that unlike previous techniques, our approach is effective for videos with both scene and camera motions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Auer, A. H. Jr. (1972). Distribution of graupel and hail with size. Monthly Weather Review, 100(5), 325–328.

    Google Scholar 

  • Auer, A. H. Jr., & Veal, D. L. (1970). The dimension of ice crystals in natural clouds. Journal of the Atmospheric Sciences, 27(6), 919–926.

    Article  Google Scholar 

  • Barnum, P., Kanade, T., & Narasimhan, S. (2007). Spatio-temporal frequency analysis for removing rain and snow from videos. In Workshop on photometric analysis for computer vision, in conjunction with international conference on computer vision.

  • Böhm, H. P. (1989). A general equation for the terminal fall speed of solid hydrometeors. Journal of the Atmospheric Sciences, 46, 2419–27.

    Article  Google Scholar 

  • Bouguet, J.-Y. (2000). Pyramidal implementation of the Lucas Kanade feature tracker. Intel Corporation, Microprocessor Research Labs.

  • Cozman, F., & Krotkov, E. (1997). Depth from scattering. In International conference on computer vision.

  • de la Torre, F., & Black, M. L. (2001). Robust principal component analysis for computer vision. In International conference on computer vision.

  • Desaulniers-Soucy, N. (1999). Empirical test of the multifractal continuum limit in rain. PhD thesis, Mcgill.

  • Desaulniers-Soucy, N., Lovejoy, S., & Schertzer, D. (2001). The HYDROP experiment: an empirical method for the determination of the continuum limit in rain. Atmospheric Research, 59–60, 163–197.

    Article  Google Scholar 

  • Elgammal, A., Harwood, D., & Davis, L. (2000). Non-parametric model for background subtraction. In European conference on computer vision.

  • Feingold, G., & Levin, Z. (1986). The lognormal fit to raindrop spectra from frontal convective clouds in Israel. Journal of Climate and Applied Meteorology, 25, 1346–63.

    Article  Google Scholar 

  • Fishler, M., & Bolles, R. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. In Communications of the ACM.

  • Foote, G. B., & duToit, P. S. (1969). Terminal velocity of raindrops aloft. Journal of Applied Meteorology, 8(2), 249–53.

    Article  Google Scholar 

  • Garg, K., & Nayar, S. K. (2004). Detection and removal of rain from videos. In Computer vision and pattern recognition.

  • Garg, K., & Nayar, S. K. (2005). When does a camera see rain? In International conference on computer vision.

  • Garg, K., & Nayar, S. K. (2006). Photorealistic rendering of rain streaks. In SIGGRAPH.

  • Gonzalez, R. C., & Woods, R. E. (2002). Digital image processing (2nd ed.). New York: Prentice Hall.

    Google Scholar 

  • Gunn, K., & Marshall, J. (1958). The distribution with size of aggregate snowflakes. Journal of Meteorology, 15, 452–461.

    Google Scholar 

  • Hase, H., Miyake, K., & Yoneda, M. (1999). Real-time snowfall noise elimination.

  • Heeger, D. (1987). Optical flow from spatiotemporal filters. In International conference on computer vision.

  • Jameson, A. R., & Kostinski, A. B. (2001). What is a raindrop size distribution? Bulletin of the American Meteorological Society, 8(6), 1169–1177.

    Article  Google Scholar 

  • Jameson, A., & Kostinski, A. (2002). When is rain steady? Journal of Applied Meteorology, 41(1), 83–90.

    Article  Google Scholar 

  • Ke, Q., & Kanade, T. (2002). A robust subspace approach to layer extraction. In IEEE workshop on motion and video computing.

  • Kubesh, R. J., & Beard, K. (1993). Laboratory measurements of spontaneous oscillations of moderate-size raindrops. Journal of the Atmospheric Sciences, 50, 1089–1098.

    Article  Google Scholar 

  • Langer, M., & Mann, R. (2003). Optical snow. International Journal of Computer Vision, 55(1), 55–71.

    Article  Google Scholar 

  • Langer, M., & Zhang, Q. (2003). Rendering falling snow using and inverse Fourier transform. In ACM SIGGRAPH technical sketches program.

  • Langer, M. S., Zhang, L., Klein, A., Bhatia, A., Pereira, J., & Rekhi, D. (2004). A spectral-particle hybrid method for rendering falling snow. In Eurographics symposium on rendering.

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 20, 91–110.

    Article  Google Scholar 

  • Magono, C., & Nakamura, T. (1965). Aerodynamic studies of falling snowflakes. Journal of the Meteorological Society of Japan, 43, 139–147.

    Google Scholar 

  • Marshall, J., & Palmer, W. (1948). The distribution of raindrops with size. Journal of Meteorology, 5, 165–166.

    Google Scholar 

  • Mikolajczyk, K., & Schmid, C. (2001). Indexing based on scale invariant interest points. In International conference on computer vision.

  • Narasimhan, S. G., & Nayar, S. K. (2002). Vision and the atmosphere. International Journal of Computer Vision, 48(3), 233–254.

    Article  MATH  Google Scholar 

  • Nayar, S. K., & Narasimhan, S. G. (1999). Vision in bad weather. In International conference on computer vision.

  • Ohtake, T. (1965). Preliminary observations on size distribution of snowflakes and raindrops at just above and below the melting layer. In International conference on cloud physics.

  • Pruppacher, H. R., & Klett, J. D. (1997). Microphysics of clouds and precipitation. Amsterdam: Kluwer Academic. Second revised and enlarged edition.

    Google Scholar 

  • Reeves, W. T. (1983). Particle systems—a technique for modeling a class of fuzzy objects. ACM Transactions on Graphics, 2(2), 91–108.

    Article  Google Scholar 

  • Sand, P., & Teller, S. (2006). Particle video: long-range motion estimation using point trajectories. In Computer vision and pattern recognition.

  • Shi, J., & Tomasi, C. (1994). Good features to track. In Computer vision and pattern recognition.

  • Starik, S., & Werman, M. (2003). Simulation of rain in videos. In International workshop on texture analysis and synthesis.

  • Stauffer, C., & Grimson, W. (1998). Adaptive background mixture models for real-time tracking. In Computer vision and pattern recognition.

  • Tariq, S. (2007). Rain (Technical report). NVIDIA.

  • Tatarchuk, N., & Isidoro, J. (2006). Artist-directable real-time rain rendering in city environments. In Eurographics workshop on natural phenomena.

  • Tokay, A., & Beard, K. (1996). A field study of raindrop oscillations. Part I: Observation of size spectra and evaluation of oscillation causes. Journal of Applied Meteorology, 35, 1671–1687.

    Article  Google Scholar 

  • Torr, P. H. S., Szeliski, R., & Anandan, P. (1999). An integrated Bayesian approach to layer extraction from image sequences. In International conference on computer vision.

  • Ulbrich, C. W. (1983). Natural variations in the analytical form of the raindrop size distribution. Journal of Applied Meteorology, 22(10), 1764–1775.

    Article  Google Scholar 

  • Van de Hulst, H. (1957). Light scattering by small particles. New York: Wiley.

    Google Scholar 

  • Zelnik-Manor, L., Machline, M., & Irani, M. (2006). Multi-body factorization with uncertainty: Revisiting motion consistency. International Journal of Computer Vision, 68(1), 27–41.

    Article  Google Scholar 

  • Zhang, X., Li, H., Qi, Y., Kheng, W., & Ng, T. K. (2006). Rain removal in video by combining temporal and chromatic properties. In International conference on multimedia and expo.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Peter C. Barnum.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Barnum, P.C., Narasimhan, S. & Kanade, T. Analysis of Rain and Snow in Frequency Space. Int J Comput Vis 86, 256–274 (2010). https://doi.org/10.1007/s11263-008-0200-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-008-0200-2

Keywords

Navigation