Abstract
Dynamic weather such as rain and snow causes complex spatio-temporal intensity fluctuations in videos. Such fluctuations can adversely impact vision systems that rely on small image features for tracking, object detection and recognition. While these effects appear to be chaotic in space and time, we show that dynamic weather has a predictable global effect in frequency space. For this, we first develop a model of the shape and appearance of a single rain or snow streak in image space. Detecting individual streaks is difficult even with an accurate appearance model, so we combine the streak model with the statistical characteristics of rain and snow to create a model of the overall effect of dynamic weather in frequency space. Our model is then fit to a video and is used to detect rain or snow streaks first in frequency space, and the detection result is then transferred to image space. Once detected, the amount of rain or snow can be reduced or increased. We demonstrate that our frequency analysis allows for greater accuracy in the removal of dynamic weather and in the performance of feature extraction than previous pixel-based or patch-based methods. We also show that unlike previous techniques, our approach is effective for videos with both scene and camera motions.
Similar content being viewed by others
References
Auer, A. H. Jr. (1972). Distribution of graupel and hail with size. Monthly Weather Review, 100(5), 325–328.
Auer, A. H. Jr., & Veal, D. L. (1970). The dimension of ice crystals in natural clouds. Journal of the Atmospheric Sciences, 27(6), 919–926.
Barnum, P., Kanade, T., & Narasimhan, S. (2007). Spatio-temporal frequency analysis for removing rain and snow from videos. In Workshop on photometric analysis for computer vision, in conjunction with international conference on computer vision.
Böhm, H. P. (1989). A general equation for the terminal fall speed of solid hydrometeors. Journal of the Atmospheric Sciences, 46, 2419–27.
Bouguet, J.-Y. (2000). Pyramidal implementation of the Lucas Kanade feature tracker. Intel Corporation, Microprocessor Research Labs.
Cozman, F., & Krotkov, E. (1997). Depth from scattering. In International conference on computer vision.
de la Torre, F., & Black, M. L. (2001). Robust principal component analysis for computer vision. In International conference on computer vision.
Desaulniers-Soucy, N. (1999). Empirical test of the multifractal continuum limit in rain. PhD thesis, Mcgill.
Desaulniers-Soucy, N., Lovejoy, S., & Schertzer, D. (2001). The HYDROP experiment: an empirical method for the determination of the continuum limit in rain. Atmospheric Research, 59–60, 163–197.
Elgammal, A., Harwood, D., & Davis, L. (2000). Non-parametric model for background subtraction. In European conference on computer vision.
Feingold, G., & Levin, Z. (1986). The lognormal fit to raindrop spectra from frontal convective clouds in Israel. Journal of Climate and Applied Meteorology, 25, 1346–63.
Fishler, M., & Bolles, R. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. In Communications of the ACM.
Foote, G. B., & duToit, P. S. (1969). Terminal velocity of raindrops aloft. Journal of Applied Meteorology, 8(2), 249–53.
Garg, K., & Nayar, S. K. (2004). Detection and removal of rain from videos. In Computer vision and pattern recognition.
Garg, K., & Nayar, S. K. (2005). When does a camera see rain? In International conference on computer vision.
Garg, K., & Nayar, S. K. (2006). Photorealistic rendering of rain streaks. In SIGGRAPH.
Gonzalez, R. C., & Woods, R. E. (2002). Digital image processing (2nd ed.). New York: Prentice Hall.
Gunn, K., & Marshall, J. (1958). The distribution with size of aggregate snowflakes. Journal of Meteorology, 15, 452–461.
Hase, H., Miyake, K., & Yoneda, M. (1999). Real-time snowfall noise elimination.
Heeger, D. (1987). Optical flow from spatiotemporal filters. In International conference on computer vision.
Jameson, A. R., & Kostinski, A. B. (2001). What is a raindrop size distribution? Bulletin of the American Meteorological Society, 8(6), 1169–1177.
Jameson, A., & Kostinski, A. (2002). When is rain steady? Journal of Applied Meteorology, 41(1), 83–90.
Ke, Q., & Kanade, T. (2002). A robust subspace approach to layer extraction. In IEEE workshop on motion and video computing.
Kubesh, R. J., & Beard, K. (1993). Laboratory measurements of spontaneous oscillations of moderate-size raindrops. Journal of the Atmospheric Sciences, 50, 1089–1098.
Langer, M., & Mann, R. (2003). Optical snow. International Journal of Computer Vision, 55(1), 55–71.
Langer, M., & Zhang, Q. (2003). Rendering falling snow using and inverse Fourier transform. In ACM SIGGRAPH technical sketches program.
Langer, M. S., Zhang, L., Klein, A., Bhatia, A., Pereira, J., & Rekhi, D. (2004). A spectral-particle hybrid method for rendering falling snow. In Eurographics symposium on rendering.
Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 20, 91–110.
Magono, C., & Nakamura, T. (1965). Aerodynamic studies of falling snowflakes. Journal of the Meteorological Society of Japan, 43, 139–147.
Marshall, J., & Palmer, W. (1948). The distribution of raindrops with size. Journal of Meteorology, 5, 165–166.
Mikolajczyk, K., & Schmid, C. (2001). Indexing based on scale invariant interest points. In International conference on computer vision.
Narasimhan, S. G., & Nayar, S. K. (2002). Vision and the atmosphere. International Journal of Computer Vision, 48(3), 233–254.
Nayar, S. K., & Narasimhan, S. G. (1999). Vision in bad weather. In International conference on computer vision.
Ohtake, T. (1965). Preliminary observations on size distribution of snowflakes and raindrops at just above and below the melting layer. In International conference on cloud physics.
Pruppacher, H. R., & Klett, J. D. (1997). Microphysics of clouds and precipitation. Amsterdam: Kluwer Academic. Second revised and enlarged edition.
Reeves, W. T. (1983). Particle systems—a technique for modeling a class of fuzzy objects. ACM Transactions on Graphics, 2(2), 91–108.
Sand, P., & Teller, S. (2006). Particle video: long-range motion estimation using point trajectories. In Computer vision and pattern recognition.
Shi, J., & Tomasi, C. (1994). Good features to track. In Computer vision and pattern recognition.
Starik, S., & Werman, M. (2003). Simulation of rain in videos. In International workshop on texture analysis and synthesis.
Stauffer, C., & Grimson, W. (1998). Adaptive background mixture models for real-time tracking. In Computer vision and pattern recognition.
Tariq, S. (2007). Rain (Technical report). NVIDIA.
Tatarchuk, N., & Isidoro, J. (2006). Artist-directable real-time rain rendering in city environments. In Eurographics workshop on natural phenomena.
Tokay, A., & Beard, K. (1996). A field study of raindrop oscillations. Part I: Observation of size spectra and evaluation of oscillation causes. Journal of Applied Meteorology, 35, 1671–1687.
Torr, P. H. S., Szeliski, R., & Anandan, P. (1999). An integrated Bayesian approach to layer extraction from image sequences. In International conference on computer vision.
Ulbrich, C. W. (1983). Natural variations in the analytical form of the raindrop size distribution. Journal of Applied Meteorology, 22(10), 1764–1775.
Van de Hulst, H. (1957). Light scattering by small particles. New York: Wiley.
Zelnik-Manor, L., Machline, M., & Irani, M. (2006). Multi-body factorization with uncertainty: Revisiting motion consistency. International Journal of Computer Vision, 68(1), 27–41.
Zhang, X., Li, H., Qi, Y., Kheng, W., & Ng, T. K. (2006). Rain removal in video by combining temporal and chromatic properties. In International conference on multimedia and expo.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Barnum, P.C., Narasimhan, S. & Kanade, T. Analysis of Rain and Snow in Frequency Space. Int J Comput Vis 86, 256–274 (2010). https://doi.org/10.1007/s11263-008-0200-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-008-0200-2