Abstract
Edge/structure-preserving operations for images aim to smooth images without blurring the edges/structures. Many exemplary edge-preserving filtering methods have recently been proposed to reduce the computational complexity and/or separate structures of different scales. They normally adopt a user-selected scale measurement to control the detail smoothing. However, natural photos contain objects of different sizes, which cannot be described by a single scale measurement. On the other hand, contour analysis is closely related to edge-preserving filtering, and significant progress has recently been achieved. Nevertheless, the majority of state-of-the-art filtering techniques have ignored the successes in this area. Inspired by the fact that learning-based edge detectors significantly outperform traditional manually-designed detectors, this paper proposes a learning-based edge-preserving filtering technique. It synergistically combines the differential operations in edge-preserving filters with the effectiveness of the recent edge detectors for scale-aware filtering. Unlike previous filtering methods, the proposed filters can efficiently extract subjectively meaningful structures from natural scenes containing multiple-scale objects.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
The structure-preserving filtering can be considered as a special design of edge-preserving filtering to deal with its limitation in handling textures. In most cases, this paper adopts the phrase “edge-preserving” for a broader concept.
The implementations with the default parameters published by the authors were employed.
Experiments conducted in this study use two iterations, except for the large-scale texture removal task in Sect. 4.1, which requires more iterations to smooth large-scale highly-textured images.
Please note that when a spatial parameter is a fractional number, it represents the percentage of width/height of the image.
References
Adams, A., Baek, J., & Davis, A. (2010). Fast high-dimensional filtering using the permutohedral lattice. CGF, 29(2), 753–762.
Adams, A., Gelfand, N., Dolson, J., & Levoy, M. (2009). Gaussian kd-trees for fast high-dimensional filtering. ACM TOG (SIGGRAPH), 28, 21:1–21:12.
An, X., & Pellacini, F. (2008). Appprop: All-pairs appearance-space edit propagation. ACM TOG (SIGGRAPH Asia), 27(3), 40:1–40:9.
Arbelaez, P., Maire, M., Fowlkes, C., & Malik, J. (2011). Contour detectionand hierarchical image segmentation. IEEE TPAMI, 33, 898–916.
Arbeláez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014). Multiscale combinatorial grouping. In CVPR.
Arnheim, R. (1956). Art and visual perception: A psychology of the creative eye. Berkeley: University of California Press.
Aujol, J., Gilboa, G., Chan, T., & Osher, S. (2006). Structure–texture image decomposition-modeling, algorithms, and parameter selection. IJCV, 67(1), 111–136.
Avidan, S., & Shamir, A. (2007). Seam carving for content-aware image resizing. ACM TOG (SIGGRAPH), 26(3), 10.
Bertasius, G., Shi, J., & Torresani, L. (2015a). Deepedge: A multi-scale bifurcated deep network for top-down contour detection. In CVPR.
Bertasius, G., Shi, J., & Torresani, L. (2015b). High-for-low and low-for-high: Efficient boundary detection from deep object features and its applications to high-level vision. In ICCV.
Bousseau, A., Paris, S., & Durand, F. (2009). User-assisted intrinsic images. ACM TOG (SIGGRAPH Asia), 28, 130:1–130:10.
Boyadzhiev, I., Bala, K., Paris, S., & Durand, F. (2012). User-guided white balance for mixed lighting conditions. ACM TOG (SIGGRAPH Asia), 31(6), 200:1–200:10.
Buades, A., & Lisani, J. L. (2016). Directional filters for color cartoon+texture image and video decomposition. Journal of Mathematical Imaging and Vision, 55(1), 125–135.
Canny, J. (1986). A computational approach to edge detection. In IEEE TPAMI.
Catanzaro, B., Su, B. Y., Sundaram, N., Lee, Y., Murphy, M., & Keutzer, K. (2009). Efficient, high-quality image contour detection. In ICCV.
Chambolle, A., & Darbon, J. (2009). On total variation minimization and surface evolution using parametric maximum flows. IJCV, 84(3), 288–307.
Chen, J., Paris, S., & Durand, F. (2007). Real-time edge-aware image processing with the bilateral grid. ACM TOG (SIGGRAPH), 26(3), 103.
Cho, H., Lee, H., Kang, H., & Lee, S. (2014). Bilateral texture filtering. ACM TOG (SIGGRAPH), 33(4), 128:1–128:8.
Criminisi, A., Sharp, T., Rother, C., & Perez, P. (2010). Geodesic image and video editing. ACM TOG, 29(5), 134.
Dani, A. L., Lischinski, D., & Weiss, Y. (2004). Colorization using optimization. ACM TOG (SIGGRAPH), 23, 689–694.
Dollár, P., Tu, Z., & Belongie, S. (2006). Supervised learning of edges and object boundaries. In CVPR.
Dollár, P., & Zitnick, C. L. (2013). Structured forests for fast edge detection. In ICCV.
Dollár, P., & Zitnick, C. L. (2015). Fast edge detection using structured forests. IEEE TPAMI.
Donoho, D., Chui, C., Coifman, R. R., & Lafon, S. (2006). Diffusion maps. Applied and Computational Harmonic Analysis, 21(1), 5–30.
Duda, R. O., & Hart, P. E. (1973). Pattern classification and scene analysis. New York: Wiley.
Durand, F., & Dorsey, J. (2002). Fast bilateral filtering for the display of high-dynamic-range images. ACM TOG (SIGGRAPH), 21(3), 257–266.
Eisemann, E., & Durand, F. (2004). Flash photography enhancement via intrinsic relighting. ACM TOG (SIGGRAPH), 23(3), 673–678.
Farbman, Z., Fattal, R., & Lischinski, D. (2010). Diffusion maps for edge-aware image editing. ACM TOG (SIGGRAPH Asia), 29(6), 145:1–145:10.
Farbman, Z., Fattal, R., Lischinski, D., & Szeliski, R. (2008). Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM TOG (SIGGRAPH), 27(3), 67.
Fattal, R. (2009). Edge-avoiding wavelets and their applications. ACM TOG (SIGGRAPH), 28(3), 1–10.
Gastal, E., & Oliveira, M. (2011). Domain transform for edge-aware image and video processing. ACM TOG (SIGGRAPH), 30(4), 69:1–69:12.
Gastal, E., & Oliveira, M. (2012). Adaptive manifolds for real-time high-dimensional filtering. ACM TOG (SIGGRAPH), 31(4), 33:1–33:13.
Gilboa, G. (2014). A total variation spectral framework for scale and texture analysis. SIAM Journal of Imaging Sciences, 7(4), 1937–1961.
Gupta, S., Arbeláez, P. A., Girshick, R. B., & Malik, J. (2015). Indoor scene understanding with RGB-D images: Bottom-up segmentation, object detection and semantic segmentation. IJCV, 112(2), 133–149.
He, K., Sun, J., & Tang, X. (2013). Guided image filtering. IEEE TPAMI, 35, 1397–1409.
Karacan, L., Erdem, E., & Erdem, A. (2013). Structure-preserving image smoothing via region covariances. ACM TOG (SIGGRAPH Asia), 32(6), 176:1–176:11.
Kivinen, J. J., Williams, C. K., & Heess, N. (2014). Visual boundary prediction: A deep neural prediction network and quality dissection. In AISTATS.
Kyprianidis, J. E., & Döllner, J. (2008). Image abstraction by structure adaptive filtering. In Proceedings of EG UK theory and practice of computer graphics, Manchester, United Kingdom, 2008 (pp. 51–58).
Kyprianidis, J. E., & Kang, H. (2011). Image and video abstraction by coherence-enhancing filtering. Computer Graphics Forum, 30(2), 593–602.
Levin, A., Lischinski, D., & Weiss, Y. (2006). A closed form solution to natural image matting. In CVPR.
Lim, J., Zitnick, C. L., & Dollár, P. (2013). Sketch tokens: A learned mid-level representation for contour and object detection. In CVPR.
Lischinski, D., Farbman, Z., Uyttendaele, M., & Szeliski, R. (2006). Interactive local adjustment of tonal values. ACM TOG (SIGGRAPH), 25(3), 646–653.
Margolin, R., Zelnik-Manor, L., & Tal, A. (2014). How to evaluate foreground maps. In CVPR.
Meyer, Y. (2001). Oscillating Patterns in Image Processing and Nonlinear Evolution Equations: The Fifteenth Dean Jacqueline B. Lewis Memorial Lectures. Providence: American Mathematical Society.
Min, D., Choi, S., Lu, J., Ham, B., Sohn, K., & Do, M. N. (2014). Fast global image smoothing based on weighted least squares. IEEE TIP, 23(12), 5638–5653.
Paris, S., & Durand, F. (2009). A fast approximation of the bilateral filter using a signal processing approach. IJCV, 81, 24–52.
Paris, S., Kornprobst, P., Tumblin, J., & Durand, F. (2009). Bilateral filtering: Theory and applications. Foundations and Trends in Computer Graphics and Vision, 4(1), 1–73.
Parisand, S., Hasinoff, S. W., & Kautz, J. (2011). Local laplacian filters: Edge-aware image processing with a Laplacian pyramid. ACM TOG (SIGGRAPH), 30(4), 68:1–68:12.
Perazzi, F., Krahenbuhl, P., Pritch, Y., & Hornung, A. (2012). Saliency filters: Contrast based filtering for salient region detection. In CVPR (pp. 733–740).
Perona, P., & Malik, J. (1990). Scale-space and edge detection using anisotropic diffusion. IEE TPAMI, 12, 629–639.
Perreault, S., & Hbert, P. (2007). Median filtering in constant time. IEEE TIP, 16(9), 2389–2394.
Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., & Toyama, K. (2004). Digital photography with flash and no-flash image pairs. ACM TOG (SIGGRAPH), 23(3), 664–672.
Pham, T. Q., & van Vliet, L. J. (2005). Separable bilateral filtering for fast video preprocessing. In ICME.
Porikli, F. (2008). Constant time o(1) bilateral filtering. In CVPR.
Rhemann, C., Hosni, A., Bleyer, M., Rother, C., & Gelautz, M. (2011). Fast cost-volume filtering for visual correspondence and beyond. In CVPR.
Ren, X., & Liefeng, B. (2012). Discriminatively trained sparse code gradients for contour detection. In NIPS.
Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60(1–4), 259–268.
Shen, W., Wang, X., Wang, Y., Bai, X., & Zhang, Z. (2015). Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. In CVPR (pp. 3982–3991).
Subr, K., Soler, C., & Durand, F. (2009). Edge-preserving multiscale image decomposition based on local extrema. ACM ToG (SIGGRAPH Asia), 28(5), 147.
Tomasi, C., & Manduchi, R. (1998). Bilateral filtering for gray and color images. In ICCV (pp. 839–846).
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE TIP, 13(4), 600–612.
Weickert, J. (1999). Coherence-enhancing diffusion filtering. IJCV, 31(2–3), 111–127.
Weiss, B. (2006). Fast median and bilateral filtering. ACM TOG (SIGGRAPH), 25(3), 519–526.
Xie, S., & Tu, Z. (2015). Holistically-nested edge detection. In Proceedings of IEEE international conference on computer vision.
Xu, K., Li, Y., Ju, T., Hu, S. M., & Liu, T. Q. (2009). Efficient affinity-based edit propagation using k-d tree. ACM ToG (SIGGRAPH Asia), 28(5), 118:1–118:6.
Xu, L., Lu, C., Xu, Y., & Jia, J. (2011). Image smoothing via l0 gradient minimization. ACM TOG (SIGGRAPH Asia), 36(6), 174.
Xu, L., Yan, Q., & Jia, J. (2013). A sparse control model for image and video editing. ACM TOG (SIGGRAPH Asia), 32(6), 197.
Xu, L., Yan, Q., Xia, Y., & Jia, J. (2012). Structure extraction from texture via relative total variation. ACM TOG (SIGGRAPH Asia), 31(6), 139.
Yan, Q., Xu, L., Shi, J., & Jia, J. (2013). Hierarchical saliency detection. In CVPR.
Yang, J., Price, B., Cohen, S., Lee, H., & Yang, M. H. (2016). Object contour detection with a fully convolutional encoder-decoder network. In CVPR.
Yang, Q. (2012). Recursive bilateral filtering. In ECCV (pp. 399–413).
Yang, Q. (2016). Semantic filtering. In CVPR.
Yang, Q., Ahuja, N., & Tan, K. (2015). Constant time median and bilateral filtering. IJCV, 112(3), 307–318.
Yang, Q., Tan, K. H., & Ahuja, N. (2009). Real-time o(1) bilateral filtering. In CVPR (pp. 557–564).
Yang, Q., Wang, S., & Ahuja, N. (2010). Svm for edge-preserving filtering. In CVPR (pp. 1775–1782).
Yin, W., Goldfarb, D., & Osher, S. (2005). Image cartoon-texture decomposition and feature selection using the total variation regularized l1 functional. In VLSM (pp. 73–84).
Yoon, K. J., & Kweon, I. S. (2006). Adaptive support-weight approach for correspondence search. IEEE TPAMI, 28(4), 650–656.
Zeune, L., van Dalum, G., Terstappen, L. W. M. M., van Gils, S. A., & Brune, C. (2016). Multiscale segmentation via Bregman distances and nonlinear spectral analysis. CoRR arXiv:1604.06665.
Zhang, J., Sclaroff, S., Lin, Z., Shen, X., Price, B., & Mech, R. (2015). Minimum barrier salient object detection at 80 fps. In ICCV.
Zhang, Q., Shen, X., Xu, L., & Jia, J. (2014). Rolling guidance filter. In ECCV.
Zheng, S., Tu, Z., & Yuille, A. (2007). Detecting object boundaries using low-,mid-, and high-level information. In CVPR.
Ziou, D., & Tabbone, S. (1998). Edge detection techniques: An overview. IEEE TPAMI, 8, 537–559.
Zitnick, C. L., & Dollár, P. (2014). Edge boxes: Locating object proposals from edges. In ECCV.
Zitnick, C. L., & Parikh, D. (2012). The role of image understanding in contour detection. In CVPR.
Acknowledgements
We thank all the reviewers for valuable comments. This work was supported by the National Basic Research Program of China (Grant No. 2015CB351705), the State Key Program of National Natural Science Foundation of China (Grant No. 61332018).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by S. Soatto.
Rights and permissions
About this article
Cite this article
Wei, X., Yang, Q. & Gong, Y. Joint Contour Filtering. Int J Comput Vis 126, 1245–1265 (2018). https://doi.org/10.1007/s11263-018-1091-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-018-1091-5