Skip to main content
SpringerLink
Log in

Joint Image Denoising and Disparity Estimation via Stereo Structure PCA and Noise-Tolerant Cost

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Stereo cameras are now commonly available on cars and mobile phones. However, the captured images may suffer from low image quality under noisy conditions, producing inaccurate disparity. In this paper, we aim at jointly restoring a clean image pair and estimating the corresponding disparity. To this end, we propose a new joint framework that iteratively optimizes these two different tasks in a multiscale fashion. First, structure information between the stereo pair is utilized to denoise the images using a non-local means strategy. Second, a new noise-tolerant cost function is proposed for noisy stereo matching. These two terms are integrated into a multiscale framework in which cross-scale information is leveraged to further improve both denoising and stereo matching. Extensive experiments on datasets captured from indoor, outdoor, and low-light conditions show that the proposed method achieves superior performance than the state-of-the-art image denoising and disparity estimation methods. While it outperforms multi-image denoising methods by about 2 dB on average, it achieves a 50% error reduction over radiometric-change-robust stereo matching on the challenging KITTI dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  • Alter, F., Matsushita, Y., & Tang, X. (2006). An intensity similarity measure in low-light conditions. In ECCV.

  • Birchfield, S., & Tomasi, C. (1998). A pixel dissimilarity measure that is insensitive to image sampling. IEEE TPAMI, 20(4), 401–406.

    Article  Google Scholar 

  • Buades, A., Coll, B., & Morel, J. (2005). A non-local algorithm for image denoising. In CVPR (pp. 60–65).

  • Buades, A., Coll, B., & Morel, J. M. (2005). A review of image denoising algorithms, with a new one. Multiscale Modeling and Simulation, 4(2), 490–530.

    Article  MathSciNet  MATH  Google Scholar 

  • Buades, A., Coll, B., & Morel, J. M. (2008). Nonlocal image and movie denoising. IJCV, 76(2), 123–139.

    Article  Google Scholar 

  • Burger, H., Schuler, C., & Harmeling, S. (2012). Image denoising: Can plain neural networks compete with BM3D? In CVPR (pp. 2392–2399).

  • Chan, S., Zickler, T., & Lu, Y. (2014). Monte Carlo non-local means: Random sampling for large-scale image filtering. IEEE TIP, 23(8), 3711–3725.

    MathSciNet  Google Scholar 

  • Chen, F., Zhang, L., & Yu, H. (2015). External patch prior guided internal clustering for image denoising. In ICCV(pp. 603–611).

  • Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE TIP, 16(8), 2080–2095.

    MathSciNet  Google Scholar 

  • De-Maeztu, L., Villanueva, A., & Cabeza, R. (2011). Stereo matching using gradient similarity and locally adaptive support-weight. Pattern Recognition Letters, 32(13), 1643–1651.

    Article  Google Scholar 

  • Deledalle, C., Salmon, J., & Dalalyan, A. (2011). Image denoising with patch based PCA: Local versus global. In BMVC (pp. 425–455).

  • Dong, W., Li, G., Shi, G., Li, X., & Ma, Y. (2015). Low-rank tensor approximation with Laplacian scale mixture modeling for multiframe image denoising. In ICCV (pp. 442–449).

  • Felzenszwalb, P., & Huttenlocher, D. (2006). Efficient belief propagation for early vision. IJCV, 70(1), 41–54.

    Article  Google Scholar 

  • Fu, Y., Lam, A., Sato, I., & Sato, Y. (2016). Adaptive spatial-spectral dictionary learning for hyperspectral image restoration. IJCV, 122, 1–18.

  • Healey, G., & Kondepudy, R. (1994). Radiometric CCD camera calibration and noise estimation. IEEE TPAMI, 16(3), 267–276.

    Article  Google Scholar 

  • Heo, Y., Lee, K., & Lee, S. (2007). Simultaneous depth reconstruction and restoration of noisy stereo images using non-local pixel distribution. In CVPR (pp. 1–8).

  • Heo, Y., Lee, K., & Lee, S. (2011). Robust stereo matching using adaptive normalized cross-correlation. IEEE TPAMI, 33(4), 807–822.

    Article  Google Scholar 

  • Hirschmüller, H. (2008). Stereo processing by semiglobal matching and mutual information. IEEE TPAMI, 30(2), 328–341.

    Article  Google Scholar 

  • Hirschmüller, H., Innocent, P., & Garibaldi, J. (2002). Real-time correlation-based stereo vision with reduced border errors. IJCV, 47(1), 229–246.

    Article  MATH  Google Scholar 

  • Hirschmüller, H., & Scharstein, D. (2009). Evaluation of stereo matching costs on images with radiometric differences. IEEE TPAMI, 31(9), 1582–1599.

    Article  Google Scholar 

  • Honda, H., Timofte, R., & Gool, L.V. (2015). Make my day—high-fidelity color denoising with near-infrared. In CVPR workshops (pp. 82–90).

  • Joshi, N., & Cohen, M. (2010). Seeing Mt. Rainier: Lucky imaging for multi-image denoising, sharpening, and haze removal. In ICCP (pp. 1–8).

  • Jung, I., Sim, J., Kim, C., & Lee, S. (2013). Robust stereo matching under radiometric variations based on cumulative distributions of gradients. In ICIP (pp. 2082–2085).

  • Kim, Y., Koo, J., & Lee, S. (2016). Adaptive descriptor-based robust stereo matching under radiometric changes. Pattern Recognition Letters, 78, 41–47.

    Article  Google Scholar 

  • Levin, A., & Nadler, B. (2011). Natural image denoising: Optimality and inherent bounds. In CVPR (pp. 2833–2840).

  • Levin, A., Nadler, B., Durand, F., & Freeman, W. (2012). Patch complexity, finite pixel correlations and optimal denoising. In ECCV (pp. 73–86).

  • Liu, C., & Freeman, W. (2010). A high-quality video denoising algorithm based on reliable motion estimation. In ECCV (pp. 706–719).

  • Lu, X., Lin, Z., Jin, H., Yang, J., & Wang, J. (2015). Image-specific prior adaptation for denoising. IEEE TIP, 24(12), 5469–5478.

    MathSciNet  Google Scholar 

  • Luo, E., Chan, S., & Nguyen, T. (2015). Adaptive image denoising by targeted databases. IEEE TIP, 24(7), 2167–2181.

    MathSciNet  Google Scholar 

  • Luo, E., Chan, S., Pan, S., & Nguyen, T. (2013). Adaptive non-local means for multiview image denoising: Searching for the right patches via a statistical approach. In ICIP (pp. 543–547).

  • Maggioni, M., Katkovnik, V., Egiazarian, K., & Foi, A. (2013). Nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE TIP, 22(1), 119–133.

    MathSciNet  Google Scholar 

  • Menz, M., & Freeman, R. (2003). Stereoscopic depth processing in the visual cortex: a coarse-to-fine mechanism. Nature Neuroscience, 6(1), 59–65.

    Article  Google Scholar 

  • Menze, M., & Geiger, A. (2015). Object scene flow for autonomous vehicles. In CVPR (pp. 3061–3070).

  • Mosseri, I., Zontak, M., & Irani, M. (2013). Combining the power of internal and external denoising. In ICIP (pp. 1–9).

  • Muresan, D., & Parks, T. (2003). Adaptive principal components and image denoising. In ICIP (pp. 101–104).

  • Nir, T., Kimmel, R., & Bruckstein, A. (2005). Variational approach for joint optic-flow computation and video restoration. Technical report CIS200503, Technion, Israel Institute of Technology.

  • Park, B., Lee, K., & Lee, S. (2006). A new similarity measure for random signatures: Perceptually modified Hausdorff distance. In ACIVS (pp. 990–1001).

  • Romeny, B. M. T. H., & Florack, L. (1993). A multiscale geometric model of human vision. In W. R. Hendee & P. N. T. Wells (Eds.), Perception of visual information, Chap. 4 (pp. 73–114).

  • Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV, 47(1), 7–42.

    Article  MATH  Google Scholar 

  • Scharstein, D., & Szeliski, R. (2007). Middlebury stereo dataset. http://vision.middlebury.edu/stereo/data/.

  • Shao, L., Yan, R., & Li, X. (2014). From heuristic optimization to dictionary learning: A review and comprehensive comparison of image denoising algorithms. IEEE Transsction on Cybernetics, 44(7), 1001–1013.

    Article  Google Scholar 

  • Shen, X., Yan, Q., Xu, L., Ma, L., & Jia, J. (2015). Multispectral joint image restoration via optimizing a scale map. IEEE TPAMI, 37(12), 2518–2530.

    Article  Google Scholar 

  • Tan, X., Sun, C., Wang, D., Guo, Y., & Pham, T. (2014). Soft cost aggregation with multi-resolution fusion. In ECCV (pp. 17–32).

  • Vemulapalli, R., Tuzel, O., & Liu, M. (2015). Deep Gaussian conditional random field network: A model-based deep network for discriminative denoising. arXiv:1511.04067.

  • Vu, D., Chidester, B., Yang, H., Do, M., & Lu, J. (2014). Efficient hybrid tree-based stereo matching with applications to postcapture image refocusing. IEEE TIP, 23(8), 3428–3442.

    MathSciNet  Google Scholar 

  • Xu, J., Yang, Q., Tang, J., & Feng, Z. (2016). Linear time illumination invariant stereo matching. IJCV, 119, 179–193.

    Article  MathSciNet  Google Scholar 

  • Xu, J., Zhang, L., Zuo, W., Zhang, D., & Feng, X. (2015). Patch group based nonlocal self-similarity prior learning for image denoising. In ICCV (pp. 244–252).

  • Xu, L., & Jia, J. (2010). Two-phase kernel estimation for robust motion deblurring. In ECCV (pp. 157–170).

  • Yue, H., Sun, X., Yang, J., & Wu, F. (2014). Cid: Combined image denoising in spatial and frequency domains using web images. In CVPR (pp. 2933–2940).

  • Yue, H., Sun, X., Yang, J., & Wu, F. (2015). Image denoising by exploring external and internal correlations. IEEE TIP, 24(6), 1967–1982.

    MathSciNet  Google Scholar 

  • Zabih, R., & Woodfill, J. (1994). Non-parametric local transforms for computing visual correspondence. In ECCV (pp. 151–158).

  • Zbontar, J., & LeCun, Y. (2016). Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 17(1), 2287–2318.

    MATH  Google Scholar 

  • Zhang, K., Fang, Y., Min, D., Sun, L., Yang, S., Yan, S., et al. (2014). Cross-scale cost aggregation for stereo matching. In CVPR (pp. 1590–1597).

  • Zhang, L., Dong, W., Zhang, D., & Shi, G. (2010). Two-stage image denoising by principal component analysis with local pixel grouping. Pattern Recognition, 43(4), 1531–1549.

    Article  MATH  Google Scholar 

  • Zhang, L., Vaddadi, S., Jin, H., & Nayar, S. (2009). Multiple view image denoising. In CVPR (pp. 1542–1549).

  • Zontak, M., Mosseri, I., & Irani, M. (2013). Separating signal from noise using patch recurrence across scales. In CVPR (pp. 1195–1202).

  • Zoran, D., & Weiss, Y. (2011) From learning models of natural image patches to whole image restoration. In ICCV (pp. 479–486).

Download references

Acknowledgements

We would thank the anonymous reviewers for their constructive suggestions and insightful comments. This work is partially supported by the Hong Kong PhD Fellowship Scheme (HKPFS) from the RGC of Hong Kong, a SRG grant from City University of Hong Kong (No. 7004416), and a GRF grant from the RGC of Hong Kong (PolyU 52124/15E).

Author information

Authors and Affiliations

  1. Department of Computer Science, City University of Hong Kong, Hong Kong, China

    Jianbo Jiao & Rynson W. H. Lau

  2. School of Information Science and Technology, University of Science and Technology of China, Hefei, Anhui, China

    Qingxiong Yang

  3. School of Computer Science and Engineering, South China University of Technology, Guangzhou, China

    Shengfeng He

  4. Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China

    Shuhang Gu & Lei Zhang

Authors
  1. Jianbo Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingxiong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengfeng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuhang Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rynson W. H. Lau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingxiong Yang.

Additional information

Communicated by Scharstein.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiao, J., Yang, Q., He, S. et al. Joint Image Denoising and Disparity Estimation via Stereo Structure PCA and Noise-Tolerant Cost. Int J Comput Vis 124, 204–222 (2017). https://doi.org/10.1007/s11263-017-1015-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-017-1015-9

Keywords

Search

Navigation