Skip to main content
SpringerLink
Log in

Unsupervised RGB-T saliency detection by node classification distance and sparse constrained graph learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Saliency detection methods which center on RGB images are sensitive to surrounding environments. Fusing complementary RGB and thermal infrared (RGB-T) images is an effective way to promote the final saliency performance. However, there are relatively few datasets and algorithms for RGB-T saliency detection, which is the prominent problem in this research field. Therefore, an unsupervised method that does not require a large amount of labeled data is proposed for RGB-T image saliency detection in this paper. At first, we construct the graph model in which the superpixels are treated as graph nodes. Instead of utilizing the Euclidean distance to construct initial affinity matrix, a novel node classification distance is designed to explore the local relationship and graph geometrical structure of nodes. Additionally, the advantageous constraint is proposed to increase the sparsity of the image, which not only makes the initial affinity matrix sparse and accurate but also enhances the foreground or background consistency during the graph learning. Furthermore, an adaptive ranking algorithm fusing classification distance and sparse constraint is used to unify the graph affinity learning and the computation of saliency values, which helps to generate more accurate saliency results. Experiments on two public RGB-T datasets demonstrate that the applied method performs desirably against the state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Yu L, Xia X, Zhou K (2019) Traffic sign detection based on visual co-saliency in complex scenes. Appl Intell 49(2):764–790

    Article  Google Scholar 

  2. Bi HB, Lu D, Zhu HH, Yang LN, Guan HP (2020) STA-Net: spatial-temporal attention network for video salient object detection, Appl Intell, 1–10

  3. Xia C, Gao X, Li KC, Zhao Q, Zhang S (2020) Salient object detection based on distribution-edge guidance and iterative Bayesian optimization. Appl Intell 50:2977–2990

    Article  Google Scholar 

  4. Zhu H, Wang B, Zhang X, Liu J (2020) Semantic image segmentation with shared decomposition convolution and boundary reinforcement structure, Appl Intell, 1–14

  5. Guo C, Zhang L (2009) A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans Image Process 19(1):185–198

    MathSciNet  MATH  Google Scholar 

  6. Mehmood Z, Mahmood T, Javid MA (2018) Content-based image retrieval and semantic automatic image annotation based on the weighted average of triangular histograms using support vector machine. Appl Intell 48(1):166–181

    Article  Google Scholar 

  7. Wang G, Li C, Ma Y, Zheng A, Tang J, Luo B (2018) Rgb-t saliency detection benchmark: Dataset, baselines, analysis and a novel approach, In Chinese Conference on Image and Graphics Technologies, pp 359–369

  8. Tu Z, Xia T, Li C, Wang X, Ma Y, Tang J (2019) RGB-T image saliency detection via collaborative graph learning. IEEE Trans Multimed 22(1):160–173

  9. Huang L, Song K, Gong A, Liu C, Yan Y (2020) RGB-T saliency detection via low-rank tensor learning and unified collaborative ranking. IEEE Sig Process Lett 27:1585–1589

    Article  Google Scholar 

  10. Yang C, Zhang L, Lu H, Ruan X, Yang, MH (2013) Saliency detection via graph-based manifold ranking, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3166–3173

  11. Peng H, Li B, Ling H, Hu W, Xiong W, Maybank SJ (2016) Salient object detection via structured matrix decomposition. IEEE Trans Pattern Anal Mach Intell 39(4):881–832

  12. Zhang L, Ai J, Jiang B, Lu H, Li X (2018) Saliency detection via absorbing Markov chain with learnt transition probability. IEEE Trans Image Process 27(2):987–998

    Article  MathSciNet  Google Scholar 

  13. Li X, Lu H, Zhang L, Ruan X, Yang MH (2013) Saliency detection via dense and sparse reconstruction, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2976–2983

  14. Kim J, Han D, Tai YW, Kim J (2015) Salient region detection via high-dimensional color transform and local spatial support. IEEE Trans Image Process 25(1):9–23

    Article  MathSciNet  Google Scholar 

  15. Zhou L, Yang Z, Zhou Z, Hu D (2017) Salient region detection using diffusion process on a two-layer sparse graph. IEEE Trans Image Process 26(12):5882–5894

    Article  MathSciNet  Google Scholar 

  16. Zhuge Y, Yang G, Zhang P, Lu H (2018) Boundary-guided feature aggregation network for salient object detection. IEEE Sig Process Lett 25(12):1800–1804

    Article  Google Scholar 

  17. Pan J, Sayrol E, Giro-i-Nieto X, McGuinness K, O'Connor NE (2016) Shallow and deep convolutional networks for saliency prediction, In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp 598–606

  18. Yang J, Yang M (2016) Top-down visual saliency via joint CRF and dictionary learning. IEEE Trans Pattern Anal Mach Intell 39(3):576–588

    Article  Google Scholar 

  19. Peng H, Li B, Xiong W, Hu W, Ji R (2014) RGBD salient object detection: A benchmark and algorithms, In European Conference on Computer Vision, pp 92–109

  20. Guo J, Ren T, Bei J (2016) Salient object detection for RGB-D image via saliency evolution, In IEEE International Conference on Multimedia and Expo, pp 1–6

  21. Cong R, Lei J, Fu H, Hou J, Huang Q, Kwong S (2020) Going from RGB to RGBD saliency: a depth-guided transformation model. IEEE Trans Cybern 50(8):3627–3639

    Article  Google Scholar 

  22. Yang S, Luo B, Li C, Wang G, Tang J (2018) Fast grayscale-thermal foreground detection with collaborative low-rank decomposition. IEEE Trans Circuits Syst Video Technol 28(10):2574–2585

    Article  Google Scholar 

  23. Li C, Sun X, Wang X, Zhang L, Tang J (2017) Grayscale-thermal object tracking via multitask laplacian sparse representation. IEEE Trans Syst Man Cybern -Syst 47(4):673–681

    Article  Google Scholar 

  24. Zhang Q, Huang N, Yao L, Zhang D, Shan C, Han J (2020) Rgb-t salient object detection via fusing multi-level CNN features. IEEE Trans Image Process 29:3321–3335

    Article  Google Scholar 

  25. Tu Z, Ma Y, Li Z, Li C, Xu J, Liu Y (2020) Rgbt salient object detection: A large-scale dataset and benchmark, arXiv:2007.03262v2

  26. Zhang Q, Xiao T, Huang N, Zhang D, Han J (2020) Revisiting feature fusion for rgb-t salient object detection. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2020.3014663

  27. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282

    Article  Google Scholar 

  28. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3431–3440

  29. Lang C, Liu G, Yu J, Yan S (2012) Saliency detection by multitask sparsity pursuit. IEEE Trans Image Process 21(3):1327–1338

    Article  MathSciNet  Google Scholar 

  30. Boyd S, Parikh N, Chu E (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers, Now Publishers Inc

  31. Huang F, Qi J, Lu H, Zhang L, Ruan X (2017) Salient object detection via multiple instance learning. IEEE Trans Image Process 26(4):1911–1922

    Article  MathSciNet  Google Scholar 

  32. Yuan Y, Li C, Kim J, Cai W, Feng DD (2017) Reversion correction and regularized random walk ranking for saliency detection. IEEE Trans Image Process 27(3):1311–1322

  33. Tang J, Fan D, Wang X, Tu Z, Li C (2019) RGBT salient object detection: benchmark and a novel cooperative ranking approach,” IEEE Trans Circuits Syst Video Technol 30(12):4421–4433

  34. Liu JJ, Hou Q, Cheng MM, Feng J, Jiang J (2019) A simple pooling-based design for real-time salient object detection,In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3917–3926

  35. Chen Q, Liu Z, Zhang Y, Fu K, Zhao Q, Du H (2021) RGB-D salient object detection via 3D convolutional neural networks, arXiv:2101.10241v1

  36. Tu Z, Li Z, Li C, Lang Y, Tang J (2020) Multi-interactive encoder-decoder network for RGBT salient object detection, arXiv:2005.02315v1

  37. Margolin R, Zelnik-Manor L, Tal A (2014) How to evaluate foreground maps?, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 248–255

Download references

Acknowledgments

This work was supported by Independent Project of State Key Laboratory on Structural Mechanical Behavior and System Safety of Traffic Engineering (ZZ2020-37).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering and Automation, Northeastern University, Shenyang, Liaoning, China

    Aojun Gong, Liming Huang & Jiashun Shi

  2. School of Mechanical Engineering, Shijiazhuang Tiedao University, Shijiazhuang, Hebei, China

    Chuang Liu

Authors
  1. Aojun Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liming Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiashun Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chuang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liming Huang.

Ethics declarations

Conflict of Interests

The authors declare no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gong, A., Huang, L., Shi, J. et al. Unsupervised RGB-T saliency detection by node classification distance and sparse constrained graph learning. Appl Intell 52, 1030–1043 (2022). https://doi.org/10.1007/s10489-021-02434-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02434-y

Keywords

Search

Navigation