Skip to main content
SpringerLink
Log in

Over-sampling strategy-based class-imbalanced salient object detection and its application in underwater scene

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

One major branch of bottom-up salient object detection methods is machine learning-based methods which learn to classify salient object(positive) and background(negative) based on various learning algorithms. Generally, each input image corresponds to a training set which is determined by prior knowledge. However, we find that the class-imbalance problem (i.e., positive and negative data are seriously imbalanced in quantity) is inevitable for these methods due to the existence of various factors. Imbalanced training set might fail to make learning process succeed. To solve above problem, we propose a novel bottom-up Salient object detection algorithm based on Class-Imbalance Learning(CILS). For the input image, our goal is to ensure that a robust saliency classifier is well learned even if class-imbalance problem occurs in training set. To this end, we propose a novel over-sampling strategy which concentrates on constructing synthetic samples to make two classes be balanced. As a result, the balanced training data could be better applied to subsequent learning and classification process. Finally, the saliency map obtained by CILS is further refined by a novel optimization framework via foreground consistency and background consistency. Our proposed algorithm and other state-of-the-art algorithms are tested on three datasets. Results demonstrate adequately that our model produces better performance than other compared algorithms. Furthermore, we apply our salient object detection algorithm to underwater object recognition task and recognition accuracy could be further improved by introducing saliency cues.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Gong, X., Liu, X., Li, Y., Li, H.: A novel co-attention computation block for deep learning based image co-segmentation. Image Vis. Comput. 101, 103973 (2020)

    Article  Google Scholar 

  2. Wei, G., Wei, Z., Huang, L., Nie, J., Li, X.: Center-aligned domain adaptation network for image classification. Expert Syst. Appl. 168, 114381 (2021)

    Article  Google Scholar 

  3. Song, G., Song, K., Yan, Y.: Saliency detection for strip steel surface detects using multiple constraints and improved texture features. Opt. Lasers Eng. 128, 106000 (2020)

    Article  Google Scholar 

  4. Zhao, X., Pang, Y., Zhang, L., Lu, H., Zhang, L.: Suppress and balance: A simple gated network for salient object detection, In: Proc. Euro Conf. Comput. Vis., pp. 35-51 (2015)

  5. He, S., Lau, R., Liu, W.: SuperCNN: A superpixel wise convolutional neural netowrk for salient object detection. Int. J. Comput. Vis. 115(3), 330–334 (2015)

    Article  MathSciNet  Google Scholar 

  6. Zeng, Y., Zhuge, Y., Lu, H., Zhang, L., Qian, M., Yu, Y.: Multi-source weak supervision for saliency detection, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 6067-6076 (2019)

  7. Qian, M., Qi, J., Zhang, L., Feng, M., Lu, H.: Language-aware weak supervision for salient object detection. Pattern Recognit. 96, 106955 (2018)

    Article  Google Scholar 

  8. Tong, N., Lu, H., Ruan, X., Yang, M.: Salient object detection via bootstrap learning, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1884-1892 (2015)

  9. Zhang, M., Wu, Y., Du, Y., Fang, L., Pang, Y.: Saliency detection integrating global and local information. J. Vis. Commun. Image Rrepresent. 53, 215–223 (2018)

    Article  Google Scholar 

  10. Gong, C., Tao, D., Liu, W., Maybank, S.J., Fang, M.: Saliency propagation from simple to difficult, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 2531-2539 (2015)

  11. Sun, J., Lu, H., Liu, X.: Saliency region detection based on Markov absorption probabilities. IEEE Trans. Image Process. 24(5), 1639–1649 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  12. Qin, Y., Feng, M., Lu, H., Cottrell, G.W.: Hierarchical cellular automata for visual saliency. Int. J. Comput. Vis. 126(7), 752–770 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  13. Achanta, R., Hemami, S., Estrada, F.: Frequency-tuned salient region detection, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1597-1604 (2009)

  14. Zhou, Y., Mao, A., Huo, S., Lei, J., Kung, S.: Salient object detection via fuzzy theory and object-level enhancement. IEEE Trans. Multimedia. 21(1), 74–85 (2019)

    Article  Google Scholar 

  15. Zeng, Y., Feng, M., Lu, H., Yang, G., Borji, A.: An unsupervised game-theoretic approach to saliency detection. IEEE Trans. Image Process. 27(9), 4545–4554 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  16. Chawla, N.V., Japkowicz, N., Kolcz, A.: Editorial: Special issue on learning from imbalanced data sets, ACM SIGKDD Explorat

  17. Liu, X., Wu, J., Zhou, Z.: Exploratory undersampling for class-imbalanced learning. IEEE Trans. Sys man, Cyb, Part-B(Cyb). 39(2), 539–550 (2009)

    Article  Google Scholar 

  18. Chawla, N., Bowyer, K., Hall, L., Kegelmeyer, W.: SMOTE: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    Article  MATH  Google Scholar 

  19. Zhang, M., Li, J., Ji, W., Piao, Y., Lu, H.: Memory-oriented decoder for light field salient object detection, In: Proc. Adv. Neural Inf. Process. Syst. (2019)

  20. Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.: Learning to detect salient objects with image-level supervision, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 3796-3805 (2017)

  21. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.: Saliency detection via graph-based manifold ranking, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 3166-3173 (2013)

  22. Lu, H., Li, X., Zhang, L., Ruan, X., Yang, M.: Dense and sparse reconstruction error based saliency descriptor. IEEE Trans. Image Process. 25(4), 1592–1603 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  23. Lin, X., Wang, Z., Ma, L., Wu, X.: Saliency detection via multi-scale global cues. IEEE Trans. Multimedia. 21(7), 1646–1659 (2019)

    Article  Google Scholar 

  24. He, H., Bai, Y., Garcia, E., Li, S.: ADASYN: Adaptive synthetic sampling approach for imbalanced learning. Int. J. Conf. Neur Netw. (2008)

  25. Achanta, R., shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)

    Article  Google Scholar 

  26. Tong, N., Lu, H., Zhang, Y., Ruan, X.: Salient object detection via global and local cues. Pattern Recognit. 48(10), 3258–3267 (2015)

    Article  Google Scholar 

  27. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, In: Proc. Int. Conf. Learn. Representat., pp. 1-14 (2015)

  28. Hwang, I., Lee, S.H., Park, J.S., Cho, N.I.: Saliency detection based on seed propagation in a multilayer graph. Multimed. Tools Appl. 76(2), 2111–2129 (2017)

    Article  Google Scholar 

  29. Qin, Y., Lu, H., Xu, Y., Wang, H.: Saliency detection via cellular automata, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 110-119 (2015)

  30. Fang, S., Li, J., Tian, Y., Huang, T., Chen, X.: Learning discriminative subspaces on random contrasts for image saliency analysis. IEEE Trans. Neural Netw. Learn. Sys. 28(5), 1095–1108 (2017)

    Article  Google Scholar 

  31. Tu, W.C., He, S., Yang, Q., Chien, S.Y.: Real-time salient object detection with a minimum spanning tree, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 2334-2342 (2016)

  32. Li, H., Lu, H., Lin, Z., Shen, X., Price, B.: Inner and inter label propagation: salient object detection in the wild. IEEE Trans. Image Process. 24(10), 3167–3186 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  33. Huang, F., Jinqing, Q., Lu, H., Zhang, L., Ruan, X.: Salient object detection via multiple instance learning. J. Vis. 8(7), 1911–1922 (2017)

    MathSciNet  MATH  Google Scholar 

  34. Peng, H., Li, B., Ling, H., Hu, W., Xiong, W.: Salient object detection via structured matrix decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 818–832 (2017)

    Article  Google Scholar 

  35. Liu, G., Yang, J.: Exploiting color volume and color difference for salient region detection. IEEE Trans. Image Process. 28(1), 6–16 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  36. Zhang, L., Ai, J., Jiang, B., Lu, H., Li, X.: Saliency detection via absorbing markov chain with learnt transition probability. IEEE Trans. Image Process. 27(2), 987–998 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  37. Zhang, L., Sun, J., Wang, T., Min, Y., Lu, H.: Visual saliency detection via kernelized subspace ranking with active learning. IEEE Trans. Image Process. 29, 2258–2270 (2020)

    Article  MATH  Google Scholar 

  38. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1155-1162 (2013)

  39. Movahedi, V., Elder, J.H.: Design and perceptual validation of performance measures for salient object segmentation, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshop., pp. 49-56 (2010)

  40. Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2189–2202 (2012)

    Article  Google Scholar 

  41. Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. 81(1), 24–52 (2009)

    Article  Google Scholar 

  42. Zhang, M., Pang, Y., Wu, Y., Du, Y., Sun, H., Zhang, K.: Saliency detection via local structure propagation. J. Vis. Commun. Image Rrepresent 52, 131–142 (2018)

    Article  Google Scholar 

  43. Deng, C., Yang, X., Nie, F., Tao, D.: Saliency detection via a multiple self-weighted graph-based manifold ranking. IEEE Trans. multimedia. 22(4), 885–896 (2020)

    Article  Google Scholar 

  44. Tu, Z., Xia, T., Li, C., Wang, X., Ma, Y., Tang, J.: RGB-T image saliency detection via collaborative graph learning. IEEE Trans. multimedia. 22(1), 160–173 (2020)

    Article  Google Scholar 

  45. Pang, J., Chen, K., Shi, J., Feng, H., Quyang, W., Lin, D.: Libra R-CNN: Towards balanced learning for object detection, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 821-830 (2019)

  46. Li, Y., Wang, T., Kang, B., Tang, S., Wang, C., Li, J., Feng, J.: Overcoming classifier imbalance for long-tail object detection with balanced group softmax, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (2020)

  47. Lin, T., Goyal, P., Doallr, P.: Focal loss for dense object detection, in Proc. IEEE Conf. Comput. Vis., pp. 2999-3007 (2017)

  48. Cao, K., Wei, C., Gaidon, A.: Learning imbalanced datasets with label-distribution-aware margin loss, In: Proc. Adv. Neural Inf. Process. Syst., vol.32 (2019)

  49. Liu, R., Fan, X., Zhu, M., Hou, M., Luo, Z.: Real-world underwater enhancement: Challenges, benchmarks, and solutions under natural light. IEEE Trans. Circuits Syst. Video Technol. 30(12), 4861–4875 (2020)

    Article  Google Scholar 

  50. Geng, X.: Label Distribution Learning. IEEE Trans. Knowl. Data Eng. 28(7), 1734–1748 (2016)

    Article  Google Scholar 

  51. Guo, T., Xu, X.: Salient object detection from low contrast images based on local contrast enhancing and non-local feature learning. Vis Comput. 37, 2069–2081 (2021)

    Article  Google Scholar 

  52. Wang, L., Lu, H., Xiang, R., Yang, M-H.: Deep networks for saliency detection via local estimation and global search, In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 3183-3192 (2015)

  53. Li, H., Wang, N., Ding, X., Yang, X., Gao, X.: Adaptively Learning Facial Expression Representation via C-F Labels and Distillation. IEEE Trans. Image Process. 30, 2016–2028 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant nos. U20A20197, 61973063, Liaoning Key Research and Development Project 2020JH2/10100040, Natural Science Foundation of Liaoning Province 2021-KF-12-01 and the Foundation of National Key Laboratory OEIP-O-202005.

Author information

Authors and Affiliations

  1. Faculty of Robot Science and Engineering, Northeastern University, Shenyang, LiaoNing, 110169, China

    Yu Pang, Chengdong Wu & Xiaosheng Yu

  2. Engineering Faculty, University of Sydney, NSW, 2006, Australia

    Hao Wu

Authors
  1. Yu Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chengdong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaosheng Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Pang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pang, Y., Wu, C., Wu, H. et al. Over-sampling strategy-based class-imbalanced salient object detection and its application in underwater scene. Vis Comput 39, 1959–1974 (2023). https://doi.org/10.1007/s00371-022-02458-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02458-6

Keywords

Search

Navigation