Abstract
Active learning (AL) in open set scenarios presents a novel challenge of identifying the most valuable examples in an unlabeled data pool that comprises data from both known and unknown classes. Traditional methods prioritize selecting informative examples with low confidence, with the risk of mistakenly selecting unknown-class examples with similarly low confidence. Recent methods favor the most probable known-class examples, with the risk of picking simple already mastered examples. In this paper, we attempt to query examples that are both likely from known classes and highly informative, and propose a Bidirectional Uncertainty-based Active Learning (BUAL) framework. Specifically, we achieve this by first pushing the unknown class examples toward regions with high-confidence predictions, i.e., the proposed Random Label Negative Learning method. Then, we propose a Bidirectional Uncertainty sampling strategy by jointly estimating uncertainty posed by both positive and negative learning to perform consistent and stable sampling. BUAL successfully extends existing uncertainty-based AL methods to complex open-set scenarios. Extensive experiments on multiple datasets with varying openness demonstrate that BUAL achieves state-of-the-art performance. The code is available at this link.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671 (2019)
Balcan, M.-F., Broder, A., Zhang, T.: Margin based active learning. In: Bshouty, N.H., Gentile, C. (eds.) COLT 2007. LNCS (LNAI), vol. 4539, pp. 35–50. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72927-3_5
Du, P., Zhao, S., Chen, H., Chai, S., Chen, H., Li, C.: Contrastive coding for active learning under class distribution mismatch. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8927–8936 (2021)
Feng, L., Kaneko, T., Han, B., Niu, G., An, B., Sugiyama, M.: Learning with multiple complementary labels. In: International Conference on Machine Learning, pp. 3072–3081. PMLR (2020)
Fu, Y., Zhu, X., Li, B.: A survey on instance selection for active learning. Knowl. Inf. Syst. 35(2), 249–283 (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Holub, A., Perona, P., Burl, M.C.: Entropy-based active learning for object recognition. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8. IEEE (2008)
Huang, S.J., Jin, R., Zhou, Z.H.: Active learning by querying informative and representative examples. Adv. Neural Inf. Process. Syst. 23 (2010)
Huang, S.J., Zong, C.C., Ning, K.P., Ye, H.B.: Asynchronous active learning with distributed label querying. In: IJCAI, pp. 2570–2576 (2021)
Ishida, T., Niu, G., Hu, W., Sugiyama, M.: Learning from complementary labels. Adv. Neural Inf. Process. Syst. 30 (2017)
Kim, Y., Yim, J., Yun, J., Kim, J.: NLNL: negative learning for noisy labels. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 101–110 (2019)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25 (2012)
Lee, J.-H., Astrid, M., Zaheer, M.Z., Lee, S.-I.: Deep visual anomaly detection with negative learning. In: Jeong, H., Sumi, K. (eds.) IW-FCV 2021. CCIS, vol. 1405, pp. 218–232. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-81638-4_18
Li, M., Sethi, I.K.: Confidence-based active learning. IEEE Trans. Pattern Anal. Mach. Intell. 28(8), 1251–1261 (2006)
Luo, X., Chen, W., Tan, Y., Li, C., He, Y., Jia, X.: Exploiting negative learning for implicit pseudo label rectification in source-free domain adaptive semantic segmentation. arXiv preprint arXiv:2106.12123 (2021)
Mahmood, R., Fidler, S., Law, M.T.: Low budget active learning via Wasserstein distance: an integer programming approach. arXiv preprint arXiv:2106.02968 (2021)
Moon, W., Park, J., Seong, H.S., Cho, C.H., Heo, J.P.: Difficulty-aware simulator for open set recognition. arXiv preprint arXiv:2207.10024 (2022)
Ning, K.P., Tao, L., Chen, S., Huang, S.J.: Improving model robustness by adaptively correcting perturbation levels with active queries. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 9161–9169 (2021)
Ning, K.P., Zhao, X., Li, Y., Huang, S.J.: Active learning for open-set annotation. arXiv preprint arXiv:2201.06758 (2022)
Ren, P., et al.: A survey of deep active learning. ACM Comput. Surv. (CSUR) 54(9), 1–40 (2021)
Roy, N., McCallum, A.: Toward optimal active learning through sampling estimation of error reduction. In: International Conference On Machine Learning (2001)
Salehi, M., Mirzaei, H., Hendrycks, D., Li, Y., Rohban, M.H., Sabokrou, M.: A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110.14051 (2021)
Scheirer, W.J., de Rezende Rocha, A., Sapkota, A., Boult, T.E.: Toward open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1757–1772 (2012)
Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach. arXiv preprint arXiv:1708.00489 (2017)
Settles, B.: Active learning literature survey (2009)
Sinha, S., Ebrahimi, S., Darrell, T.: Variational adversarial active learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5972–5981 (2019)
Yao, L., Miller, J.: Tiny imagenet classification with convolutional neural networks. CS 231N 2(5), 8 (2015)
Yoo, D., Kweon, I.S.: Learning loss for active learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 93–102 (2019)
You, X., Wang, R., Tao, D.: Diverse expected gradient active learning for relative attributes. IEEE Trans. Image Process. 23(7), 3203–3217 (2014)
Yu, X., Liu, T., Gong, M., Tao, D.: Learning with biased complementary labels. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 69–85. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_5
Zong, C.C., et al.: Noise-robust bidirectional learning with dynamic sample reweighting. arXiv preprint arXiv:2209.01334 (2022)
Zong, C.C., Wang, Y.W., Xie, M.K., Huang, S.J.: Dirichlet-based prediction calibration for learning with noisy labels. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 17254–17262 (2024)
Acknowledgements
This work was supported by the Natural Science Foundation of Jiangsu Province of China (BK20222012, BK20211517), the National Key R&D Program of China (2020AAA0107000), and NSFC (62222605).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zong, CC., Wang, YW., Ning, KP., Ye, HB., Huang, SJ. (2025). Bidirectional Uncertainty-Based Active Learning for Open-Set Annotation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15086. Springer, Cham. https://doi.org/10.1007/978-3-031-73390-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-73390-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73389-5
Online ISBN: 978-3-031-73390-1
eBook Packages: Computer ScienceComputer Science (R0)