Abstract
Active learning has achieved remarkable success in minimizing labeling costs for classification tasks with all data samples drawn from known classes. However, in real scenarios, most active learning methods fail when encountering open-set annotation (OSA) problem, i.e., numerous samples from unknown classes. The main reason for such failure comes from existing query strategies that are unavoidable to select unknown class samples. To tackle such problem and select the most informative samples, we propose a novel active learning framework named OSA-CQ, which simplifies the detection work of samples from known classes and enhances the classification performance with an effective contrastive query strategy. Specifically, OSA-CQ firstly adopts an auxiliary network to distinguish samples using confidence scores, which can dynamically select samples with the highest probability from known classes in the unlabeled set. Secondly, by comparing the predictions between auxiliary network, classification, and feature similarity, OSA-CQ designs a contrastive query strategy to select these most informative samples from unlabeled and known classes set. Experimental results on CIFAR10, CIFAR100 and Tiny-ImageNet show the proposed OSA-CQ can select samples from known classes with high information, and achieve higher classification performance with lower annotation cost than state-of-the-art active learning algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Yao, L., Miller, J.: Tiny ImageNet classification with convolutional neural networks (2015)
Settles, B.: Active learning literature survey (2009)
Wang, D., Shang, Y.: A new active labeling method for deep learning. In: 2014 International Joint Conference on Neural Networks (IJCNN), pp. 112–119. IEEE, Beijing, China (2014). https://doi.org/10.1109/IJCNN.2014.6889457
Roth, D., Small, K.: Margin-based active learning for structured output spaces. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS (LNAI), vol. 4212, pp. 413–424. Springer, Heidelberg (2006). https://doi.org/10.1007/11871842_40
Gal, Y., Islam, R., Ghahramani, Z.: Deep Bayesian active learning with image data. In: International Conference on Machine Learning, pp. 1183–1192. PMLR (2017)
Wan, F., Yuan, T., Fu, M., Ji, X., Huang, Q., Ye, Q.: Nearest neighbor classifier embedded network for active learning. In: AAAI, vol. 35, pp. 10041–10048 (2021). https://doi.org/10.1609/aaai.v35i11.17205
Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach (2018). http://arxiv.org/abs/1708.00489
Parvaneh, A., Abbasnejad, E., Teney, D., Haffari, R., Van Den Hengel, A., Shi, J.Q.: Active learning by feature mixing. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12227–12236. IEEE, New Orleans, LA, USA (2022). https://doi.org/10.1109/CVPR52688.2022.01192
Yoo, D., Kweon, I.S.: Learning loss for active learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 93–102. IEEE, Long Beach, CA, USA (2019). https://doi.org/10.1109/CVPR.2019.00018
Vodrahalli, K., Li, K., Malik, J.: Are all training examples created equal? An empirical study. http://arxiv.org/abs/1811.12569 (2018)
Zhu, J.-J., Bento, J.: Generative adversarial active learning. http://arxiv.org/abs/1702.07956 (2017)
Sinha, S., Ebrahimi, S., Darrell, T.: Variational adversarial active learning. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5971–5980. IEEE, Seoul, Korea (South) (2019). https://doi.org/10.1109/ICCV.2019.00607
Yuan, T., et al.: Multiple instance active learning for object detection. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5326–5335. IEEE, Nashville, TN, USA (2021). https://doi.org/10.1109/CVPR46437.2021.00529
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. http://arxiv.org/abs/1609.02907 (2017)
Caramalau, R., Bhattarai, B., Kim, T.-K.: Sequential graph convolutional network for active learning. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9578–9587. IEEE, Nashville, TN, USA (2021). https://doi.org/10.1109/CVPR46437.2021.00946
Mnih, V., et al.: Playing Atari with deep reinforcement learning. http://arxiv.org/abs/1312.5602, (2013)
Haussmann, M., Hamprecht, F.A., Kandemir, M.: Deep active learning with adaptive acquisition. http://arxiv.org/abs/1906.11471 (2019)
Du, P., Zhao, S., Chen, H., Chai, S., Chen, H., Li, C.: Contrastive coding for active learning under class distribution mismatch. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8907–8916. IEEE, Montreal, QC, Canada (2021). https://doi.org/10.1109/ICCV48922.2021.00880
Ning, K.-P., Zhao, X., Li, Y., Huang, S.-J.: Active learning for open-set annotation. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 41–49. IEEE, New Orleans, LA, USA (2022). https://doi.org/10.1109/CVPR52688.2022.00014
Coleman, C., et al.: Similarity search for efficient active learning and search of rare concepts. In: AAAI, vol. 36, pp. 6402–6410 (2022). https://doi.org/10.1609/aaai.v36i6.20591
Geng, C., Huang, S.-J., Chen, S.: Recent advances in open set recognition: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3614–3631 (2021). https://doi.org/10.1109/TPAMI.2020.2981604
Bendale, A., Boult, T.E.: Towards open set deep networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1563–1572. IEEE, Las Vegas, NV, USA (2016). https://doi.org/10.1109/CVPR.2016.173
Dhamija, A.R., Günther, M., Boult, T.: Reducing network agnostophobia (2018)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. http://arxiv.org/abs/1503.02531 (2015)
Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020. OpenReview.net (2020). https://openreview.net/forum?id=ryghZJBKPS
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Las Vegas, NV, USA (2016). https://doi.org/10.1109/CVPR.2016.90
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Han, P., Chen, Z., Jiang, F., Si, J. (2024). Active Learning for Open-Set Annotation Using Contrastive Query Strategy. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14452. Springer, Singapore. https://doi.org/10.1007/978-981-99-8076-5_2
Download citation
DOI: https://doi.org/10.1007/978-981-99-8076-5_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8075-8
Online ISBN: 978-981-99-8076-5
eBook Packages: Computer ScienceComputer Science (R0)