Abstract
Open-Set Recognition (OSR) is a problem with mainly practical applications. However, recent evaluations have largely focused on small-scale data and tuning thresholds over the test set, which disregard the real-world operational needs of parameter selection. Thus, we revisit the original goals of OSR and propose a new evaluation metric, Operational Open-Set Accuracy (OOSA), which requires predicting an operationally relevant threshold from a validation set with known and a surrogate set with unknown samples, and then applying this threshold during testing. With this new measure in mind, we develop a large-scale evaluation protocol suited for operational scenarios. Additionally, we introduce the novel PostMax algorithm that performs post-processing refinement of the logit of the maximal class. This refinement involves normalizing logits by deep feature magnitudes and utilizing an extreme-value-based generalized Pareto distribution to map them into proper probabilities. We evaluate multiple pre-trained deep networks, including leading transformer and convolution-based architectures, on different selections of large-scale surrogate and test sets. Our experiments demonstrate that PostMax advances the state of the art in open-set recognition, showing statistically significant improvements in our novel OOSA metric as well as in previously used metrics such as AUROC, FPR95, and others.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bendale, A., Boult, T.E.: Towards open set deep networks. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2016)
Bitterwolf, J., Mueller, M., Hein, M.: In or out? Fixing ImageNet out-of-distribution detection evaluation. In: ICLR Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models (2023)
Bodesheim, P., Freytag, A., Rodner, E., Denzler, J.: Local novelty detection in multi-class recognition problems. In: Winter Conference on Applications of Computer Vision (WACV) (2015)
Boult, T.E., Cruz, S., Dhamija, A.R., Günther, M., Henrydoss, J., Scheirer, W.J.: Learning and the unknown: surveying steps toward open world recognition. In: AAAI Conference on Artificial Intelligence, vol. 33, pp. 9801–9807 (2019)
Cevikalp, H., Uzun, B., Salk, Y., Saribas, H., Köpüklü, O.: From anomaly detection to open set recognition: bridging the gap. Pattern Recogn. 138, 109385 (2023)
Chen, G., Peng, P., Wang, X., Tian, Y.: Adversarial reciprocal points learning for open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 44(11), 8065–8081 (2021)
Chen, G., et al.: Learning open set network with discriminative reciprocal points. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12348, pp. 507–522. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58580-8_30
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2014)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255. IEEE (2009)
Dhamija, A.R., Günther, M., Boult, T.: Reducing network agnostophobia. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 31 (2018)
Ge, Z., Demyanov, S., Garnavi, R.: Generative OpenMax for multi-class open set classification. In: British Machine Vision Conference (BMVC) (2017)
Gumbel, E.J.: Statistical Theory of Extreme Values and Some Practical Applications: A Series of Lectures, vol. 33. US Government Printing Office (1954)
Günther, M., Dhamija, A.R., Boult, T.E.: Watchlist adaptation: protecting the innocent. In: International Conference of the Biometrics Special Interest Group (BIOSIG) (2020)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000–16009 (2022)
Hendrycks, D., et al.: Scaling out-of-distribution detection for real-world settings. In: International Conference on Machine Learning (ICML). PMLR (2022)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Learning Representations (ICLR) (2017)
van Horn, G., et al.: The INaturalist species classification and detection dataset. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Huang, R., Geng, A., Li, Y.: On the importance of gradients for detecting distributional shifts in the wild. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 677–689 (2021)
Huang, R., Li, Y.: MOS: towards scaling out-of-distribution detection for large semantic space. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8710–8719 (2021)
Kong, S., Ramanan, D.: Opengan: open-set recognition via open data generation. In: International Conference on Computer Vision (ICCV), pp. 813–822 (2021)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Liu, X., Lochman, Y., Zach, C.: GEN: pushing the limits of softmax-based out-of-distribution detection. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23946–23955 (2023)
Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11976–11986 (2022)
Lu, J., Xu, Y., Li, H., Cheng, Z., Niu, Y.: PMAL: open set recognition via robust prototype mining. In: AAAI Conference on Artificial Intelligence, vol. 36:2, pp. 1872–1880 (2022)
Lyu, Z., Gutierrez, N.B., Beksi, W.J.: Metamax: improved open-set deep neural networks via weibull calibration. In: Winter Conference on Applications of Computer Vision Workshops (WACVW) (2023)
Moon, W., Park, J., Seong, H.S., Cho, C.H., Heo, J.P.: Difficulty-aware simulator for open set recognition. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13685, pp. 365–381. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19806-9_21
Neal, L., Olson, M., Fern, X., Wong, W.K., Li, F.: Open set learning with counterfactual images. In: European Conference on Computer Vision (ECCV) (2018)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)
Palechor, A., Bhoumik, A., Günther, M.: Large-scale open-set classification protocols for imagenet. In: Winter Conference on Applications of Computer Vision (WACV), pp. 42–51. CVF/IEEE (2023)
Park, J., Chai, J.C.L., Yoon, J., Teoh, A.B.J.: Understanding the feature norm for out-of-distribution detection. In: International Conference on Computer Vision (ICCV), pp. 1557–1567 (2023)
Park, J., Jung, Y.G., Teoh, A.B.J.: Nearest neighbor guidance for out-of-distribution detection. In: International Conference on Computer Vision (ICCV), pp. 1686–1695 (2023)
Pickands, J., III.: Statistical inference using extreme order statistics. Ann. Stat. 3(1), 119–131 (1975)
Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do Imagenet classifiers generalize to Imagenet? In: International Conference on Machine Learning (ICML), pp. 5389–5400. PMLR (2019)
Rudd, E.M., Jain, L.P., Scheirer, W.J., Boult, T.E.: The extreme value machine. Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(3), 762–768 (2017)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision (IJCV) 115(3), 211–252 (2015)
Ryali, C., et al.: Hiera: a hierarchical vision transformer without the bells-and-whistles. In: International Conference on Machine Learning (ICML) (2023)
Scheirer, W.J., Jain, L.P., Boult, T.E.: Probability models for open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(11), 2317–2324 (2014)
Scheirer, W.J., de Rezende Rocha, A., Sapkota, A., Boult, T.E.: Towards open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 35(7) (2013)
Shu, L., Xu, H., Liu, B.: DOC: deep open classification of text documents. In: Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics (2017)
Sun, Y., Guo, C., Li, Y.: React: out-of-distribution detection with rectified activations. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 144–157 (2021)
Vareto, R.H., Linghu, Y., Boult, T.E., Schwartz, W.R., Günther, M.: Open-set face recognition with maximal entropy and Objectosphere loss. Image Vis. Comput. (IMAVIS) 141 (2024)
Vareto, R.H., Schwartz, W.R., Günther, M.: Toward open-set face recognition with neural ensemble, maximal entropy loss and feature augmentation. In: Conference on Graphics, Patterns and Images (SIBGRAPI) (2023)
Vaze, S., Han, K., Vedaldi, A., Zissermann, A.: Open-set recognition: a good closed-set classifier is all you need? In: International Conference on Learning Representations (ICLR) (2022)
Wang, H., Li, Z., Feng, L., Zhang, W.: Vim: out-of-distribution with virtual-logit matching. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4921–4930 (2022)
Wang, Z., Xu, Q., Yang, Z., He, Y., Cao, X., Huang, Q.: OpenAUC: towards AUC-oriented open-set recognition. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 25033–25045 (2022)
Wilson, S., Fischer, T., Dayoub, F., Miller, D., Sünderhauf, N.: Safe: sensitivity-aware features for out-of-distribution object detection. In: International Conference on Computer Vision (ICCV), pp. 23565–23576 (2023)
Woo, S., et al.: Convnext v2: co-designing and scaling convnets with masked autoencoders. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16133–16142 (2023)
Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485–3492. IEEE (2010)
Xu, K., Chen, R., Franchi, G., Yao, A.: Scaling for training time and post-hoc out-of-distribution detection enhancement. In: International Conference on Learning Representations (ICLR) (2024)
Yang, H.M., Zhang, X.Y., Yin, F., Yang, Q., Liu, C.L.: Convolutional prototype network for open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 44(5), 2358–2370 (2020)
Yang, J., et al.: OpenOOD: benchmarking generalized out-of-distribution detection. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 32598–32611 (2022)
Yoshihashi, R., Shao, W., Kawakami, R., You, S., Iida, M., Naemura, T.: Classification-reconstruction learning for open-set recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Zhang, J., et al.: OpenOOD v1.5: enhanced benchmark for out-of-distribution detection. In: NeurIPS Workshop on Distribution Shifts: New Frontiers with Foundation Models (2023)
Zhang, X., Cheng, X., Zhang, D., Bonnington, P., Ge, Z.: Learning network architecture for open-set recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 3, pp. 3362–3370 (2022)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(6), 1452–1464 (2017)
Zhou, D.W., Ye, H.J., Zhan, D.C.: Learning placeholders for open-set recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cruz, S., Rabinowitz, R., Günther, M., Boult, T.E. (2025). Operational Open-Set Recognition and PostMax Refinement. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15064. Springer, Cham. https://doi.org/10.1007/978-3-031-72658-3_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-72658-3_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72657-6
Online ISBN: 978-3-031-72658-3
eBook Packages: Computer ScienceComputer Science (R0)