Skip to main content
Springer Nature Link
Log in

Operational Open-Set Recognition and PostMax Refinement

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15064))

Included in the following conference series:

  • 323 Accesses

Abstract

Open-Set Recognition (OSR) is a problem with mainly practical applications. However, recent evaluations have largely focused on small-scale data and tuning thresholds over the test set, which disregard the real-world operational needs of parameter selection. Thus, we revisit the original goals of OSR and propose a new evaluation metric, Operational Open-Set Accuracy (OOSA), which requires predicting an operationally relevant threshold from a validation set with known and a surrogate set with unknown samples, and then applying this threshold during testing. With this new measure in mind, we develop a large-scale evaluation protocol suited for operational scenarios. Additionally, we introduce the novel PostMax algorithm that performs post-processing refinement of the logit of the maximal class. This refinement involves normalizing logits by deep feature magnitudes and utilizing an extreme-value-based generalized Pareto distribution to map them into proper probabilities. We evaluate multiple pre-trained deep networks, including leading transformer and convolution-based architectures, on different selections of large-scale surrogate and test sets. Our experiments demonstrate that PostMax advances the state of the art in open-set recognition, showing statistically significant improvements in our novel OOSA metric as well as in previously used metrics such as AUROC, FPR95, and others.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/Vastlab/PostMax-OOSA.

  2. 2.

    https://zjysteven.github.io/OpenOOD.

References

  1. Bendale, A., Boult, T.E.: Towards open set deep networks. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2016)

    Google Scholar 

  2. Bitterwolf, J., Mueller, M., Hein, M.: In or out? Fixing ImageNet out-of-distribution detection evaluation. In: ICLR Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models (2023)

    Google Scholar 

  3. Bodesheim, P., Freytag, A., Rodner, E., Denzler, J.: Local novelty detection in multi-class recognition problems. In: Winter Conference on Applications of Computer Vision (WACV) (2015)

    Google Scholar 

  4. Boult, T.E., Cruz, S., Dhamija, A.R., Günther, M., Henrydoss, J., Scheirer, W.J.: Learning and the unknown: surveying steps toward open world recognition. In: AAAI Conference on Artificial Intelligence, vol. 33, pp. 9801–9807 (2019)

    Google Scholar 

  5. Cevikalp, H., Uzun, B., Salk, Y., Saribas, H., Köpüklü, O.: From anomaly detection to open set recognition: bridging the gap. Pattern Recogn. 138, 109385 (2023)

    Article  Google Scholar 

  6. Chen, G., Peng, P., Wang, X., Tian, Y.: Adversarial reciprocal points learning for open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 44(11), 8065–8081 (2021)

    Google Scholar 

  7. Chen, G., et al.: Learning open set network with discriminative reciprocal points. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12348, pp. 507–522. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58580-8_30

    Chapter  Google Scholar 

  8. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

    Google Scholar 

  9. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255. IEEE (2009)

    Google Scholar 

  10. Dhamija, A.R., Günther, M., Boult, T.: Reducing network agnostophobia. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 31 (2018)

    Google Scholar 

  11. Ge, Z., Demyanov, S., Garnavi, R.: Generative OpenMax for multi-class open set classification. In: British Machine Vision Conference (BMVC) (2017)

    Google Scholar 

  12. Gumbel, E.J.: Statistical Theory of Extreme Values and Some Practical Applications: A Series of Lectures, vol. 33. US Government Printing Office (1954)

    Google Scholar 

  13. Günther, M., Dhamija, A.R., Boult, T.E.: Watchlist adaptation: protecting the innocent. In: International Conference of the Biometrics Special Interest Group (BIOSIG) (2020)

    Google Scholar 

  14. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000–16009 (2022)

    Google Scholar 

  15. Hendrycks, D., et al.: Scaling out-of-distribution detection for real-world settings. In: International Conference on Machine Learning (ICML). PMLR (2022)

    Google Scholar 

  16. Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Learning Representations (ICLR) (2017)

    Google Scholar 

  17. van Horn, G., et al.: The INaturalist species classification and detection dataset. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  18. Huang, R., Geng, A., Li, Y.: On the importance of gradients for detecting distributional shifts in the wild. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 677–689 (2021)

    Google Scholar 

  19. Huang, R., Li, Y.: MOS: towards scaling out-of-distribution detection for large semantic space. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8710–8719 (2021)

    Google Scholar 

  20. Kong, S., Ramanan, D.: Opengan: open-set recognition via open data generation. In: International Conference on Computer Vision (ICCV), pp. 813–822 (2021)

    Google Scholar 

  21. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)

    Google Scholar 

  22. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  23. Liu, X., Lochman, Y., Zach, C.: GEN: pushing the limits of softmax-based out-of-distribution detection. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23946–23955 (2023)

    Google Scholar 

  24. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11976–11986 (2022)

    Google Scholar 

  25. Lu, J., Xu, Y., Li, H., Cheng, Z., Niu, Y.: PMAL: open set recognition via robust prototype mining. In: AAAI Conference on Artificial Intelligence, vol. 36:2, pp. 1872–1880 (2022)

    Google Scholar 

  26. Lyu, Z., Gutierrez, N.B., Beksi, W.J.: Metamax: improved open-set deep neural networks via weibull calibration. In: Winter Conference on Applications of Computer Vision Workshops (WACVW) (2023)

    Google Scholar 

  27. Moon, W., Park, J., Seong, H.S., Cho, C.H., Heo, J.P.: Difficulty-aware simulator for open set recognition. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13685, pp. 365–381. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19806-9_21

    Chapter  Google Scholar 

  28. Neal, L., Olson, M., Fern, X., Wong, W.K., Li, F.: Open set learning with counterfactual images. In: European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

  29. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)

    Google Scholar 

  30. Palechor, A., Bhoumik, A., Günther, M.: Large-scale open-set classification protocols for imagenet. In: Winter Conference on Applications of Computer Vision (WACV), pp. 42–51. CVF/IEEE (2023)

    Google Scholar 

  31. Park, J., Chai, J.C.L., Yoon, J., Teoh, A.B.J.: Understanding the feature norm for out-of-distribution detection. In: International Conference on Computer Vision (ICCV), pp. 1557–1567 (2023)

    Google Scholar 

  32. Park, J., Jung, Y.G., Teoh, A.B.J.: Nearest neighbor guidance for out-of-distribution detection. In: International Conference on Computer Vision (ICCV), pp. 1686–1695 (2023)

    Google Scholar 

  33. Pickands, J., III.: Statistical inference using extreme order statistics. Ann. Stat. 3(1), 119–131 (1975)

    MathSciNet  Google Scholar 

  34. Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do Imagenet classifiers generalize to Imagenet? In: International Conference on Machine Learning (ICML), pp. 5389–5400. PMLR (2019)

    Google Scholar 

  35. Rudd, E.M., Jain, L.P., Scheirer, W.J., Boult, T.E.: The extreme value machine. Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(3), 762–768 (2017)

    Article  Google Scholar 

  36. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision (IJCV) 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  37. Ryali, C., et al.: Hiera: a hierarchical vision transformer without the bells-and-whistles. In: International Conference on Machine Learning (ICML) (2023)

    Google Scholar 

  38. Scheirer, W.J., Jain, L.P., Boult, T.E.: Probability models for open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(11), 2317–2324 (2014)

    Article  Google Scholar 

  39. Scheirer, W.J., de Rezende Rocha, A., Sapkota, A., Boult, T.E.: Towards open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 35(7) (2013)

    Google Scholar 

  40. Shu, L., Xu, H., Liu, B.: DOC: deep open classification of text documents. In: Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics (2017)

    Google Scholar 

  41. Sun, Y., Guo, C., Li, Y.: React: out-of-distribution detection with rectified activations. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 144–157 (2021)

    Google Scholar 

  42. Vareto, R.H., Linghu, Y., Boult, T.E., Schwartz, W.R., Günther, M.: Open-set face recognition with maximal entropy and Objectosphere loss. Image Vis. Comput. (IMAVIS) 141 (2024)

    Google Scholar 

  43. Vareto, R.H., Schwartz, W.R., Günther, M.: Toward open-set face recognition with neural ensemble, maximal entropy loss and feature augmentation. In: Conference on Graphics, Patterns and Images (SIBGRAPI) (2023)

    Google Scholar 

  44. Vaze, S., Han, K., Vedaldi, A., Zissermann, A.: Open-set recognition: a good closed-set classifier is all you need? In: International Conference on Learning Representations (ICLR) (2022)

    Google Scholar 

  45. Wang, H., Li, Z., Feng, L., Zhang, W.: Vim: out-of-distribution with virtual-logit matching. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4921–4930 (2022)

    Google Scholar 

  46. Wang, Z., Xu, Q., Yang, Z., He, Y., Cao, X., Huang, Q.: OpenAUC: towards AUC-oriented open-set recognition. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 25033–25045 (2022)

    Google Scholar 

  47. Wilson, S., Fischer, T., Dayoub, F., Miller, D., Sünderhauf, N.: Safe: sensitivity-aware features for out-of-distribution object detection. In: International Conference on Computer Vision (ICCV), pp. 23565–23576 (2023)

    Google Scholar 

  48. Woo, S., et al.: Convnext v2: co-designing and scaling convnets with masked autoencoders. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16133–16142 (2023)

    Google Scholar 

  49. Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485–3492. IEEE (2010)

    Google Scholar 

  50. Xu, K., Chen, R., Franchi, G., Yao, A.: Scaling for training time and post-hoc out-of-distribution detection enhancement. In: International Conference on Learning Representations (ICLR) (2024)

    Google Scholar 

  51. Yang, H.M., Zhang, X.Y., Yin, F., Yang, Q., Liu, C.L.: Convolutional prototype network for open set recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 44(5), 2358–2370 (2020)

    Google Scholar 

  52. Yang, J., et al.: OpenOOD: benchmarking generalized out-of-distribution detection. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 32598–32611 (2022)

    Google Scholar 

  53. Yoshihashi, R., Shao, W., Kawakami, R., You, S., Iida, M., Naemura, T.: Classification-reconstruction learning for open-set recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  54. Zhang, J., et al.: OpenOOD v1.5: enhanced benchmark for out-of-distribution detection. In: NeurIPS Workshop on Distribution Shifts: New Frontiers with Foundation Models (2023)

    Google Scholar 

  55. Zhang, X., Cheng, X., Zhang, D., Bonnington, P., Ge, Z.: Learning network architecture for open-set recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 3, pp. 3362–3370 (2022)

    Google Scholar 

  56. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(6), 1452–1464 (2017)

    Article  Google Scholar 

  57. Zhou, D.W., Ye, H.J., Zhan, D.C.: Learning placeholders for open-set recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Notre Dame, Notre Dame, USA

    Steve Cruz

  2. University of Colorado Colorado Springs, Colorado Springs, USA

    Ryan Rabinowitz & Terrance E. Boult

  3. University of Zurich, Zürich, Switzerland

    Manuel Günther

Authors
  1. Steve Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryan Rabinowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manuel Günther
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Terrance E. Boult
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steve Cruz .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3951 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cruz, S., Rabinowitz, R., Günther, M., Boult, T.E. (2025). Operational Open-Set Recognition and PostMax Refinement. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15064. Springer, Cham. https://doi.org/10.1007/978-3-031-72658-3_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72658-3_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72657-6

  • Online ISBN: 978-3-031-72658-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation