Abstract
Supervised contrastive learning has shown promising results in image classification tasks where the representations are pulled together if they share same labels or otherwise pushed apart. Such dispersion process in the representation space benefits the downstream classification tasks. However, when applied to regression tasks directly, such dispersion lacks guidance of the relationship among target labels (i.e. the label distances), which leads to the disalignment between representation distances and label distances. Achieving such alignment without compromising the dispersion of learned representations is challenging. In this paper, we propose a Ranking Enhanced Supervised Contrastive Loss (RESupCon) to empower the representation dispersion process with ranking alignment between representation distances and label distances in a controlled fashion. We demonstrate the effectiveness of our method in image regression tasks on four real-world datasets with various interests, including meteorological, medical and human facial data. Experimental results of our method show that representations with better ranking are learned and improvements are made over other baselines in terms of RMSE on all four datasets.
This work was supported by the National Key Research and Development Program of China (2022YFC3004102) and Qinghai Kunlun Talents Program.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Note that they are outputs of the projection head which is omitted after the contrastive training. Network before the projection head is called the encoder, whose outputs we denote by “representations”. We generally refer to both as “feature”.
References
Blondel, M., Teboul, O., Berthet, Q., Djolonga, J.: Fast differentiable sorting and ranking. In: International Conference on Machine Learning, pp. 950–959. PMLR (2020)
Chen, B., Chen, B.F., Lin, H.T.: Rotation-blended CNNs on a new open dataset for tropical cyclone image-to-intensity regression. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 90–99 (2018)
Chen, M., Fu, D.Y., Narayan, A., Zhang, M., Song, Z., Fatahalian, K., Ré, C.: Perfectly balanced: improving transfer and robustness of supervised contrastive learning. In: International Conference on Machine Learning, pp. 3090–3122. PMLR (2022)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9620–9629. IEEE (2021). https://doi.org/10.1109/ICCV48922.2021.00950
Cuturi, M., Teboul, O., Vert, J.P.: Differentiable ranking and sorting using optimal transport. In: Advances in Neural Information Processing Systems. vol. 32 (2019)
Dai, W., Li, X., Chiu, W.H.K., Kuo, M.D., Cheng, K.T.: Adaptive contrast for image regression in computer-aided disease assessment. IEEE Trans. Med. Imaging 41(5), 1255–1268 (2021)
Graf, F., Hofer, C., Niethammer, M., Kwitt, R.: Dissecting supervised contrastive learning. In: International Conference on Machine Learning, pp. 3821–3830. PMLR (2021)
Halabi, S.S., et al.: The RSNA pediatric bone age machine learning challenge. Radiology 290(2), 498–503 (2019)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729–9738 (2020)
Khosla, P., et al.: Supervised contrastive learning. Adv. Neural. Inf. Process. Syst. 33, 18661–18673 (2020)
Le-Khac, P.H., Healy, G., Smeaton, A.F.: Contrastive representation learning: a framework and review. IEEE Access 8, 193907–193934 (2020)
McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426 (2020)
Microsoft: Neural Network Intelligence (2021). https://github.com/microsoft/nni. Accessed 02 Aug 2023
Moschoglou, S., Papaioannou, A., Sagonas, C., Deng, J., Kotsia, I., Zafeiriou, S.: AgeDB: the first manually collected, in-the-wild age database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, vol. 2, p. 5 (2017)
Qin, T., Liu, T.Y., Xu, J., Li, H.: LETOR: a benchmark collection for research on learning to rank for information retrieval. Inf. Retrieval 13, 346–374 (2010)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)
Rothe, R., Timofte, R., Van Gool, L.: DEX: deep expectation of apparent age from a single image. In: Proceedings of the IEEE international conference on computer vision workshops, pp. 10–15 (2015)
Taylor, M., Guiver, J., Robertson, S., Minka, T.: SoftRank: optimizing non-smooth rank metrics. In: Proceedings of the 2008 International Conference on Web Search and Data Mining, pp. 77–86 (2008)
Zha, K., Cao, P., Son, J., Yang, Y., Katabi, D.: Rank-n-contrast: learning continuous representations for regression. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhou, Z., Zhao, Y., Zuo, H., Chen, W. (2024). Ranking Enhanced Supervised Contrastive Learning for Regression. In: Yang, DN., Xie, X., Tseng, V.S., Pei, J., Huang, JW., Lin, J.CW. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2024. Lecture Notes in Computer Science(), vol 14646. Springer, Singapore. https://doi.org/10.1007/978-981-97-2253-2_2
Download citation
DOI: https://doi.org/10.1007/978-981-97-2253-2_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-2252-5
Online ISBN: 978-981-97-2253-2
eBook Packages: Computer ScienceComputer Science (R0)