Abstract
Human-centric perception (e.g. detection, segmentation, pose estimation, and attribute analysis) is a long-standing problem for computer vision. This paper introduces a unified and versatile framework (HQNet) for single-stage multi-person multi-task human-centric perception (HCP). Our approach centers on learning a unified human query representation, denoted as Human Query, which captures intricate instance-level features for individual persons and disentangles complex multi-person scenarios. Although different HCP tasks have been well-studied individually, single-stage multi-task learning of HCP tasks has not been fully exploited in the literature due to the absence of a comprehensive benchmark dataset. To address this gap, we propose COCO-UniHuman benchmark to enable model development and comprehensive evaluation. Experimental results demonstrate the proposed method’s state-of-the-art performance among multi-task HCP models and its competitive performance compared to task-specific HCP models. Moreover, our experiments underscore Human Query’s adaptability to new HCP tasks, thus demonstrating its robust generalization capability. Codes and data are available at https://github.com/lishuhuai527/COCO-UniHuman.
S. Jin and S. Li—Equal contribution.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
UniHead trains separate models for different HCP tasks.
References
Abdulnabi, A.H., Wang, G., Lu, J., Jia, K.: Multi-task cnn model for attribute prediction. IEEE Trans. Multimedia 17(11), 1949–1959 (2015)
Agustsson, E., Timofte, R., Escalera, S., Baro, X., Guyon, I., Rothe, R.: Apparent and real age estimation in still images with deep residual regressors on appa-real database. In: IEEE International Conference on Automation Face & Gesture Recognition, pp. 87–94 (2017)
Alp Güler, R., Neverova, N., Kokkinos, I.: Densepose: dense human pose estimation in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
Andriluka, M., et al.: Posetrack: a benchmark for human pose estimation and tracking. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: new benchmark and state of the art analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (2014)
Bolya, D., Zhou, C., Xiao, F., Lee, Y.J.: Yolact: real-time instance segmentation. In: International Conference on Computer Vision, pp. 9157–9166 (2019)
Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-End object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Chen, W., et al.: Beyond appearance: a semantic controllable self-supervised learning framework for human-centric visual tasks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 15050–15061 (2023)
Cheng, B., Xiao, B., Wang, J., Shi, H., Huang, T.S., Zhang, L.: Higherhrnet: scale-aware representation learning for bottom-up human pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5386–5395 (2020)
Ci, Y., et al.: Unihcp: a unified model for human-centric perceptions. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 17840–17852 (2023)
Deng, Y., Luo, P., Loy, C.C., Tang, X.: Pedestrian attribute recognition at far distance. In: ACM International Conference on Multimedia, pp. 789–792 (2014)
Doering, A., Chen, D., Zhang, S., Schiele, B., Gall, J.: Posetrack21: a dataset for person search, multi-object tracking and multi-person pose tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 20963–20972 (2022)
Dollár, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: a benchmark. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 304–311 (2009)
Dosovitskiy, A., et al.: An image is worth 16\(\times \)16 words: transformers for image recognition at scale. In: International Conference on Learning Representation (2021)
Ge, Z., Liu, S., Wang, F., Li, Z., Sun, J.: Yolox: exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430 (2021)
Geng, Z., Sun, K., Xiao, B., Zhang, Z., Wang, J.: Bottom-up human pose estimation via disentangled keypoint regression. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 14676–14686 (2021)
Gong, K., Liang, X., Li, Y., Chen, Y., Yang, M., Lin, L.: Instance-level human parsing via part grouping network. In: European Conference on Computer Vision, pp. 770–785 (2018)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: International Conference Computer Vision, pp. 2961–2969 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
Hong, F., Pan, L., Cai, Z., Liu, Z.: Versatile multi-modal pre-training for human-centric perception. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 16156–16166 (2022)
Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schiele, B.: DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 34–50. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_3
Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6m: large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (2013)
Jia, J., Huang, H., Yang, W., Chen, X., Huang, K.: Rethinking of pedestrian attribute recognition: realistic datasets with efficient method. arXiv preprint arXiv:2005.11909 (2020)
Jiang, W., Jin, S., Liu, W., Qian, C., Luo, P., Liu, S.: Posetrans: a simple yet effective pose transformation augmentation for human pose estimation. In: European Conference on Computer Vision, pp. 643–659 (2022)
Jin, S., Liu, W., Ouyang, W., Qian, C.: Multi-person articulated tracking with spatial and temporal embeddings. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5664–5673 (2019)
Jin, S., Liu, W., Xie, E., Wang, W., Qian, C., Ouyang, W., Luo, P.: Differentiable hierarchical graph grouping for multi-person pose estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12352, pp. 718–734. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58571-6_42
Jin, S., et al.: Whole-body human pose estimation in the wild. In: European Conference on Computer Vision (2020)
Jin, S., et al.: Unifs: universal few-shot instance perception with point representations. In: European Conference on Computer Vision (2024)
Joo, H., Neverova, N., Vedaldi, A.: Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation. In: International Conference on 3D Vision, pp. 42–52. IEEE (2021)
Ju, X., Zeng, A., Wang, J., Xu, Q., Zhang, L.: Human-art: a versatile human-centric dataset bridging natural and artificial scenes. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 618–629 (2023)
Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7122–7131 (2018)
Kirillov, A., Levinkov, E., Andres, B., Savchynskyy, B., Rother, C.: Instancecut: from edges to instances with multicut. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5008–5017 (2017)
Kong, S., Fowlkes, C.C.: Recurrent pixel embedding for instance grouping. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9018–9028 (2018)
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L.M., Zhang, L.: Dn-detr: accelerate detr training by introducing query denoising. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 13619–13627 (2022)
Li, F., et al.: Mask dino: towards a unified transformer-based framework for object detection and segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3041–3050 (2023)
Li, J., et al.: Multiple-human parsing in the wild. arXiv preprint arXiv:1705.07206 (2017)
Li, K., Wang, S., Zhang, X., Xu, Y., Xu, W., Tu, Z.: Pose recognition with cascade transformers. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944–1953 (2021)
Li, Y., Huang, C., Loy, C.C., Tang, X.: Human attribute recognition by deep hierarchical contexts. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 684–700. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_41
Liang, J., Song, G., Leng, B., Liu, Y.: Unifying visual perception by dispersible points learning. In: European Conference on Computer Vision, pp. 439–456 (2022)
Lin, J., Zeng, A., Wang, H., Zhang, L., Li, Y.: One-stage 3d whole-body mesh recovery with component aware transformer. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 21159–21168 (2023)
Lin, K., Wang, L., Liu, Z.: End-to-end human pose and mesh reconstruction with transformers. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1954–1963 (2021)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Lin, Y., Shen, J., Wang, Y., Pantic, M.: Fp-age: leveraging face parsing attention for facial age estimation in the wild. IEEE Trans. Image Process (2022)
Liu, S., et al.: Dab-detr: dynamic anchor boxes are better queries for detr. In: International Conference on Learning Representation (2022)
Liu, X., et al.: Hydraplus-net: attentive deep features for pedestrian analysis. In: International Conference on Computer Vision, pp. 1–9 (2017)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: International Conference on Computer Vision, pp. 10012–10022 (2021)
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: International Conference on Computer Vision (2015)
Luo, Z., Wang, Z., Huang, Y., Wang, L., Tan, T., Zhou, E.: Rethinking the heatmap regression for bottom-up human pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 13264–13273 (2021)
Mao, W., Tian, Z., Wang, X., Shen, C.: Fcpose: fully convolutional multi-person pose estimation with dynamic instance-aware convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9034–9043 (2021)
Newell, A., Huang, Z., Deng, J.: Associative embedding: end-to-end learning for joint detection and grouping. Adv. Neural Inform. Process. Syst. (2017)
Nie, X., Feng, J., Yan, S.: Mutual learning to adapt for joint human parsing and pose estimation. In: European Conference on Computer Vision, pp. 502–517 (2018)
Nie, X., Feng, J., Zhang, J., Yan, S.: Single-stage multi-person pose machines. In: International Conference on Computer Vision, pp. 6951–6960 (2019)
Nie, X., Feng, J., Zuo, Y., Yan, S.: Human pose estimation with parsing induced learner. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2100–2108 (2018)
Pang, H.E., Cai, Z., Yang, L., Zhang, T., Liu, Z.: Benchmarking and analyzing 3d human pose and shape estimation beyond algorithms. Adv. Neural. Inf. Process. Syst. 35, 26034–26051 (2022)
Papandreou, G., Zhu, T., Chen, L.C., Gidaris, S., Tompson, J., Murphy, K.: Personlab: person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In: European Conference on Computer Vision, pp. 269–286 (2018)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. (2015)
Rothe, R., Timofte, R., Van Gool, L.: Dex: deep expectation of apparent age from a single image. In: International Conference on Computer Vision Workshop, pp. 10–15 (2015)
Shao, S., Zhao, Z., Li, B., Xiao, T., Yu, G., Zhang, X., Sun, J.: Crowdhuman: a benchmark for detecting human in a crowd. arXiv preprint arXiv:1805.00123 (2018)
Shi, D., Wei, X., Li, L., Ren, Y., Tan, W.: End-to-end multi-person pose estimation with transformers. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11069–11078 (2022)
Shi, D., Wei, X., Yu, X., Tan, W., Ren, Y., Pu, S.: Inspose: instance-aware networks for single-stage multi-person pose estimation. In: ACM International Conference on Multimedia, pp. 3079–3087 (2021)
Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)
Sun, Y., Bao, Q., Liu, W., Fu, Y., Black, M.J., Mei, T.: Monocular, one-stage, regression of multiple 3d people. In: International Conference on Computer Vision, pp. 11179–11188 (2021)
Tang, S., et al.: Humanbench: towards general human-centric perception with projector assisted pretraining. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 21970–21982 (2023)
Tian, Y., Luo, P., Wang, X., Tang, X.: Pedestrian detection aided by deep learning semantic tasks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5079–5087 (2015)
Tian, Z., Chen, H., Shen, C.: Directpose: direct end-to-end multi-person pose estimation. arXiv preprint arXiv:1911.07451 (2019)
Tian, Z., Shen, C., Chen, H.: Conditional convolutions for instance segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 282–298. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_17
Wang, D., Zhang, S.: Contextual instance decoupling for robust multi-person pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11060–11068 (2022)
Wang, J., et al.: Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. (2020)
Wang, Z., Zheng, L., Liu, Y., Li, Y., Wang, S.: Towards real-time multi-object tracking. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 107–122. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_7
Wei, F., Sun, X., Li, H., Wang, J., Lin, S.: Point-set anchors for object detection, instance segmentation and pose estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12355, pp. 527–544. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58607-2_31
Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: IEEE International Conference on Image Processing, pp. 3645–3649 (2017)
Xiao, B., Wu, H., Wei, Y.: Simple baselines for human pose estimation and tracking. In: European Conference on Computer Vision (2018)
Xie, E., et al.: Polarmask: single shot instance segmentation with polar representation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 12193–12202 (2020)
Xu, L., et al.: Vipnas: efficient video pose estimation via neural architecture search. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 16072–16081 (2021)
Xu, L., et al.: Zoomnas: searching for whole-body human pose estimation in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 45(4), 5296–5313 (2022)
Xu, Y., Zhang, J., Zhang, Q., Tao, D.: Vitpose: simple vision transformer baselines for human pose estimation. Adv. Neural Inform. Process. Syst. 35, 38571–38584 (2022)
Xue, N., Wu, T., Xia, G.S., Zhang, L.: Learning local-global contextual adaptation for multi-person pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 13065–13074 (2022)
Yang, J., Zeng, A., Liu, S., Li, F., Zhang, R., Zhang, L.: Explicit box detection unifies end-to-end multi-person pose estimation. In: International Conference on Learning Representation (2023)
Zeng, W., et al.: Not all tokens are equal: human-centric visual analysis via token clustering transformer. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11101–11111 (2022)
Zeng, W., Ouyang, W., Luo, P., Liu, W., Wang, X.: 3d human mesh regression with dense correspondence. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7054–7063 (2020)
Zhang, H., et al.: Dino: detr with improved denoising anchor boxes for end-to-end object detection. In: International Conference on Learning Representation (2023)
Zhang, N., Paluri, M., Ranzato, M., Darrell, T., Bourdev, L.: Panda: pose aligned networks for deep attribute modeling. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1644 (2014)
Zhang, R., Tian, Z., Shen, C., You, M., Yan, Y.: Mask encoding for single shot instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 10226–10235 (2020)
Zhang, S., Benenson, R., Schiele, B.: Citypersons: a diverse dataset for pedestrian detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3221 (2017)
Zhang, S.H., et al.: Pose2seg: detection free human instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 889–898 (2019)
Zhang, X., Ma, B., Chang, H., Shan, S., Chen, X.: Location sensitive network for human instance segmentation. IEEE Trans. Image Process. 30, 7649–7662 (2021)
Zhang, Y., Zeng, W., Jin, S., Qian, C., Luo, P., Liu, W.: When pedestrian detection meets multi-modal learning: Generalist model and benchmark dataset. In: European Conference on Computer Vision (2024)
Zhang, Y., Wang, C., Wang, X., Zeng, W., Liu, W.: Fairmot: on the fairness of detection and re-identification in multiple object tracking. Int. J. Comput. Vis. 129, 3069–3087 (2021)
Zhang, Y., Liu, L., Li, C., Loy, C.C.: Quantifying facial age by posterior of age comparisons. In: British Machine Vision Conference (2017)
Zheng, A., Zhang, Y., Zhang, X., Qi, X., Sun, J.: Progressive end-to-end object detection in crowded scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 857–866 (2022)
Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: deformable transformers for end-to-end object detection. In: International Conference on Learning Representation (2021)
Acknowledgement
This paper is partially supported by the National Key R&D Program of China No.2022ZD0161000 and the General Research Fund of Hong Kong No.17200622 and 17209324.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Jin, S., Li, S., Li, T., Liu, W., Qian, C., Luo, P. (2025). You Only Learn One Query: Learning Unified Human Query for Single-Stage Multi-person Multi-task Human-Centric Perception. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15076. Springer, Cham. https://doi.org/10.1007/978-3-031-72649-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-72649-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72648-4
Online ISBN: 978-3-031-72649-1
eBook Packages: Computer ScienceComputer Science (R0)