Abstract
Human-Object Interaction (HOI) detection is a core task for high-level image understanding. Recently, Detection Transformer (DETR)-based HOI detectors have become popular due to their superior performance and efficient structure. However, these approaches typically adopt fixed HOI queries for all testing images, which is vulnerable to the location change of objects in one specific image. Accordingly, in this paper, we propose to enhance DETR’s robustness by mining hard-positive queries, which are forced to make correct predictions using partial visual cues. First, we explicitly compose hard-positive queries according to the ground-truth (GT) position of labeled human-object pairs for each training image. Specifically, we shift the GT bounding boxes of each labeled human-object pair so that the shifted boxes cover only a certain portion of the GT ones. We encode the coordinates of the shifted boxes for each labeled human-object pair into an HOI query. Second, we implicitly construct another set of hard-positive queries by masking the top scores in cross-attention maps of the decoder layers. The masked attention maps then only cover partial important cues for HOI predictions. Finally, an alternate strategy is proposed that efficiently combines both types of hard queries. In each iteration, both DETR’s learnable queries and one selected type of hard-positive queries are adopted for loss computation. Experimental results show that our proposed approach can be widely applied to existing DETR-based HOI detectors. Moreover, we consistently achieve state-of-the-art performance on three benchmarks: HICO-DET, V-COCO, and HOI-A. Code is available at https://github.com/MuchHair/HQM.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Gupta, S., Malik, J.: Visual semantic role labeling. arXiv preprint arXiv:1505.04474 (2015)
Chao, Y., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect human-object interactions. In: WACV (2018)
Ji, J., Krishna, R., Fei-Fei, L., Niebles, J.: Action genome: Actions as compositions of spatio-temporal scene graphs. In: CVPR (2020)
Tamura, M., Ohashi, H., Yoshinaga, T.: QPIC: query-based pairwise human-object interaction detection with image-wide contextual information. In: CVPR (2021)
Kim, B., Lee, J., Kang, J., Kim, E., Kim, H.: HOTR: end-to-end human-object interaction detection with transformers. In: CVPR (2021)
Zou, C., et al.: End-to-end human object interaction detection with hoi transformer. In: CVPR (2021)
Zhang, A., et al.: Mining the Benefits of Two-stage and One-stage HOI Detection. In: NeurIPS (2021)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: NeurIPS (2015)
Li, Y., et al.: Transferable Interactiveness Knowledge for Human-Object Interaction Detection. In: CVPR (2019)
Gupta, T., Schwing, A., Hoiem, D.: No-frills human-object interaction detection: factorization, layout encodings, and training techniques. In: ICCV (2019)
Wang, T., Yang, T., Danelljan, M., Khan, F., Zhang, X., Sun, J.: Learning human-object interaction detection using interaction points. In: CVPR (2020)
Liao, Y., Liu, S., Wang, F., Chen, Y., Qian, C., Feng, J.: Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In: CVPR (2020)
Ulutan, O., Iftekhar, A., Manjunath, B.: VSGNet: Spatial attention network for detecting human object interactions using graph convolutions. In: CVPR (2020)
Li, Y.: Detailed 2D–3D joint representation for human-object interaction. In: CVPR (2020)
Zhong, X., Ding, C., Qu, X., Tao, D.: Polysemy deciphering network for human-object interaction detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 69–85. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_5
Zhong, X., Ding, C., Qu, X., Tao, D.: Polysemy deciphering network for robust human-object interaction detection. In: IJCV (2021)
Gao, C., Xu, J., Zou, Y., Huang, J.-B.: DRG: Dual relation graph for human-object interaction detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 696–712. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_41
Hou, Z., Peng, X., Qiao, Yu., Tao, D.: Visual compositional learning for human-object interaction detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12360, pp. 584–600. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58555-6_35
Kim, D.-J., Sun, X., Choi, J., Lin, S., Kweon, I.S.: Detecting human-object interactions with action co-occurrence priors. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 718–736. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_43
Zhou, P., Chi, M.: Relation parsing neural network for human-object interaction detection. In: ICCV (2019)
Liu, Y., Chen, Q., Zisserman, A.: Amplifying key cues for human-object-interaction detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12359, pp. 248–265. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58568-6_15
Liu, Y., Yuan, J., Chen, C.: ConsNet: learning consistency graph for zero-shot human-object interaction detection. In: ACM MM (2020)
Wan, B., Zhou, D., Liu, Y., Li, R., He, X.: Pose-aware Multi-level Feature Network for Human Object Interaction Detection. In: ICCV (2019)
Gao, C., Zou, Y., Huang, J.: ican: Instance-centric attention network for human-object interaction detection. In: BMVC (2018)
Wang, T., et al.: Deep contextual attention for human-object interaction detection. In: ICCV (2019)
Gkioxari, G., Girshick, R.: Detecting and recognizing human-object interactions. In: CVPR (2018)
Zhong, X., Qu, X., Ding, C., Tao, D.: Glance and gaze: inferring action-aware points for one-stage human-object interaction detection. In: CVPR (2021)
Kim, B., Choi, T., Kang, J., Kim, H.J.: Uniondet: Union-level detector towards real-time human-object interaction detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12360, pp. 498–514. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58555-6_30
Chen, M., Liao, Y., Liu, S., Chen, Z., Wang, F., Qian, C.: Reformulating hoi detection as adaptive set prediction. In: CVPR (2021)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.: Attention is all you need. In: NeurIPS (2017)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Kuhn, H.: The Hungarian method for the assignment problem. In: Naval Research Logistics Quarterly (2020)
Ghiasi, G., Lin, T., Le, Q.: Dropblock: A regularization method for convolutional networks. In: Wiley Online Library (1955)
Zhou, T., Wang, W., Qi, S., Ling, H., Shen, J.: Cascaded human-object interaction recognition. In: CVPR (2020)
Pic leaderboard (2019). http://www.picdataset.com/challenge/leaderboard/hoi2019,
Meng, D.: Conditional DETR for fast training convergence. In: ICCV (2021)
Gao, P., Zheng, M., Wang, X., Dai, J., Li, H.: Fast convergence of DETR with spatially modulated CoAttention. In: ICCV (2021)
Dai, X., Chen, Y., Yang, J., Zhang, P., Yuan, L., Zhang, L.: Dynamic DETR: end-to-end object detection with dynamic attention. In: ICCV (2021)
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: deformable transformers for end- to-end object detection. In: ICLR (2020)
Liu, S., et al.: DAB-DETR: dynamic anchor boxes are better queries for DETR. In: ICLR (2022)
Yuan, H., Wang, M., Ni, D., Xu, L.: Detecting human-object interactions with object-guided cross-modal calibrated semantics. In: AAAI (2022)
Li, Z., Zou, C., Zhao, Y., Li, B., Zhong, S.: Improving human-object interaction detection via phrase learning and label composition. In: AAAI (2022)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2018)
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: A metric and a loss for bounding box regression. In: CVPR (2019)
Lin, T., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: ICCV (2017)
Wang, X., Shrivastava, A., Gupta, A.: A-fast-rcnn: Hard positive generation via adversary for object detection. arXiv preprint arXiv:2201.12329 (2022)
Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: CVPR (2017)
Wang, K., Wang, P., Ding, C., Tao, D.: Batch coherence-driven network for part-aware person re-identification. In: TIP (2021)
Qu, X., Ding, C., Li, X., Zhong, X., Tao, D.: Distillation using oracle queries for transformer-based human-object interaction detection. In: CVPR (2022)
Lin, X., Ding, C., Zhang, J., Zhan, Y., Tao, D.: RU-Net: regularized unrolling network for scene graph generation. In: CVPR (2022)
Lin, X., Ding, C., Zhan, Y., Li, Z., Tao, D.: HL-Net: Heterophily learning network for scene graph generation. In: CVPR (2022)
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L., Zhang, L.: Dn-detr: Accelerate detr training by introducing query denoising. In: CVPR (2022)
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant 62076101 and 61702193, the Program for Guangdong Introducing Innovative and Entrepreneurial Teams under Grant 2017ZT07X183, Guangdong Basic and Applied Basic Research Foundation under Grant 2022A1515011549, and Guangdong Provincial Key Laboratory of Human Digital Twin under Grant 2022B1212010004.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhong, X., Ding, C., Li, Z., Huang, S. (2022). Towards Hard-Positive Query Mining for DETR-Based Human-Object Interaction Detection. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13687. Springer, Cham. https://doi.org/10.1007/978-3-031-19812-0_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-19812-0_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19811-3
Online ISBN: 978-3-031-19812-0
eBook Packages: Computer ScienceComputer Science (R0)