Skip to main content

Advertisement

SpringerLink
Log in

HF-SRGR: a new hybrid feature-driven social relation graph reasoning model

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Social relations and interactions between persons form the foundation of human society. Effective recognition of social relationships has great potential for understanding and improving people’s psychology and behaviors, e.g., mental health and activity analysis, and further improving social resilience. Existing work of social relation recognition (SRR) mainly focuses on exploiting two or three types of features to recognize social relations without considering the relations between features. In this paper, we proposed a new framework for extraction and fusion of the hybrid features, namely Social Relation Graph Reasoning model driven by Hybrid-Features (HF-SRGR). For the proposed method, a social relation graph was constructed first using relation and scene features as nodes. An attention mechanism was then designed to incorporate into graph neural networks (GNNs), generating inter-pair features and interactions between relation nodes and the scene node, respectively. Besides, the propagation of scene information further strengthens the rationality of interaction reasoning. Extensive experiments on PISC and PIPA datasets show that our proposed approach achieves better performance over the state-of-the-art methods in terms of accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Bugental, D.B.: Acquisition of the algorithms of social life: a domain-based approach. Psychol. Bull. 126(2), 187–219 (2000)

    Article  Google Scholar 

  2. Platek, S.M., Raines, D.M., Gallup, G.G., Mohamed, F.B., Thomson, J.W., Myers, T.E., Panyavin, I.S., Levin, S.L., Davis, J.A., Fonteyn, L.CMa.: Reactions to children’s faces: males are more affected by resemblance than females are, and so are their brains. Evol. Hum. Behav. 25(6), 394–405 (2004)

    Article  Google Scholar 

  3. Biddle, B.J.: Recent development in role theory. Annu. Rev. Sociol. 12(1), 1267–1292 (1986)

    Article  Google Scholar 

  4. Boonstra, T.W., Werner-Seidler, A., O”Dea, B., Larsen, M.E., Christensen, H.: Smartphone app to investigate the relationship between social connectivity and mental health. In: 39-th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 287–290 (2017)

  5. Kawachi, I., Berkman, L.F.: Social ties and mental health. Urban Health 78, 458–467 (2001)

    Article  Google Scholar 

  6. Zhou, X., Lu, J., Hu, J., Shang, Y.: Gabor-based gradient orientation pyramid for kinship verification under uncontrolled environments. In: Proceedings of the 20th ACM International Conference on Multimedia, pp. 725–728 (2012)

  7. Yoo, H., Eom, T., Seo, J., Choi, S.I.: Detection of interaction groups based on geometric and social relations between individuals in an image. Pattern Recognit. 93, 498–507 (2019)

    Article  Google Scholar 

  8. Ramanathan, V., Yao, B., Li, F.F.: Social role discovery in human events. In: IEEE International Conference on Computer Vision (ICCV), pp. 1497–1504 (2013)

  9. Wang, M., Du, X., Xiangbo, S., Wang, X., Tang, J.: Deep supervised feature selection for social relationship recognition. Pattern Recognit. Lett. 138, 410–416 (2020)

    Article  Google Scholar 

  10. Fang, R., Tang, K.D., Snavely, N., Chen, T.: Towards computational models of kinship verification. In: Proceedings of the International Conference on Image Processing, ICIP 2010, September 26–29, Hong Kong, China, pp. 1577–1580 (2010)

  11. Wang, G., Gallagher, A.C., Luo, J., Forsyth, D.A.: Seeing people in social context: Recognizing people and social relationships. In: European Conference on Computer Vision (ECCV) (2010)

  12. Xia, S., Shao, M., Luo, J., Fu, Y.: Understanding kin relationships in a photo. IEEE Trans. Multimedia 14(4), 1046–1056 (2012)

    Article  Google Scholar 

  13. Dibeklioglu, H., Salah, A.A., Gevers, T.: Like father, like son: facial expression dynamics for kinship verification. In: IEEE International Conference on Computer Vision (ICCV), pp. 1497–1504 (2013)

  14. Lu, J., Zhou, X., Tan, Y.P., Shang, Y.: Neighborhood repulsed metric learning for kinship verification. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 331–345 (2014)

    Article  Google Scholar 

  15. Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Learning social relation traits from face images. In: IEEE International Conference on Computer Vision (ICCV), pp. 3631–3639 (2015)

  16. Li, W., Zhang, Y., Lv, K., Lu, J., Zhou, J.: Graph-based kinship reasoning network. In: IEEE International Conference on Multimedia and Expo (ICME), London, United Kingdom, pp. 1–6 (2020)

  17. Li, J., Wong, Y., Zhao, Q., Kankanhalli, M.S.: Dual-glance model for deciphering social relationships. In: IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pp. 2669–2678 (2017)

  18. Sun, Q., Schiele, B., Fritz, M.: A domain based approach to social relation recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 435–444 (2017)

  19. Wang, Z., Chen, T., Ren, J., Yu, W., Cheng, H., Lin, L.: Deep reasoning with knowledge graph for social relationship understanding. In: International Joint Conferences on Artificial Intelligence (IJCAI), pp. 1021–1028 (2018)

  20. Zhang, M., Liu, X., Liu, W., Zhou, A., Ma, H., Mei, T.: Multi-granularity reasoning for social relation recognition from images. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 1618–1623 (2019)

  21. Gao, J., Qing, L., Li, L., Cheng, Y., Peng, Y.: Multi-scale features based interpersonal relation recognition using higher-order graph neural network. Neurocomputing. https://doi.org/10.1016/j.neucom.2021.05.097

  22. Morris, C., Ritzert, M., Fey, M., Hamilton, W., Lenssen, J., Rattan, G., Grohe, M.: Weisfeiler and leman go neural: higher-order graph neural networks. pp. 4602–4609 (2019)

  23. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding, pp. 675–678 (2014)

  24. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

  25. Wang, M., Shu, X., Feng, J., Wang, X., Tang, J.: Deep multi-person kinship matching and recognition for family photos. Pattern Recognit. 105, 111–21 (2020)

    Article  Google Scholar 

  26. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)

  27. He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask r-cnn. In: IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017)

  28. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)

    Article  Google Scholar 

  29. Gori, M., Monfardini, G., Scarselli, F.: A new model for learning in graph domains. In: IEEE International Joint Conference on Neural Networks, Montreal, Que 2, pp. 729–734 (2005)

  30. Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2009)

    Article  Google Scholar 

  31. Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. In: International Conference on Learning Representations (ICLR) (2016)

  32. Kipf, T., Max, W.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (ICLR) (2017)

  33. Velič ković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (ICLR) (2018)

  34. Zhang, W., Lin, Z., Cheng, J., Ma, C., Deng, X., Wang, H.: STA-GCN: two-stream graph convolutional network with spatial-temporal attention for hand gesture recognition. Vis. Comput. 36(10–12), 2433–2444 (2020)

    Article  Google Scholar 

  35. Qin, Y., Mo, L., Li, C., Luo, J.: Skeleton-based action recognition by part-aware graph convolutional networks. Vis. Comput. 36(3), 621–631 (2020)

    Article  Google Scholar 

  36. Liu, X., Zhang, M., Liu, W., Song, J., Mei, T.: BraidNet: braiding semantics and details for accurate human parsing. In: Proceedings of the 27th ACM International Conference on Multimedia (MM’19), pp. 338–346 (2019)

  37. Liu, X., Liu, W., Zheng, J., Yan, C., Mei, T.: Beyond the parts: Learning multi-view cross-part correlation for vehicle re-identification. In: Proceedings of the 27th ACM International Conference on Multimedia (MM’20), pp. 907–912 (2020)

  38. Liu, X., Liu, W., Zhagn, M., Mei, J.C.L.G.C.Y.T.: Social relation recognition from videos via multi-scale spatial-temporal reasoning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3561–3569 (2019)

  39. Deng, J., Dong, W., Socher, R., Li, L.J., Li, F.F.: Imagenet: A large-scale hierarchical image database. In: IEEE Conference on Computer Vision & Pattern Recognition, pp. 248–255 (2009)

  40. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2018)

    Article  Google Scholar 

  41. Kim, J.H., On, K., Kim, J., Ha, J.W., Zhang, B.T.: Hadamard product for low-rank bilinear pooling. In: International Conference on Learning Representations (ICLR) (2017)

  42. Zhang, N., Paluri, M., Taigman, Y., Fergus, R., Bourdev, L.: Beyond frontal faces: Improving person recognition using multiple cues. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4804–4813 (2015)

  43. Hammond, D.K., Vandergheynst, P., Gribonval, R.: Wavelets on graphs via spectral graph theory. Appl. Comput. Harmonic Anal. 30(2), 129–150 (2011)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Nature Science Foundation of China under Grant 61871278 and the Sichuan Science and Technology Program under Grant 2018HH0143.

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Sichuan University, Chengdu, 610065, China

    Lindong Li, Linbo Qing, Yuchen Wang & Jie Su

  2. Department of Computer Science and Technology, University of Hull, Hull, HU67RX, UK

    Yongqiang Cheng

  3. Department of Computing and Mathematics, Manchester Metropolitan University, Manchester, M156BH, UK

    Yonghong Peng

Authors
  1. Lindong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Linbo Qing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuchen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongqiang Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yonghong Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linbo Qing.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, L., Qing, L., Wang, Y. et al. HF-SRGR: a new hybrid feature-driven social relation graph reasoning model. Vis Comput 38, 3979–3992 (2022). https://doi.org/10.1007/s00371-021-02244-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02244-w

Keywords

Search

Navigation