Abstract
The diversity of action possibilities offered by an environment, a.k.a affordances, cannot be addressed in a scalable manner simply from object categories or semantics, which are limitless. To this end, we present a one-shot learning approach that trains on one or a handful of human-scene interaction samples. Then, given a previously unseen scene, we can predict human affordances and generate the associated articulated 3D bodies. Our experiments show that our approach generates physically plausible interactions that are perceived as more natural in 60–70% of the comparisons with other methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chang, A., et al.: Matterport3D: learning from RGB-D data in indoor environments. In: International Conference on 3D Vision (3DV) (2017)
Fouhey, D.F., Wang, X., Gupta, A.: In defense of the direct perception of affordances. arXiv preprint arXiv:1505.01085 (2015). https://doi.org/10.1002/eji.201445290
Gibson, J.J.: The theory of affordances. In: Perceiving, Acting and Knowing. Toward and Ecological Psychology. Lawrence Eribaum Associates (1977)
Gupta, A., Satkin, S., Efros, A.A., Hebert, M.: From 3D scene geometry to human workspace. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1961–1968, June 2011. IEEE. https://doi.org/10.1109/CVPR.2011.5995448. http://ieeexplore.ieee.org/document/5995448/
Hassan, M., Choutas, V., Tzionas, D., Black, M.J.: Resolving 3D human pose ambiguities with 3D scene constraints. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2282–2292 (2019)
Hassan, M., Ghosh, P., Tesch, J., Tzionas, D., Black, M.J.: Populating 3D scenes by learning human-scene interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14708–14718 (2021)
Jiang, Y., Koppula, H.S., Saxena, A.: Modeling 3D environments through hidden human context. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2040–2053 (2016). https://doi.org/10.1109/TPAMI.2015.2501811
Li, X., Liu, S., Kim, K., Wang, X., Yang, M.H., Kautz, J.: Putting humans in a scene: learning affordance in 3D indoor environments. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12368–12376 (2019)
Luddecke, T., Worgotter, F.: Learning to segment affordances. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 769–776. IEEE, October 2017. https://doi.org/10.1109/ICCVW.2017.96. http://ieeexplore.ieee.org/document/8265305/
Pavlakos, G., et al.: Expressive body capture: 3D hands, face, and body from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10975–10985 (2019)
Peternell, M.: Geometric properties of bisector surfaces. Graph. Models 62(3), 202–236 (2000). https://doi.org/10.1006/gmod.1999.0521
Ruiz, E., Mayol-Cuevas, W.: Geometric affordance perception: leveraging deep 3D saliency with the interaction tensor. Front. Neurorobot. 14, 45 (2020)
Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33715-4_54
Straub, J., et al.: The replica dataset: a digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 (2019)
Wang, X., Girdhar, R., Gupta, A.: Binge watching: scaling affordance learning from sitcoms. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2596–2605 (2017)
Yuksel, C.: Sample elimination for generating poisson disk sample sets. In: Computer Graphics Forum, vol. 34, pp. 25–32 (2015). https://doi.org/10.1111/cgf.12538
Zhang, S., Zhang, Y., Ma, Q., Black, M.J., Tang, S.: PLACE: proximity learning of articulation and contact in 3D environments. In: 8th International Conference on 3D Vision (3DV 2020) (virtual) (2020)
Zhang, Y., Hassan, M., Neumann, H., Black, M.J., Tang, S.: Generating 3D people in scenes without people. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020). https://github.com/yz-cnsdqz/PSI-release/
Zhao, X., Wang, H., Komura, T.: Indexing 3D scenes using the interaction bisector surface. ACM Trans. Graph. 33(3), 1–14 (2014). https://doi.org/10.1145/2574860. http://dl.acm.org/citation.cfm?doid=2631978.2574860
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ADE20K dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Acknowledgments
Abel Pacheco-Ortega thanks the Mexican Council for Science and Technology (CONACYT) for the scholarship provided for his studies with the scholarship number 709908. Walterio Mayol-Cuevas thanks the visual egocentric research activity partially funded by UK EPSRC EP/N013964/1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pacheco-Ortega, A., Mayol-Cuervas, W. (2023). One-Shot Learning for Human Affordance Detection. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13803. Springer, Cham. https://doi.org/10.1007/978-3-031-25066-8_46
Download citation
DOI: https://doi.org/10.1007/978-3-031-25066-8_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25065-1
Online ISBN: 978-3-031-25066-8
eBook Packages: Computer ScienceComputer Science (R0)