Valentine Bernasconi
ABSTRACT
Hands are an important tool for our daily communication with our peers and the world. They allow us to convey information through particular gestures that are either the product of social conventions or personal expressions. Thanks to the sophistication of sensing and computer vision technologies over the past decade, automated hand recognition can now be more easily used and integrated in simple web applications. In a context of digital artworks collections, it means that gestures can now be envisioned as a new browsing tool that goes beyond simple movements to navigate through a 3D digital space. The paper presents Gestures for Artwork Browsing (GAB), a web application which proposes to use hand motions as a way to directly query pictorial hand gestures from the past. Based on materials from a digitized collection of Renaissance paintings, GAB enables users to record a sequence with the hand movement of their choice and outputs an animation reproducing that same sequence with painted hands. Fostering new research possibilities, the project is a novelty in terms of art database browsing and human-computer interaction, as it does not require traditional search tools such as text-based inputs based on metadata, and allows a direct communication with the content of the artworks.
- Kai Uwe Barthel, Nico Hezel, and Klaus Jung. 2017. Visually Browsing Millions of Images Using Image Graphs. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval(ICMR ’17). Association for Computing Machinery, New York, NY, USA, 475–479. https://doi.org/10.1145/3078971.3079016Google Scholar
Digital Library
- Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2021. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 1 (Jan. 2021), 172–186. https://doi.org/10.1109/TPAMI.2019.2929257 Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.Google ScholarDigital Library
- Lins Derry, Douglas Duhaime, Jordan Kruguer, Dario Rodighiero, Jeffrey Schnapp, and Christopher Pietsch. 2021. Surprise Machines. https://vimeo.com/595473865Google Scholar
- Conghui Hu, Yongxin Yang, Yunpeng Li, Timothy M. Hospedales, and Yi-Zhe Song. 2021. Towards Unsupervised Sketch-based Image Retrieval. arxiv:2105.08237 [cs.CV]Google Scholar
- Leonardo Impett. 2020. Analyzing Gesture in Digital Art History. In The Routledge Companion to Digital Humanities and Art History. Routledge. Num Pages: 22.Google Scholar
- Yanir Kleiman, Joel Lanir, Dov Danon, Yasmin Felberbaum, and Daniel Cohen-Or. 2015. DynamicMaps: Similarity-based Browsing through a Massive Set of Images. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). Association for Computing Machinery, New York, NY, USA, 995–1004. https://doi.org/10.1145/2702123.2702224Google ScholarDigital Library
- Panayiotis Koutsabasis and Chris K. Domouzis. 2016. Mid-Air Browsing and Selection in Image Collections. In Proceedings of the International Working Conference on Advanced Visual Interfaces(AVI ’16). Association for Computing Machinery, New York, NY, USA, 21–27. https://doi.org/10.1145/2909132.2909248Google ScholarDigital Library
- Sreejith M, Siddharth Rakesh, Samik Gupta, Samprit Biswas, and Partha Pratim Das. 2015. Real-time hands-free immersive image navigation system using Microsoft Kinect 2.0 and Leap Motion Controller. In 2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG). 1–4. https://doi.org/10.1109/NCVPRIPG.2015.7489999Google ScholarCross Ref
- Vivien Petras, Timothy Hill, Juliane Stiller, and Maria Gäde. 2017. Europeana – a Search Engine for Digitised Cultural Heritage Material. Datenbank-Spektrum 17, 1 (2017), 41–46. https://doi.org/10.1007/s13222-016-0238-1Google ScholarCross Ref
- Benoit Seguin. 2018. The Replica Project: Building a visual search engine for art historians. XRDS: Crossroads, The ACM Magazine for Students 24, 3 (April 2018), 24–29. https://doi.org/10.1145/3186653Google ScholarDigital Library
- Xi Shen, Alexei A. Efros, and Mathieu Aubry. 2019. Discovering Visual Patterns in Art Collections With Spatially-Consistent Feature Learning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Long Beach, CA, USA, 9270–9279. https://doi.org/10.1109/CVPR.2019.00950Google ScholarCross Ref
- Tijana Vuletic, Alex Duffy, Laura Hay, Chris McTeague, Gerard Campbell, and Madeleine Grealy. 2019. Systematic literature review of hand gestures used in human computer interaction interfaces. International Journal of Human-Computer Studies 129 (2019), 74–94. https://doi.org/10.1016/j.ijhcs.2019.03.011Google ScholarDigital Library
- Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. arXiv:2006.10214 [cs] (June 2020). http://arxiv.org/abs/2006.10214 arXiv:2006.10214.Google Scholar
- Wengang Zhou, Houqiang Li, and Qi Tian. 2017. Recent Advance in Content-based Image Retrieval: A Literature Survey. arxiv:1706.06064 [cs.MM]Google Scholar
Index Terms
- GAB - Gestures for Artworks Browsing
Recommendations
Designing and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices
Hand gestures provide a natural and easy-to-use way to input commands. However, few works have studied the design space of bimanual hand gestures or attempted to infer gestures that involve devices on both hands. We explore the design space of hand-to-...
Human-Machine Interaction based on Hand Gesture Recognition using Skeleton Information of Kinect Sensor
ICAIT'2018: Proceedings of the 3rd International Conference on Applications in Information TechnologyThe hand gesture provides a natural and intuitive communication medium for the human and machine interaction. Because, it can use in virtual reality, language detection, computer games, and other human-computer or human-machine instruction applications. ...
Puppeteer: Manipulating Human Avatar Actions with Intuitive Hand Gestures and Upper-Body Postures
UIST '22 Adjunct: Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and TechnologyWe present Puppeteer, an input prototype system that allows players directly control their avatars through intuitive hand gestures and upper-body postures. We selected 17 avatar actions discovered in the pilot study and conducted a gesture elicitation ...
Comments