Skip to main content
SpringerLink
Log in

SEQUENCE: a remote control technique to select objects by matching their rhythm

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

We present SEQUENCE, a novel interaction technique for selecting objects from a distance. Objects display different rhythmic patterns by means of animated dots, and users can select one of them by matching the pattern through a sequence of taps on a smartphone. The technique works by exploiting the temporal coincidences between patterns displayed by objects and sequences of taps performed on a smartphone: if a sequence matches with the pattern displayed by an object, the latter is selected. We propose two different alternatives for displaying rhythmic sequences associated with objects: the first one uses fixed dots (FD), the second one rotating dots (RD). Moreover, we performed two evaluations on such alternatives. The first evaluation, carried out with five participants, was aimed to discover the most appropriate speed for displaying animated rhythmic patterns. The second evaluation, carried out on 12 participants, was aimed to discover errors (i.e., activation of unwanted objects), missed activations (within a certain time), and time of activations. Overall, the proposed design alternatives perform in similar ways (errors, 2.8% for FD and 3.7% for RD; missed, 1.3% for FD and 0.9% for RD; time of activation, 3862 ms for FD and 3789 ms for RD).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. Conceptually, there are no limits of division level. At any rate, in music the highest subdivision is given by the 256th note (which is rare and split a measure in 256 time units).

  2. Practically every song has this rhythmic pattern in one way or in another.

  3. From the system perspective, any sequence has starting and ending points, but in circular TUBSs, they are not identifiable by users.

  4. These users belong to the close circle of the author of the paper. Therefore, these tests cannot have a formal validity for clear reasons, but they were extremely useful to better reflect on the different possibilities of design of SEQUENCE.

  5. An “experiment conducted by Michotte and reported by Card, Moran and Newell (1983) [11] shows that humans perceive two events as connected by immediate causality if the delay between the events is less than 50 ms” [35].

References

  1. Ackad C, Clayphan A, Tomitsch M, Kay J (2015) An in-the-wild study of learning mid-air gestures to browse hierarchical information at a large interactive public display. Proceedings of the Joint International Conference on Pervasive and Ubiquitous Computing and the International Symposium on Wearable Computers (Ubicomp/ISWC’15): 1227–1238. https://doi.org/10.1145/2750858.2807532

  2. Baldauf M, Fröhlich P (2013) The augmented video wall: multi-user AR interaction with public displays. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’13): 3015–3018. https://doi.org/10.1145/2468356.2479599

  3. Bar M, Neta M (2006) Humans prefer curved visual objects. Psychol Sci 17(8):645–648. https://doi.org/10.1111/j.1467-9280.2006.01759.x

    Article  Google Scholar 

  4. Barbara N, Camilleri TA (2017) Interfacing with a speller using EOG glasses. In 2016 I.E. International Conference on Systems, Man, and Cybernetics, SMC 2016 - Conference Proceedings, 1069–1074. https://doi.org/10.1109/SMC.2016.7844384

  5. Baudel T, Beaudouin-lafon M (1993) CHARADE : remote control of objects using free-hand gestures. Commun ACM 36(7):28–35. https://doi.org/10.1145/159544.159562

    Article  Google Scholar 

  6. Bellino A (2015) Enhancing pinch-drag-flick paradigm with two new gestures: two-finger-tap for tablets and tap&tap for smartphones. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 534–551. https://doi.org/10.1007/978-3-319-22698-9_37

  7. Bellino A, Cabitza F, De Michelis G, De Paoli F (2016) Touch&Screen. In Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia - MUM ‘16, 25–35. https://doi.org/10.1145/3012709.3012736

  8. Boring S, Jurmu M, Butz A (2009) Scroll, tilt or move it: using mobile phones to continuously control pointers on large public displays. Conference of the Australian Computer-Human Interaction Special Interest Group: 161–168. https://doi.org/10.1145/1738826.1738853

  9. Boring S, Baur D, Butz A, Gustafson S, Baudisch P (2010) Touch projector: mobile interaction through video. SIGCHI Conference on Human Factors in Computing Systems (CHI’10): 2287–2296. https://doi.org/10.1145/1753326.1753671

  10. Bradski G, Kaehler A, Bradski G (2013) Learning OpenCV. https://doi.org/10.1017/CBO9781107415324.004

    Google Scholar 

  11. Card SK, Newell A, Moran TP (1983) The psychology of human-computer interaction. L. Erlbaum Associates Inc., Hillsdale

    Google Scholar 

  12. Carter M, Velloso E, Downs J, Sellen A, O’Hara K, Vetere F. 2016. PathSync: multi-user gestural interaction with touchless rhythmic path mimicry. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ‘16: 3415–3427. https://doi.org/10.1145/2858036.2858284

  13. Chen M-Y, Mummert L, Pillai P, Hauptmann A, Sukthankar R (2010) Controlling your TV with gestures. Proceedings of the international conference on Multimedia information retrieval - MIR ‘10: 405. https://doi.org/10.1145/1743384.1743453

  14. Clarke C, Bellino A, Esteves A, Velloso E, Gellersen H (2016) TraceMatch : a computer vision technique for user input by tracing of animated controls. In UbiComp ‘16, 1–6. https://doi.org/10.1145/2971648.2971714

  15. Clarke C, Bellino A, Esteves A, Gellersen H (2017) Using body movements in synchrony with orbiting widgets for remote control : an evaluation of TraceMatch. Proc ACM Interact Mobile Wearable Ubiquitous Technol 1:3

    Google Scholar 

  16. Dachselt R, User Interface, Engineering Group, Robert Buchholz, and Security Group (2009) Natural throw and tilt interaction between mobile phones and distant displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI EA ’09), 3253–3258. https://doi.org/10.1145/1520340.1520467

  17. Fekete JD, Elmqvist N, Guiard Y (2009) Motion-pointing: target selection using elliptical motions. Proceedings of the SIGCHI conference on Human Factors in computing systems - CHI ‘09: 289–298. https://doi.org/10.1145/1518701.1518748

  18. Forth J, Wiggins GA, McLean A (2010) Unifying conceptual spaces: concept formation in musical creative systems. Mind Mach 20(4):503–532. https://doi.org/10.1007/s11023-010-9207-x

    Article  Google Scholar 

  19. Gallo L, Placitelli AP, Ciampi M (2011) Controller-free exploration of medical image data: experiencing the Kinect. In Proceedings - IEEE Symposium on Computer-Based Medical Systems https://doi.org/10.1109/CBMS.2011.5999138

  20. Ghomi E, Faure G, Huot S, Chapuis O, Beaudouin-Lafon M (2012) Using rhythmic patterns as an input method. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI ‘12: 1253–1262. https://doi.org/10.1145/2207676.2208579

  21. Gonzaiez C (1996) Does animation in user interfaces improve decision making ? In CHI 96, 27–34. https://doi.org/10.1145/238386.238396

  22. Grahn JA, Brett M (2007) Rhythm and beat perception in motor areas of the brain. J Cogn Neurosci 19(5):893–906. https://doi.org/10.1162/jocn.2007.19.5.893

    Article  Google Scholar 

  23. Guiard Y (1993) On Fitts’s and Hooke’s laws: simple harmonic movement in upper-limb cyclical aiming. Acta Psychol 82(1–3):139–159. https://doi.org/10.1016/0001-6918(93)90009-G

    Article  Google Scholar 

  24. Guinness D, Jude A, Michael Poor G, Dover A (2015) Models for rested touchless gestural interaction. SUI 2015 - Proceedings of the 3rd ACM Symposium on Spatial User Interaction: 34–43. https://doi.org/10.1145/2788940.2788948

  25. Heer J, Robertson GG (2007) Animated transitions in statistical data graphics. IEEE Trans Vis Comput Graph 13(6):1240–1247. https://doi.org/10.1109/TVCG.2007.70539

    Article  Google Scholar 

  26. Hincapié-Ramos JD, Guo X, Moghadasian P, Irani P (2014) Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ‘14: 1063–1072. https://doi.org/10.1145/2556288.2557130

  27. Hwang I, Kim H-C, Cha J, Ahn C, Kim K, Park J-I (2015) A gesture based TV control interface for visually impaired: Initial design and user study. Frontiers of Computer Vision (FCV), 2015 21st Korea-Japan Joint Workshop on: 1–5. https://doi.org/10.1109/FCV.2015.7103730

  28. Iannarilli F, Pesce C, Persichini C, Capranica L (2013) Age-related changes of rhythmic ability in musically trained and untrained individuals. Sport Sci Health 9(2):43–50. https://doi.org/10.1007/s11332-013-0144-y

    Article  Google Scholar 

  29. Jeong S, Jin J, Song T, Kwon K, Jeon J (2012) Single-camera dedicated television control system using gesture drawing. IEEE Trans Consum Electron 58(4):1129–1137. https://doi.org/10.1109/TCE.2012.6414977

    Article  Google Scholar 

  30. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1867–1874. https://doi.org/10.1109/CVPR.2014.241

  31. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758. https://doi.org/10.1145/1577069.1755843

    Google Scholar 

  32. Koskela T, Väänänen-Vainio-mattila K (2004) Evolution towards smart home environments: empirical evaluation of three user interfaces. Pers Ubiquit Comput 8:234–240. https://doi.org/10.1007/s00779-004-0283-x

    Article  Google Scholar 

  33. Launay J, Grube M, Stewart L (2014) Dysrhythmia: a specific congenital rhythm perception deficit. Front Psychol 5. https://doi.org/10.3389/fpsyg.2014.00018

  34. Leiva L, Böhmer M, Gehring S, Krüger A (2012) Back to the app: the costs of mobile application interruptions. In Proceedings of the 14th international conference on human-computer interaction with mobile devices and services, 291–294. https://doi.org/10.1145/2371574.2371617

  35. Mäki-Patola T, Hämäläinen P (2004) Latency tolerance for gesture controlled continuous sound instrument without tactile feedback. Proceedings ICMC 2004, 1998. Retrieved from http://www.mcpatola.com/publications/icmcarticlefinal10.pdf

  36. Malacria S, Lecolinet E, Guiard Y (2010) Clutch-free panning and integrated pan-zoom control on touch-sensitive surfaces: the cyclostar approach. Conference on Human Factors in Computing Systems: 2615–2624. https://doi.org/10.1145/1753326.1753724

  37. Mayer RE, Moreno R (2002) Animation as an aid to multimedia learning. Educ Psychol Rev 14:87–99. https://doi.org/10.1023/A:1013184611077

    Article  Google Scholar 

  38. Mayrhofer R, Gellersen H (2009) Shake well before use: intuitive and secure pairing of mobile devices. IEEE Trans Mob Comput 8(6):792–806. https://doi.org/10.1109/TMC.2009.51

    Article  Google Scholar 

  39. Mccallum DC, Irani PP (2009) ARC-Pad : Absolute + Relative Cursor Positioning for Large Displays with a Mobile Touchscreen. Human Factors: 153–156. https://doi.org/10.1145/1622176.1622205

  40. Nancel M, Chapuis O, Pietriga E, Yang X-D, Irani PP, Beaudouin-Lafon M (2013) High-precision pointing on large wall displays using small handheld devices. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ‘13: 831–840. https://doi.org/10.1145/2470654.2470773

  41. Norman DA, Nielsen J (2010) Gestural interfaces: a step backward in usability. Interactions 17:46–49. https://doi.org/10.1145/1836216.1836228

    Article  Google Scholar 

  42. Oakley I, Lee DY, Rasel Islam MD, Esteves A (2015) Beats. In Proceedings of the 33rd annual ACM conference on human factors in computing systems—CHI ‘15, 1237–1246. https://doi.org/10.1145/2702123.2702226

  43. Pal M, Banerjee A, Datta S, Konar A, Tibarewala DN, Janarthanan R (2014) Electrooculography based blink detection to prevent Computer Vision Syndrome. In IEEE CONECCT 2014 - 2014 I.E. International Conference on Electronics, Computing and Communication Technologies. https://doi.org/10.1109/CONECCT.2014.6740337

  44. Pérez-Quiñones MA, Sibert JL (1996) A collaborative model of feedback in human-computer interaction. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 316–323. https://doi.org/10.1145/238386.238535

  45. Phillips-Silver J, Toiviainen P, Gosselin N, Piché O, Nozaradan S, Palmer C, Peretz I (2011) Born to dance but beat deaf: a new form of congenital amusia. Neuropsychologia 49(5):961–969. https://doi.org/10.1016/j.neuropsychologia.2011.02.002

    Article  Google Scholar 

  46. Reyes G, Jason W, Juneja N, Goldshtein M, Keith Edwards W, Abowd GD, Starner T (2017) SynchroWatch : one-handed synchronous smartwatch gestures. Proc ACM Interact Mobile Wearable Ubiquitous Technol 1(4):1–26

    Google Scholar 

  47. Seifert J, Bayer A, Rukzio E (2013) PointerPhone: using mobile phones for direct pointing interactions with remote displays. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 18–35. https://doi.org/10.1007/978-3-642-40477-1_2

  48. Sharp T, Keskin C, Robertson D, Taylor J, Shotton J, Kim D, Rhemann C, Leichter I, Vinnikov A, Wei Y, Freedman D, Kohli P, Krupka E, Fitzgibbon A, Izadi S (2015) Accurate, robust, and flexible real-time hand tracking. ACM Conference on Human Factors in Computing Systems (CHI): 3633–3642. https://doi.org/10.1145/2702123.2702179

  49. Shotton J, Fitzgibbon A, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2013) Real-time human pose recognition in parts from single depth images. Stud Comput Intell 411:119–135. https://doi.org/10.1007/978-3-642-28661-2-5

    Google Scholar 

  50. Soukupová T, Jan Č (2016) Real-time eye blink detection using facial landmarks. 21st Computer Vision Winter Workshop - Rimske Toplice

  51. Sowiński J, Bella SD (2013) Poor synchronization to the beat may result from deficient auditory-motor mapping. Neuropsychologia 51(10):1952–1963. https://doi.org/10.1016/j.neuropsychologia.2013.06.027

    Article  Google Scholar 

  52. Stellmach S, Dachselt R (2012) Look & Touch : gaze-supported target acquisition. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: 2981–2990. https://doi.org/10.1145/2207676.2208709

  53. Stellmach S, Dachselt R (2013) Still looking: investigating seamless gaze-supported selection, positioning, and manipulation of distant targets. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ‘13: 285. https://doi.org/10.1145/2470654.2470695

  54. Stowell D, Plumbley MD (2008) Characteristics of the beatboxing vocal style. Electron Eng:1–4 https://doi.org/C4DM-TR-08-01

  55. Szentgyorgyi C, Lank E (2007) Five-key text input using rhythmic mappings. Proceedings of the ninth international conference on Multimodal interfaces - ICMI ‘07: 118. https://doi.org/10.1145/1322192.1322214

  56. Vajk T, Coulton P, Bamford W, Edwards R (2008) Using a mobile phone as a "Wii-like" controller for playing games on a large public display. Int J Comput Games Technol 2008. https://doi.org/10.1155/2008/539078

  57. Velloso E, Carter M, Newn J, Esteves A, Clarke C, Gellersen H (2017) Motion correlation: selecting objects by matching their movement. ACM Trans Comput Human Interact 24(3):1–35. https://doi.org/10.1145/3064937

    Article  Google Scholar 

  58. Walter R, Bailly G, Müller J (2013) StrikeAPose: revealing mid-air gestures on public displays. SIGCHI Conference on Human Factors in Computing Systems (CHI’13): 841–850. https://doi.org/10.1145/2470654.2470774

  59. Wei T, Lee B, Qiao Y, Kitsikidis A, Dimitropoulos K, Grammalidis N (2015) Experimental study of skeleton tracking abilities from microsoft kinect non-frontal views. In 3DTV-Conference. https://doi.org/10.1109/3DTV.2015.7169367

  60. Winold A (1975) Rhythm in twentieth-century music. Aspects of Twentieth-Century Music

  61. Wobbrock JO (2009) TapSongs : Tapping Rhythm-Based Passwords on a Single Binary Sensor. October: 93–96. https://doi.org/10.1145/1622176.1622194

Download references

Funding

Many thanks to Giorgio De Michelis for his comments and pieces of advice on the initial and revised version of the paper, and Daniela Bascuñán for helping me to improve English. Thanks also to my daughter Eleonora, who was born during the revision period of this paper. Despite that, she allowed me to work on it and send it before the deadline. Finally, thanks to friends and colleagues who gave me suggestions and discussed critically the different alternatives of design of SEQUENCE.

Author information

Authors and Affiliations

  1. Università di Milano-Bicocca, Milan, Italy

    Alessio Bellino

Authors
  1. Alessio Bellino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessio Bellino.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bellino, A. SEQUENCE: a remote control technique to select objects by matching their rhythm. Pers Ubiquit Comput 22, 751–770 (2018). https://doi.org/10.1007/s00779-018-1129-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-018-1129-2

Keywords

Search

Navigation