Abstract
Most smartphones today have a rich set of sensors that could be used to infer input (e.g., accelerometer, gyroscope, microphone); however, the primary mode of interaction is still limited to the front-facing touchscreen and several physical buttons on the case. To investigate the potential opportunities for interactions supported by built-in sensors, we present the implementation and evaluation of BeyondTouch, a family of interactions to extend and enrich the input experience of a smartphone. Using only existing sensing capabilities on a commodity smartphone, we offer the user a wide variety of additional inputs on the case and the surface adjacent to the smartphone. Although most of these interactions are implemented with machine learning methods, compact and robust rule-based detection methods can also be applied for recognizing some interactions by analyzing physical characteristics of tapping events on the phone. This article is an extended version of Zhang et al. [2015], which solely covered gestures implemented by machine learning methods. We extended our previous work by adding gestures implemented with rule-based methods, which works well with different users across devices without collecting any training data. We outline the implementation of both machine learning and rule-based methods for these interaction techniques and demonstrate empirical evidence of their effectiveness and usability. We also discuss the practicality of BeyondTouch for a variety of application scenarios and compare the two different implementation methods.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Beyond the Touchscreen: An Exploration of Extending Interactions on Commodity Smartphones
- Patrick Baudisch and Gerry Chu. 2009. Back-of-device interaction allows creating very small touch devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1923--1932. Google ScholarDigital Library
- Liang Cai and Hao Chen. 2011. TouchLogger: Inferring keystrokes on touch screen from smartphone motion. In Proceedings of the 6th USENIX Conference on Hot Topics in Security (HotSec'11). 9. Google ScholarDigital Library
- Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-Hyuck Lee, and Shwetak Patel. 2014. AirLink: sharing files between multiple devices using in-air gestures. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, New York, NY, 565--569. Google ScholarDigital Library
- Mayank Goel, Brendan Lee, Md. Tanvir Islam Aumi, Shwetak Patel, Gaetano Borriello, Stacie Hibino, and Bo Begole. 2014. SurfaceLink: Using inertial and acoustic sensing to enable multi-device interaction on a surface. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, 1387--1396. Google ScholarDigital Library
- Mayank Goel, Jacob Wobbrock, and Shwetak Patel. 2012. GripSense: Using built-in sensors to detect hand posture and pressure on commodity mobile phones. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 545--554. Google ScholarDigital Library
- Chris Harrison and Scott E. Hudson. 2008. Scratch input: Creating large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 205--208. Google ScholarDigital Library
- Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing finger interaction on touch surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 627--636. Google ScholarDigital Library
- Seongkook Heo and Geehyuk Lee. 2011. Forcetap: Extending the input vocabulary of mobile touch screens by adding tap gestures. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services. ACM, New York, NY, 113--122. Google ScholarDigital Library
- Ken Hinckley, Jeff Pierce, Mike Sinclair, and Eric Horvitz. 2000. Sensing techniques for mobile interaction. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 91--100. Google ScholarDigital Library
- Ken Hinckley and Hyunyoung Song. 2011. Sensor synaesthesia: Touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 801--810. Google ScholarDigital Library
- Shigeo Hiraoka, Isshin Miyamoto, and Kiyoshi Tomimatsu. 2003. Behind Touch: A text input method for mobile phones by the back and tactile sense interface. In Proceedings of the 9th IFIP TC13 International Conference on Human-Computer Interaction (Interact'03). 131--138.Google Scholar
- Scott E. Hudson, Chris Harrison, Beverly L. Harrison, and Anthony LaMarca. 2010. Whack Gestures: Inexact and inattentive interaction with mobile devices. In Proceedings of the 4th International Conference on Tangible, Embedded, and Embodied Interaction. ACM, New York, NY, 109--112. Google ScholarDigital Library
- Kevin A. Li, Patrick Baudisch, and Ken Hinckley. 2008. Blindsight: Eyes-free access to mobile phones. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1389--1398. Google ScholarDigital Library
- William McGrath and Yang Li. 2014. Detecting tapping motion on the side of mobile devices by probabilistically combining hand postures. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 215--219. Google ScholarDigital Library
- Emiliano Miluzzo, Alexander Varshavsky, Suhrid Balakrishnan, and Romit Roy Choudhury. 2012. TapPrints: Your finger taps have fingerprints. In Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services. ACM, New York, NY, 323--336. Google ScholarDigital Library
- Donald A. Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books, New York, NY.Google Scholar
- Emmanuel Owusu, Jun Han, Sauvik Das, Adrian Perrig, and Joy Zhang. 2012. ACCessory: Password inference using accelerometers on smartphones. In Proceedings of the 12th Workshop on Mobile Computing Systems and Applications. ACM, New York, NY, 9. Google ScholarDigital Library
- Joseph A. Paradiso, Che King Leo, Nisha Checka, and Kaijen Hsiao. 2002. Passive acoustic knock tracking for interactive windows. In CHI'02 Extended Abstracts on Human Factors in Computing Systems. ACM, New York, NY, 732--733. Google ScholarDigital Library
- Simon Robinson, Nitendra Rajput, Matt Jones, Anupam Jain, Shrey Sahay, and Amit Nanavati. 2011. TapBack: Towards richer mobile interfaces in impoverished contexts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 2733--2736. Google ScholarDigital Library
- Sami Ronkainen, Jonna Häkkilä, Saana Kaleva, Ashley Colley, and Jukka Linjama. 2007. Tap input as an embedded interaction method for mobile devices. In Proceedings of the 1st International Conference on Tangible and Embedded Interaction. ACM, New York, NY, 263--270. Google ScholarDigital Library
- T. Scott Saponas, Chris Harrison, and Hrvoje Benko. 2011. PocketTouch: Through-fabric capacitive touch input. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 303--308. Google ScholarDigital Library
- Paul Schmieder, John Hosking, Andrew Luxton-Reilly, and Beryl Plimmer. 2013. Thumbs up: 3D gesture input on mobile phones using the front facing camera. In Human-Computer Interaction—INTERACT 2013. Lecture Notes in Computer Science, Vol. 8118. Springer, 318--336.Google Scholar
- Carsten Schwesig, Ivan Poupyrev, and Eijiro Mori. 2004. Gummi: A bendable computer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 263--270. Google ScholarDigital Library
- James Scott, Shahram Izadi, Leila Sadat Rezai, Dominika Ruszkowski, Xiaojun Bi, and Ravin Balakrishnan. 2010. RearType: Text entry using keys on the back of a device. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services. ACM, New York, NY, 171--180. Google ScholarDigital Library
- Masanori Sugimoto and Keiichi Hiroki. 2006. HybridTouch: An intuitive manipulation technique for PDAs using their front and rear surfaces. In Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, New York, NY, 137--140. Google ScholarDigital Library
- Weka. n.d. Weka 3: Data Mining Software in Java. Retrieved June 17, 2016, from http://www.cs.waikato.ac.nz/ml/weka/.Google Scholar
- Piotr Wendykier. n.d. JTransforms. Retrieved June 17, 2016, from https://sites.google.com/site/piotrwendykier/software/jtransforms.Google Scholar
- J. Wiese, T. S. Saponas, and A. J. Brush. 2013. Phoneprioception: Enabling mobile phones to infer where they are kept. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 2157--2166. Google ScholarDigital Library
- Daniel Wigdor, Clifton Forlines, Patrick Baudisch, John Barnwell, and Chia Shen. 2007. Lucid touch: A see-through mobile device. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 269--278. Google ScholarDigital Library
- Jacob O. Wobbrock, Brad A. Myers, and John A. Kembel. 2003. EdgeWrite: A stylus-based text entry method designed for high accuracy and stability of motion. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 61--70. Google ScholarDigital Library
- Zhi Xu, Kun Bai, and Sencun Zhu. 2012. Taplogger: Inferring user inputs on smartphone touchscreens using on-board motion sensors. In Proceedings of the 5th ACM Conference on Security and Privacy in Wireless and Mobile Networks. ACM, New York, NY, 113--124. Google ScholarDigital Library
- Cheng Zhang. 2013. BeyondTouch: A framework for extending input on commodity smartphones. In Proceedings of the Adjunct Publication of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp'13 Adjunct Publication).Google Scholar
- Cheng Zhang, Aman Parnami, Caleb Southern, Edison Thomaz, Gabriel Reyes, Rosa Arriaga, and Gregory D. Abowd. 2013. BackTap: Robust four-point tapping on the back of an off-the-shelf smartphone. In Proceedings of the Adjunct Publication of the 26th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 111--112. Google ScholarDigital Library
- Cheng Zhang, Anhong Guo, Dingtian Zhang, Caleb Southern, Rosa Arriaga, and Gregory Abowd. 2015. BeyondTouch: Extending the input language with built-in sensors on commodity smartphones. In Proceedings of the 20th International Conference on Intelligent User Interfaces. ACM, New York, NY, 67--77. Google ScholarDigital Library
Index Terms
- Beyond the Touchscreen: An Exploration of Extending Interactions on Commodity Smartphones
Recommendations
BeyondTouch: Extending the Input Language with Built-in Sensors on Commodity Smartphones
IUI '15: Proceedings of the 20th International Conference on Intelligent User InterfacesWhile most smartphones today have a rich set of sensors that could be used to infer input (.e.g., accelerometer, gyroscope, microphone), the primary mode of interaction is still limited to the front-facing touchscreen and several physical buttons on the ...
Detecting repackaged smartphone applications in third-party android marketplaces
CODASPY '12: Proceedings of the second ACM conference on Data and Application Security and PrivacyRecent years have witnessed incredible popularity and adoption of smartphones and mobile devices, which is accompanied by large amount and wide variety of feature-rich smartphone applications. These smartphone applications (or apps), typically organized ...
Learning Processes of Touchscreen Gesture Interaction in Older Adults and Children
HCI International 2023 – Late Breaking PapersAbstractTouchscreen mobile computers, including smartphones, are widely used today by people of all ages across the globe, from children to senior citizens. In Japan, significant numbers of seniors, such as active ones, have begun to learn how to use ...
Comments