research-article

Public Access

Exploring Context-Aware User Interfaces for Smartphone-Smartwatch Cross-Device Interaction

Authors:

Ryosuke Takada,

Buntarou Shizuki,

Shin TakahashiAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 3

Article No.: 69, Pages 1 - 21

https://doi.org/10.1145/3130934

Published: 11 September 2017 Publication History

Abstract

In this study, we explore context-aware cross-device interactions between a smartphone and smartwatch. We present 24 contexts, and then examine and prioritize suitable user interfaces (UIs) for each. In addition, we present example applications, including a map, notification management system, multitasking application, music player, and video chat application, each of which has its own context-aware UIs. To support these context-aware UIs, we investigate the performance of our context recognizer in which recognition is based on machine-learning using the accelerometers in a smartphone and smartwatch. We conduct seven different evaluations using four machine-learning algorithms: J48 decision tree, sequential minimal optimization (SMO)-based support vector machine (SVM), random forest, and multilayer perceptron. With each algorithm, we conduct a long-interval experiment to examine the level of accuracy at which each context is recognized using data previously collected for training. The results show that SMO-based SVM is suitable for recognizing the 24 contexts considered in this study.

Supplementary Material

kubo (kubo.zip)

Supplemental movie, appendix, image and software files for, Exploring Context-Aware User Interfaces for Smartphone-Smartwatch Cross-Device Interaction

Download
33.95 MB

References

[1]

Ian Anderson, Julie Maitland, Scott Sherwood, Louise Barkhuus, Matthew Chalmers, Malcolm Hall, Barry Brown, and Henk Muller. 2007. Shakra: Tracking and Sharing Daily Activity Levels with Unaugmented Mobile Phones. Mobile Networks and Applications 12, 2 (2007), 185--s199.

Digital Library

[2]

Ling Bao and Stephen S. Intille. 2004. Activity Recognition from User-Annotated Acceleration Data. In Proceedings of the 2nd International Conference on Pervasive Computing (Pervasive ’04). Springer, 1--17.

[3]

Sebastian Boring, David Ledo, Xiang ‘Anthony’ Chen, Nicolai Marquardt, Anthony Tang, and Saul Greenberg. 2012. The Fat Thumb: Using the Thumb’s Contact Size for Single-handed Mobile Interaction. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI ’12). ACM, New York, NY, USA, 39--48.

Digital Library

[4]

Agata Brajdic and Robert Harle. 2013. Walk Detection and Step Counting on Unconstrained Smartphones. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’13). ACM, New York, NY, USA, 225--234.

Digital Library

[5]

Xiang ‘Anthony’ Chen, Tovi Grossman, Daniel J. Wigdor, and George Fitzmaurice. 2014. Duet: Exploring Joint Interactions on a Smart Phone and a Smart Watch. In Proceedings of the 32nd SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 159--168.

Digital Library

[6]

Lung-Pan Cheng, Hsiang-Sheng Liang, Che-Yang Wu, and Mike Y. Chen. 2013. iGrasp: Grasp-based Adaptive Keyboard for Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). ACM, New York, NY, USA, 3037--3046.

Digital Library

[7]

Rajkumar Darbar and Debasis Samanta. 2015. SurfaceSense: Smartphone Can Recognize Where It Is Kept. In Proceedings of the 7th International Conference on HCI, IndiaHCI 2015 (IndiaHCI’15). ACM, New York, NY, USA, 39--46.

Digital Library

[8]

Anind K. Dey, Gregory D. Abowd, and Daniel Salber. 2001. A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-aware Applications. Human-Computer Interaction 16, 2 (Dec. 2001), 97--166.

Digital Library

[9]

Mayank Goel, Jacob Wobbrock, and Shwetak Patel. 2012. GripSense: Using Built-in Sensors to Detect Hand Posture and Pressure on Commodity Mobile Phones. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST ’12). ACM, New York, NY, USA, 545--554.

Digital Library

[10]

Mitsuhiro Goto, Hideaki Kimata, Masahiko Toyoshi, Tatsuya Nishikiori, Kosuke Moriwaki, and Koji Nakamura. 2015. A Wearable Action Support System for Business Use by Context-aware Computing Based on Web Schedule. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (UbiComp/ISWC’15 Adjunct). ACM, New York, NY, USA, 53--56.

Digital Library

[11]

Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations Newsletter 11, 1 (Nov. 2009), 10--18.

Digital Library

[12]

Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New York, NY, USA, 627--636.

Digital Library

[13]

Seongkook Heo, Jiseong Gu, and Geehyuk Lee. 2014. Expanding Touch Input Vocabulary by using Consecutive Distant Taps. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 2597--2606.

Digital Library

[14]

Ken Hinckley, Seongkook Heo, Michel Pahud, Christian Holz, Hrvoje Benko, Abigail Sellen, Richard Banks, Kenton O’Hara, Gavin Smyth, and William Buxton. 2016. Pre-Touch Sensing for Mobile Interaction. In Proceedings of the 34th SIGCHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 2869--2881.

Digital Library

[15]

Ken Hinckley, Michel Pahud, Hrvoje Benko, Pourang Irani, François Guimbretière, Marcel Gavriliu, Xiang ‘Anthony’ Chen, Fabrice Matulic, William Buxton, and Andrew Wilson. 2014. Sensing Techniques for Tablet+Stylus Interaction. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST ’14). ACM, New York, NY, USA, 605--614.

Digital Library

[16]

Ken Hinckley, Jeff Pierce, Mike Sinclair, and Eric Horvitz. 2000. Sensing Techniques for Mobile Interaction. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST ’00). ACM, New York, NY, USA, 91--100.

Digital Library

[17]

Han-Jong Kim, Seijin Cha, Richard C. Park, Tek-Jin Nam, Woohun Lee, and Geehyuk Lee. 2016. Mo-Bi: Contextual Mobile Interfaces Through Bimanual Posture Sensing with Wrist-Worn Devices. In Proceedings of HCI Korea (HCIK ’16). Hanbit Media, Inc., South Korea, 94--99.

Digital Library

[18]

Yuki Kubo, Ryosuke Takada, Buntarou Shizuki, and Shin Takahashi. 2017. SynCro: Context-Aware User Interface System for Smartphone-Smartwatch Cross-Device Interaction. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHIEA ’17). ACM, New York, NY, USA, 1794--1801.

Digital Library

[19]

Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. 2011. Activity Recognition using Cell Phone Accelerometers. SIGKDD Explorations Newsletter 12, 2 (March 2011), 74--82.

Digital Library

[20]

Hosub Lee, Young Sang Choi, Sunjae Lee, and Eunsoo Shim. 2013. Smart Pose: Mobile Posture-aware System for Lowering Physical Health Risk of Smartphone Users. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (CHIEA ’13). ACM, New York, NY, USA, 2257--2266.

Digital Library

[21]

Seungwoo Lee, Yungeun Kim, Daye Ahn, Rhan Ha, Kyoungwoo Lee, and Hojung Cha. 2015. Non-obstructive Room-level Locating System in Home Environments using Activity Fingerprints from Smartwatch. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 939--950.

Digital Library

[22]

Hyunchul Lim, Gwangseok An, Yoonkyong Cho, Kyogu Lee, and Bongwon Suh. 2016. WhichHand: Automatic Recognition of a Smartphone’s Position in the Hand using a Smartwatch. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI ’16). ACM, New York, NY, USA, 675--681.

Digital Library

[23]

Luyang Liu, Cagdas Karatas, Hongyu Li, Sheng Tan, Marco Gruteser, Jie Yang, Yingying Chen, and Richard P. Martin. 2015. Toward Detection of Unsafe Driving with Wearables. In Proceedings of the 1st Workshop on Wearable Systems and Applications (WearSys ’15). ACM, New York, NY, USA, 27--32.

Digital Library

[24]

Bilge Mutlu, Andreas Krause, Jodi Forlizzi, Carlos Guestrin, and Jessica Hodgins. 2007. Robust, Low-cost, Non-intrusive Sensing and Recognition of Seated Postures. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST ’07). ACM, New York, NY, USA, 149--158.

Digital Library

[25]

Woori Noh, Mankyung Lee, Hyelim Cheon, Joohee Kim, Kwangjae Lee, and Jundong Cho. 2016. TakeOut: Drawing Application using Distributed User Interface for Being Close to Real Experience. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp ’16).ACM, New York, NY, USA, 173--176.

Digital Library

[26]

Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch 8 Activate: Adding Interactivity to Existing Objects using Active Acoustic Sensing. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13). ACM, New York, NY, USA, 31--40.

Digital Library

[27]

Chanho Park and Takefumi Ogawa. 2015. A Study on Grasp Recognition Independent of Users’ Situations using Built-in Sensors of Smartphones. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST ’15 Adjunct). ACM, New York, NY, USA, 69--70.

Digital Library

[28]

Jun-geun Park, Ami Patel, Dorothy Curtis, Seth Teller, and Jonathan Ledlie. 2012. Online Pose Classification and Walking Speed Estimation using Handheld Devices. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp’12). ACM, New York, NY, USA, 113--122.

Digital Library

[29]

Anand Ranganathan and Roy H. Campbell. 2003. An Infrastructure for Context-awareness Based on First Order Logic. Personal Ubiquitous Comput. 7, 6 (Dec. 2003), 353--364.

Digital Library

[30]

Juhi Ranjan and Kamin Whitehouse. 2015. Object Hallmarks: Identifying Object Users using Wearable Wrist Sensors. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 51--61.

Digital Library

[31]

Jun Rekimoto. 1996. Tilting Operations for Small Screen Interfaces. In Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology (UIST ’96). ACM, New York, NY, USA, 167--168.

Digital Library

[32]

Jun Rekimoto. 1997. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (UIST ’97). ACM, New York, NY, USA, 31--39.

Digital Library

[33]

Nazir Saleheen, Amin Ahsan Ali, Syed Monowar Hossain, Hillol Sarker, Soujanya Chatterjee, Benjamin Marlin, Emre Ertin, Mustafa al’Absi, and Santosh Kumar. 2015. puffMarker: A Multi-sensor Approach for Pinpointing the Timing of First Lapse in Smoking Cessation. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 999--1010.

Digital Library

[34]

Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY, USA, 483--492.

Digital Library

[35]

Bill N. Schilit, Norman Adams, and Roy Want. 1994. Context-Aware Computing Applications. In Proceedings of the 1994 First Workshop on Mobile Computing Systems and Applications (WMCSA ’94). IEEE Computer Society, Washington, DC, USA, 85--90.

Digital Library

[36]

Albrecht Schmidt, Kofi Asante Aidoo, Antti Takaluoma, Urpo Tuomela, Kristof Van Laerhoven, and Walter Van de Velde. 1999. Advanced Interaction in Context. In Proceedings of the 1st International Symposium on Handheld and Ubiquitous Computing (HUC ’99). Springer-Verlag, London, UK, UK, 89--101. http://dl.acm.org/citation.cfm?id=647985.743849

Digital Library

[37]

Dominik Schmidt, Julian Seifert, Enrico Rukzio, and Hans Gellersen. 2012. A Cross-device Interaction Style for Mobiles and Surfaces. In Proceedings of the Designing Interactive Systems Conference (DIS ’12). ACM, New York, NY, USA, 318--327.

Digital Library

[38]

Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I Am a Smartwatch and I Can Track My User’s Arm. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ’16). ACM, New York, NY, USA, 85--96.

Digital Library

[39]

Katsuma Tanaka, Shoya Ishimaru, Koichi Kise, Kai Kunze, and Masahiko Inami. 2015. Nekoze!: Monitoring and Detecting Head Posture While Working with Laptop and Mobile Phone. In Proceedings of the 9th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth ’15). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 237--240. http://dl.acm.org/citation.cfm?id=2826165.2826203

Digital Library

[40]

Edison Thomaz, Irfan Essa, and Gregory D. Abowd. 2015. A Practical Approach for Recognizing Eating Moments with Wrist-mounted Inertial Sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 1029--1040.

Digital Library

[41]

Raphael Wimmer and Sebastian Boring. 2009. HandSense: Discriminating Different Ways of Grasping and Holding a Tangible User Interface. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (TEI ’09). ACM, New York, NY, USA, 359--362.

Digital Library

[42]

Yoonsik Yang, Seungho Chae, Jinwook Shim, and Tack-Don Han. 2015. EMG Sensor-based Two-Hand Smart Watch Interaction. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST ’15 Adjunct). ACM, New York, NY, USA, 73--74.

Digital Library

[43]

Dongwook Yoon, Ken Hinckley, Hrvoje Benko, François Guimbretière, Pourang Irani, Michel Pahud, and Marcel Gavriliu. 2015. Sensing Tablet Grasp + Micro-mobility for Active Reading. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST ’15). ACM, New York, NY, USA, 477-487.

Digital Library

Cited By

Chang TSu LLin YTsai JTang ZChang Y(2024)"I Want Lower Tone for Work-Related Notifications": Exploring the Effectiveness of User-Assigned Notification Alerts in Improving User Speculation of and Attendance to Mobile NotificationsProceedings of the ACM on Human-Computer Interaction10.1145/36765128:MHCI(1-25)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676512
Deng HJiang JYu ZOuyang JWu D(2024)CrossGAIProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435428:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643542
Cheng QTian XLiao FChen TChen XWang ZDong YLiu Y(2024)Enhancing Mobile Interaction: Practical Insights from Smartphone and Smartwatch IntegrationAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680451(1-9)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3640471.3680451
Show More Cited By

Index Terms

Exploring Context-Aware User Interfaces for Smartphone-Smartwatch Cross-Device Interaction
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. Activity centered design
      2. Contextual design
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices
      1. Mobile devices

Recommendations

SynCro: Context-Aware User Interface System for Smartphone-Smartwatch Cross-Device Interaction
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

We present a context-aware user interface system, called SynCro, comprising a smartphone and a smartwatch. SynCro provides the user with context-dependent user interfaces (UIs), and will synthesize layouts, feedback, or input methods of these devices ...
Post-transitioning user performance on cross-device menu interfaces
Highlights
- First study to investigate user performance with multiple menu interfaces across devices with focusing on performance after transitioning from one device to ...
Abstract
Multi-device user-interaction behavior creates a need to design cross-device menus that support users to re-locate menu items efficiently and effectively after the transition from one device to another. We conducted two laboratory ...
Exploring Interactions with a Flexible Tactile Device for Multi-Context Interaction
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

The use of touch-based interaction now dominates the market as it enables a more natural form of interaction with our devices, but is largely limited to flat and rigid surfaces. The increasing availability of flexible interactive technologies offers us ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 3

September 2017

2023 pages

EISSN:2474-9567

DOI:10.1145/3139486

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2017

Accepted: 01 June 2017

Revised: 01 May 2017

Received: 01 February 2017

Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Japan Science and Technology Agency

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
1,461
Total Downloads

Downloads (Last 12 months)195
Downloads (Last 6 weeks)16

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chang TSu LLin YTsai JTang ZChang Y(2024)"I Want Lower Tone for Work-Related Notifications": Exploring the Effectiveness of User-Assigned Notification Alerts in Improving User Speculation of and Attendance to Mobile NotificationsProceedings of the ACM on Human-Computer Interaction10.1145/36765128:MHCI(1-25)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676512
Deng HJiang JYu ZOuyang JWu D(2024)CrossGAIProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435428:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643542
Cheng QTian XLiao FChen TChen XWang ZDong YLiu Y(2024)Enhancing Mobile Interaction: Practical Insights from Smartphone and Smartwatch IntegrationAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680451(1-9)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3640471.3680451
Yin YXie LJiang ZXiao FCao JLu S(2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2024.3357591
Lu JLiu YShen SRen H(2024)Exploration of a method for insight into accessibility design flaws based on touch dynamicsHumanities and Social Sciences Communications10.1057/s41599-024-03234-811:1Online publication date: 1-Jun-2024
https://doi.org/10.1057/s41599-024-03234-8
Ale LZhang NKing SChen D(2024)Empowering generative AI through mobile edge computingNature Reviews Electrical Engineering10.1038/s44287-024-00053-61:7(478-486)Online publication date: 14-Jun-2024
https://doi.org/10.1038/s44287-024-00053-6
Liu NGong Y(2024)Seamless switching, feedforward, and feedback mechanisms: Enhancing task performance and user perception in device switchInternational Journal of Industrial Ergonomics10.1016/j.ergon.2024.103649104(103649)Online publication date: Nov-2024
https://doi.org/10.1016/j.ergon.2024.103649
Inácio ECapani F(2023)New therapies in the treatment of Perinatal Asphyxia: A systematic reviewSCT Proceedings in Interdisciplinary Insights and Innovations10.56294/piii20231691(169)Online publication date: 18-Nov-2023
https://doi.org/10.56294/piii2023169
Markopoulos PKhanshan ABormans STisza GKang LVan Gorp P(2023)Comparative Evaluation of Touch-Based Input Techniques for Experience Sampling on SmartwatchesProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627768(74-86)Online publication date: 3-Dec-2023
https://dl.acm.org/doi/10.1145/3626705.3627768
Chen TZhu KYang M(2023)Deep-learning-based unobtrusive handedness prediction for one-handed smartphone interactionMultimedia Tools and Applications10.1007/s11042-021-11844-682:4(4941-4964)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1007/s11042-021-11844-6
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents