Thomas Weber
Sven Mayer
ABSTRACT
Highly complex systems, such as software development tools, constantly gain features and, consequently, complexity and, thus, risk overwhelming or distracting the user. We argue that automation and adaptation could help users to focus on their work. However, the challenge is to correctly and promptly determine when to adapt what, as often the users’ intent is unclear. To assist software developers, we build a gaze-adaptive integrated development environment using the developers’ gaze as the source for learning appropriate adaptation. Beyond our experience of using gaze for an adaptive user interface, we also report first feedback from developers regarding the desirability of such a user interface, which indicated that adaptations for development tools need to strike a careful balance between automation and user control. Nonetheless, the developers see the value in a gaze-based adaptive user interface and how it could improve software development tools going forward.
- Pierre A. Akiki, Arosha K. Bandara, and Yijun Yu. 2014. Adaptive Model-Driven User Interface Development Systems. ACM Comput. Surv. 47, 1 (2014), 9:1–9:33. https://doi.org/10.1145/2597999Google Scholar
Digital Library
- Sven Amann, Sebastian Proksch, and Sarah Nadi. 2016. FeedBaG: An interaction tracker for Visual Studio. In 24th IEEE International Conference on Program Comprehension, ICPC 2016, Austin, TX, USA, May 16-17, 2016. IEEE Computer Society, New York, NY, USA, 1–3. https://doi.org/10.1109/ICPC.2016.7503741Google ScholarCross Ref
- Roman Bednarik. 2012. Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. Int. J. Hum. Comput. Stud. 70, 2 (2012), 143–155. https://doi.org/10.1016/j.ijhcs.2011.09.003Google ScholarDigital Library
- Roman Bednarik and Markku Tukiainen. 2006. An eye-tracking methodology for characterizing program comprehension processes. In Proceedings of the Eye Tracking Research & Application Symposium, ETRA 2006, San Diego, California, USA, March 27-29, 2006, Kari-Jouko Räihä and Andrew T. Duchowski (Eds.). ACM, 125–132. https://doi.org/10.1145/1117309.1117356Google ScholarDigital Library
- David Benyon. 1993. Accommodating Individual Differences through an Adaptive User Interface. Human Factors in Information Technology 10 (1993), 149–149.Google Scholar
- Tyson Bulmer, Lloyd Montgomery, and Daniela Damian. 2018. Predicting Developers’ IDE Commands with Machine Learning. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). ACM, New York, NY, USA, 82–85. https://doi.org/10.1145/3196398.3196459Google ScholarDigital Library
- Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye Movements in Code Reading: Relaxing the Linear Order. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension(ICPC ’15). IEEE Computer Society, USA, 255–265. https://doi.org/10.1109/ICPC.2015.36Google ScholarDigital Library
- Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. 2003. A Unifying Reference Framework for multi-target user interfaces. Interact. Comput. 15, 3 (2003), 289–308. https://doi.org/10.1016/S0953-5438(03)00010-9Google ScholarCross Ref
- Jose Manuel Cantera Fonseca. 2010. Model-Based UI XG Final Report.Google Scholar
- Mon Chu Chen, John R. Anderson, and Myeong Ho Sohn. 2001. What Can a Mouse Cursor Tell Us More? Correlation of Eye/Mouse Movements on Web Browsing. In CHI ’01 Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington) (CHI EA ’01). ACM, New York, NY, USA, 281–282. https://doi.org/10.1145/634067.634234Google ScholarDigital Library
- Cristina Conati, Christina Merten, Saleema Amershi, and Kasia Muldner. 2007. Using Eye-Tracking Data for High-Level User Modeling in Adaptive Interfaces. In Proceedings of the 22nd National Conference on Artificial Intelligence - Volume 2 (Vancouver, British Columbia, Canada) (AAAI’07). AAAI Press, California, USA, 1614–1617.Google Scholar
- Edwin S. Dalmaijer, Sebastiaan Mathôt, and Stefan Van der Stigchel. 2014. PyGaze: An open-source, cross-platform toolbox for minimal-effort programming of eyetracking experiments. Behavior Research Methods 46, 4 (01 Dec 2014), 913–921. https://doi.org/10.3758/s13428-013-0422-2Google ScholarCross Ref
- Kostadin Damevski, Hui Chen, David C. Shepherd, Nicholas A. Kraft, and Lori Pollock. 2018. Predicting Future Developer Behavior in the IDE Using Topic Models. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). ACM, New York, NY, USA, 932. https://doi.org/10.1145/3180155.3182541Google ScholarDigital Library
- Charles-Eric Dessart, Vivian Genaro Motti, and Jean Vanderdonckt. 2011. Showing User Interface Adaptivity by Animated Transitions. In Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Pisa, Italy) (EICS ’11). ACM, New York, NY, USA, 95–104. https://doi.org/10.1145/1996461.1996501Google ScholarDigital Library
- Steven M. Drucker, Danyel Fisher, and Sumit Basu. 2011. Helping Users Sort Faster with Adaptive Machine Learning Recommendations. In Human-Computer Interaction – INTERACT 2011. Springer Berlin Heidelberg, Berlin, Heidelberg, 187–203. https://doi.org/10.1007/978-3-642-23765-2_13Google ScholarCross Ref
- Leah Findlater and Krzysztof Z. Gajos. 2009. Design Space and Evaluation Challenges of Adaptive Graphical User Interfaces. AI Mag. 30, 4 (2009), 68–73. https://doi.org/10.1609/aimag.v30i4.2268Google ScholarDigital Library
- Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S. Weld. 2006. Exploring the Design Space for Adaptive Graphical User Interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (Venezia, Italy) (AVI ’06). ACM, New York, NY, USA, 201–208. https://doi.org/10.1145/1133265.1133306Google ScholarDigital Library
- Marko Gasparic, Tural Gurbanov, and Francesco Ricci. 2017. Context-aware integrated development environment command recommender systems. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 688–693. https://doi.org/10.1109/ASE.2017.8115679Google ScholarCross Ref
- Christoph Gebhardt, Brian Hecox, Bas van Opheusden, Daniel Wigdor, James Hillis, Otmar Hilliges, and Hrvoje Benko. 2019. Learning Cooperative Personalized Policies from Gaze Data. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). ACM, New York, NY, USA, 197–208. https://doi.org/10.1145/3332165.3347933Google ScholarDigital Library
- Panagiotis Germanakos, Marios Belk, Argyris Constantinides, and George Samaras. 2015. The PersonaWeb System: Personalizing E-Commerce Environments based on Human Factors. In Posters, Demos, Late-breaking Results and Workshop Proceedings of the 23rd Conference on User Modeling, Adaptation, and Personalization(UMAP’15 Workshop Proceedings, Vol. 1388), Alexandra I. Cristea, Judith Masthoff, Alan Said, and Nava Tintarev (Eds.). CEUR-WS.org, Online, 4.Google Scholar
- Fabian Göbel, Peter Kiefer, Ioannis Giannopoulos, Andrew T. Duchowski, and Martin Raubal. 2018. Improving Map Reading with Gaze-Adaptive Legends. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (Warsaw, Poland) (ETRA ’18). ACM, New York, NY, USA, Article 29, 9 pages. https://doi.org/10.1145/3204493.3204544Google ScholarDigital Library
- Gunnar Harboe and Elaine M. Huang. 2015. Real-World Affinity Diagramming Practices: Bridging the Paper-Digital Gap. In Proc. 33rd Annual ACM Conf. Human Factors in Computing Systems. ACM, New York, NY, USA, 95–104. https://doi.org/10.1145/2702123.2702561Google ScholarDigital Library
- Prateek Hejmady and N. Hari Narayanan. 2012. Visual attention patterns during program debugging with an IDE. In Proceedings of the 2012 Symposium on Eye-Tracking Research and Applications, ETRA 2012, Santa Barbara, CA, USA, March 28-30, 2012, Carlos Hitoshi Morimoto, Howell O. Istance, Stephen N. Spencer, Jeffrey B. Mulligan, and Pernilla Qvarfordt (Eds.). ACM, 197–200. https://doi.org/10.1145/2168556.2168592Google ScholarDigital Library
- Anthony David Jameson. 2009. Understanding and Dealing With Usability Side Effects of Intelligent Processing. AI Mag. 30, 4 (2009), 23–40. https://doi.org/10.1609/aimag.v30i4.2274Google ScholarDigital Library
- Simeon Keates, P. John Clarkson, Lee-Anne Harrison, and Peter Robinson. 2000. Towards a Practical Inclusive Design Approach. In Proceedings on the 2000 Conference on Universal Usability (Arlington, Virginia, USA) (CUU ’00). ACM, New York, NY, USA, 45–52. https://doi.org/10.1145/355460.355471Google ScholarDigital Library
- Rex Bryan Kline and Ahmed Seffah. 2005. Evaluation of integrated software development environments: Challenges and results from three empirical studies. Int. J. Hum. Comput. Stud. 63, 6 (2005), 607–627. https://doi.org/10.1016/j.ijhcs.2005.05.002Google ScholarDigital Library
- Sucheta V. Kolekar, Radhika M. Pai, and M. M. Manohara Pai. 2019. Rule based adaptive user interface for adaptive E-learning system. Educ. Inf. Technol. 24, 1 (2019), 613–641. https://doi.org/10.1007/s10639-018-9788-1Google ScholarDigital Library
- Pat Langley. 1997. Machine Learning for Adaptive User Interfaces. In KI-97: Advances in Artificial Intelligence, 21st Annual German Conference on Artificial Intelligence, Freiburg, Germany, September 9-12, 1997, Proceedings(Lecture Notes in Computer Science, Vol. 1303). Springer, Cham, Switzerland, 53–62. https://doi.org/10.1007/3540634932_3Google ScholarCross Ref
- Jiming Liu, Kelvin Chi Kuen Wong, and Ka Keung Hui. 2003. An Adaptive User Interface Based On Personalized Learning. IEEE Intell. Syst. 18, 2 (2003), 52–57. https://doi.org/10.1109/MIS.2003.1193657Google ScholarDigital Library
- Ronald Mace. 1991. Accessible Environments : Toward Universal Design. Design Interventions : Toward A More Humane Architecture 156 (1991), 2.Google Scholar
- James S. Magnuson. 2005. Moving hand reveals dynamics of thought. Proceedings of the National Academy of Sciences 102, 29 (2005), 9995–9996. https://doi.org/10.1073/pnas.0504413102 arXiv:https://www.pnas.org/content/102/29/9995.full.pdfGoogle ScholarCross Ref
- Nesrine Mezhoudi. 2013. User Interface Adaptation Based on User Feedback and Machine Learning. In Proceedings of the Companion Publication of the 2013 International Conference on Intelligent User Interfaces Companion (Santa Monica, California, USA) (IUI ’13 Companion). ACM, New York, NY, USA, 25–28. https://doi.org/10.1145/2451176.2451184Google ScholarDigital Library
- J. Mitchell and B. Shneiderman. 1989. Dynamic versus Static Menus: An Exploratory Comparison. SIGCHI Bull. 20, 4 (apr 1989), 33–37. https://doi.org/10.1145/67243.67247Google ScholarDigital Library
- Emerson Murphy-Hill, Rahul Jiresal, and Gail C. Murphy. 2012. Improving Software Developers’ Fluency by Recommending Development Environment Commands. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (Cary, North Carolina) (FSE ’12). ACM, New York, NY, USA, Article 42, 11 pages. https://doi.org/10.1145/2393596.2393645Google ScholarDigital Library
- Jeanne Nakamura and Mihaly Csikszentmihalyi. 2014. The Concept of Flow. In Flow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi. Springer Netherlands, Dordrecht, 239–263. https://doi.org/10.1007/978-94-017-9088-8_16Google ScholarCross Ref
- Matthias Peissner and Rob Edlin-White. 2013. User Control in Adaptive User Interfaces for Accessibility. In Human-Computer Interaction – INTERACT 2013. Springer Berlin Heidelberg, Berlin, Heidelberg, 623–640. https://doi.org/10.1007/978-3-642-40483-2_44Google ScholarCross Ref
- Norman Peitek, Janet Siegmund, and Sven Apel. 2020. What Drives the Reading Order of Programmers? An Eye Tracking Study. In Proceedings of the 28th International Conference on Program Comprehension (Seoul, Republic of Korea) (ICPC ’20). ACM, New York, NY, USA, 342–353. https://doi.org/10.1145/3387904.3389279Google ScholarDigital Library
- Ken Pfeuffer, Yasmeen Abdrabou, Augusto Esteves, Radiah Rivu, Yomna Abdelrahman, Stefanie Meitner, Amr Saadi, and Florian Alt. 2021. ARtention: A design space for gaze-adaptive user interfaces in augmented reality. Comput. Graph. 95 (2021), 1–12. https://doi.org/10.1016/j.cag.2021.01.001Google ScholarCross Ref
- Ken Pfeuffer, Jason Alexander, Ming Ki Chong, Yanxia Zhang, and Hans Gellersen. 2015. Gaze-Shifting: Direct-Indirect Input with Pen and Touch Modulated by Gaze. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 373–383. https://doi.org/10.1145/2807442.2807460Google ScholarDigital Library
- Paige Rodeghero, Collin McMillan, Paul W. McBurney, Nigel Bosch, and Sidney K. D’Mello. 2014. Improving automated source code summarization via an eye-tracking study of programmers. In 36th International Conference on Software Engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014, Pankaj Jalote, Lionel C. Briand, and André van der Hoek (Eds.). ACM, 390–401. https://doi.org/10.1145/2568225.2568247Google ScholarDigital Library
- Jean Michel Rouly, Jonathan D. Orbeck, and Eugene Syriani. 2014. Usability and Suitability Survey of Features in Visual Ides for Non-Programmers. In Proceedings of the 5th Workshop on Evaluation and Usability of Programming Languages and Tools (Portland, Oregon, USA) (PLATEAU ’14). ACM, New York, NY, USA, 31–42. https://doi.org/10.1145/2688204.2688207Google ScholarDigital Library
- Mazeiar Salehie and Ladan Tahvildari. 2009. Self-adaptive software: Landscape and research challenges. ACM Trans. Auton. Adapt. Syst. 4, 2 (2009), 14:1–14:42. https://doi.org/10.1145/1516533.1516538Google ScholarDigital Library
- Matthias Schmidmaier, Zhiwei Han, Thomas Weber, Yuanting Liu, and Heinrich Hußmann. 2019. Real-Time Personalization in Adaptive IDEs. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization (Larnaca, Cyprus) (UMAP’19 Adjunct). ACM, New York, NY, USA, 81–86. https://doi.org/10.1145/3314183.3324975Google ScholarDigital Library
- Timothy R. Shaffer, Jenna L. Wise, Braden M. Walters, Sebastian C. Müller, Michael Falcone, and Bonita Sharif. 2015. ITrace: Enabling Eye Tracking on Software Artifacts within the IDE to Support Software Engineering Tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). ACM, New York, NY, USA, 954–957. https://doi.org/10.1145/2786805.2803188Google ScholarDigital Library
- Daniel J. Simons and Daniel T. Levin. 1997. Change blindness. Trends in Cognitive Sciences 1, 7 (01 Oct 1997), 261–267. https://doi.org/10.1016/S1364-6613(97)01080-2Google ScholarCross Ref
- Constantine Stephanidis. 1997. Towards the Next Generation of UIST: Developing for All Users. In Proceedings of the Seventh International Conference on Human-Computer Interaction-Volume 1 - Volume I(HCI International ’97). Elsevier Science Inc., USA, 473–476.Google Scholar
- Tensorflow. 2022. Time Series Forecasting.Google Scholar
- Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. 2021. Adapting User Interfaces with Model-Based Reinforcement Learning. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). ACM, New York, NY, USA, Article 573, 13 pages. https://doi.org/10.1145/3411764.3445497Google ScholarDigital Library
- Thomas Weber and Heinrich Hußmann. 2022. Tooling for Developing Data-Driven Applications: Overview and Outlook. In Proceedings of Mensch Und Computer 2022 (Darmstadt, Germany) (MuC ’22). Association for Computing Machinery, New York, NY, USA, 66–77. https://doi.org/10.1145/3543758.3543779Google ScholarDigital Library
- Thomas Weber, Christina Winiker, and Heinrich Hussmann. 2021. A Closer Look at Machine Learning Code. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). ACM, New York, NY, USA, Article 338, 6 pages. https://doi.org/10.1145/3411763.3451679Google ScholarDigital Library
- Daniel S. Weld, Corin Anderson, Pedro Domingos, Oren Etzioni, Krzysztof Gajos, Tessa Lau, and Steve Wolfman. 2003. Automatically Personalizing User Interfaces. In Proceedings of the 18th International Joint Conference on Artificial Intelligence (Acapulco, Mexico) (IJCAI’03). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1613–1619.Google ScholarDigital Library
- Enes Yigitbas, Stefan Sauer, and Gregor Engels. 2017. Adapt-UI: An IDE Supporting Model-Driven Development of Self-Adaptive UIs. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Lisbon, Portugal) (EICS ’17). ACM, New York, NY, USA, 99–104. https://doi.org/10.1145/3102113.3102144Google ScholarDigital Library
- Enes Yigitbas, Hagen Stahl, Stefan Sauer, and Gregor Engels. 2017. Self-adaptive UIs: Integrated Model-Driven Development of UIs and Their Adaptations. In Modelling Foundations and Applications, Anthony Anjorin and Huáscar Espinoza (Eds.). Springer International Publishing, Cham, 126–141.Google Scholar
- Guanhua Zhang, Susanne Hindennach, Jan Leusmann, Felix Bühler, Benedict Steuerlein, Sven Mayer, Mihai Bâce, and Andreas Bulling. 2022. Predicting Next Actions and Latent Intents during Text Formatting. In Proceedings of the CHI Workshop Computational Approaches for Understanding, Generating, and Adapting User Interfaces. University of Stuttgart, Stuttgart, Germany, 1–6.Google Scholar
- Nan Zheng, Aaron Paloski, and Haining Wang. 2016. An Efficient User Verification System Using Angle-Based Mouse Movement Biometrics. ACM Trans. Inf. Syst. Secur. 18, 3 (2016), 11:1–11:27. https://doi.org/10.1145/2893185Google ScholarDigital Library
Index Terms
- Supporting Software Developers Through a Gaze-Based Adaptive IDE
Recommendations
The Eye in Extended Reality: A Survey on Gaze Interaction and Eye Tracking in Head-worn Extended Reality
With innovations in the field of gaze and eye tracking, a new concentration of research in the area of gaze-tracked systems and user interfaces has formed in the field of Extended Reality (XR). Eye trackers are being used to explore novel forms of spatial ...
"Leagile" software development
In recent years there has been a noticeable shift in attention from those who use agile software development toward lean software development, often labelled as a shift "from agile to lean". However, the reality may not be as simple or linear as this ...
Investigating the interplay between developers and automation
ICSE '21: Proceedings of the 43rd International Conference on Software Engineering: Companion ProceedingsContinuous practices are a staple of the modern software development workflow. Automation, in particular, is widely adopted due to its benefits related to quality and productivity. However, automation, similarly to all other aspects of the software ...
Comments