Graham Dove
Kim Halskov
John Zimmerman
ABSTRACT
Machine learning (ML) is now a fairly established technology, and user experience (UX) designers appear regularly to integrate ML services in new apps, devices, and systems. Interestingly, this technology has not experienced a wealth of design innovation that other technologies have, and this might be because it is a new and difficult design material. To better understand why we have witnessed little design innovation, we conducted a survey of current UX practitioners with regards to how new ML services are envisioned and developed in UX practice. Our survey probed on how ML may or may not have been a part of their UX design education, on how they work to create new things with developers, and on the challenges they have faced working with this material. We use the findings from this survey and our review of related literature to present a series of challenges for UX and interaction design research and education. Finally, we discuss areas where new research and new curriculum might help our community unlock the power of design thinking to re-imagine what ML might be and might do.
- James F. Allen, Curry I. Guinn, and Eric Horovitz. 1999. Mixed-initiative interaction: IEEE Intelligent Systems and their Applications: 14--23. Google Scholar
Digital Library
- Robbert-Jan Beun, Eveliene De Vos, and Cilia Witteman. 2003. Embodied conversational agents: effects on memory performance and anthropomorphisation. In International Workshop on Intelligent Virtual Agents, 315--319. Google ScholarCross Ref
- Nick Bostrom, and Eliezer Yudkowsky. 2014. The ethics of artificial intelligence. In The Cambridge Handbook of Artificial Intelligence, Keith Frankish and William M Ramsey (eds), Cambridge University Press, Cambridge: 316--334 Google ScholarCross Ref
- Sheryl Brahnam, and Antonella De Angeli. 2012. Gender affordances of conversational agents: Interacting With Computers: 139--153. Google ScholarDigital Library
- Virginia Braun, and Victoria Clarke. 2006. Using thematic analysis in psychology: Qualitative research in psychology 3, 2: 77--101. Google ScholarCross Ref
- John Brownlee. 2015. Apple finaly learns AI is the new UI. Retrieved August 1, 2016 from http://www.fastcodesign.com/3047199/apple-finallylearns-ai-is-the-new-uiGoogle Scholar
- Marion Buchenau, and Jane Fulton Suri. 2000. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, 424433. Google ScholarDigital Library
- Bill Buxton, 2007. Sketching user experiences. Morgan Kaufmann,.Google Scholar
- Stephen M Casner, , Edwin L Hutchins, and Don Norman. 2016 The challenges of partially automated driving. Communications of the ACM 59, 5: 70--77.Google ScholarDigital Library
- Justine Cassell. 2000. Emboddied conversational interface agents. Communications of the ACM 43, 4: 70--78. Google ScholarDigital Library
- Guanling Chen and David Kotz. 2000 A survey of context-aware mobile computing research. Dept. of Computer Science, Dartmouth College, Dartmouth.Google Scholar
- Giles Colborne. 2016. Interaction design in the age of algorithms. Retrieved August 8, 2016 from https://www.cxpartners.co.uk/our-thinking/interactiondesign-in-the-age-of-algorithms/Google Scholar
- Alan Cooper, Robert Reimann, David Cronin, and Christopher Noessel. 2014. About face: the essentials of interaction design. John Wiley & Sons,.Google ScholarDigital Library
- Scott Davidoff, Brian D Ziebart, John Zimmerman, and Anind K Dey. 2011. Learning patterns of pick-ups and drop-offs to support busy family coordination. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.(CHI'11): 1175--1184.Google ScholarDigital Library
- Eva Deckers, Pierre Levy, Stephan Wensveen, Rene Ahn, and Kees Overbeeke. 2012. Designing for perceptual crossing: Applying and evaluating design notions. International Journal of Design 6, 3: 41--55.Google Scholar
- Pedro Domingos. 2012. A few useful things to know about machine learning. Communications of the ACM 55,10: 78--87. Google ScholarDigital Library
- Afsaneh Doryab, Jun Ki Min, Jason Wiese, John Zimmerman, and Jason I Hong. 2014. Detection of behavior change in people with depression. In Proceedings of AAAI Workshop on Modern Artificial Intelligence for Health Analytics (MAIHA).Google Scholar
- Graham Dove, and Sara Jones. 2014. Using information visualization to stimulate creativity in service design workshops.In Proceedings of the fourth Service Design and Service Innovation Conference. (ServDes14): 281--290.Google Scholar
- Pelle Ehn, and Morten Kyng. 1991. Cardboard computers. In Design at Work, Joan Greenbaum and Morten Kyng (eds) Lawrence Erlbaum Associates: 169--197.Google Scholar
- Harry Enten. 2016. Election update: where polls and demographics disagree. Retrieved September 14, 2016 from http://fivethirtyeight.com/features/electionupdate-where-polls-and-demographics-disagree/Google Scholar
- Peter Flach. 2012. Machine learning: the art and science of algorithms that make sense of data. Cambridge University Press. Google ScholarCross Ref
- Jodi Forlizzi, John Zimmerman, Vince Mancuso, and Sonya Kwak. 2007. How interface agents affect interaction between humans and computers. In Proceedings of the 2007 conference on Designing pleasurable products and interfaces: 209--221. Google ScholarDigital Library
- Marco Gillies, et al. 2016. Human-centered machine learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI'16): 3558--3565.Google ScholarDigital Library
- Google. 2016. Breakthroughs in Machine Learning Google I/O 2016. Retrieved August 1, 2016 from https://www.youtube.com/watch?v=sphFCJE1HkIGoogle Scholar
- Jonathan Grudin. 2006. Turing maturing: the separation of artificial intelligence and humancomputer interaction. In Interactions: 54--57. Google ScholarDigital Library
- Bjarki Hallgrimsson. 2012. Prototyping and modelmaking for product design. Laurence King,.Google Scholar
- Harrison, Chris, Desney Tan, and Dan Morris. "Skinput: appropriating the body as an input surface." Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Atlanta: ACM, 2010. 453--462. Google ScholarDigital Library
- Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. (CSCW): 241--250.Google ScholarDigital Library
- Geoffrey E Hinton. 2007. Learning multiple layers of representation. Trends in Cognitive Science 11,10: 428434.Google ScholarCross Ref
- Chien-Ming Huang, and Bilge Mutlu. 2012. Robot behavior toolkit: generating effective social behaviors for robots. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. (HRI'12): 25--32. Google ScholarDigital Library
- Scott E Hudson, et al. 2003. Predicting human interruptibility with sensors: a wizard of oz feasibility study. InThe CHI 2003 New Horizons Conference Proceedings: Conference on Human Factors in Computing Systems. (CHI'03): 257--264.Google ScholarDigital Library
- Francisco Iacobelli, and Justine Cassell. 2007. Ethnic identity and engagement in embodied conversational agents. In International Workshop on Intelligent Virtual Agents: 57--63. Google ScholarDigital Library
- Rabia Kahn and Antonella De Angeli. 2009. The attractiveness stereotype in the evaluation of embodied conversational agents. IFIP Conference on HumanComputer Interaction (INTERACT 2009): 85--97. Google ScholarDigital Library
- Yunkyung Kim, and Bilge Mutlu. 2014. How social distance shapes human-robot interaction. International Journal of Human-Computer Studies 72: 783--795. Google ScholarCross Ref
- Kenneth R Koedinger, and A Corbett. 2006. Cognitive tutors. In The Cambridge handbook of the learning sciences, Keith R Sawyer (ed) Cambridge University Press: 61--77.Google Scholar
- Pat Langley. 1997. Machine learning for adaptive user interfaces. In Annual conference on Artificial Intelligence: 53--62. Google ScholarCross Ref
- Min Kyung Lee, Sara Kiesler, Jodi Forlizzi, Siddhartha Srinivasa, and Paul E Rybski. 2010. Gracefully mitigating breakdowns in robotic services. In 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI): 203--210. Google ScholarCross Ref
- Ewa Luger, and Abigail Sellen. 2016. Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI'16): 5286--5297. Google ScholarDigital Library
- Media Lab Helsiki. A quick primer for ethics in design. Retrieved September 15 2016 from http://mlab.uiah.fi/polut/Yhteiskunnalliset/lisatieto_ethi cs_primer.htmlGoogle Scholar
- Camille Mousette, and Richard Banks. 2011. Designing through making: exploring the simple haptic design space. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction: 279--282. Google ScholarDigital Library
- Clifford Nass, Jonathan Steuer, and Ellen R Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. (CHI'94): 72--78.Google ScholarDigital Library
- William Odom, John Zimmerman, Scott Davidoff, Jodi Forlizzi, Anind K Dey, and Min Kyung Lee. 2012. A fieldwork of the future with user enactments. In Proceedings of the Designing Interactive Systems Conference. (DIS'12): 338--347.Google ScholarDigital Library
- Fatih Kursat Ozene, , Miso Kim, John Zimmerman, Stephen Oney, and Brad Myers. 2010. How to support designers in getting hold of the immaterial material of software. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (CHI'10): 2513--2522.Google ScholarDigital Library
- Victor Papanek. 1995. The Green Imperative: Ecology and Ethics in Design and Architecture. Thames and Hudson.Google Scholar
- Pamela Pavliscak. 2016. Algorithms as the new material of design. Retrieved September 14th from http://www.uxmatters.com/mt/archives/2016/06/algorit hms-as-the-new-material-of-design.phpGoogle Scholar
- Jenny Preece, Helen Sharp, and Yvonne Rogers. 2015. Interaction design: beyond human-computer interaction. John Wiley & Sons.Google ScholarDigital Library
- Frank Rosenblatt. 1961. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Cornell Aeronautical Lab.Google Scholar
- Arthur L Samuel,.1959. Some studies in machine learning using the game of checkers. IBM Journal of research and development 3, 3: 210--229.Google Scholar
- Ben Shneiderman, and Patti Maes. 1997. Direct manipulation vs. interface agents. Interactions: 42--61.Google Scholar
- Yale Song, David Demirdjian, and Randall Davis. 2012. Continuous body and hand gesture recognition for natural human-computer interaction. ACM Transactions on Interactive Intelligent Systems 2, 1: 5.Google ScholarDigital Library
- Aaron Spaulding, , Krzystof Z Gajos, Anthony Jameson, Per Ola Kristensson, Andrea Bunt, and Will Haines. 2009. Usable intelligent interactive systems. In CHI'09 Extended Abstracts on Human Factors in Computing Systems.Google Scholar
- Swedish ICT. Retrieved September 15th 2016 from https://www.tii.se/projects/ittextilesGoogle Scholar
- Terry Winograd. 2006. Shifting viewpoints: artificial intelligence and human-computert interaction. Artificial intelligence: 1256--1258. Google ScholarDigital Library
- Ian Witten, Eibe Frank, and Mark Hall. 2011. Data mining: practical machine learning tools and techniques. Morgan Kauffman.Google Scholar
- Allison Woodruff, Sally Augustin, and Brooke Foucault. 2007. Sabbath day home automation: it's like mixing technology and religion. In Proceedings of the SIGCHI conference on Human factors in computing systems. (CHI'07): 527--536. Google ScholarDigital Library
- Qian Yang, John Zimmerman, Aaron Steinfeld, and Anthony Tomasic. 2016. Planning adaptive mobile experiences when wireframing. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. (DIS'16): 565--576. Google ScholarDigital Library
- Rayoung Yang, and Mark W Newman. 2013. Learning from a learning thermostat: lessons for intelligent systems for the home. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing: 93--102. Google ScholarDigital Library
- Rayoung Yang, Mark W Newman, and Jodi Forlizzi. 2014. Making sustainability sustainable: challenges in the design of eco-interaction technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (CHI'14) 823--832.Google ScholarDigital Library
- John Zimmerman. 2005. Video Sketches: Exploring pervasive computing interaction designs. IEEE pervasive computing 4, 4: 91--94. Google ScholarDigital Library
- John Zimmerman, , et al. 2007. VIO: a mixed-initiative approach to learning and automating procedural update tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (CHI'07) 14451454. Google ScholarDigital Library
- John Zimmerman, Kaushal Kauapati, Anna L Buczak, Dave Schaffer, Srinivas Gutta, and Jacquelyn Martino. 2004. TV personalization system. In Personalized Digital Television, Liliana Ardissono, Alfred Kobsa and Mark Maybury, (eds) Kluwer Academic Publishers: 27--51.Google Scholar
Index Terms
- UX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Recommendations
Investigating How Experienced UX Designers Effectively Work with Machine Learning
DIS '18: Proceedings of the 2018 Designing Interactive Systems ConferenceMachine learning (ML) plays an increasingly important role in improving a user's experience. However, most UX practitioners face challenges in understanding ML's capabilities or envisioning what it might be. We interviewed 13 designers who had many ...
Exploring the Design Context of AI-Powered Services: A Qualitative Investigation of Designers’ Experiences with Machine Learning
Artificial Intelligence in HCIAbstractArtificial Intelligence (AI) has provided user experience (UX) designers with a richer toolset. To use technologies such as Machine Learning (ML) that can expand their creative capacity to design intelligent services. ML has the capability to ...
Using Design Fiction Memos to Analyze UX Professionals’ Values Work Practices: A Case Study Bridging Ethnographic and Design Futuring Methods
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsMultiple methods have been used to study how social values and ethics are implicated in technology design and use, including empirical qualitative studies of technologists’ work. Recently, more experimental approaches such as design fiction explore ...
Comments