skip to main content
10.1145/3025453.3025739acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

UX Design Innovation: Challenges for Working with Machine Learning as a Design Material

Published: 02 May 2017 Publication History

Abstract

Machine learning (ML) is now a fairly established technology, and user experience (UX) designers appear regularly to integrate ML services in new apps, devices, and systems. Interestingly, this technology has not experienced a wealth of design innovation that other technologies have, and this might be because it is a new and difficult design material. To better understand why we have witnessed little design innovation, we conducted a survey of current UX practitioners with regards to how new ML services are envisioned and developed in UX practice. Our survey probed on how ML may or may not have been a part of their UX design education, on how they work to create new things with developers, and on the challenges they have faced working with this material. We use the findings from this survey and our review of related literature to present a series of challenges for UX and interaction design research and education. Finally, we discuss areas where new research and new curriculum might help our community unlock the power of design thinking to re-imagine what ML might be and might do.

References

[1]
James F. Allen, Curry I. Guinn, and Eric Horovitz. 1999. Mixed-initiative interaction: IEEE Intelligent Systems and their Applications: 14--23.
[2]
Robbert-Jan Beun, Eveliene De Vos, and Cilia Witteman. 2003. Embodied conversational agents: effects on memory performance and anthropomorphisation. In International Workshop on Intelligent Virtual Agents, 315--319.
[3]
Nick Bostrom, and Eliezer Yudkowsky. 2014. The ethics of artificial intelligence. In The Cambridge Handbook of Artificial Intelligence, Keith Frankish and William M Ramsey (eds), Cambridge University Press, Cambridge: 316--334
[4]
Sheryl Brahnam, and Antonella De Angeli. 2012. Gender affordances of conversational agents: Interacting With Computers: 139--153.
[5]
Virginia Braun, and Victoria Clarke. 2006. Using thematic analysis in psychology: Qualitative research in psychology 3, 2: 77--101.
[6]
John Brownlee. 2015. Apple finaly learns AI is the new UI. Retrieved August 1, 2016 from http://www.fastcodesign.com/3047199/apple-finallylearns-ai-is-the-new-ui
[7]
Marion Buchenau, and Jane Fulton Suri. 2000. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, 424433.
[8]
Bill Buxton, 2007. Sketching user experiences. Morgan Kaufmann,.
[9]
Stephen M Casner, Edwin L Hutchins, and Don Norman. 2016 The challenges of partially automated driving. Communications of the ACM 59, 5: 70--77.
[10]
Justine Cassell. 2000. Emboddied conversational interface agents. Communications of the ACM 43, 4: 70--78.
[11]
Guanling Chen and David Kotz. 2000 A survey of context-aware mobile computing research. Dept. of Computer Science, Dartmouth College, Dartmouth.
[12]
Giles Colborne. 2016. Interaction design in the age of algorithms. Retrieved August 8, 2016 from https://www.cxpartners.co.uk/our-thinking/interactiondesign-in-the-age-of-algorithms/
[13]
Alan Cooper, Robert Reimann, David Cronin, and Christopher Noessel. 2014. About face: the essentials of interaction design. John Wiley & Sons,.
[14]
Scott Davidoff, Brian D Ziebart, John Zimmerman, and Anind K Dey. 2011. Learning patterns of pick-ups and drop-offs to support busy family coordination. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.(CHI'11): 1175--1184.
[15]
Eva Deckers, Pierre Levy, Stephan Wensveen, Rene Ahn, and Kees Overbeeke. 2012. Designing for perceptual crossing: Applying and evaluating design notions. International Journal of Design 6, 3: 41--55.
[16]
Pedro Domingos. 2012. A few useful things to know about machine learning. Communications of the ACM 55,10: 78--87.
[17]
Afsaneh Doryab, Jun Ki Min, Jason Wiese, John Zimmerman, and Jason I Hong. 2014. Detection of behavior change in people with depression. In Proceedings of AAAI Workshop on Modern Artificial Intelligence for Health Analytics (MAIHA).
[18]
Graham Dove, and Sara Jones. 2014. Using information visualization to stimulate creativity in service design workshops.In Proceedings of the fourth Service Design and Service Innovation Conference. (ServDes14): 281--290.
[19]
Pelle Ehn, and Morten Kyng. 1991. Cardboard computers. In Design at Work, Joan Greenbaum and Morten Kyng (eds) Lawrence Erlbaum Associates: 169--197.
[20]
Harry Enten. 2016. Election update: where polls and demographics disagree. Retrieved September 14, 2016 from http://fivethirtyeight.com/features/electionupdate-where-polls-and-demographics-disagree/
[21]
Peter Flach. 2012. Machine learning: the art and science of algorithms that make sense of data. Cambridge University Press.
[22]
Jodi Forlizzi, John Zimmerman, Vince Mancuso, and Sonya Kwak. 2007. How interface agents affect interaction between humans and computers. In Proceedings of the 2007 conference on Designing pleasurable products and interfaces: 209--221.
[23]
Marco Gillies, et al. 2016. Human-centered machine learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI'16): 3558--3565.
[24]
Google. 2016. Breakthroughs in Machine Learning Google I/O 2016. Retrieved August 1, 2016 from https://www.youtube.com/watch?v=sphFCJE1HkI
[25]
Jonathan Grudin. 2006. Turing maturing: the separation of artificial intelligence and humancomputer interaction. In Interactions: 54--57.
[26]
Bjarki Hallgrimsson. 2012. Prototyping and modelmaking for product design. Laurence King,.
[27]
Harrison, Chris, Desney Tan, and Dan Morris. "Skinput: appropriating the body as an input surface." Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Atlanta: ACM, 2010. 453--462.
[28]
Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. (CSCW): 241--250.
[29]
Geoffrey E Hinton. 2007. Learning multiple layers of representation. Trends in Cognitive Science 11,10: 428434.
[30]
Chien-Ming Huang, and Bilge Mutlu. 2012. Robot behavior toolkit: generating effective social behaviors for robots. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. (HRI'12): 25--32.
[31]
Scott E Hudson, et al. 2003. Predicting human interruptibility with sensors: a wizard of oz feasibility study. InThe CHI 2003 New Horizons Conference Proceedings: Conference on Human Factors in Computing Systems. (CHI'03): 257--264.
[32]
Francisco Iacobelli, and Justine Cassell. 2007. Ethnic identity and engagement in embodied conversational agents. In International Workshop on Intelligent Virtual Agents: 57--63.
[33]
Rabia Kahn and Antonella De Angeli. 2009. The attractiveness stereotype in the evaluation of embodied conversational agents. IFIP Conference on HumanComputer Interaction (INTERACT 2009): 85--97.
[34]
Yunkyung Kim, and Bilge Mutlu. 2014. How social distance shapes human-robot interaction. International Journal of Human-Computer Studies 72: 783--795.
[35]
Kenneth R Koedinger, and A Corbett. 2006. Cognitive tutors. In The Cambridge handbook of the learning sciences, Keith R Sawyer (ed) Cambridge University Press: 61--77.
[36]
Pat Langley. 1997. Machine learning for adaptive user interfaces. In Annual conference on Artificial Intelligence: 53--62.
[37]
Min Kyung Lee, Sara Kiesler, Jodi Forlizzi, Siddhartha Srinivasa, and Paul E Rybski. 2010. Gracefully mitigating breakdowns in robotic services. In 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI): 203--210.
[38]
Ewa Luger, and Abigail Sellen. 2016. Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI'16): 5286--5297.
[39]
Media Lab Helsiki. A quick primer for ethics in design. Retrieved September 15 2016 from http://mlab.uiah.fi/polut/Yhteiskunnalliset/lisatieto_ethi cs_primer.html
[40]
Camille Mousette, and Richard Banks. 2011. Designing through making: exploring the simple haptic design space. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction: 279--282.
[41]
Clifford Nass, Jonathan Steuer, and Ellen R Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. (CHI'94): 72--78.
[42]
William Odom, John Zimmerman, Scott Davidoff, Jodi Forlizzi, Anind K Dey, and Min Kyung Lee. 2012. A fieldwork of the future with user enactments. In Proceedings of the Designing Interactive Systems Conference. (DIS'12): 338--347.
[43]
Fatih Kursat Ozene, Miso Kim, John Zimmerman, Stephen Oney, and Brad Myers. 2010. How to support designers in getting hold of the immaterial material of software. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (CHI'10): 2513--2522.
[44]
Victor Papanek. 1995. The Green Imperative: Ecology and Ethics in Design and Architecture. Thames and Hudson.
[45]
Pamela Pavliscak. 2016. Algorithms as the new material of design. Retrieved September 14th from http://www.uxmatters.com/mt/archives/2016/06/algorit hms-as-the-new-material-of-design.php
[46]
Jenny Preece, Helen Sharp, and Yvonne Rogers. 2015. Interaction design: beyond human-computer interaction. John Wiley & Sons.
[47]
Frank Rosenblatt. 1961. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Cornell Aeronautical Lab.
[48]
Arthur L Samuel,.1959. Some studies in machine learning using the game of checkers. IBM Journal of research and development 3, 3: 210--229.
[49]
Ben Shneiderman, and Patti Maes. 1997. Direct manipulation vs. interface agents. Interactions: 42--61.
[50]
Yale Song, David Demirdjian, and Randall Davis. 2012. Continuous body and hand gesture recognition for natural human-computer interaction. ACM Transactions on Interactive Intelligent Systems 2, 1: 5.
[51]
Aaron Spaulding, Krzystof Z Gajos, Anthony Jameson, Per Ola Kristensson, Andrea Bunt, and Will Haines. 2009. Usable intelligent interactive systems. In CHI'09 Extended Abstracts on Human Factors in Computing Systems.
[52]
Swedish ICT. Retrieved September 15th 2016 from https://www.tii.se/projects/ittextiles
[53]
Terry Winograd. 2006. Shifting viewpoints: artificial intelligence and human-computert interaction. Artificial intelligence: 1256--1258.
[54]
Ian Witten, Eibe Frank, and Mark Hall. 2011. Data mining: practical machine learning tools and techniques. Morgan Kauffman.
[55]
Allison Woodruff, Sally Augustin, and Brooke Foucault. 2007. Sabbath day home automation: it's like mixing technology and religion. In Proceedings of the SIGCHI conference on Human factors in computing systems. (CHI'07): 527--536.
[56]
Qian Yang, John Zimmerman, Aaron Steinfeld, and Anthony Tomasic. 2016. Planning adaptive mobile experiences when wireframing. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. (DIS'16): 565--576.
[57]
Rayoung Yang, and Mark W Newman. 2013. Learning from a learning thermostat: lessons for intelligent systems for the home. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing: 93--102.
[58]
Rayoung Yang, Mark W Newman, and Jodi Forlizzi. 2014. Making sustainability sustainable: challenges in the design of eco-interaction technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (CHI'14) 823--832.
[59]
John Zimmerman. 2005. Video Sketches: Exploring pervasive computing interaction designs. IEEE pervasive computing 4, 4: 91--94.
[60]
John Zimmerman, et al. 2007. VIO: a mixed-initiative approach to learning and automating procedural update tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (CHI'07) 14451454.
[61]
John Zimmerman, Kaushal Kauapati, Anna L Buczak, Dave Schaffer, Srinivas Gutta, and Jacquelyn Martino. 2004. TV personalization system. In Personalized Digital Television, Liliana Ardissono, Alfred Kobsa and Mark Maybury, (eds) Kluwer Academic Publishers: 27--51.

Cited By

View all
  • (2025)Human leading or following preferencesRobotics and Autonomous Systems10.1016/j.robot.2024.104821183:COnline publication date: 1-Jan-2025
  • (2024)Challenges and Responsibilities in the UX Design of Text-to-Image AI Models: A Discussion Through a Comparative Heuristics EvaluationReptes i responsabilitats del disseny d’UX en models d’IA de text a imatge: Discussió a través d’una avaluació heurística comparativaRetos y responsabilidades del diseño de UX en modelos de IA de texto a imagen: Discusión a través de una evaluación heurística comparativaTemes de Disseny10.46467/TdD40.2024.156-175(156-175)Online publication date: 13-Nov-2024
  • (2024)Beyond the ScreenNavigating Usability and User Experience in a Multi-Platform World10.4018/979-8-3693-2337-3.ch001(1-18)Online publication date: 15-Nov-2024
  • Show More Cited By

Index Terms

  1. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. design material
    2. interaction design
    3. machine learning
    4. ux practice

    Qualifiers

    • Research-article

    Funding Sources

    • The Danish Innovation Foundation

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)833
    • Downloads (Last 6 weeks)64
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Human leading or following preferencesRobotics and Autonomous Systems10.1016/j.robot.2024.104821183:COnline publication date: 1-Jan-2025
    • (2024)Challenges and Responsibilities in the UX Design of Text-to-Image AI Models: A Discussion Through a Comparative Heuristics EvaluationReptes i responsabilitats del disseny d’UX en models d’IA de text a imatge: Discussió a través d’una avaluació heurística comparativaRetos y responsabilidades del diseño de UX en modelos de IA de texto a imagen: Discusión a través de una evaluación heurística comparativaTemes de Disseny10.46467/TdD40.2024.156-175(156-175)Online publication date: 13-Nov-2024
    • (2024)Beyond the ScreenNavigating Usability and User Experience in a Multi-Platform World10.4018/979-8-3693-2337-3.ch001(1-18)Online publication date: 15-Nov-2024
    • (2024)Preparing humane ML experts for a better future. Experiments with design and engineering studentsJournal of Entrepreneurial Researchers10.29073/jer.v2i1.212:1(027-048)Online publication date: 9-Jul-2024
    • (2024)Research on the Integration and Innovation of Artificial Intelligence Technology in Modern Brand DesignApplied Mathematics and Nonlinear Sciences10.2478/amns-2024-31679:1Online publication date: 11-Nov-2024
    • (2024)Prototyping with Uncertainties: Data, Algorithms, and Research through DesignACM Transactions on Computer-Human Interaction10.1145/370232231:6(1-21)Online publication date: 6-Dec-2024
    • (2024)A Hybrid Prototype Method Combining Physical Models and Generative Artificial Intelligence to Support Creativity in Conceptual DesignACM Transactions on Computer-Human Interaction10.1145/368943331:5(1-34)Online publication date: 2-Sep-2024
    • (2024)Analyzing Collaborative Challenges and Needs of UX Practitioners when Designing with AI/MLProceedings of the ACM on Human-Computer Interaction10.1145/36869868:CSCW2(1-25)Online publication date: 8-Nov-2024
    • (2024)Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AIProceedings of the ACM on Human-Computer Interaction10.1145/36869628:CSCW2(1-44)Online publication date: 8-Nov-2024
    • (2024)Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language ModelsProceedings of the Halfway to the Future Symposium10.1145/3686169.3686185(1-8)Online publication date: 21-Oct-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media