skip to main content
10.1145/3565472.3595612acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
research-article

Human Expectations and Perceptions of Learning in Machine Teaching

Published: 19 June 2023 Publication History

Abstract

Interactive interfaces in tandem with Machine Learning (ML) models support user understanding of model uncertainty, build confidence, improve predictive accuracy and enable users to teach application-specific concepts that are difficult for the model to learn otherwise. These systems offer empirically proven benefits due to tightly coupled feedback loops and workflow scaffolding. However, deployment with ML non-experts who cannot manage the complex, expertise-heavy process remains challenging. Through deployment with non-expert users in a common classification task, we investigate the impact of human factors of machine teaching interfaces such as user expectations, their perceptions of the learning process and user engagement with respect to teaching process and outcomes. We measure how affective and performance attributes shape the success or failure of the process. Finally, we reflect on how intelligent user interfaces can be designed to accommodate these factors for successful deployment with a broad spectrum of human adjudicators.

References

[1]
Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. 2011. Improved Algorithms for Linear Stochastic Bandits. In Advances in Neural Information Processing Systems (Granada, Spain) (NIPS’11), J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.). Curran Associates Inc., Red Hook, NY, USA, 2312–2320.
[2]
BS Abhigna, Nitasha Soni, and Shilpa Dixit. 2018. Crowdsourcing–A step towards advanced machine learning. Procedia computer science 132 (2018), 632–642.
[3]
Hillary Abraham, Bobbie Seppelt, Bruce Mehler, and Bryan Reimer. 2017. What’s in a Name: Vehicle Technology Branding & Consumer Expectations for Automation. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Oldenburg, Germany) (AutomotiveUI ’17). Association for Computing Machinery, New York, NY, USA, 226–234. https://doi.org/10.1145/3122986.3123018
[4]
Amol Agrawal. 2016. Clickbait detection using deep learning. In 2016 2nd International Conference on Next Generation Computing Technologies (NGCT). IEEE, India, 268–272. https://doi.org/10.1109/NGCT.2016.7877426
[5]
Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. Ai Magazine 35, 4 (2014), 105–120.
[6]
Saleema Amershi, James Fogarty, and Daniel Weld. 2012. Regroup: Interactive Machine Learning for on-Demand Group Creation in Social Networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/2207676.2207680
[7]
Aman Anand. 2020. Clickbait dataset. https://www.kaggle.com/datasets/amananandrai/clickbait-dataset Accessed Oct 2021.
[8]
Alejandro Correa Bahnsen, Djamia Aouada, and Björn Ottersten. 2014. Example-Dependent Cost-Sensitive Logistic Regression for Credit Scoring. In 2014 13th International Conference on Machine Learning and Applications. ICML, USA, 263–269. https://doi.org/10.1109/ICMLA.2014.48
[9]
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 610–623. https://doi.org/10.1145/3442188.3445922
[10]
Francisco Bernardo, Michael Zbyszynski, Rebecca Fiebrink, and Mick Grierson. 2016. Interactive Machine Learning for End-User Innovation. In Designing the User Experience of Machine Learning Systems. American Association for Artificial Intelligence (AAAI), USA, 369–375. https://research.gold.ac.uk/id/eprint/19767/
[11]
Anol Bhattacherjee. 2001. Understanding Information Systems Continuance: An Expectation-Confirmation Model. MIS Quarterly 25, 3 (2001), 351–370. http://www.jstor.org/stable/3250921
[12]
Alessandro Bondielli and Francesco Marcelloni. 2019. A survey on fake news and rumour detection techniques. Information Sciences 497 (2019), 38–55.
[13]
Léon Bottou. 2012. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade. Springer, 421–436.
[14]
Daniel S. Brown and Scott Niekum. 2019. Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (Jul. 2019), 7749–7758. https://doi.org/10.1609/aaai.v33i01.33017749
[15]
Susan A Brown, Viswanath Venkatesh, and Sandeep Goyal. 2012. Expectation confirmation in technology use. Information Systems Research 23, 2 (2012), 474–487.
[16]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
[17]
Katherine A Burson, Richard P Larrick, and Joshua Klayman. 2006. Skilled or unskilled, but still unaware of it: how perceptions of difficulty drive miscalibration in relative comparisons.Journal of personality and social psychology 90, 1 (2006), 60.
[18]
Maya Cakmak and Andrea L. Thomaz. 2014. Eliciting Good Teaching from Humans for Machine Learners. Artif. Intell. 217, C (dec 2014), 198–215. https://doi.org/10.1016/j.artint.2014.08.005
[19]
Davide Calvaresi, Giovanni Ciatto, Amro Najjar, Reyhan Aydoğan, Leon Van der Torre, Andrea Omicini, and Michael Schumacher. 2021. Expectation: Personalized Explainable Artificial Intelligence for Decentralized Agents with Heterogeneous Knowledge. In Explainable and Transparent AI and Multi-Agent Systems: Third International Workshop, EXTRAAMAS 2021, Virtual Event, May 3–7, 2021, Revised Selected Papers. Springer-Verlag, Berlin, Heidelberg, 331–343. https://doi.org/10.1007/978-3-030-82017-6_20
[20]
Abhijnan Chakraborty, Rajdeep Sarkar, Ayushi Mrigen, and Niloy Ganguly. 2017. Tabloids in the Era of Social Media? Understanding the Production and Consumption of Clickbaits in Twitter. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 30 (dec 2017), 21 pages. https://doi.org/10.1145/3134665
[21]
Angelos Chatzimparmpas, Rafael Messias Martins, Ilir Jusufi, Kostiantyn Kucher, Fabrice Rossi, and Andreas Kerren. 2020. The state of the art in enhancing trust in machine learning models with the use of visualizations. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 713–756.
[22]
Ana Paula Chaves and Marco Aurelio Gerosa. 2021. How Should My Chatbot Interact? A Survey on Social Characteristics in Human–Chatbot Interaction Design. International Journal of Human–Computer Interaction 37, 8 (2021), 729–758. https://doi.org/10.1080/10447318.2020.1841438 arXiv:https://doi.org/10.1080/10447318.2020.1841438
[23]
Maria D. Molina, S Shyam Sundar, Md Main Uddin Rony, Naeemul Hassan, Thai Le, and Dongwon Lee. 2021. Does clickbait actually attract more clicks? Three clickbait studies you must read. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–19.
[24]
Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan. 2022. A review of some techniques for inclusion of domain-knowledge into deep neural networks. Scientific Reports 12, 1 (2022), 1–15.
[25]
Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 278–288. https://doi.org/10.1145/3025453.3025739
[26]
Zhengfang Duanmu, Kede Ma, and Zhou Wang. 2018. Quality-of-experience for adaptive streaming videos: An expectation confirmation theory motivated approach. IEEE Transactions on Image Processing 27, 12 (2018), 6135–6146.
[27]
John J Dudley and Per Ola Kristensson. 2018. A review of user interface design for interactive machine learning. ACM Transactions on Interactive Intelligent Systems (TiiS) 8, 2 (2018), 1–37.
[28]
Kirstin Early, Stephen E. Fienberg, and Jennifer Mankoff. 2016. Test Time Feature Ordering with FOCUS: Interactive Predictions with Minimal User Burden. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 992–1003. https://doi.org/10.1145/2971648.2971748
[29]
Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 147–156. https://doi.org/10.1145/1978942.1978965
[30]
Wilbert O Galitz. 2007. The essential guide to user interface design: an introduction to GUI design principles and techniques. John Wiley & Sons.
[31]
Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2020. Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience. arXiv (2020), arXiv–2001.
[32]
Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers. Proc. ACM Hum.-Comput. Interact. 4, CSCW3, Article 235 (jan 2021), 28 pages. https://doi.org/10.1145/3432934
[33]
Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable active learning (xal) toward ai explanations as interfaces for machine teachers. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1–28.
[34]
Sally A Goldman and H David Mathias. 1996. Teaching a smarter learner. J. Comput. System Sci. 52, 2 (1996), 255–267.
[35]
Alex Groce, Todd Kulesza, Chaoqiang Zhang, Shalini Shamasunder, Margaret Burnett, Weng-Keen Wong, Simone Stumpf, Shubhomoy Das, Amber Shinsel, Forrest Bice, and Kevin McIntosh. 2014. You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems. IEEE Transactions on Software Engineering 40, 3 (2014), 307–323. https://doi.org/10.1109/TSE.2013.59
[36]
Anil Gupta, Neeraj Dhiman, Anish Yousaf, and Neelika Arora. 2021. Social comparison and continuance intention of smart fitness wearables: An extended expectation confirmation theory perspective. Behaviour & Information Technology 40, 13 (2021), 1341–1354.
[37]
Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
[38]
Jani Heikkinen, Thomas Olsson, and Kaisa Väänänen-Vainio-Mattila. 2009. Expectations for User Experience in Haptic Communication with Mobile Devices. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (Bonn, Germany) (MobileHCI ’09). Association for Computing Machinery, New York, NY, USA, Article 28, 10 pages. https://doi.org/10.1145/1613858.1613895
[39]
Andreas Holzinger, Markus Plass, Michael Kickmeier-Rust, Katharina Holzinger, Gloria Cerasela Crişan, Camelia-M. Pintea, and Vasile Palade. 2019. Interactive Machine Learning: Experimental Evidence for the Human in the Algorithmic Loop. Applied Intelligence 49, 7 (jul 2019), 2401–2414. https://doi.org/10.1007/s10489-018-1361-5
[40]
Jonggi Hong, Kyungjun Lee, June Xu, and Hernisa Kacorri. 2020. Crowdsourcing the perception of machine teaching. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
[41]
Eva Hudlicka. 2003. To feel or not to feel: The role of affect in human–computer interaction. International Journal of Human-Computer Studies 59, 1 (2003), 1–32. https://doi.org/10.1016/S1071-5819(03)00047-8 Applications of Affective Computing in Human-Computer Interaction.
[42]
K. Höök. 2000. Steps to take before intelligent user interfaces become real. Interacting with Computers 12, 4 (2000), 409–426. https://doi.org/10.1016/S0953-5438(99)00006-5
[43]
Luis-Daniel Ibáñez, Neal Reeves, and Elena Simperl. 2020. Crowdsourcing and Human-in-the-Loop for IoT. The Internet of Things: From Data to Insight (2020), 91–105.
[44]
Liu Jiang, Shixia Liu, and Changjian Chen. 2019. Recent research advances on interactive machine learning. Journal of Visualization 22, 2 (2019), 401–417.
[45]
Muneo Kitajima and Peter G. Polson. 1992. A Computational Model of Skilled Use of a Graphical User Interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey, California, USA) (CHI ’92). Association for Computing Machinery, New York, NY, USA, 241–249. https://doi.org/10.1145/142750.142803
[46]
Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems. 453–456.
[47]
David G Kleinbaum, K Dietz, M Gail, Mitchel Klein, and Mitchell Klein. 2002. Logistic regression. Springer.
[48]
Reed Larson and Mihaly Csikszentmihalyi. 2014. The experience sampling method. In Flow and the foundations of positive psychology. Springer, 21–34.
[49]
Mingkun Li and Ishwar K Sethi. 2006. Confidence-based active learning. IEEE transactions on pattern analysis and machine intelligence 28, 8 (2006), 1251–1261.
[50]
Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 384, 23 pages. https://doi.org/10.1145/3491102.3501825
[51]
Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. 2017. Iterative machine teaching. In International Conference on Machine Learning. PMLR, 2149–2158.
[52]
Weiyang Liu, Bo Dai, Xingguo Li, Zhen Liu, James Rehg, and Le Song. 2018. Towards black-box iterative machine teaching. In International Conference on Machine Learning. PMLR, 3141–3149.
[53]
Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5286–5297. https://doi.org/10.1145/2858036.2858288
[54]
Oisin Mac Aodha, Vassilios Stathopoulos, Gabriel J Brostow, Michael Terry, Mark Girolami, and Kate E Jones. 2014. Putting the scientist in the loop–Accelerating scientific progress with interactive machine learning. In 2014 22nd International Conference on Pattern Recognition. IEEE, 9–17.
[55]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (2019), 1–38. https://doi.org/10.1016/j.artint.2018.07.007
[56]
Swati Mishra and Jeffrey M Rzeszotarski. 2021. Crowdsourcing and evaluating concept-driven explanations of machine learning models. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–26.
[57]
Swati Mishra and Jeffrey M Rzeszotarski. 2021. Designing Interactive Transfer Learning Tools for ML Non-Experts. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
[58]
Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019. Explaining Explanations in AI. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 279–288. https://doi.org/10.1145/3287560.3287574
[59]
Robert Munro and Robert Monarch. 2021. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster.
[60]
Christine Murad and Cosmin Munteanu. 2019. "I Don’t Know What You’re Talking about, HALexa": The Case for Voice User Interface Guidelines. In Proceedings of the 1st International Conference on Conversational User Interfaces (Dublin, Ireland) (CUI ’19). Association for Computing Machinery, New York, NY, USA, Article 9, 3 pages. https://doi.org/10.1145/3342775.3342795
[61]
Bilal Naeem, Aymen Khan, Mirza Omer Beg, and Hasan Mujtaba. 2020. A deep learning framework for clickbait detection on social area network using natural language cues. Journal of Computational Social Science 3, 1 (2020), 231–243.
[62]
Hieu T Nguyen and Arnold Smeulders. 2004. Active learning using pre-clustering. In Proceedings of the twenty-first international conference on Machine learning. 79.
[63]
J. Nielsen. 1993. Iterative user-interface design. Computer 26, 11 (1993), 32–41. https://doi.org/10.1109/2.241424
[64]
Jakob Nielsen. 1994. Usability Inspection Methods. In Conference Companion on Human Factors in Computing Systems (Boston, Massachusetts, USA) (CHI ’94). Association for Computing Machinery, New York, NY, USA, 413–414. https://doi.org/10.1145/259963.260531
[65]
Simon Nusinovici, Yih Chung Tham, Marco Yu Chak Yan, Daniel Shu Wei Ting, Jialiang Li, Charumathi Sabanayagam, Tien Yin Wong, and Ching-Yu Cheng. 2020. Logistic regression was as good as machine learning for predicting major chronic diseases. Journal of Clinical Epidemiology 122 (2020), 56–69. https://doi.org/10.1016/j.jclinepi.2020.03.002
[66]
Richard L Oliver. 1980. A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of marketing research 17, 4 (1980), 460–469.
[67]
Sunjeong Park and Youn-kyung Lim. 2020. Investigating User Expectations on the Roles of Family-Shared AI Speakers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376450
[68]
Reid Porter, James Theiler, and Don Hush. 2013. Interactive Machine Learning in Data Exploitation. Computing in Science & Engineering 15, 5 (2013), 12–20. https://doi.org/10.1109/MCSE.2013.74
[69]
Abinash Pujahari and Dilip Singh Sisodia. 2021. Clickbait detection using multiple categorisation techniques. Journal of Information Science 47, 1 (2021), 118–128.
[70]
Anant Raj and Francis Bach. 2022. Convergence of uncertainty sampling for active learning. In International Conference on Machine Learning. PMLR, 18310–18331.
[71]
Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 33–44.
[72]
Maria Riveiro and Serge Thill. 2021. “That’s (not) the output I expected!” On the role of end user expectations in creating explanations of AI systems. Artificial Intelligence 298 (2021), 103507. https://doi.org/10.1016/j.artint.2021.103507
[73]
Dominik Sacha, Matthias Kraus, Daniel A. Keim, and Min Chen. 2019. VIS4ML: An Ontology for Visual Analytics Assisted Machine Learning. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 385–395. https://doi.org/10.1109/TVCG.2018.2864838
[74]
Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Müller. 2021. Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications. Proc. IEEE 109, 3 (2021), 247–278. https://doi.org/10.1109/JPROC.2021.3060483
[75]
BEN SHNEIDERMAN. 1982. The future of interactive systems and the emergence of direct manipulation. Behaviour & Information Technology 1, 3 (1982), 237–256. https://doi.org/10.1080/01449298208914450 arXiv:https://doi.org/10.1080/01449298208914450
[76]
Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, and John Wernsing. 2017. Machine Teaching: A New Paradigm for Building Machine Learning Systems. Technical Report MSR-TR-2017-26. https://www.microsoft.com/en-us/research/publication/machine-teaching-new-paradigm-building-machine-learning-systems/
[77]
Herbert A Simon. 1955. A behavioral model of rational choice. The quarterly journal of economics 69, 1 (1955), 99–118.
[78]
Ilia Stepin, Jose M. Alonso, Alejandro Catala, and Martín Pereira-Fariña. 2021. A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence. IEEE Access 9 (2021), 11974–12001. https://doi.org/10.1109/ACCESS.2021.3051315
[79]
Simone Stumpf, Vidya Rajaram, Lida Li, Weng-Keen Wong, Margaret Burnett, Thomas Dietterich, Erin Sullivan, and Jonathan Herlocker. 2009. Interacting meaningfully with machine learning systems: Three experiments. International journal of human-computer studies 67, 8 (2009), 639–662.
[80]
S Shyam Sundar. 2020. Rise of Machine Agency: A Framework for Studying the Psychology of Human–AI Interaction (HAII). Journal of Computer-Mediated Communication 25, 1 (01 2020), 74–88. https://doi.org/10.1093/jcmc/zmz026 arXiv:https://academic.oup.com/jcmc/article-pdf/25/1/74/32961171/zmz026.pdf
[81]
Ashlesha Vaidya. 2017. Predictive and probabilistic approach using logistic regression: Application to prediction of loan approval. In 2017 8th International Conference on Computing, Communication and Networking Technologies (ICCCNT). 1–6. https://doi.org/10.1109/ICCCNT.2017.8203946
[82]
Aikaterini C Valvi and Douglas C West. 2013. E-loyalty is not all about trust, price also matters: extending expectation-confirmation theory in bookselling websites. Journal of Electronic Commerce Research 14, 1 (2013), 99.
[83]
Niels Van Berkel, Jorge Goncalves, Danula Hettiachchi, Senuri Wijenayake, Ryan M Kelly, and Vassilis Kostakos. 2019. Crowdsourcing perceptions of fair predictors for machine learning: A recidivism case study. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–21.
[84]
Cornelis Joost Van Rijsbergen. 1977. A theoretical basis for the use of co-occurrence data in information retrieval. Journal of documentation (1977).
[85]
Kiri L. Wagstaff. 2012. Machine Learning That Matters. In Proceedings of the 29th International Coference on International Conference on Machine Learning (Edinburgh, Scotland) (ICML’12). Omnipress, Madison, WI, USA, 1851–1856.
[86]
MALCOLM WARE, EIBE FRANK, GEOFFREY HOLMES, MARK HALL, and IAN H WITTEN. 2001. Interactive machine learning: letting users build classifiers. International Journal of Human-Computer Studies 55, 3 (2001), 281–292. https://doi.org/10.1006/ijhc.2001.0499
[87]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6
[88]
Tongshuang Wu, Daniel S Weld, and Jeffrey Heer. 2019. Local decision pitfalls in interactive machine learning: An investigation into feature selection in sentiment analysis. ACM Transactions on Computer-Human Interaction (TOCHI) 26, 4 (2019), 1–27.
[89]
Doris Xin, Litian Ma, Jialin Liu, Stephen Macke, Shuchen Song, and Aditya Parameswaran. 2018. Accelerating Human-in-the-Loop Machine Learning: Challenges and Opportunities. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning (Houston, TX, USA) (DEEM’18). Association for Computing Machinery, New York, NY, USA, Article 9, 4 pages. https://doi.org/10.1145/3209889.3209897
[90]
Luyao Yuan, Dongruo Zhou, Junhong Shen, Jingdong Gao, Jeffrey L Chen, Quanquan Gu, Ying Nian Wu, and Song-Chun Zhu. 2021. Iterative Teacher-Aware Learning. Advances in Neural Information Processing Systems 34 (2021), 29231–29245.
[91]
Lijun Zhang, Rong Jin, Chun Chen, Jiajun Bu, and Xiaofei He. 2021. Efficient Online Learning for Large-Scale Sparse Kernel Logistic Regression. Proceedings of the AAAI Conference on Artificial Intelligence 26, 1 (Sep. 2021), 1219–1225. https://doi.org/10.1609/aaai.v26i1.8300
[92]
Zhigang Zhang, Wangshu Cheng, and Zhenyu Gu. 2016. User experience studies based on expectation dis-confirmation theory. In International Conference of Design, User Experience, and Usability. Springer, 670–677.
[93]
Jingbo Zhu, Huizhen Wang, Benjamin K. Tsou, and Matthew Ma. 2010. Active Learning With Sampling by Uncertainty and Density for Data Annotations. IEEE Transactions on Audio, Speech, and Language Processing 18, 6 (2010), 1323–1331. https://doi.org/10.1109/TASL.2009.2033421
[94]
Xiaojin Zhu. 2015. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29.
[95]
Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N Rafferty. 2018. An overview of machine teaching. arXiv preprint arXiv:1801.05927 (2018).
[96]
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. 2020. A comprehensive survey on transfer learning. Proc. IEEE 109, 1 (2020), 43–76.

Cited By

View all
  • (2024)Devising Scrutable User Models for Time Management AssistantsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3665182(250-255)Online publication date: 27-Jun-2024
  • (2024)Interaction Visualization for Analysing and Improving User ModelsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664877(160-163)Online publication date: 27-Jun-2024
  • (2024)Teachable Facets: A Framework of Interactive Machine Teaching for Information FilteringProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638289(178-188)Online publication date: 10-Mar-2024

Index Terms

  1. Human Expectations and Perceptions of Learning in Machine Teaching

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UMAP '23: Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization
    June 2023
    333 pages
    ISBN:9781450399326
    DOI:10.1145/3565472
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 June 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Expectations
    2. Interactive Machine Teaching
    3. User Experience

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    UMAP '23
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 162 of 633 submissions, 26%

    Upcoming Conference

    UMAP '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)101
    • Downloads (Last 6 weeks)9
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Devising Scrutable User Models for Time Management AssistantsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3665182(250-255)Online publication date: 27-Jun-2024
    • (2024)Interaction Visualization for Analysing and Improving User ModelsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664877(160-163)Online publication date: 27-Jun-2024
    • (2024)Teachable Facets: A Framework of Interactive Machine Teaching for Information FilteringProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638289(178-188)Online publication date: 10-Mar-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media