Skip to main content
Springer Nature Link
Log in

Evaluation of Interactive Machine Learning Systems

  • Chapter
  • First Online:
Human and Machine Learning

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

The evaluation of interactive machine learning systems remains a difficult task. These systems learn from and adapt to the human, but at the same time, the human receives feedback and adapts to the system. Getting a clear understanding of these subtle mechanisms of co-operation and co-adaptation is challenging. In this chapter, we report on our experience in designing and evaluating various interactive machine learning applications from different domains. We argue for coupling two types of validation: algorithm-centred analysis, to study the computational behaviour of the system; and human-centred evaluation, to observe the utility and effectiveness of the application for end-users. We use a visual analytics application for guided search, built using an interactive evolutionary approach, as an exemplar of our work. Our observation is that human-centred design and evaluation complement algorithmic analysis, and can play an important role in addressing the “black-box” effect of machine learning. Finally, we discuss research opportunities that require human-computer interaction methodologies, in order to support both the visible and hidden roles that humans play in interactive machine learning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Amershi, S., Fogarty, J., Weld, D.: Regroup: interactive machine learning for on-demand group creation in social networks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, pp. 21–30. ACM, New York, NY, USA (2012)

    Google Scholar 

  2. Azuan, N., Embury, S., Paton, N.: Observing the data scientist: Using manual corrections as implicit feedback. In: Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics, HILDA’17, pp. 13:1–13:6. ACM, New York, NY, USA (2017)

    Google Scholar 

  3. Bach, B., Spritzer, A., Lutton, E., Fekete, J.D.: Interactive random graph generation with evolutionary algorithms. In: Graph Drawing. Lecture Notes in Computer Science. Springer, Berlin (2012)

    Google Scholar 

  4. Banzhaf, W.: Handbook of Evolutionary Computation. Oxford University Press, Oxford (1997)

    Google Scholar 

  5. Behrisch, M., Korkmaz, F., Shao, L., Schreck, T.: Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. In: 2014 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 43–52 (2014)

    Google Scholar 

  6. Boukhelifa, N., Cancino, W., Bezerianos, A., Lutton, E.: Evolutionary visual exploration: evaluation with expert users. Comput. Graph. Forum 32(3), 31–40 (2013)

    Article  Google Scholar 

  7. Boukhelifa, N., Bezerianos, A., Lutton, E.: A mixed approach for the evaluation of a guided exploratory visualization system. In: Aigner, W., Rosenthal, P., Scheidegger, C. (eds.) EuroVis Workshop on Reproducibility, Verification, and Validation in Visualization (EuroRV3). The Eurographics Association, Spain (2015)

    Google Scholar 

  8. Boukhelifa, N., Bezerianos, A., Tonda, A., Lutton, E.: Research prospects in the design and evaluation of interactive evolutionary systems for art and science. In: CHI Workshop on Human Centred Machine Learning. San Jose, United States (2016)

    Google Scholar 

  9. Boukhelifa, N., Bezerianos, A., Cancino, W., Lutton, E.: Evolutionary visual exploration: evaluation of an iec framework for guided visual search. Evol. Comput. 25(1), 55–86 (2017)

    Article  Google Scholar 

  10. Boukhelifa, N., Perrin, M.E., Huron, S., Eagan, J.: How data workers cope with uncertainty: a task characterisation study. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, pp. 3645–3656. ACM, New York, NY, USA (2017)

    Google Scholar 

  11. Brown, E., Liu, J., Brodley, C., Chang, R.: Dis-function: Learning distance functions interactively. In: 2012 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 83–92 (2012)

    Google Scholar 

  12. Bryan, N., Mysore, G., Wang, G.: ISSE: an interactive source separation editor. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, pp. 257–266. ACM, New York, NY, USA (2014)

    Google Scholar 

  13. Cancino, W., Boukhelifa, N., Lutton, E.: Evographdice: interactive evolution for visual analytics. In: IEEE Congress on Evolutionary Computation, pp. 1–8. IEEE (2012)

    Google Scholar 

  14. Cancino, W., Boukhelifa, N., Bezerianos, A., Lutton, E.: Evolutionary visual exploration: experimental analysis of algorithm behaviour. In: Blum, C., Alba, E. (eds.) GECCO (Companion), pp. 1373–1380. ACM (2013)

    Google Scholar 

  15. Carpendale, S.: Information visualization. Evaluating Information Visualizations, pp. 19–45. Springer, Berlin (2008)

    Google Scholar 

  16. Cherry, E., Latulipe, C.: Quantifying the creativity support of digital tools through the creativity support index. ACM Trans. Comput.-Hum. Interact. 21(4), 21:1–21:25 (2014)

    Article  Google Scholar 

  17. Chilana, P., Wobbrock, J., Andrew, J.: Understanding usability practices in complex domains. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, pp. 2337–2346. ACM, New York, NY, USA (2010)

    Google Scholar 

  18. Choo, J., Lee, C., Reddy, C., Park, H.: Utopian: user-driven topic modeling based on interactive nonnegative matrix factorization. IEEE Trans. Vis. Comput. Graph. 19(12), 1992–2001 (2013)

    Google Scholar 

  19. Cortellessa, G., Cesta, A.: Evaluating mixed-initiative systems: an experimental approach. ICAPS 6, 172–181 (2006)

    Google Scholar 

  20. Dabek, F., Caban, J.: A grammar-based approach for modeling user interactions and generating suggestions during the data exploration process. IEEE Trans. Vis. Comput. Graph. 23(1), 41–50 (2017)

    Article  Google Scholar 

  21. Ehrenberg, H., Shin, J., Ratner, A., Fries, J., Ré, C.: Data programming with DDLite: putting humans in a different part of the loop. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA ’16, pp. 13:1–13:6. ACM, New York, NY, USA (2016)

    Google Scholar 

  22. Endert, A., Fiaux, P., North, C.: Semantic interaction for visual text analytics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, pp. 473–482. ACM, New York, NY, USA (2012)

    Google Scholar 

  23. Gao, L., Cao, Y., Lai, Y., Huang, H., Kobbelt, L., Hu, S.: Active exploration of large 3d model repositories. IEEE Trans. Vis. Comput. Graph. 21(12), 1390–1402 (2015)

    Article  Google Scholar 

  24. Goldberg, D.: Genetic Algorithms in Search, Optimization and Machine Learning, 1st edn. Addison-Wesley Longman Publishing Co., Inc., Boston (1989)

    MATH  Google Scholar 

  25. Grinstein, G.: Harnessing the human in knowledge discovery. In: Simoudis, E., Han, J., Fayyad, U.M. (eds.) KDD, pp. 384–385. AAAI Press (1996)

    Google Scholar 

  26. Healey, C., Dennis, B.: Interest driven navigation in visualization. IEEE Trans. Vis. Comput. Graph. 18(10), 1744–1756 (2012)

    Article  Google Scholar 

  27. Heimerl, F., Koch, S., Bosch, H., Ertl, T.: Visual classifier training for text document retrieval. IEEE Trans. Vis. Comput. Graph. 18(12), 2839–2848 (2012)

    Article  Google Scholar 

  28. Holzinger, A.: Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform. 3(2), 119–131 (2016)

    Article  Google Scholar 

  29. Horvitz, E.: Principles of mixed-initiative user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’99, pp. 159–166. ACM, New York, NY, USA (1999)

    Google Scholar 

  30. Kelly, D., Teevan, J.: Implicit feedback for inferring user preference: a bibliography. SIGIR Forum 37(2), 18–28 (2003)

    Article  Google Scholar 

  31. Koyama, Y., Sakamoto, D., Igarashi, T.: Selph: Progressive learning and support of manual photo color enhancement. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pp. 2520–2532. ACM, New York, NY, USA (2016)

    Google Scholar 

  32. Landrin-Schweitzer, Y., Collet, P., Lutton, E.: Introducing lateral thinking in search engines. Genet. Program. Evolvable Hardw. J. 1(7), 9–31 (2006)

    Article  Google Scholar 

  33. Legg, P., Chung, D., Parry, M., Bown, R., Jones, M., Griffiths, I., Chen, M.: Transformation of an uncertain video search pipeline to a sketch-based visual analytics loop. IEEE Trans. Vis. Comput. Graph. 19(12), 2109–2118 (2013)

    Article  Google Scholar 

  34. Legrand, P., Bourgeois-Republique, C., Pean, V., Harboun-Cohen, E., Lévy Véhel, J., Frachet, B., Lutton, E., Collet, P.: Interactive evolution for cochlear implants fitting. GPEM 8(4), 319–354 (2007)

    Google Scholar 

  35. Lin, H., Gao, S., Gotz, D., Du, F., He, J., Cao, N.: RCLens: interactive rare category exploration and identification. IEEE Trans. Vis. Comput. Graph. PP(99), 1–1 (2017)

    Google Scholar 

  36. Lutton, E.: Evolution of fractal shapes for artists and designers. IJAIT Int. J. Artif. Intell. Tools 15(4), 651–672 (2006) (Special Issue on AI in Music and Art)

    Article  Google Scholar 

  37. Mackay, W.: Responding to cognitive overhead: co-adaptation between users and technology. Intellectica 30(1), 177–193 (2000)

    Google Scholar 

  38. North, C., Endert, A., Fiaux, P.: Semantic interaction for sensemaking: inferring analytical reasoning for model steering. IEEE Trans. Vis. Comput. Graph. 18, 2879–2888 (2012)

    Article  Google Scholar 

  39. Poli, R., Cagnoni, S.: Genetic programming with user-driven selection: experiments on the evolution of algorithms for image enhancement. In: Genetic Programming Conference, pp. 269–277. Morgan Kaufmann (1997)

    Google Scholar 

  40. Sacha, D., Sedlmair, M., Zhang, L., Lee, J., Weiskopf, D., North, S., Keim, D.: Human-centered machine learning through interactive visualization. ESANN (2016)

    Google Scholar 

  41. Saraiya, P., North, C., Duca, K.: An insight-based methodology for evaluating bioinformatics visualizations. IEEE Trans. Vis. Comput. Graph. 11(4), 443–456 (2005)

    Article  Google Scholar 

  42. Sedlmair, M., Brehmer, M., Ingram, S., Munzner, T.: Dimensionality reduction in the wild: Gaps and guidance. Department of Computer Science, University British Columbia, Vancouver, BC, Canada, Technical Report TR-2012-03 (2012)

    Google Scholar 

  43. Song, Y., Pickup, D., Li, C., Rosin, P., Hall, P.: Abstract art by shape classification. IEEE Trans. Vis. Comput. Graph. 19(8), 1252–1263 (2013)

    Article  Google Scholar 

  44. Stanley, K., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolut. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  45. Takagi, H.: Interactive evolutionary computation: system optimisation based on human subjective evaluation. In: Proceedings of Intelligent Engineering Systems (INES’98). IEEE (1998)

    Google Scholar 

  46. Tonda, A., Spritzer, A., Lutton, E.: Balancing user interaction and control in Bayesian network structure learning. In: Artificial Evolution Conference. LNCS, vol. 8752. Springer, Berlin (2013)

    Google Scholar 

  47. Valigiani, G., Lutton, E., Jamont, Y., Biojout, R., Collet, P.: Automatic rating process to audit a man-hill. WSEAS Trans. Adv. Eng. Educ. 3(1), 1–7 (2006)

    Google Scholar 

  48. Wenskovitch, J., North, C.: Observation-level interaction with clustering and dimension reduction algorithms. In: Proceedings of the 2Nd Workshop on Human-In-the-Loop Data Analytics, HILDA’17, pp. 14:1–14:6. ACM, New York, NY, USA (2017)

    Google Scholar 

  49. Wilkinson, L., Anand, A., Grossman, R.: Graph-theoretic scagnostics (2005)

    Google Scholar 

  50. Yaochu, J., Branke, J.: Evolutionary optimization in uncertain environments-a survey. IEEE Trans. Evolut. Comput. 9(3), 303–317 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. INRA, Université Paris-Saclay, 1 av. Brétignières, 78850, Thiverval-Grignon, France

    Nadia Boukhelifa & Evelyne Lutton

  2. Univ Paris-Sud, CNRS, INRIA, Université Paris-Saclay, Orsay, 91405, France

    Anastasia Bezerianos

Authors
  1. Nadia Boukhelifa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anastasia Bezerianos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Evelyne Lutton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nadia Boukhelifa .

Editor information

Editors and Affiliations

  1. DATA61, CSIRO, Eveleigh, New South Wales, Australia

    Jianlong Zhou

  2. DATA61, CSIRO, Eveleigh, New South Wales, Australia

    Fang Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Boukhelifa, N., Bezerianos, A., Lutton, E. (2018). Evaluation of Interactive Machine Learning Systems. In: Zhou, J., Chen, F. (eds) Human and Machine Learning. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-90403-0_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-90403-0_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-90402-3

  • Online ISBN: 978-3-319-90403-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation