Skip to main content

Evaluation of Visualization Systems with Long-Term Case Studies

  • Chapter
  • First Online:
Foundations of Data Visualization
  • 2387 Accesses

Abstract

New visualization systems need to be evaluated ideally with participants representing the target user group doing real tasks. Most evaluations are short, i.e. participants receive an instruction and use the system only once. These short evaluations have many drawbacks, e.g., participants do not get familiar with the system. Long-term case studies that involve the regular use of a system for at least several weeks lead to more reliable assessments of user acceptance. We discuss strategies to plan and conduct long-term case studies and present examples that demonstrate the feasibility of this evaluation strategy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bentley, R., Hughes, J.A., Randall, D., Rodden, T., Sawyer, P., Shapiro, D., Sommerville, I.: Ethnographically-informed systems design for air traffic control. In: Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work, pp. 123–129 (1992)

    Google Scholar 

  2. Breakwell, G.M., Hammond, S., Fife-Scha, C.: Research Methods in Psychology. Sage Publications, Thousand Oaks (1995)

    Google Scholar 

  3. Carpendale, S.: Evaluating information visualizations. Information Visualization, pp. 19–45. Springer, Berlin (2008)

    Google Scholar 

  4. Crabtree, A., Rodden, T., Tolmie, P., Button, G.: Ethnography considered harmful. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 879–888 (2009)

    Google Scholar 

  5. Elmqvist, N., Yi, J.S.: Patterns for visualization evaluation. Inf. Vis. 14(3), 250–269 (2015)

    Article  Google Scholar 

  6. González, V., Kobsa, A.: A workplace study of the adoption of information visualization systems. Proceedings of I-KNOW, vol. 3, pp. 92–102 (2003)

    Google Scholar 

  7. Hughes, J., King, V., Rodden, T., Andersen, H.: The role of ethnography in interactive systems design. Interactions 2(2), 56–65 (1995)

    Google Scholar 

  8. Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., Möller, T.: A systematic review on the practice of evaluating visualization. IEEE Trans. Vis. Comput. Graph. 19(12), 2818–2827 (2013)

    Google Scholar 

  9. Joshi, A., Papanastassiou, A., Vives, K., Spencer, D., Staib, L., Papademetris, X.: Light-sensitive visualization of multimodal data for neurosurgical applications. In: IEEE International Symposium on Biomedical Imaging (ISBI) (2010)

    Google Scholar 

  10. Joshi, A., Qian, X., Dione, D.P., Bulsara, K.R., Breuer, C.K., Sinusas, A.J., Papademetris, X.: Effective visualization of complex vascular structures using a non-parametric vessel detection method. IEEE Trans. Vis. Comput. Graph. (VIS 2008), 14(6), (2008)

    Google Scholar 

  11. Joshi, A., Scheinost, D., Okuda, H., Murphy, I., Staib, L.H., Papademetris, X.: Unified framework for development, deployment and testing of image analysis algorithms. In: MICCAI Workshop on Systems and Architectures for Computer Assisted Interventions (2009)

    Google Scholar 

  12. Joshi, A., Scheinost, D., Spann, M., Papademetris, X.: Evaluation of multi-viewport based visualization for electrode navigation during stereotactic image guided neurosurgery. In: International Brain Mapping and Interoperative Surgical Planning Society’s 6th World Congress for Brain Mapping and Image Guided Therapy (2009)

    Google Scholar 

  13. Joshi, A., Scheinost, D., Vives, K.P., Spencer, D.D., Staib, L.H., Papademetris, X.: Novel interaction techniques for neurosurgical planning and stereotactic navigation. IEEE Trans. Vis. Comput. Graph. (VIS 2008), 14(6), (2008)

    Google Scholar 

  14. Kang, Y.A., Gorg, C., Stasko, J.: Evaluating visual analytics systems for investigative analysis: deriving design principles from a case study. In: Proceedings of IEEE Visual Analytics Science and Technology, pp. 139–146 (2009)

    Google Scholar 

  15. Kang, Y.A., Stasko, J.: Examining the use of a visual analytics system for sensemaking tasks: case studies with domain experts. IEEE Trans. Vis. Comput. Graph. 18(12), 2869–2878 (2012)

    Google Scholar 

  16. Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S.: Empirical studies in information visualization: seven scenarios. IEEE Trans. Vis. Comput. Graph. 18(9), 1520–1536 (2012)

    Google Scholar 

  17. Ohly, S., Sonnentag, S., Niessen, C., Zapf, D.: Diary studies in organizational research: an introduction and some practical recommendations. J. Pers. Psychol. 9, 79–93 (2010)

    Article  Google Scholar 

  18. Perer, A., Shneiderman, B.: Integrating statistics and visualization: case studies of gaining clarity during exploratory data analysis. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 265–274 (2008)

    Google Scholar 

  19. Preim, B., Baer, A., Cunningham, D., Isenberg, T., Ropinski, T.: A survey of perceptually motivated 3d visualization of medical image data. Comput. Graph. Forum 35(3), 501–525 (2016)

    Article  Google Scholar 

  20. Preim, B., Isenberg, P., Ropinski, T.: A critical analysis of the evaluation practice in medical visualization. In: Proceedings of the EG Workshop on Visual Computing in Biology and Medicine (2018)

    Google Scholar 

  21. Saalfeld, P., Luz, M., Berg, P., Preim, B., Saalfeld, S.: Guidelines for quantitative evaluation of medical visualizations on the example of 3d aneurysm surface comparisons. Comput. Graph. Forum 37, (2018)

    Google Scholar 

  22. Saraiya, P., North, C., Lam, V., Duca, K.A.: An insight-based longitudinal study of visual analytics. IEEE Trans. Vis. Comput. Graph. 12(6), 1511–1522 (2006)

    Google Scholar 

  23. Seo, J., Shneiderman, B.: A rank-by-feature framework for unsupervised multidimensional data exploration using low dimensional projections. In: Proceedings of IEEE Symposium on Information Visualization, pp. 65–72 (2004)

    Google Scholar 

  24. Seo, J., Shneiderman, B.: Knowledge discovery in high-dimensional data: case studies and a user survey for the rank-by-feature framework. IEEE Trans. Vis. Comput. Graph. 12(3), 311–322 (2006)

    Google Scholar 

  25. Shneiderman, B., Plaisant, C.: Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. In: Proceedings of the Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (2006)

    Google Scholar 

  26. Stasko, J.T., Görg, C., Liu, Z., Singhal, K.: Jigsaw: supporting investigative analysis through interactive visualization. In: Proceedings of the IEEE Symposium on Visual Analytics Science and Technology, pp. 131–138 (2007)

    Google Scholar 

  27. Valiati, E.R., Freitas, C.M., Pimenta, M.S.: Using multi-dimensional in-depth long-term case studies for information visualization evaluation. In: Proceedings of the Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization, p. 9 (2008)

    Google Scholar 

  28. Whittaker, S., Sidner, C.L.: Email overload: exploring personal information management of email. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 276–283 (1996)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Bernhard Preim .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Preim, B., Joshi, A. (2020). Evaluation of Visualization Systems with Long-Term Case Studies. In: Chen, M., Hauser, H., Rheingans, P., Scheuermann, G. (eds) Foundations of Data Visualization. Springer, Cham. https://doi.org/10.1007/978-3-030-34444-3_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-34444-3_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-34443-6

  • Online ISBN: 978-3-030-34444-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics