Skip to main content
SpringerLink
Log in

Empirical Evaluations with Domain Experts

  • Chapter
  • First Online:
Book cover Foundations of Data Visualization

Abstract

Over the past thirty years, the visualization community has developed theories and models to explain visualization as a technology that augments human cognition by enabling the efficient, accurate, and timely discovery of meaningful information in data. Along the way, practitioners have also debated theories and practices for visualization evaluation: How do we generate durable, reliable evidence that a visualization is effective? Interestingly, there is still no consensus in the visualization research community how to evaluate visualization methods. The goal of this chapter is to rise awareness of still open issues in the visualization evaluation and to discuss appropriate evaluations suitable for different visualization approaches. This includes user studies and best practices to conduct them but also other approaches for suitable evaluation of visualization. The chapter is structured as a moderated dialog of two visualization experts.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdul-Rahman, A., Chen, M., Laidlaw, D.H.: A survey of variables used in empirical studies for visualization. In: Chen, M., Hauser, H., Rheingans, P., Scheuermann, G. (eds.) Foundations of Data Visualization. Springer, Berlin (2019)

    Google Scholar 

  2. Chen, M., Ebert, D.S.: An ontological framework for supporting the design and evaluation of visual analytics systems. Comput. Graph. Forum 38(3), 131–144. https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13677 (2019). https://doi.org/10.1111/cgf.13677

  3. Demiralp, C., Jackson, C., Karelitz, D., Zhang, S., Laidlaw, D.H.: Cave and fishtank virtual-reality displays: a qualitative and quantitative comparison. IEEE Trans. Vis. Comput. Graph. 12(3), 323–330 (2006)

    Article  Google Scholar 

  4. Gillmann, C., Wischgoll, T., Hamann, B., Hagen, H.: Accurate and reliable extraction of surfaces from image data using a multi-dimensional uncertainty model. Graph. Model. 99, 13–21. http://www.sciencedirect.com/science/article/pii/S1524070318300365 (2018). https://doi.org/10.1016/j.gmod.2018.07.004

  5. Glendenning, K., Wischgoll, T., Harris, J., Vickery, R., Blaha, L.: Parameter space visualization for large-scale datasets using parallel coordinate plots. J. Imaging Sci. Technol. 60(1), 10,406–1–10,406–8 (2016)

    Google Scholar 

  6. Gomez, S.R., Guo, H., Ziemkiewicz, C., Laidlaw, D.H.: An insight- and task-based methodology for evaluating spatiotemporal visual analytics. In: Proceedings of IEEE VAST (2014)

    Google Scholar 

  7. Guo, H., Gomez, S.R., Ziemkiewicz, C., Laidlaw, D.H.: A case study using visualization interaction logs and insight metrics to understand how analysts arrive at insights. In: Proceedings of IEEE VAST (2015)

    Google Scholar 

  8. Holten, D., Van Wijk, J.J.: Force-directed edge bundling for graph visualization. Comput. Graph. Forum 28(3), 983–990 (2009). https://doi.org/10.1111/j.1467-8659.2009.01450.x

    Article  Google Scholar 

  9. Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., Möller, T.: A systematic review on the practice of evaluating visualization. IEEE Trans. Vis. Comput. Graph. 19(12), 2818–2827 (2013). https://doi.org/10.1109/TVCG.2013.126

    Article  Google Scholar 

  10. Keefe, D., Acevedo, D., Moscovich, T., Laidlaw, D.H., LaViola, J.: Cavepainting: a fully immersive 3D artistic medium and interactive experience. In: Proceedings of ACM Symposium on Interactive 3D Graphics, pp. 85–93 (2001)

    Google Scholar 

  11. Kirby, M., Marmanis, H., Laidlaw, D.H.: Visualizing multivalued data from 2D incompressible flows using concepts from painting. Proc. IEEE Vis. 1999, 333–340 (1999)

    Google Scholar 

  12. Koehler, C., Wischgoll, T., Dong, H., Gaston, Z.: Vortex visualization in ultra low reynolds number insect flight. IEEE Trans. Vis. Comput. Graph. 17(12), 2071–2079 (2011). https://doi.org/10.1109/TVCG.2011.260

    Article  Google Scholar 

  13. Kosara, R., Healey, C.G., Interrante, V., Laidlaw, D.H., Ware, C.: User studies: why, how, and when. Comput. Graph. Appl. 23(4), 20–25 (2003)

    Article  Google Scholar 

  14. Laidlaw, D.H., Kirby, M., Jackson, C., Davidson, J.S., Miller, T., DaSilva, M., Warren, W., Tarr, M.: Comparing 2D vector field visualization methods: a user study. IEEE Trans. Vis. Comput. Graph. 11(1), 59–70 (2005)

    Article  Google Scholar 

  15. Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S.: Empirical studies in information visualization: seven scenarios. IEEE Trans. Vis. Comput. Graph. 18(9), 1520–1536 (2012). https://doi.org/10.1109/TVCG.2011.279

    Article  Google Scholar 

  16. Lam, H., Tory, M., Munzner, T.: Bridging from goals to tasks with design study analysis reports. IEEE Trans. Vis. Comput. Graph. 24(1), 435–445 (2018). https://doi.org/10.1109/TVCG.2017.2744319

    Article  Google Scholar 

  17. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. SIGGRAPH Comput. Graph. 21(4), 163–169 (1987). https://doi.org/10.1145/37402.37422

    Article  Google Scholar 

  18. Matkovic, K., Gracanin, D., Jelovic, M., Hauser, H.: Interactive visual steering - rapid visual prototyping of a common rail injection system. IEEE Trans. Vis. Comput. Graph. 14(6), 1699–1706 (2008). https://doi.org/10.1109/TVCG.2008.145

    Article  Google Scholar 

  19. Meyer, M., Sedlmair, M., Munzner, T.: The four-level nested model revisited: blocks and guidelines. In: Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization, BELIV ’12, pp. 11:1–11:6. ACM, New York, NY, USA (2012). https://doi.org/10.1145/2442576.2442587

  20. Munzner, T.: A nested model for visualization design and validation. IEEE Trans. Vis. Comput. Graph. 15(6), 921–928 (2009). https://doi.org/10.1109/TVCG.2009.111

    Article  Google Scholar 

  21. Sedlmair, M., Meyer, M., Munzner, T.: Design study methodology: reflections from the trenches and the stacks. IEEE Trans. Vis. Comput. Graph. 18(12), 2431–2440 (2012). https://doi.org/10.1109/TVCG.2012.213

    Article  Google Scholar 

  22. Smith, G.C.S., Pell, J.P.: Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ 327(7429), 1459–1461. https://www.bmj.com/content/327/7429/1459 (2003). https://doi.org/10.1136/bmj.327.7429.1459

  23. Upson, C., Faulhaber, T.A., Kamins, D., Laidlaw, D., Schlegel, D., Vroom, J., Gurwitz, R., van Dam, A.: The application visualization system: a computational environment for scientific visualization. IEEE Comput. Graph. Appl. 9(4), 30–42 (1989). https://doi.org/10.1109/38.31462

    Article  Google Scholar 

  24. van Dam, A., Laidlaw, D.H., Simpson, R.M.: Experiments in immersive virtual reality for scientific visualization. Comput. Graph. 26(4), 535–555 (2002)

    Article  Google Scholar 

  25. Weber, G.H., Carpendale, S., Ebert, D., Fisher, B., Hagen, H., Shneiderman, B., Ynnerman, A.: Apply or die: on the role and assessment of application papers in visualization. IEEE Comput. Graph. Appl. 37(3), 96–104 (2017). https://doi.org/10.1109/MCG.2017.51

    Article  Google Scholar 

  26. Wischgoll, T., Glines, M., Whitlock, T., Guthrie, B.R., Mowrey, C.M., Parikh, P.J., Flach, J.: Display infrastructure for virtual environments (dive). J. Imaging Sci. Technol. 61(6), 60,406–1–60,406–11 (2017)

    Google Scholar 

  27. Wischgoll, T., Scheuermann, G.: Detection and visualization of closed streamlines in planar flows. IEEE Trans. Vis. Comput. Graph. 7(2), 165–172 (2001). https://doi.org/10.1109/2945.928168

    Article  Google Scholar 

  28. Ziemkiewicz, C., Chen, M., Laidlaw, D., Preim, B., Weiskopf, D.: Open challenges in empirical visualization research. In: Chen, M., Hauser, H., Rheingans, P., Scheuermann, G. (eds.) Foundations of Data Visualization. Springer, Berlin (2019)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank Laura McNamara for numerous discussions. Her input was very valuable and it helped improve the chapter considerably.

Author information

Authors and Affiliations

  1. VRVis Zentrum für Virtual Reality und Visualisierung Forschungs-GmbH, Vienna, Austria

    Krešimir Matković

  2. Wright State University, Dayton, OH, USA

    Thomas Wischgoll

  3. Brown University, Providence, RI, USA

    David H. Laidlaw

Authors
  1. Krešimir Matković
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Wischgoll
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David H. Laidlaw
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Krešimir Matković .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Min Chen

  2. Department of Informatics, University of Bergen, Bergen, Norway

    Helwig Hauser

  3. University of Maryland, Baltimore County, Baltimore, MD, USA

    Penny Rheingans

  4. Institut für Informatik, Universität Leipzig, Leipzig, Germany

    Gerik Scheuermann

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Matković, K., Wischgoll, T., Laidlaw, D.H. (2020). Empirical Evaluations with Domain Experts. In: Chen, M., Hauser, H., Rheingans, P., Scheuermann, G. (eds) Foundations of Data Visualization. Springer, Cham. https://doi.org/10.1007/978-3-030-34444-3_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-34444-3_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-34443-6

  • Online ISBN: 978-3-030-34444-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation