skip to main content
10.1145/3025453.3025882acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Empirical Analysis of the Subjective Impressions and Objective Measures of Domain Scientists' Visual Analytic Judgments

Published: 02 May 2017 Publication History

Abstract

Scientists often use specific data analysis and presentation methods familiar within their domain. But does high familiarity drive better analytical judgment? This question is especially relevant when familiar methods themselves can have shortcomings: many visualizations used conventionally for scientific data analysis and presentation do not follow established best practices. This necessitates new methods that might be unfamiliar yet prove to be more effective. But there is little empirical understanding of the relationships between scientists' subjective impressions about familiar and unfamiliar visualizations and objective measures of their visual analytic judgments. To address this gap and to study these factors, we focus on visualizations used for comparison of climate model performance. We report on a comprehensive survey-based user study with 47 climate scientists and present an analysis of: i) relationships among scientists' familiarity, their perceived levels of comfort, confidence, accuracy, and objective measures of accuracy, and ii) relationships among domain experience, visualization familiarity, and post-study preference.

Supplementary Material

suppl.mov (pn3272.mp4)
Supplemental video
suppl.mov (pn3272p.mp4)
Supplemental video

References

[1]
Jürgen Bernard, Martin Steiger, Sven Widmer, Hendrik Lücke-Tieke, Thorsten May, and Jörn Kohlhammer. 2014. Visual-interactive Exploration of Interesting Multivariate Relations in Mixed Research Data Sets. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 291--300.
[2]
Jacques Bertin. 1983. Semiology of graphics: diagrams, networks, maps. University of Wisconsin press.
[3]
Matthew Brehmer, Jocelyn Ng, Kevin Tate, and Tamara Munzner. 2016. Matches, Mismatches, and Methods: Multiple-View Workflows for Energy Portfolio Analysis. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 449--458.
[4]
Sabrina Bresciani and Martin J Eppler. 2009. The benefits of synchronous collaborative information visualization: Evidence from an experimental evaluation. IEEE transactions on visualization and computer graphics 15, 6 (2009), 1073--1080.
[5]
Bertjan Broeksema, Alexandru C Telea, and Thomas Baudel. 2013. Visual Analysis of Multi-Dimensional Categorical Data Sets. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 158--169.
[6]
Curt Covey, Krishna M AchutaRao, Michael Fiorino, Peter J Gleckler, Karl E Taylor, and Michael F Wehner. 2002. Intercomparison of climate data sets as a measure of observational uncertainty. UCRL.
[7]
Aritra Dasgupta and Robert Kosara. 2010. Pargnostics: Screen-Space Metrics for Parallel Coordinates. Transactions on Visualization and Computer Graphics 16, 6 (2010), 1017--26.
[8]
Aritra Dasgupta, Joon-Yong Lee, Ryan Wilson, Robert A Lafrance, Nick Cramer, Kristin Cook, and Samuel Payne. 2017. Familiarity Vs Trust: A Comparative Study of Domain Scientists' Trust in Visual Analytics and Conventional Analysis Methods. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017), 271--280.
[9]
Aritra Dasgupta, Jorge Poco, Yaxing Wei, Robert Cook, Enrico Bertini, and Claudio T. Silva. 2015. Bridging Theory with Practice: An Exploratory Study of Visualization Use and Design for Climate Model Comparison. IEEE Transactions on Visualization and Computer Graphics 21, 9 (2015), 996--1014.
[10]
Michael Friendly and SAS Institute. 2000. Visualizing categorical data. Sas Institute Cary, NC.
[11]
David Gefen. 2000. E-commerce: the role of familiarity and trust. Omega 28, 6 (2000), 725--737.
[12]
Peter J Gleckler, Karl E Taylor, and Charles Doutriaux. 2008. Performance metrics for climate models. Journal of Geophysical Research: Atmospheres 113, D6 (2008).
[13]
Michael Gleicher, Danielle Albers, Rick Walker, Ilir Jusufi, Charles D. Hansen, and Jonathan C. Roberts. 2011. Visual comparison for information visualization. Information Visualization 10, 4 (2011), 289--309.
[14]
Mark Harrower and Cynthia A Brewer. 2003. ColorBrewer. org: an online tool for selecting colour schemes for maps. The Cartographic Journal 40, 1 (2003), 27--37.
[15]
Yosef Hochberg. 1988. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 4 (1988), 800--802.
[16]
James W. Hurrell, M.M. Holland, P. R. Gent, S. Ghan, J. E. Kay, P. J. Kushner, J.-F. Lamarque, W.G. Large, D. Lawrence, K. Lindsay, W. H. Lipscomb, M. C. Long, N. Mahowald, D. R. Marsh, R. B. Neale, P. J. Rasch, S. Vavrus, M. Vertenstein, D. Bader, W.D. Collins, J.J. Hack, J. Kiehl, and S. Marshall. 2013. The Community Earth System Model: A Framework for Collaborative Research. Bulletin of the American Meteorological Society (2013), 1339--1360.
[17]
Alfred Inselberg and Bernard Dimsdale. 1990. Parallel Coordinates: A Tool for Visualizing Multi-Dimensional Geometry. In IEEE Visualization. IEEE CS Press, 361--378.
[18]
Kurt Koffka. 2013. Principles of Gestalt psychology. Vol. 44. Routledge.
[19]
Heidi Lam, Enrico Bertini, Petra Isenberg, Catherine Plaisant, and Sheelagh Carpendale. 2012. Empirical studies in information visualization: Seven scenarios. IEEE Transactions on Visualization and Computer Graphics 18, 9 (2012), 1520--1536.
[20]
Daniel J McAllister. 1995. Affect-and cognition-based trust as foundations for interpersonal cooperation in organizations. Academy of management journal 38, 1 (1995), 24--59.
[21]
T. Munzner. 2009. A nested process model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics 15, 6 (2009), 921--928.
[22]
Allan H Murphy. 1988. Skill scores based on the mean square error and their relationships to the correlation coefficient. Monthly Weather Review 116, 12 (1988), 2417--2424.
[23]
R B. Neale, C. C. Chen, A. Gettelman, P. H. Lauritzen, S. Park, D. L. Williamson, A. J. Conley, R. Garcia, D. Kinnison, J-F. Lamarque, D. Marsh, M. Mills, A. K. Smith, S. TIlmes, F. Vitt, H. Morrison, P. Cameron-Smith, W. D. Collins, M. J. Iacono, R. C. Easter, S. J. Ghan ans X. Liu, P. J. Rasch, and M. Taylor. 2012. Description of the NCAR Community Atmosphere Model: CAM5.0. Technical Report NCAR/TN-486+STR. National Center for Atmospheric Research, Boulder, Colorado, USA. 268 pp., URL = http://www.ccsm.ucar.edu/models/atm-cam.
[24]
Intergovernmental Panel on Climate Change. 2015. Climate Change 2014: Mitigation of Climate Change. Vol. 3. Cambridge University Press.
[25]
Catherine Plaisant. 2004. The challenge of information visualization evaluation. In Proceedings of the working conference on Advanced visual interfaces. ACM, 109--116.
[26]
Dominik Sacha, Hansi Senaratne, Bum Chul Kwon, Geoffrey Ellis, and Daniel A Keim. 2016. The role of uncertainty, awareness, and trust in visual analytics. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 240--249.
[27]
Leila Takayama and Eser Kandogan. 2006. Trust as an underlying factor of system administrator interface choice. In CHI'06 extended abstracts on Human factors in computing systems. ACM, 1391--1396.
[28]
Karl E Taylor. 2001. Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research: Atmospheres 106, D7 (2001), 7183--7192.
[29]
Edward R. Tufte. 2001. The Visual Display of Quantitative Information (2nd ed.). Graphics Press.
[30]
Ananth Uggirala, Anand K Gramopadhye, Brain J Melloy, and Joe E Toler. 2004. Measurement of trust in complex and dynamic systems using a quantitative approach. International Journal of Industrial Ergonomics 34, 3 (2004), 175--186.
[31]
Stef van den Elzen and Jarke J van Wijk. 2013. Small multiples, large singles: A new approach for visual data exploration. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 191--200.
[32]
Edward Vonesh and Vernon M Chinchilli. 1996. Linear and nonlinear models for the analysis of repeated measurements. CRC press.
[33]
C. Ware. 2004. Information visualization: Perception for Design. Morgan Kaufmann.

Cited By

View all
  • (2024)Struggles and Strategies in Understanding Information VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338856030:6(3035-3048)Online publication date: 15-Apr-2024
  • (2023)Trust in maps: what we know and what we need to knowCartography and Geographic Information Science10.1080/15230406.2023.228130652:1(1-18)Online publication date: 30-Nov-2023
  • (2022)Exploring how Temporal Framing Affects Trust with Time-series Visualizations2022 International Conference on Graphics and Interaction (ICGI)10.1109/ICGI57174.2022.9990428(1-8)Online publication date: 3-Nov-2022
  • Show More Cited By

Index Terms

  1. Empirical Analysis of the Subjective Impressions and Objective Measures of Domain Scientists' Visual Analytic Judgments

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. climate
    2. information visualization
    3. preference
    4. slope plot
    5. taylor plot
    6. trust
    7. visual comparison

    Qualifiers

    • Research-article

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)48
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Struggles and Strategies in Understanding Information VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338856030:6(3035-3048)Online publication date: 15-Apr-2024
    • (2023)Trust in maps: what we know and what we need to knowCartography and Geographic Information Science10.1080/15230406.2023.228130652:1(1-18)Online publication date: 30-Nov-2023
    • (2022)Exploring how Temporal Framing Affects Trust with Time-series Visualizations2022 International Conference on Graphics and Interaction (ICGI)10.1109/ICGI57174.2022.9990428(1-8)Online publication date: 3-Nov-2022
    • (2021)User Trust in Assisted Decision-Making Using Miniaturized Near-Infrared SpectroscopyProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445710(1-16)Online publication date: 6-May-2021
    • (2021)Comparative Layouts Revisited: Design Space, Guidelines, and Future DirectionsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.303041927:2(1525-1535)Online publication date: Feb-2021
    • (2020)Knowing what to look for: A Fact-Evidence Reasoning Framework for Decoding Communicative Visualization2020 IEEE Visualization Conference (VIS)10.1109/VIS47514.2020.00053(231-235)Online publication date: Oct-2020
    • (2020)Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing ModelsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2019.293454026:1(1043-1053)Online publication date: Jan-2020
    • (2020)The Effect of Color Scales on Climate Scientists’ Objective and Subjective Performance in Spatial Data Analysis TasksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2018.287653926:3(1577-1591)Online publication date: 1-Mar-2020
    • (2020)A Task-Based Taxonomy of Cognitive Biases for Information VisualizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2018.287257726:2(1413-1432)Online publication date: 1-Feb-2020
    • (2019)Evaluating Multi-Dimensional Visualizations for Understanding Fuzzy ClustersIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2018.286502025:1(12-21)Online publication date: 1-Jan-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media