skip to main content
10.1145/3025453.3025998acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Increasing Users' Confidence in Uncertain Data by Aggregating Data from Multiple Sources

Published: 02 May 2017 Publication History

Abstract

We often base our decisions on uncertain data - for instance, when consulting the weather forecast before deciding what to wear. Due to their uncertainty, such forecasts can differ by provider. To make an informed decision, many people compare several forecasts, which is a time-consuming and cumbersome task. To facilitate comparison, we identified three aggregation mechanisms for forecasts: manual comparison and two mechanisms of computational aggregation. In a survey, we compared the mechanisms using different representations. We then developed a weather application to evaluate the most promising candidates in a real-world study. Our results show that aggregation increases users' confidence in uncertain data, independent of the type of representation. Further, we find that for daily events, users prefer to use computationally aggregated forecasts. However, for high-stakes events, they prefer manual comparison. We discuss how our findings inform the design of improved interfaces for comparison of uncertain data, including non-weather purposes.

Supplementary Material

ZIP File (pn4441-file4.zip)

References

[1]
Nadia Boukhelifa and David J. Duke. 2009. Uncertainty Visualization: Why Might It Fail?. In CHI '09 Extended Abstracts on Human Factors in Computing Systems. ACM, New York, NY, USA, 4051--4056.
[2]
David V. Budescu, Stephen Broomell, and Han-Hui Por. 2009. Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change. Psychological Science 20, 3 (2009), 299--308. //dx.doi.org/10.1111/j.1467--9280.2009.02284.x
[3]
Michael Correll and Michael Gleicher. 2014. Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error. Visualization and Computer Graphics, IEEE Transactions on 20, 12 (Dec 2014), 2142--2151.
[4]
AMS Council. 2008. Enhancing weather information with probability forecasts. Bulletin of the American Meteorological Society 89 (2008), 1049--1053.
[5]
National Research Council. 2003. Communicating Uncertainties in Weather and Climate Information: A Workshop Summary. The National Academies Press.
[6]
National Research Council. 2006. Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts. The National Academies Press.
[7]
David Demeritt, Hannah Cloke, Florian Pappenberger, Jutta Thielen, Jens Bartholmes, and Maria-Helena Ramos. 2007. Ensemble predictions and perceptions of risk, uncertainty, and error in flood forecasting. Environmental Hazards 7, 2 (2007), 115 -- 127.
[8]
Nivan Ferreira, Danyel Fisher, and Arnd C. Konig. 2014. Sample-oriented Task-driven Visualizations: Allowing Users to Make Better, More Confident Decisions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 571--580.
[9]
J. Frick and C. Hegg. 2011. Can end-users' flood management decision making be improved by information about forecast uncertainty? Atmospheric Research 100, 23 (2011), 296 -- 303. //dx.doi.org/10.1016/j.atmosres.2010.12.006
[10]
Susan Joslyn, Karla Pak, David Jones, John Pyles, and Earl Hunt. 2007. The effect of probabilistic information on threshold forecasts. Weather and Forecasting 22, 4 (2007), 804--812.
[11]
Susan Joslyn and Sonia Savelli. 2010. Communicating forecast uncertainty: public perception of weather forecast uncertainty. Meteorological Applications 17, 2 (2010), 180--195.
[12]
Susan L. Joslyn and Jared E. LeClerc. 2012. Uncertainty forecasts improve weather-related decisions and attenuate the effects of forecast error. Journal of Experimental Psychology: Applied 18, 1 (2012), 126--140.
[13]
Malte F. Jung, David Sirkin, Turgut M. Gür, and Martin Steinert. 2015. Displayed Uncertainty Improves Driving Experience and Behavior: The Case of Range Anxiety in an Electric Car. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2201--2210.
[14]
Daniel Kahneman and Amos Tversky. 1979. Prospect Theory: An Analysis of Decision under Risk. Econometrica 47, 2 (1979), 263--291.
[15]
Matthew Kay, Tara Kola, Jessica R. Hullman, and Sean A. Munson. 2016. When (Ish) is My Bus?: User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5092--5103.
[16]
Matthew Kay, Dan Morris, M.C. Schraefel, and Julie A. Kientz. 2013. There's No Such Thing As Gaining a Pound: Reconsidering the Bathroom Scale User Interface. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 401--410.
[17]
Matthew Kay, Shwetak N. Patel, and Julie A. Kientz. 2015. How Good is 85 Classifier Evaluation to Acceptability of Accuracy. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 347--356.
[18]
H. Kootval. 2008. Guidelines on communicating forecast uncertainty. World Meteorological Organization/Technical Document 4122 (2008).
[19]
Jeffrey K. Lazo, Rebecca E. Morss, and Julie L. Demuth. 2009. 300 Billion Served. Bulletin of the American Meteorological Society 90, 6 (2009), 785--798.
[20]
Isaac M. Lipkus, Greg Samsa, and Barbara K. Rimer. 2001. General Performance on a Numeracy Scale among Highly Educated Samples. Medical Decision Making 21, 1 (2001), 37--44.
[21]
Alan M. MacEachren, Anthony Robinson, Susan Hopper, Steven Gardner, Robert Murray, Mark Gahegan, and Elisabeth Hetzler. 2005. Visualizing Geospatial Information Uncertainty: What We Know and What We Need to Know. Cartography and Geographic Information Science 32, 3 (2005), 139--160.
[22]
Rebecca E. Morss, Julie L. Demuth, and Jeffrey K. Lazo. 2008. Communicating Uncertainty in Weather Forecasts: A Survey of the U.S. Public. Weather and forecasting 23, 5 (2008), 974--991.
[23]
Rebecca E. Morss, Jeffrey K. Lazo, and Julie L. Demuth. 2010. Examining the use of weather forecasts in decision scenarios: results from a US survey with implications for uncertainty communication. Meteorological Applications 17, 2 (2010), 149--162.
[24]
Limor Nadav-Greenberg and Susan L. Joslyn. 2009. Uncertainty Forecasts Improve Decision Making Among Nonexperts. Journal of Cognitive Engineering and Decision Making 3, 3 (2009), 209--227.
[25]
Neville Nicholls. 1999. Cognitive Illusions, Heuristics, and Climate Prediction. Bulletin of the American Meteorological Society 80, 7 (1999), 1385--1397. 080<1385:CIHACP>2.0.CO;2
[26]
Chris Olston and Jock D. Mackinlay. 2002. Visualizing data with bounded uncertainty. In Information Visualization, 2002. INFOVIS 2002. IEEE Symposium on. 37--40.
[27]
Alex T. Pang, Craig M. Wittenbrink, and Suresh K. Lodha. 1997. Approaches to uncertainty visualization. The Visual Computer 13, 8 (1997), 370--390.
[28]
Kristin Potter, Joe Kniss, Richard Riesenfeld, and Chris.R. Johnson. 2010. Visualizing Summary Statistics and Uncertainty. Computer Graphics Forum 29, 3 (2010), 823--832. //dx.doi.org/10.1111/j.1467--8659.2009.01677.x
[29]
Kristin Potter, Paul Rosen, and Chris R. Johnson. 2012. From quantification to visualization: A taxonomy of uncertainty visualization approaches. In Uncertainty Quantification in Scientific Computing. Vol. 377. Springer, 226--249.
[30]
Mark S. Roulston, Gary E. Bolton, Andrew N. Kleit, and Addison L. Sears-Collins. 2006. A laboratory study of the benefits of including uncertainty information in weather forecasts. Weather and Forecasting 21, 1 (2006), 116--122.
[31]
Mark S. Roulston and Todd R. Kaplan. 2009. A laboratory-based study of understanding of uncertainty in 5-day site-specific temperature forecasts. Meteorological Applications 16, 2 (2009), 237--244.
[32]
Orit Shaer, Oded Nov, Johanna Okerlund, Martina Balestra, Elizabeth Stowell, Lauren Westendorf, Christina Pollalis, Jasmine Davis, Liliana Westort, and Madeleine Ball. 2016. GenomiX: A Novel Interaction Tool for Self-Exploration of Personal Genomic Data. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 661--672.
[33]
Michael Siegrist, Heinz Gutscher, and Timothy C. Earle. 2005. Perception of risk: the influence of general trust, and general confidence. Journal of Risk Research 8, 2 (2005), 145--156.
[34]
Meredith Skeels, Bongshin Lee, Greg Smith, and George G. Robertson. 2010. Revealing Uncertainty for Information Visualization. Information Visualization 9, 1 (2010), 70--81.
[35]
Paul Slovic. 1987. Perception of risk. Science (Washington, D.C.); (United States) 236 (Apr 1987).
[36]
Susanne Tak, Alexander Toet, and Jan van Erp. 2014. The Perception of Visual Uncertainty Representation by Non-Experts. Visualization and Computer Graphics, IEEE Transactions on 20, 6 (June 2014), 935--943.
[37]
Judi Thomson, Elizabeth Hetzler, Alan MacEachren, Mark Gahegan, and Misha Pavel. 2005. A typology for visualizing uncertainty. In Electronic imaging 2005. International Society for Optics and Photonics, 146--157.
[38]
Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185, 4157 (1974), 1124--1131. http://www.jstor.org/stable/1738360
[39]
Thomas S. Wallsten, David V. Budescu, Amnon Rapoport, Rami Zwick, and Barbara Forsyth. 1986. Measuring the vague meanings of probability terms. Journal of Experimental Psychology: General 115, 4 (1986), 348--365.
[40]
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 143--146.
[41]
Jianlong Zhou, Constant Bridon, Fang Chen, Ahmad Khawaji, and Yang Wang. 2015. Be Informed and Be Involved: Effects of Uncertainty and Correlation on User's Confidence in Decision Making. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15). ACM, New York, NY, USA, 923--928.
[42]
Torre Zuk and Sheelagh Carpendale. 2006. Theoretical analysis of uncertainty visualizations. Proc. SPIE 6060 (2006), 606007--606007--14.

Cited By

View all
  • (2023)Communicating Uncertainty and Risk in Air Quality MapsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.317144329:9(3746-3757)Online publication date: 1-Sep-2023
  • (2023)Combining uncertainty information with AI recommendations supports calibration with domain knowledgeJournal of Risk Research10.1080/13669877.2023.225940626:10(1137-1152)Online publication date: 13-Oct-2023
  • (2023)Visualizing Uncertainty in Different Domains: Commonalities and Potential Impacts on Human Decision-MakingVisualization Psychology10.1007/978-3-031-34738-2_14(331-369)Online publication date: 7-Nov-2023
  • Show More Cited By

Index Terms

  1. Increasing Users' Confidence in Uncertain Data by Aggregating Data from Multiple Sources

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    • Honorable Mention

    Author Tags

    1. aggregation
    2. comparison
    3. multiple sources
    4. uncertainty
    5. weather forecast

    Qualifiers

    • Research-article

    Funding Sources

    • Cluster of Excellence - SimTech

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)62
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Communicating Uncertainty and Risk in Air Quality MapsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.317144329:9(3746-3757)Online publication date: 1-Sep-2023
    • (2023)Combining uncertainty information with AI recommendations supports calibration with domain knowledgeJournal of Risk Research10.1080/13669877.2023.225940626:10(1137-1152)Online publication date: 13-Oct-2023
    • (2023)Visualizing Uncertainty in Different Domains: Commonalities and Potential Impacts on Human Decision-MakingVisualization Psychology10.1007/978-3-031-34738-2_14(331-369)Online publication date: 7-Nov-2023
    • (2022)Threddy: An Interactive System for Personalized Thread-based Exploration and Organization of Scientific LiteratureProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545660(1-15)Online publication date: 29-Oct-2022
    • (2022)Visual Analytics of Multivariate Intensive Care Time Series DataComputer Graphics Forum10.1111/cgf.1449841:6(273-286)Online publication date: 13-Apr-2022
    • (2022)Implicit Error, Uncertainty and Confidence in Visualization: An Archaeological Case StudyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.308833928:12(4389-4402)Online publication date: 1-Dec-2022
    • (2022)Communicating qualitative uncertainty in data visualizationInformation Design Journal10.1075/idj.22014.pan27:1(52-63)Online publication date: 7-Nov-2022
    • (2021)A Content Analysis Review of Literature to Create a Useable Framework for Reputation Risk ManagementHandbook of Research on Multidisciplinary Perspectives on Managerial and Leadership Psychology10.4018/978-1-7998-3811-1.ch006(91-133)Online publication date: 2021
    • (2020)Understanding Preferences of Delhi Metro Users Using Choice-Based Conjoint AnalysisIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2019.295825922:1(384-393)Online publication date: 24-Dec-2020
    • (2019)Telling the Story RightProceedings of the 13th EAI International Conference on Pervasive Computing Technologies for Healthcare10.1145/3329189.3329239(435-443)Online publication date: 20-May-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media