Elsevier

Decision Support Systems

Volume 52, Issue 2, January 2012, Pages 331-341
Decision Support Systems

Design of an information volatility measure for health care decision making

https://doi.org/10.1016/j.dss.2011.08.009Get rights and content

Abstract

Health care decision makers and researchers often use reporting tools (e.g. Online Analytical Processing (OLAP)) that present data aggregated from multiple medical registries and electronic medical records to gain insights into health care practices and to understand and improve patient outcomes and quality of care. An important limitation is that the data are usually displayed as point estimates without full description of the instability of the underlying data, thus decision makers are often unaware of the presence of outliers or data errors. To manage this problem, we propose an Information Volatility Measure (IVM) to complement business intelligence (BI) tools when considering aggregated data (intra-cell) or when observing trends in data (inter-cell). The IVM definitions and calculations are drawn from volatility measures found in the field of finance, since the underlying data in both arenas display similar behaviors. The presentation of the IVM is supplemented with three types of benchmarking to support improved user understanding of the measure: numerical benchmarking, graphical benchmarking, and categorical benchmarking. The IVM is designed and evaluated using exploratory and confirmatory focus groups.

Highlights

► Information Volatility Measure describes instability of data in BI tools. ► IVM is designed and evaluated using feedback from expert focus groups. ► IVM definitions and calculations are drawn from volatility measures in finance. ► IVM includes three types of benchmarking: numerical, graphical, and categorical.

Introduction

Health care decision makers and researchers often use aggregated data from centralized repositories and/or data warehouses to gain insights into health care practices and to understand and improve patient outcomes and quality of care. For example, policy makers may analyze aggregated data to point out significant variations in hospitalization rates for expensive medical interventions [13], or researchers may compare cancer mortality rates in similar communities by investigating average tumor size at detection.

These aggregated data are often created by combining administrative data collected across several sources as a patient traverses the health care system. Data are collected from the moment a patient enrolls in a health plan and along each step as he or she seeks care as health care providers and payers approve expenditures and track service utilization and monitor cost and performance. Administrative data are submitted when billing for care. These data are not explicitly collected to examine the health or healthcare of populations, but offer important advantages for this purpose [24].

Unfortunately, in the United States the health care delivery system is fragmented which impedes the creation of longitudinal, population-based databases. Efforts have been made to centralize this information in medical registries. Thanks to increased availability of data and new health information exchange initiatives with centralized repositories, data from multiple medical registries and from electronic medical records can be stored in data warehouses and, thus are available to create meaningful insights [6], [18]. Data from data warehouses are presented to decision makers using reporting tools (often referred to as business intelligence (BI) tools) such as Online Analytical Processing (OLAP) which display aggregated data to help decision makers make comparisons and observe trends [42]. One challenge to decision makers is that the data are usually displayed as point estimates (typically mean values) and do not contain information about instability (such as variation around the mean), so decision makers often are unaware of the presence of outliers or data errors. When aggregating data from several medical registries, the possibility of type mismatches and other integration problems, and well documented problems with data quality from the original data sources have the potential to create the illusion of data trends or data shifts where none exist. In these cases, a descriptive analysis of the data can often provide an understanding of any unusual patterns. Yet, the impact and importance of the variability of a point estimate or trend on a decision is difficult to quantify, since it is highly subjective and dependent on the context of the decision being made.

To address this problem, we propose a measure of data variability termed an information volatility measure (IVM) and we introduce the notion of benchmarking the variability of data vis-à-vis a standard baseline to support better user understanding. Volatility is defined as a measure of instability of underlying data: data that are relatively stable exhibit low volatility and vice-versa. We implement the IVM design in the health care context, in particular in the area of public health decision making and evaluate the effectiveness and utility of the design with expert users, through the use of focus groups. We calculate IVM based on the underlying distribution of the point estimate. We examine the two most common distributions we have found in aggregated health care data: a normal distribution and log normal distribution. For data that are normally distributed we argue that the coefficient of variation (i.e. the standard deviation divided by the mean) is best. For log normal data (as is often the case for time series data) we borrow from finance where the term ‘volatility’ connotes the variability of stock returns over time, namely the standard deviation of log-relative stock returns.

Because the best way to present the IVM to decision makers is not self-evident, we employ the design science research paradigm [17], [21], [36] to develop a presentation method and then refine and evaluate the IVM measure. We implement the IVM using simple OLAP interfaces. The refinement and evaluation of the IVM is done by conducting four focus groups consisting of domain experts. Though the focus group technique is a qualitative approach, our epistemology is positivist. We introduce several manipulations within the focus groups to compare decisions made without the IVM measure and with several versions of the IVM. Additionally, we conduct two types of focus groups: exploratory and confirmatory [43]. The first two (exploratory) focus groups are mainly used to provide feedback to be utilized for design changes and improvement of the IVM. The second two (confirmatory) focus groups to provide evidence of utility and efficacy of the IVM metric in field settings.

The remainder of this paper is organized as follows. We begin with a discussion of related work. We follow with a detailed definition of information volatility and a description of how the information volatility measure is calculated and implemented. We describe how four focus groups were conducted and present the results of the evaluations. We conclude with a discussion of our findings and suggest future research directions.

Section snippets

Related work

Information data products are manufactured much like any other product [3], [30], [31], [34], [38], [46], [48]. Information producers generate and provide the “raw material” which is stored and maintained by information systems (or custodians) and accessed and utilized by information consumers for their tasks [38], [48], creating a data product. As in manufacturing, the data products are in turn the raw material for a different data manufacturing process [48]. Thus, just like the inputs and

Information volatility measure

There is a rich literature that documents decision making biases under uncertainty [44]. Information products inform decisions in the context of unknowns arising from complexity and inherent biases in our decision making abilities. It requires cognitive effort to reframe problems and mitigate our natural biases. Information volatility and this research agenda in general aim to develop quantitative measures that can be incorporated into information products to highlight potentially misleading

Design of the information volatility measure

The stability of data from a certain source in the information supply chain can be examined by considering the rate of change and impact of change in the values it provides over a grouping variable or by its dispersion about a central tendency. Assuming a normal distribution, a confidence interval can give a decision maker a feel for the stability of the data. For example, a large confidence interval is indicative of data that are not tightly distributed along the mean, thus displaying

Refinement and evaluation of IVM using focus groups

The design of artifacts can be described as having two phases: the development of the artifact and its evaluation. This is a process which involves frequent iteration between development and evaluation rather than a strictly procedural approach [25]. A design researcher not only designs an artifact but must provide evidence that this artifact solves a real problem. After careful consideration of several possible evaluation techniques we decided to evaluate the IVM using expert focus groups. We

Discussion and future work

In our research, we investigate an important issue: the instability of the underlying data for point estimates and trends used in tools that present aggregated data, such as OLAP, spreadsheet or reporting tools. Statisticians are well aware that aggregated values should always be presented with a measure of confidence, yet this is rarely done in practice. Decision makers rarely consider the reliability of the underlying data when they formulate their decisions. As a result, they may be making

Monica Chiarini Tremblay is an Assistant Professor in the Decision Sciences and Information Systems Department in the College of Business Administration at Florida International University in Miami. Her research interests focus on data analytics and business intelligence, data and text mining, data quality, data warehousing, decision support systems and knowledge management, particularly in the context of healthcare. Specifically, she concentrates on electronic health records, health

References (50)

  • S.S. Yoon et al.

    Analysis of data-collection methods for an acute stroke care registry

    American Journal of Preventive Medicine

    (2006)
  • Y. Adachi et al.

    Tumor size as a simple prognostic indicator for gastric carcinoma

    Annals of Surgical Oncology

    (1997)
  • D.G.T. Arts et al.

    Defining and improving data quality in medical registries: a literature review, case study, and generic framework

    Journal of the American Medical Informatics Association

    (2002)
  • D. Ballou et al.

    Modeling information manufacturing systems to determine information product quality

    Management Science

    (1998)
  • D.P. Ballou et al.

    Modeling data and process quality in multi-input

    Multi-Output Information Systems Management Science

    (1985)
  • D.P. Ballou et al.

    Modeling completeness versus consistency tradeoffs in information decision contexts

    Knowledge and Data Engineering, IEEE Transactions on

    (2003)
  • S. Croome

    Understanding volatility measurements

  • L.A. Cunningham

    How to think like Benjamin Graham and invest like Warren Buffett

    (2001)
  • M.R. Dambro et al.

    Assessing the quality of data entry in a computerized medical records system

    Journal of Medical Systems

    (1988)
  • R.A. Deyo et al.

    Cost, controversy, crisis: low back pain and the health of the public

    Annual Review of Public Health

    (1991)
  • T.J. Eggebraaten et al.

    A health-care data model based on the HL7 reference information model

    IBM Systems Journal

    (2007)
  • R.D. Gorsky et al.

    The cost effectiveness of prenatal care in reducing low birth weight in New Hampshire

    Health Services Research

    (1989)
  • S. Gregor et al.

    The anatomy of a design theory

    Journal of the Association for Information Systems

    (2007)
  • S. Hasan et al.

    Analyzing the effect of data quality on the accuracy of clinical decision support systems: a computer simulation approach

  • D. Heath et al.

    Local volatility function models under a benchmark approach

    Quantitative Finance

    (2006)
  • Cited by (31)

    • Medication adherence: A method for designing context-aware reminders

      2019, International Journal of Medical Informatics
      Citation Excerpt :

      In this section, we provide a general discussion and some observations, followed by the key contributions the paper makes. The healthcare industry poses important social challenges and interesting research possibilities [4,11,52,105–112]. From the comprehensive literature review (Section 2), we identified the need for a method to design context-aware reminders with additional special requirements of lower cost, lower side effect or higher reliability.

    • Breaking the chain: GPO changes and hospital supply cost efficiency

      2019, International Journal of Production Economics
    • Healthcare support for underserved communities using a mobile social media platform

      2017, Information Systems
      Citation Excerpt :

      These are detailed next. Tremblay et al. [52] proposed an approach of using focus group both for exploratory and confirmatory purposes of artifact evaluation. We adopted the approach for confirmatory purposes due to our project's requirements.

    View all citing articles on Scopus

    Monica Chiarini Tremblay is an Assistant Professor in the Decision Sciences and Information Systems Department in the College of Business Administration at Florida International University in Miami. Her research interests focus on data analytics and business intelligence, data and text mining, data quality, data warehousing, decision support systems and knowledge management, particularly in the context of healthcare. Specifically, she concentrates on electronic health records, health information exchanges and medical passports. Dr. Tremblay is the principal and co-investigator on several large federally and state funded grants. Her work has been published in European Journal of Information Systems, Communications of the AIS, ACM Journal of Data and Information Quality, Information Technology and Management, Decision Support Systems, Journal of Computer Information Systems, and Health Progress.

    Alan R. Hevner is an Eminent Scholar and Professor in the Information Systems and Decision Sciences Department in the College of Business at the University of South Florida. He holds the Citigroup/Hidden River Chair of Distributed Technology. Dr. Hevner's areas of research interest include information systems development, software engineering, distributed database systems, healthcare information systems, and service-oriented computing. Dr. Hevner has co-authored a 2010 book on design science research, presented seminars internationally on the topic, and co-founded an international annual conference (Design Science Research in Information Systems and Technology – DESRIST) that is in its sixth year. He has published over 150 research papers on these topics and has consulted for a number of Fortune 500 companies. Dr. Hevner received a Ph.D. in Computer Science from Purdue University. He has held faculty positions at the University of Maryland and the University of Minnesota. Dr. Hevner is a member of ACM, IEEE, AIS, and INFORMS. Recently, he served as a program manager at the U.S. National Science Foundation in the Computer and Information Science and Engineering (CISE) Directorate.

    Donald J. Berndt is an associate professor in the Information Systems Decision Sciences Department at the University of South Florida in Tampa. His areas of expertise include data mining, business intelligence, bioterrorism surveillance, and healthcare data warehousing and management. His work on business intelligence and the role of data in effective bioterrorism surveillance systems was published in Decision Support Systems: Special Issue on Cybersecurity for Homeland Security. Other papers have been published journals such as the American Journal of Preventive Medicine and Journal of Computer Information Systems. A frequent presenter at academic conferences, Berndt also coauthors for computer science textbooks and serves as chief technology officer for a healthcare information management company.

    View full text