An empirical and comparative analysis of E-government performance measurement models: Model selection via explanation, prediction, and parsimony

Highlights

• A comparison between Expectancy-Disconfirmation & Service Quality paradigms for citizen E-Government satisfaction & trust.

• The role of explanation & prediction of citizen trust as empirical modeling goals is highlighted.

• Expectancy-Disconfirmation paradigm well-suited suited for explaining citizen trust.

• A parsimonious model with an “overall quality-satisfaction-trust” link is best suited for predicting trust.

• The service quality paradigm offers the best compromise between predictive accuracy & explanatory power.

Abstract

Driven by the growing importance of the digital provision of government services (e-government), recent research has sought to develop and test conceptual models of citizen satisfaction and trust with these services. Yet, there remains little agreement on how to optimally model these relationships with regards to the somewhat divergent goals of explanation and prediction of citizen trust. In this paper, we test two prominent modeling paradigms of the e-government satisfaction-trust relationship: the “service quality” model and the “expectancy-disconfirmation” model. We compare several variations of these models for their in-sample explanatory abilities, out-of-sample predictive abilities, and parsimony. To test the models, we examine a pooled, cross-agency sample of survey data measuring citizens' experiences with and perceptions of three important and widely accessed U.S. federal e-government services—the webpages of the Social Security Administration, the Internal Revenue Service, and the U.S. Census Bureau. Our findings suggest that while the expectancy-disconfirmation paradigm performs well in explanation, a parsimonious model with an “overall quality-satisfaction-trust” link is best suited for predicting trust. In addition, the service quality paradigm offers the best compromise between predictive accuracy and explanatory power. These findings offer new insights for academic researchers, government agencies, and practitioners, especially those deciding upon an empirical model to adopt to measure e-government satisfaction and its impact upon citizen trust.

Introduction

The performance of services delivered through government information and communication technology (ICT) applications has a substantial impact on the overall performance of a government, on citizens' perceptions of that performance, and even on the success of some of the most important public policies enacted by the government. The validity of this statement is now beyond debate. The high-profile launch of the HealthCare.gov website by the U.S. federal government – and more specifically, the many failings of that website in the eyes of both the experts and the general public – evidences just how important e-government has become, not just to the quality of services offered to citizens, but also to the fundamental and foundational relationship between citizens and governments (Office of Inspector General, 2016). No longer is e-government just one among the many alternative channels of government service delivery; increasingly, it is the single most important point of contact between government and citizens, and successes and failures therein have effects that can be felt across a political system.

Consider, for instance, the significant impact of the trouble-plagued launch of the HealthCare.gov website on the political fortunes of the then U.S. President Barack Obama. The website went live on October 1, 2013 and was designed to be the primary mechanism for the implementation of the Affordable Care Act (ACA) healthcare reform law. Needless to say, it had major shortcomings immediately after its launch. President Obama himself noted: “I am the first to acknowledge that the website that was supposed to do this all in a seamless way has had way more glitches than I think are acceptable” (Schwarz, 2013).

On the day of the launch of the HealthCare.gov website, the Gallup Daily Tracking Poll (of presidential job approval) put Obama at 45% approval and 48% disapproval, a negative gap of three percentage points. On November 3, 2013, a little more than a month after the launch of the website – a month filled with intense media scrutiny regarding the website's shortcomings – that gap had grown to 14 percentage points, at just 39% approval and 53% disapproval. Most if not all of the decline in Obama's job approval ratings, that had sunken to the lowest level of his presidency to that point in most of the reputable polls, was linked to the failed launch of HealthCare.gov (Brandus, 2013; McLean, 2013). In short, the functionality problems encountered by a single e-government website – albeit a website supporting a highly visible and undeniably controversial political initiative – caused a substantial erosion in citizen trust for the U.S. president and his entire political administration. Thus, at least in this one instance, it is fair to conclude that e-government service satisfaction can be a major driver of citizen trust in government above and beyond the traditional factors such as citizen demographics, political ideology, and party affiliation. While we may not have reached this point yet, it is not inconceivable to envision a future where e-government failures contribute to the fall of an elected politician at the highest level.

Driven by events like the troubled HealthCare.gov launch and the increasing importance and prevalence of e-government overall, a substantial and growing number of academic studies investigating the adoption and impact of the performance of this service channel have emerged (Dwivedi et al., 2017). More specifically, a veritable slew of the studies has investigated the links among e-government, performance of e-government, and citizen trust in the government (Cyr, 2008; Im, Cho, Porumbescu, & Park, 2014; Kim & Lee, 2012; Lim, Tan, Cyr, Pan, & Xiao, 2012; Morgeson & Petrescu, 2011; Morgeson, VanAmburg, & Mithas, 2011; Osman et al., 2014; Parent, Vandebeek, & Gemino, 2005; Porumbescu, 2016; Robinson, Liu, Stoutenborough, & Vedlitz, 2012; Teo, Srivastava, & Jiang, 2008; Tolbert & Mossberger, 2006; Van Ryzin, 2007; Welch, Hinnant, & Moon, 2005). Lying at the intersection of the literatures on e-government and citizen-centered administrative performance measurement, many of these studies have investigated not only how e-government impacts citizens' perceptions of different government services, but also how it influences broader perceptions of a government, a political system, and even individual political actors.

Not surprisingly, given the relative “age” of this literature, extant research in this area has used a wide array of different theoretical and conceptual models to measure e-government performance, citizen satisfaction with that performance, and the consequent impact on citizen trust. However, few (or no) studies have empirically explored the tradeoffs among explanation, prediction, and parsimony across these models with the goal of finding an “optimal” model for the key outcome in this context – citizen trust in government (Shmueli & Koppius, 2011). Given the growing importance of these websites to government service delivery of all types and at all levels, the ubiquitous and long-running “crisis of trust” in Western (and other) governments, and the prospect (at least among some) that strong e-government performance might play a role in repairing citizen trust, a critical comparison of the performance of different models for assessing the linkages between e-government, citizen satisfaction, and citizen trust is long overdue.

Our goal in this study is to apply two ubiquitous citizen-centered performance measurement models linking citizen satisfaction to trust in government – the expectancy-disconfirmation model (EDM) and the service quality model (SQM) – to the e-government context, and to critically compare the relative performance of each in explaining and predicting trust. To this end, we test a variety of model specifications to identify the models that best explain and predict trust in e-government, and that do so parsimoniously. Our efforts are guided by the recommendations for “theory-pruning” suggested in recent work (Davis, 2015; Gray & Cooper, 2010; Leavitt, Mitchell, & Peterson, 2010). More specifically, we compare the models for their in-sample explanatory power, out-of-sample predictive accuracy, and parsimony. By staging a direct empirical comparison between EDM and SQM frameworks our work contributes to theoretical development by delineating the strengths and weaknesses of the models vis-à-vis their explanatory and predictive abilities (Clarke, 2007; Edwards, 2010; Gray & Cooper, 2010).

Our results suggest that while the expectancy-disconfirmation paradigm exhibits high levels of explanatory power, a parsimonious model offers the best out-of-sample predictive accuracy, which is the statistical “gold standard” for judging predictive models (Shmueli & Koppius, 2011). In addition, the service quality model offers the best balance between explanatory power and predictive accuracy and thus emerges as an optimal choice for achieving the trade-off between explanation and prediction. In sum, we conduct an exhaustive comparison between the two major e-government citizen satisfaction-trust modeling paradigms and provide relevant implications for researchers and practitioners alike. The implications and the benchmarks established will, at the very least, help inform those deciding upon a modeling approach when analyzing and forecasting with this type of data.