Elsevier

Information & Management

Volume 42, Issue 2, January 2005, Pages 275-288
Information & Management

Exploratory analysis on the halo effect of strategic goals on IOS effectiveness evaluation

https://doi.org/10.1016/j.im.2003.12.002Get rights and content

Abstract

How do managers evaluate the effectiveness of IT use? Past literature showed that the assessment of IT effectiveness is not straightforward, especially when viewed in terms of “business outcomes,” such as sales increase, firm profitability, and firm success. For inter-organizational systems (IOS), IT effectiveness evaluation is even more challenging, because these IOS are an integral part of a retail channel distribution system where several factors can influence business outcomes. Under such circumstances, IOS effectiveness evaluation is prone to a halo effect. A survey of retailers and suppliers showed that firms favorably judge IOS as long as their strategic goals are met, even when they do not recognize any IOS contribution towards accomplishing them. Conversely, IOS use was not appreciated when strategic goals were not met—even if IOS use benefits the firm at the operational or tactical level. In effect, IOS evaluation was overshadowed by the halo effects of strategic goal accomplishment.

Introduction

While the impact of IT use on organizational performance has been a major research focus, past studies “have provided findings that tend to be either mixed or inconclusive …” because “the effects of IT are indirect …” and “IT is a moderator of organizational characteristics and processes” [23]. For example, one may find a direct link between office IT use and office work productivity, but the IT use may not translate into business unit success in terms of operational cost reductions, revenue increases, or business goal accomplishments.

The strategic value of interorganizational systems (IOS) has been well recognized for order entry and distribution [18], just-in-time (JIT) manufacturing [56], reservation systems [43], quick response [29], and vendor managed inventory (VMI) [30].1 An interesting case is efficient consumer response (ECR)—an IT-enabled industry-wide supply-chain initiative that started in the early 1990s in US grocery distribution channels. ECR has received reports of “growing industry dissatisfaction in the [US] trade press” [9], though positive reports exist [31], [36]. ECR had “fallen far short of its promised efficiencies and value” primarily because the industry expectation exceeded the reality [25]. Then, how do firms assess IOS, the key enabler of initiatives like ECR, when their business expectations and business outcomes do not coincide?

The adoption of IOS generally requires substantial commitments of both financial and operational resources. While such commitments indicate a firm’s faith in the ability of IOS to help achieve its strategic goals, not all firms share the same level of “faith.” Some see the situation differently because their trading partners or market conditions have mandated such resource commitments. Sometimes the results from IOS do not make significant strategic contributions [19], [41]. Furthermore, unlike systems used only in one firm, IOS are used with multiple trading partners. How does a retailer evaluate its IOS when its benefits differ with various suppliers? It usually requires a judgment call on how much IOS contributes to the accomplishment of favorable retailing business results when considering the impact on one trading partner.

Thus, the business evaluation of IOS use is neither simple nor straightforward. There are criteria on effectiveness, such as the gains in operational efficiency from various retail distribution processes, the reduction of operational personnel and/or their time, and product sales increase due to more timely and accurate market information. There are also factors that can either limit IOS use and any resulting effectiveness; these include general market conditions, the number and quality of competitors, and the firm’s management skills.

Given such an “ambiguous” link between IOS use and business goal accomplishments, our study examines whether a halo effect exists (or even pervades) when evaluating IOS. A halo effect is a strong overall impression that blurs distinctions between dimensions or attributes [10]. For example, our first impression of a person is sometimes based on feeling about a stereotype (of the group) to which the person belongs [3].

Our question therefore is: Do firms evaluate IOS based on the status of their business goal accomplishments but not on how much IOS actually contributes to the accomplishments? To answer this question we examined the relationships between four factors—the strategic goals established for a trading relationship, the degree that each goal is achieved, IOS usage, and the perceptions of IOS effectiveness—in the context of trading relations between retailers and suppliers. By assessing these relationships, the study explored whether and how IOS are “fairly” evaluated by retailers and suppliers. Also the study explored whether and how their views were different.

At the outset, it should be noted that this is an exploratory study based on a cross-sectional survey with limited sample sizes. Also, the survey data were collected in late 1998 and this limits the generalization of the research findings. However the results of our research did not depend on the “currency” of the IOS technologies used.

Section snippets

Research background

Studies of IS effectiveness are rooted in those of IS success. Delone and McLean [22] summarized the fundamental measures of IS success as systems quality, information quality, use, user satisfaction, individual impact, and organizational impact. Yet while these measures are the direct results of IS use, we face ambiguities when we evaluate IS success against the degree that strategic goals are realized.

Dewett and Jones reviewed and summarized a broad array of IT impact literature, and their

Research framework and hypotheses

This study formulated hypotheses on the relationships between four factors: strategic goals, IOS use, goal accomplishments, and IOS effectiveness evaluation (see Fig. 1). In the model, the existence of the halo effects was evaluated in three different ways:

  • (a)

    Whether the path between goal accomplishments and IOS effectiveness evaluation was significant while the path between IOS use and goal accomplishments was not or was only marginal (a halo based on the level of goal accomplishments).

  • (b)

    Whether

Research method

We collected data from retailers and suppliers through a mail survey in late 1998. The hypotheses were then tested using path analyses, because they allow us to test not only the relationship between individual variables but also the pattern of these relationships. Thus, we could construct and test the various models of possible causal relationships based on theoretical considerations.

Results

The overall results of PLS path analyses are shown in Fig. 2, Fig. 3.

Implications

The research model tested three possible types of halo effects in evaluating IOS effectiveness: a halo from accomplishing strategic goals; a halo from having important strategic goals; and a halo from having IOS in place itself. The results were that the first two types of halo effects occurred for both retailers and suppliers. The presence of the third was not detected for either.

For retailers, path analysis showed only the second type of halo effect. They correctly evaluate IOS based on how

Limitations

This study is exploratory in nature because of convenience sampling, smaller sample sizes, limited supplier participation and IOS data prior to 2000. In addition, it had low response rates like those in other studies using unsolicited mail surveys.14 It should be noted, however, that F tests between the first and second half of the responses did not show significant non-response bias.

In this research,

Conclusion

This exploratory research study showed that firms do not necessarily evaluate the effectiveness of IOS by how much IOS use contributed to achieving strategic goals. Rather, firms gave more favorable opinions on IOS, regardless of its contributions. This is due to the two types of halo effects where firms’ IOS evaluation is overshadowed by the extent to which their strategic goals are accomplished and the extent to which those strategic goals have significance to their trading partners.

Makoto Nakayama is assistant professor at the School of Computer Science, Telecommunications and Information Systems in DePaul University. He holds a PhD from University of California, Los Angeles and an MBA from University of Texas at Austin. Prior to moving into academe, he served as a product marketing manager at Novell Japan. He also worked on business planning and operating system developments at Yokogawa Electric Corporation in Tokyo. His current research interests are in IT use in

References (59)

  • S. Asch

    Studies in the principles of judgments and attitudes II: determination of judgments by group and by ego standards

    Journal of Social Psychology

    (1940)
  • R.L. Atkinson, R.C. Atkinson, E.E. Smith, E.R. Hilgard, Introduction to Psychology, ninth ed., Hartcourt Brace...
  • S.B. Bacharach et al.

    The organizational transformation process: the micropolitics of dissonance reduction and the alignment of logics of action

    Administrative Science Quarterly

    (1996)
  • A.S. Bharadwaj

    A resource-based perspective on information capability and firm performance

    MIS Quarterly

    (2000)
  • M. Bensaou, Interorganizational cooperation: the role of information technology—an empirical comparison of US and...
  • P.M. Bentler, EQS Structural Equations Program Manual, BMDP Statistical Software, Los Angeles, CA,...
  • T.A. Brown et al.

    ECR and grocery retailing: an exploratory financial statement analysis

    Journal of Business Logistics

    (2001)
  • B. Brown et al.

    Removing the financial performance halo from fortune’s “most admired” companies

    Academy of Management Journal

    (1994)
  • R.S. Burt

    Cooptive corporate actor networks: a reconsideration of interlocking directorates involving American manufacturing

    Administrative Science Quarterly

    (1980)
  • Chain Store Age, Overview: Inventory Management, 2001, pp....
  • W.W. Chin

    Issues and opinion on structural equation modeling

    MIS Quarterly

    (1998)
  • W.W. Chin, The partial least squares approach to structural equation modeling, in: G.A. Marcoulides (Ed.), Modern...
  • W.W. Chin et al.

    On the use, usefulness, and ease of use of structural equation modeling in MIS research: a note of caution

    MIS Quarterly

    (1995)
  • P. Chwelos et al.

    Research report: empirical test of an EDI adoption model

    Information Systems Research

    (2001)
  • E.K. Clemons et al.

    Information technology and industrial cooperation: the changing economics of coordination and ownership

    Journal of Information Systems

    (1992)
  • E.K. Clemons et al.

    McKesson drug company: a case study of economist—a strategic information system

    Journal of Management Information Systems

    (1988)
  • E.K. Clemons et al.

    Limits to interfirm coordination through information technology: results of a field study in consumer packaged goods distribution

    Journal of Management Information Systems

    (1993)
  • L.J. Cronbach

    Coefficient alpha and the internal consistency of tests

    Psychometrika

    (1951)
  • W.H. Delone et al.

    Information systems success: the quest for the dependent variable

    Information Systems Research

    (1992)
  • Cited by (0)

    Makoto Nakayama is assistant professor at the School of Computer Science, Telecommunications and Information Systems in DePaul University. He holds a PhD from University of California, Los Angeles and an MBA from University of Texas at Austin. Prior to moving into academe, he served as a product marketing manager at Novell Japan. He also worked on business planning and operating system developments at Yokogawa Electric Corporation in Tokyo. His current research interests are in IT use in marketing channels and IT skills portfolio management. His papers appeared in Information & Management, Journal of Information Technology, and proceedings of national conferences.

    Norma G. Sutcliffe is assistant professor at the School of Computer Science, Telecommunications and Information Systems in DePaul University. She holds a PhD as well as an MBA from University of California, Los Angeles. With extensive experience in industry, Dr. Sutcliffe has been a consultant for many Fortune 500 firms in evaluating IT needs, IT implementations, and IT strategies. Dr. Sutcliffe has worked as systems developer on mainframe and client/server systems as well. Her current research interests are in IT enabled organizational change, IT leadership behavior, and IT skills portfolio management. Her papers appeared in Information & Management and proceedings of national conferences.

    View full text