CASE-mediated organizational and deutero learning at NASA

Abstract

Deutero learning (DL) involves the use of an organizational learning (OL) strategy for the adoption of learning behavior, or simply ‘learning to learn.’ This article describes the application of an evolutionary prototyping form of action research in a federal government department to address problems stemming from systemic bureaucracy. We developed two approaches, each mediated by computer-aided software engineering (CASE), that are relevant to initiating OL and DL. The first approach, CASE-Mediated Organizational Learning (CMOL), was found to be useful in the initiation, development, and guidance of OL behavior in the research setting. The second approach, named CASE-Mediated Deutero Learning (CMDL), is a guide for generalizing the CMOL approach to other settings. We validate both approaches using criteria drawn from the action research literature. These findings should be useful to managers interested in adopting OL and contributes to several streams of MIS literature.

Appendices

Appendix A:

Table 7 Overview of research informing this project about deutero learning

Appendix B: Methodological approach

The research project was a part of the NASA Faculty Fellowship Program (NFFP), which allows researchers from academic institutions to study agency problems in either one or two successive summers. The program allows university faculty to serve in residence at a NASA center to conduct research to advance or understand a wide range of potential strategic challenges. Thus, the NFFP is the equivalent of an action research program encouraging focused research and allowing for a range of methodological applications. Two study research questions (RQs) correspond to goals that are consistent with the missions of NASA and the NFFP: 1) organizational development and 2) scientific knowledge. As required by program policy, the researcher first approached the client before the program started to determine the level of fit between the two parties. When organizational problems surfaced in two prescreening meetings, suggestions by researchers included the application of CASE, OL, and evolutionary prototyping.

The methodology developed in this study, called evolutionary prototyping action research (EPAR), intended to address the two research questions by: 1) developing an approach (CMOL) that would be useful in solving problems within the department and 2) developing a general approach (CMDL) to CMOL adoption in other settings. EPAR combined basic action research foundations (Davison et al. 1997; Iversen et al. 2004) with evolutionary information systems prototyping “to improve user-designer communications” and “to facilitate user validation of system designs” (Baskerville and Wood-Harper 1998, p. 99). Prototyping is believed to facilitate learning at different levels: “Refining an existing technology substitutes for recognizing a better one, and vice versa” (Levinthal and March 1993, p. 101). EPAR supports the common action research project goal of creating exemplary approaches during local organizational problem solving. However, EPAR is unique in that it involves repeating a sequence of design steps in the development of a prototype information system. In this study, the method involved the iterative design, development, and validation of organizational structures represented in a CASE-based repository.

The following sections describe EPAR by detailing the five principle elements of action research established by Davison et al. (1997): 1) researcher-client agreement, 2) cyclical process model, 3) theory, 4) change through action, and 5) learning through reflection. This is followed by a discussion of the research criteria that will be the basis for evaluating the method and findings of EPAR.

Researcher-client agreement. The client-researcher relationship is important in EPAR because the responsibility for prototype design alternates between the faculty researcher and client researcher throughout the project. In this study, the OL implementation approach was iteratively assessed and judged for validity by the user(s) as recommended in literature on action research (Boar 1984; Iivari and Karjalainen 1989; Naumann and Jenkins 1982). There was no formal organizational research structure in the department prior to the researcher-client agreement. The department had conducted process change from paper-based copies of process models and descriptions.

A departmental adaptation problem emerging from initial discussions necessitated a formal agreement between the faculty researcher and the department. The faculty researchers’ application to the NFFP stated the following expertise: 1) an ability to contribute to NASA’s mission, 2) an ability to work closely with NASA personnel for the ten-week period, and 3) an ability to communicate and collaborate with other faculty from multidisciplinary backgrounds. Furthermore, the faculty member had published multiple peer-reviewed articles on OL and taught systems development methodology using CASE. This background and a desire to use the fellowship experience to publish any studies resulting from the relationship were compatible with the mission of the department. The client researcher held practical expertise in all technology transfer operations in the center, understood the organizational culture and norms, held a PhD in solar physics, and had published in that area. The fit between the organizational problems and the backgrounds of the faculty and client researchers shaped elements of the contractual agreement.

The agreement stated that the function of the faculty researcher was to improve the technology transfer department at the center. The stated source of funding and facilities was the Technology Transfer Department. Two university researchers (one faculty and one student) were to collaborate with a client researcher with the faculty and client serving as primary drivers of the research. The participants agreed to pursue several possible interventions, including identifying problems, planning interventions, designing solutions based on relevant theories, and publishing.

Cyclical process model. Action research tests theory in practice in a cyclical process with the intention of improving the organization. Table 1 shows that we based our action research methodology on the three phases provided in Iversen et al. (2004): initiating, iteration and closing. In our study, these phases comprised of a total of sixteen steps, each consisting of some form of data collection, one or more participants, and advancing practitioner and theoretical knowledge. Each phase is detailed in the remainder of this section.

Initiating. The initiating phase intended to help the faculty researcher and client researcher better appraise organizational problems and potential solutions. It began with two off-site meetings (steps 1 and 2) to determine the fit between the needs of: 1) the faculty researcher, 2) the client researcher, and 3) the department. After deliberation that included specific problem identification (3), reflection on appropriate theory and a literature review by the faculty researcher (4 and 5), and interviews with several departmental process owners (6), a defined direction was formalized between client and researcher (7) and planning of an action research project ensued to develop an approach to OL adoption within the department (8).

Iterating. The EPAR methodology has five steps during the iteration phase, which is a cycle of problem solving by means of evolutionary prototyping. This phase involved the design of two approaches, each addressing RQs 1 and 2. Each of the two approaches required multiple repetitions of the iterating phase. For each iteration, the approach was reevaluated for updates in theory (9) and design (10). The approach was then put into practice (11) and the experiences were evaluated jointly by the client and researcher to determine if the approach design was adequately mature (12). The iteration was reflected upon by the client and a determination was made as to whether an additional iteration was warranted (13). The design of the iterating phase ensured that any resulting approach would be developed based on relevant theory (provided by the previous literature review and researcher understanding) and become useful in practice (as evaluated by the client researcher).

Closing. Because social action and dynamics have no defined beginning or end, selecting the end of an action research project is an arbitrary decision (Checkland 1991; Lau 1997). The decision to exit (14) was made to allow enough time for researchers to assess the usefulness (15) of CMOL and present results to the department (16).

Theory. In this study, at least three theoretical topics that were previously nonexistent in the department were important (see Appendix A for a summary): action research, OL, and CASE. The action research literature guided researchers to test collaboratively developed approaches to generate new theory or inform existing theory (Davison et al. 1997). In EPAR, theory development is mediated by the prototype, the design of which is validated by the researcher and client and serves as an important finding of the EPAR research process. Relating the results of the study (including the final prototype) to existing theory supports the impartiality of our EPAR study (Iversen et al. 2004). Determined during Steps 5 and 6 in EPAR and reevaluated in the iterative step 9. Theory serves as the philosophical context of the study and is dependent upon the interests of the researchers (McKay and Marshall 2001). For these reasons, the development of researcher and client understanding, the prototype and the theory are discussed. Understanding and accepting OL and CASE in the department were important moderating factors during the investigation. Both theories were used to guide the development of prescribed organizational actions to influence positive change. In general, both RQs were concerned with how CASE may influence OL.

Change through action. Traditional action research involves interventions that couple action with effective change aimed at problem solving. In EPAR, these actions involve the design and validation of a prototype in a prescribed approach informed by the two aforementioned theoretical foundations. Using OL and CASE as theoretical lenses, several actions were identified to improve the problem situation. These became elements of the two resulting approaches. The implemented actions and their effects on the problem situation in the department are explicitly specified in Section 4.

Learning through reflection. Faced with “dynamic situations that consist of complex systems of changing problems” (Schön 1983, p. 16), practitioners learn through a process of reflection. In EPAR, the ‘evolution’ of understanding depends heavily upon the cognitive reflections of researchers and clients regarding the likelihood that approaches they have designed will alleviate the problem situation. The reflections of researchers will influence ongoing prototype design and its implications for practice and theory. Thus, reflection is a crucial aspect of EPAR, since prototyping encourages dialogue and interaction between systems developers (Baskerville and Stage 1996). Consequently, the expertise of those whose reflections influence suggested intervention designs is significant (Baskerville and Wood-Harper 1998).

The roles of reflection fell on two primary researchers: 1) a university faculty member specializing in information systems and 2) a managing research officer within the department. In this manuscript, these researchers are referred to as the ‘researcher’ and ‘client’, respectively. Both held PhDs—the researcher in Management Information Systems and the client in Physics. Thus, both primary researchers understood the scientific method, shared a general understanding of the philosophy of knowledge, and were interested in advancing knowledge. With a publication background in OL and teaching experience in systems development using CASE, the researcher took on the role of expert on theory. The role was collaborative and included helping guide intervention designs consistent with his experience and the relevant literature. The expertise of the client was in the departmental setting and problem domain. Also collaborative in nature, his role was to inform intervention designs based on his experience in the organization. To facilitate collaboration, the researcher was physically resident in the department throughout the project.

The lessons learned in action research studies should be explicitly stated for both practitioner and research communities (Davison et al. 1997). Lessons may accordingly be classified and described as: 1) approaches serving as exemplars (for practitioners) and 2) contributions to theory (for researchers). In both ways, the competencies of the researchers were advanced and described. These lessons are articulated in Sections 4 and 6.

Research criteria. One pitfall to action research is that the research criteria for evaluating the method and findings may change as the study progresses (Markus et al. 2002). This is necessary because designed solutions change over time and problem domains and participants are unique from one study to another (Gustavsen et al. 2007). These dynamics cause a number of limitations to the action research method (Baskerville and Stage 1996). We therefore used the six criteria explicated by Iversen et al. (2004), which drew from Davison et al. (1997), for assessing the rigor and relevance of our study. Table 2 defines each of these criteria and shows that each is used to test one of two perspectives on study quality: 1) method (using roles, documentation and control criteria) and 2) findings (usefulness, theory and transfer). The study findings are represented by two approaches—each resulting from pursuing the two research questions. Each of the criteria represents a category of detailed description that will be offered to make the case for research rigor and relevance.

Each criterion is described in Iversen et al. (2004) and demonstrated in Section 5. In the context of evaluating the EPAR method, roles include heavy collaboration and are expected to change over time with the prototype. According to Baskerville and Wood-Harper (Baskerville and Wood-Harper 1998), “The researchers’ reflection and understanding is improved by writing (with incumbent structuring of the knowledge) about the experience” (p. 101). Therefore, the documentation features were an important criterion for selecting the CASE tool used in the study. EPAR enabled the encoding of data in a CASE-mediated environment. Drawing from Fruhling and de Vreede (2006), data sources included direct observation, group interviews, usability evaluation, system documentation, and a departmental project timeline. Because EPAR is evolutionary by definition, it is important that researchers maintain control over several considerations, including the problem definition, roles, data standards and coding, and collaboration processes and outcomes.

The findings of this EPAR study include two approaches, each resulting from exploring the two research questions. The usefulness of these approaches was demonstrated in practice by the implementation of changes to the organizational structure (Checkland 1991). Once found to be useful, these approaches may be relevant to theory: “Relating results to existing frameworks supports the impartiality of…research” and “…provides a basis for discussing transferability of results” (Iversen et al. 2004, p. 409). Therefore, descriptions of the theoretical foundations and any resulting prototypes may support informing research and practice. This is done through transfer, which is the explicitation of how, and under what conditions, the approaches may (or may not) become useful in other settings. Generalizing to other settings would be the ultimate validity test for designed artifacts in action research (Van Aken 2004). Transfer is also concerned with how the approaches found to be useful in the research setting advance theory.

Appendix C: The paradigm level of organizational learning (Sources: Templeton and Snyder, 1999; 2000)

Appendix D: CMOL process with issues attached

Appendix E: The difference between radical and incremental change processes in CMOL