Integrated feedback control reporting for improving quality of technical service reporting in IT service management

Highlights

• Quality issues of technical service reporting are defined for large ITSP in ITSM.

• A process model is implemented to improve technical service reports (TSRs).

• Design principles are provided to improve TSRs.

• An ensemble artifact is implemented to deliver improvement loop for TSRs.

Abstract

Service Reporting is an essential part of Service Level Management (SLM) in IT Service Management (ITSM). These SLM reports show Service Quality status in compare to Service Level Target (SLT) based on Service Level Agreement (SLA). However, producing Technical Service Reports (TSRs) for SLM has Data Quality (DQ) challenges when IT services of a large enterprise are outsourced to a large IT service provider (ITSP). In fact, sources of technical metrics in these reports are coming from huge, unverified and non-normalized system-generated events and logs in a large enterprise environment. In addition, there are lacks of configuration items and service information meta-data that are essential for producing these SLM reports. These challenges lead to low intrinsic and contextual DQ of reports that destroy customer’s trust and management visibility which lead to Service Quality (SQ) issues and financial penalties. This research used Action Design Research (ADR) methodology and insider participant observation, focus group, interview and questionnaire data gathering and content analysis. Design, actions and data collection are done in and from a Multi-National Company (MNC) in Malaysia that provides IT services to one of the top 10 biggest enterprises in the world, during 4 years of research. The research implemented artifact is an integrated reporting system. It used Feedback Control System (FCS) theory and ITSM concepts to create Reports Data Quality Improvement Loop (RDQIL) to improve TSRs’ DQ. The research output is an ensemble artifact which formalized in the class of problems, class of solutions and set of design principles that consist of new Hybrid Normalize Data Warehouse (HNDW) architecture and Integrated Feedback Control Reporting (IFCR) process model. The importance of research emerges by improving SQ and reducing the risks of financial penalties by providing complex event processing and service reporting integrated into Operation processes. The research’s design principles and IFCR process model create RDQIL that improves TSRs’ DQ in large ITSPs which lead SQ improvement and prevent financial penalties.

Introduction

Information Technology (IT) services are part of each business and organization these days which like all other business constructs require being well managed in order to bring its highest value. Rockart (1982) listed the Critical Success Factors (CSFs) of Information Systems (IS) and addressed the most obvious CSFs for service quality (Rockart, 1982). After that and also during IT maturity in last 30 years, companies are facing challenges to move from production and software to IT operation based on service (Marrone and Kolbe, 2011). This perspective and phenomenon in IT have emerged IT Service Management (ITSM). ITSM focuses on IT service creation, design, delivery, and maintenance to implement and manage the quality of IT services in a business and to play an important role in the success of that business in using IT (Hanna, 2011). ITSM has high impact in the success of an organization which has chosen IT as a service (Marrone and Kolbe, 2011). To monitor the quality of service by both customer and supplier management, ISO/IEC 20000 introduced Service Reporting part of ITSM to monitor trends and performance against service targets based on an agreed document between two parties (IEEE Std. 20000-1, 2013).

Service Level Agreements (SLAs) are the contracted measures that describe the services to be delivered. They also specify the metrics with which the effectiveness of service activities, functions and processes will be measured, examined, changed and controlled (Maurer and Matlus, 2007). In fact, SLAs and their associated agreements are great tools for enabling customer managers to measure, audit and align their outsourcing relationships to reach a successful IT outsourcing (Maurer and Matlus, 2007). ITSM and especially Information Technology Infrastructure Library (ITIL) as a famous major implemented framework in ITSM (Galup et al., 2009) define service reporting as the main process of SLA monitoring and the one which is responsible for providing Key Performance Indicator (KPI) reports for Service Level Management (SLM) (Britain et al., 2011). These service reports deliver to the customer through SLM processes. Many IT service organizations consider the measurement of IT service management processes, especially service support processes, as a challenge (Lahtela et al., 2010).

Some companies outsource their IT services to a third organization called IT Service Provider (ITSP). ITSP is an organization supplying services to one or more internal customers or external customers (Hanna, 2011). In outsourced IT Services, ITSPs need to provide service reporting based on their SLA for the customer. However, SLA monitoring is an open issue within the IT Service Management domain (Correia and Brito e Abreu, 2010). Based on Gartner’s (2012) interaction with over 85 clients, some conflicts between green performance KPI and red business impact bring doubts to quality or usefulness of services’ reports to customers (Ackerman and Maurer, 2012). Green performance is service level upper than agreement and red is lower than minimum agreed level. These ITSM performance measurement issues have research gap which also has been addressed by a comprehensive systematic literature review in ITSM till April 2012 (Proehl et al., 2013) by reference to Gacenga researches (Gacenga et al., 2011).

In ITSM, SLA contains penalties for some defined target metric in service reports with the name of Service Level Target (SLT). In fact, when these service reports tied to financial penalties, it becomes critical and valuable because it shows the value of IT services to both business and service provider (Adams and Govekar, 2013). SLA defined earn-back and cost for both IT service provider and customer which are based on defined target metrics that contractually guarantee service provider obligation to the quality of service (IEEE Std. 20000-2, 2013). The SLT is part of service reporting, and quality of it is the concern of both service provider and customer. The quality of these reports are becoming crucial when metric did not define based on simple process or system output and also it requires lots of data gathering, verification and calculation. Even some researchers (Paschke and Schnappinger-Gerull, 2006) and ITSM frameworks like ITIL clearly suggest to avoid these kind of metrics in SLA (Hunnebeck et al., 2011) or some researchers have introduced a framework or method to define and improve performance monitoring and metrics maturity in organizations (Gacenga et al., 2011) but customers like this research case have defined these complex metrics as a must in their service requirements and contract.

Service reports that are based on technical metrics definition require system-generated logs to be produced. Technical metrics are the type of performance metrics in service reporting that is based on the end result of service which requires system-generated event logs from IT infrastructure for its calculation like “percent of backup success” and are not based on a process like “percent of successful change”. Reports that contain these explained technical metrics define as Technical Service Reports (TSRs). Generating TSRs are more complex and costly because system-generated logs need experts’ verification, data cleansing, and complex processing to fit with all business rules and condition defined in SLA. In fact, an expert must exclude testing issues or any kind of issues which are not genuine based on SLA definition or exclude log noises that are not real service result. These verifications are costly and time-consuming (Correia and Brito e Abreu, 2010) which make the report delivery and report monitoring out of expected time. However, Operation and customer need reports in a right-time that be believable (a data quality dimension). An acceptable report in service reporting is a report which shows all customer services with correct status and matches to real implementation and SLA which represent the final verified figure of a metric (Paschke and Schnappinger-Gerull, 2006). These difficulties mostly lead to manual data cleansing logs and reports data which are time-consuming and lead to a late delivery report. The process of data cleansing and data quality improvement has lots of open problems and highly is domain related and need to explore for each specific context (Müller and Freytag, 2005).

Moreover, manual report generating by spreadsheet software (like Microsoft Excel) cannot cope to some complex formal definitions in SLA and contract such as complex service level in different asset type, location, customer or situation. The manual reports mostly could be based on simple system-generated events and uptimes which mixed by testing and valid events (Correia and Brito e Abreu, 2010). These all challenges lead to manual or semi-manual generation of technical metrics for SLM purpose with a low quality which brings doubt and dispute of customer.

In addition to system-generated logs, there are other Data Sources (DSs) that must be used in the process of reports generation such as Asset Management System (AMS) and Configuration Management Database (CMDB). AMS is a system on top of Asset Management (AM) process that manages activities or processes of tracking and reporting the properties, value, and ownership of assets throughout their lifecycle (Hanna, 2011). Besides, CMDB is a database that is used to store configuration records and their attributes throughout their lifecycle (Hanna, 2011). AMS and CMDB contain some meta-data that is required for reports. Meta-data in this research refers to any complementary data that are required beside system-generated logs to produce Technical Service Reports (TSRs). These meta-data are Configuration Items (CIs), Service Catalogues Information and some service definitions, rules, and categories. CI is any component or service asset that is required to be managed and stored in order to deliver an IT service (Hanna, 2011). Besides, service catalog information or in abbreviation Service Information (SI) is structured information about live IT services. (Hanna, 2011). However, completeness and accuracy of these meta-data and DSs are depended on the maturity of their process and their adoption in the organization. In fact, process design deficiencies like “Incomplete representation”, “Ambiguous representation” and Operation deficiencies would lead actual data in reality that are required for generation of reports to be missed (Wand and Wang, 1996). Thus, because of direct dependency between Service Reports (SRs) and these data sources, Data Sources’ Data Quality (DSDQ) issues have a direct impact on Reports’ Data Quality (RDQ) issues and become a part of the problem.

Whilst, some researchers have begun to examine Data Quality (DQ) and Information Quality (IQ) (Ahn et al., 2008, Blake and Shankaranarayanan, 2015, Madnick et al., 2009, Pipino et al., 2002, Wand and Wang, 1996, Wang and Strong, 1996), but little focus has been paid to more specific area of data quality in reporting of IT services. Lots of researchers discuss data quality for Information systems, EIP, data warehouse and decision systems when data sources are other information systems or human-generated sources (Madnick et al., 2009). However, no attention has been paid for system-generated data in complex IT services. Thus, it remains to be seen how SLM reports’ data quality in this context can be improved and, DQ of Technical Service Reports (TSRs) is the gap that this research is focused on. To be more clear on the differences between terms of DQ, IQ and Report Quality (RQ), researchers followed Madnick and consider DQ and IQ in a same scope, opposite others who define DQ to be more on technical issues and IQ to be non-technical issues (Madnick et al., 2009). RQ in this research considered as an overall quality of Reports and not only data of reports that are more focuses on representation quality and considered out of this research scope. Besides, researchers adopted “fitness-of-use” concept for its quality perspective (Wang and Strong, 1996) and define Reports’ Data Quality (RDQ) as a subset of Reports’ Quality which leads to accurate SLT reporting and customer satisfaction.

During literature review, some researchers confirmed and explained quality issues in ITSM service reports (Ackerman and Maurer, 2012, Gacenga et al., 2011) case organization also faced the quality issue on technical metrics reports. In the research case organization, backup technical service reports contain around 800,000 entries which made it impossible to fully review them manually in the time of submission. On the other hand, random checks by customer showed discrepancies which lead to dispute and distrust of customer. Case investigation showed these report quality are low and trust of the customer to service report has been lost so that it leads to a dispute with the customer. This dispute could lead to 1.5 million USD penalties per month for case organization. This presented as the main problem statement of the research as “Technical service reports in IT service management for large enterprise are facing quality issues.”

This research discusses the approach of improving technical service reporting quality in Service Level Management for outsourced IT. The purpose of this study is to address the current quality issues, concerns and solution of service reporting in ITSM by developing and evaluating an ensemble service reporting artifact. Based on the literature, the case investigation, problems statement and research questions were formulated. The main research question of this research is formulated as:

• How the quality of technical service reports in ITSM context can be improved?

Based on the main research question and problem background, four sub-questions are formulated to support the main research questions. These questions are:

• What are the technical service Reports’ Data Quality (RDQ) issues in ITSM service reporting?

• What process model can be used to improve the technical service reports in large IT service providers?

• What design principles can be used to implement a service reporting system to improve the technical service reports?

• How the artifact of technical service reports generation can be designed to reach data quality improvement loop in large IT service providers?

Based on the research questions, four research objectives under a main objective are defined. The main goal of this research is “To design and implement an ensemble artifact to improve the technical service reports on outsourced IT services and ITSM context”. In fact, it is to create a data quality improvement loop that corrects data quality issues to reach customer and ITSP satisfaction with reports’ data quality. Accordingly, three objectives of this study are:

• To identify quality issues of technical service reporting for large IT service providers in IT service management.

• To design and implement a process model for improving technical service reports in large IT service providers.

• To provide design principles to improve the technical service reports in large IT service providers.

• To achieve an ensemble artifact and implement a system to deliver improvement loop for technical service reports in large IT service providers.

In reality, reaching to perfect quality when none of the stakeholders require that degree of quality is not necessary and even may not possible. Thus, improvement loop, improve quality till it is required by the customer. In fact, customer unsatisfaction or data quality issues trigger or increase the improvement loop and customer satisfaction slow or stop the loop.

This study uses a large Multi-National Company (MNC) IT Service Provider (ITSP) in Malaysia as the case for an Action Design Research (ADR). Case company (To remain anonymous is named ITSPMY in this report) provides IT services to more than 20 big companies. This research focused on ITSPMY department that service to an Oil and Gaz company which is in top 10 of Fortune Global 500 (To remain anonymous is named CUSOGT10 in this report). For this research observation (A3, A7), interview (A1, A4), focus group (A8) and questionnaire (A10) were used to collect data from a single case organization to implement an ensemble artifact as a new reporting system and processes. Also, Oracle database and its technologies were used for implementation and NVIVO software was used for categorizing and coding the collected data (part of A11).

The success of all service management processes is dependent on the use of the information provided in service reports (IEEE Std. 20000-2, 2013, p.37). ITSPMY service provider experienced Service Level Target (SLT) verification challenge that leads to heavy penalties. In addition, without a trustable report, management cannot improve operation and Service Quality. Thus, improving report quality which linked to Service Quality prevents penalties and increases customer satisfaction. This research aims underpin improved reporting and monitoring of service delivery in operation units of a large IT enterprise.

Research Design Principles (DP) output will give principle for design and implement ITSM service reporting and repository in large IT service providers to reach agreed SLA and prevent financial penalties. Besides, designed artifact provides Complex Event Processing (CEP) for operation context which increases service monitoring quality that leads to Service Quality. In addition, the definition of the class of problems gives the opportunity to other researchers to explore more in this field of study. Moreover, designed Hybrid Normalized Data Warehouse (HNDW) architecture provides a new class of solutions for a broader class of problems to reach higher data quality and utilization in comparison to current Data Warehouse (DWH) or Change Management Database (CMDB) design. Moreover, Integrated Feedback Reporting model provides an understanding of the impact of report usage and feedback on report quality in a similar context which helps an organization to reach Reports’ Data Quality Improvement (RDQI) loop in their Technical Service Reports (TSR).