Keywords

1 Context

Italian Nursing homes are a type of residential care (private and public) that provide nursing care for elderly people. Multidisciplinary teams work together in order to manage health care problems but not only. These organizations have administrative, human resources, technical and quality departments with complex issues because of many working people, processes, rules, units and service strategies.

Furthermore, some critical services are outsourced (e.g. meals, laundry service, health care staff, …) by public tenders or by contracts which define prices and quality (Service Level Agreements SLA).

The establishment of an Internal Quality Measurement (QM) [1] becomes an urgent priority because of the lack of control for some critical services.

In this scenario various challenges - both organizational and technical – are emerging:

  • each service provider and even each internal department has typically its own IT system as a “Silo (or Box)” isolated from the others (no communication);

  • there is not a shared infrastructure to coordinate them and integrate data produced by each system in each department;

  • even when there is a single IT system (e.g. maintained by a unique service provider) internal procedures are different (from one office to another one), ineffective, and they process data manually not considering IT systems (e.g. using office automation tools like spreadsheets or even paper documents). This practice increases the risk of errors, wrong data, delays and stress because involved offices don’t respect internal and external deadlines;

  • even if a shared IT system is used and could provide financial, medical, human resource reports, data provided do not support decision makers [2]. They miss the “global vision”, inter-functional processes and high data correlations (e.g. Patient Falls-Fractures-Costs related to rehabilitation). The Decision Support System (DSS) has also to include the SLA of the outsourced services which have a critical impact in terms of Quality and Cost.

In the real scenario QM is not useful because data is collected and processed manually, sometimes once a year, often for law requirements. Reports are used for financial needs (budget) and to comply with laws, and data is not considered for planning activities (PDCA) (see [3, 4]). The timing information is very critical to check and anticipate quality service problems and to establish “new relationships” with suppliers in order to improve the global quality to the end users (patients).

Furthermore, those organisations have to collect and manage very critical and sensitive data (e.g. patient’s medical data, financial reports for invoices and for public reimbursements) required by public authorities (local Government and Italian Ministry of Health): it is very important and crucial for them to extract data automatically (and not manually as it is currently done). Such reports (required by law) could be internally used to analyse/review processes combining lean [5] and Business Process Reengineering (BPR [6]) tools in order to improve their practices and their quality standards in a Continuous Quality Improvement (CQI).

In some cases, those reports need so many efforts that administrative staff spends at least 1 week/per month for this “no value” activities (NVA).

In this scenario the needs of stakeholders are:

  • To make their processes more effective and efficient starting from the possibility to organize the internal and external flow of data (local Authorities and Suppliers)

  • To have a data warehouse system capable to collect key data (not only financial but also quality data)

  • To have interactive dashboards to support decision makers for long-term and short-term planning activities in different areas: economical, human resource, finance, quality, warehouse, ….

  • To employ easily customizable procedures to prepare periodic reports required by external auditors (authorities).

Key data provided by Business Intelligence (BI) can be the first step to plan and set targets, both for quality and for economic goals. A BI/Dashboard is a starting point to go deeper to analyse the critical processes using Lean tools of problems solving as DMAIC [12, 13] for improving quality by eliminating wastes. Particularly, the possibility to access critical information timely and easily will provide new opportunities such as:

  1. 1.

    considerable saving in time by reducing the No Value Activities (NVA)

  2. 2.

    accurate control of internal standard of quality (QM)

  3. 3.

    analysis and design of new processes and new practices both for wards and for administrative area (BPR and Lean tools as PDCA-DMAIC).

We propose an inter-functional process/organizational analysis approach concerning different areas which we wanted to analyze. We combined this holistic approach of analysis with the involvement of the end user in the first step (design and KPI definition) and in the second step (validation, development, interaction). We combined lean tools (DMAIC), BSC and BPR shared with end users of different departments (financial, healthcare, human resources), paying attention to their real needs with classical Data Warehouse design and development cycle.

This method has been applied in real scenarios with medium-large nursing homes (100–600 patients with 80–120 employees located in different regions in the north of Italy: Lombardia, Emilia-Romagna and Trentino) considering the different local requirements and different regional laws.

In Sect. 2 we introduce the techniques adopted, in Sect. 3 we present our approach and in Sect. 4 how it has been conducted in the actual scenario, in Sect. 5 we summarize our results and derive the lessons learned and in Sect. 6 we draw some conclusions and future work.

2 Background and State of the Art

In this section we introduce state of the art methodologies adapted in the project. We adopted a combination of methodologies, that normally are applied to manufacturing industry and we adapted them to health care sector which need new vision and approaches. We considered Lean methodology because its core idea is to create more value for customers with fewer resources increasing quality. It focuses on: i. elimination of wastes (inefficiencies) along entire value streams (instead of isolated points); ii. creation of processes that need less human effort, less space, less capital, and less time. The goal is to make services that are less costly and with much fewer errors, compared with traditional business systems. Lean applies in all industries, services and every process. It is a way of thinking and acting for an entire organization [7].

For Nursing Homes quality, cost control and inter-functional teams are core principles because of their complexity (many services, many law requirements and less resources given by Government).

2.1 LEAN Tools: DMAIC, Root Cause Analysis, Standardization

Our approach starts from the structured method of DMAIC [12, 13] as problem solving approach of Lean Six Sigma because we wanted to check the actual processes and problems in a short time (e.g. we fix 3 meetings in 3 weeks, once a week with the working team) before developing the BI solution.

The DMAIC approach enabled quick follow up of different problems, root causes and solutions/improvements related to the 4 Key areas investigated (Financial, H.R., Medical, Process that are linked to BSC evaluation) (Fig. 1).

Fig. 1.
figure 1

DMAIC approach.

Full size image

  • Define: we define the project, goals with the project sponsor (director of the organization) and the employees of the 4 Key areas (Financial, Human Resources, Medical, Process)

  • Measure: we collect all the documentation/data necessary for the first analysis of the inter-functional processes in the 4 Key areas

  • Analyze: we analyze with flow charts processes, wastes and redundancies

  • Improvement: we identify different steps of improvements (short -3 months- and medium term -9 months)

  • Control: we validate the project with standard procedures with the identification of the “process owner”.

The focus on process analysis of each area, within the organizations, showed process redundancies (Muda) that is ‘waste’ which is the primary aim of lean manufacturing. We also try to look deeper into problems with Root Cause Analysis. We refer to this Lean Tool because we want to trace a problem to its origins, sharing those information within the group.

In the Control Phase we introduce documented procedures (Standardized Work) easy to understand by all the end users involved in the project and useful for future improvement activities.

2.2 Balanced Scorecard Approach

For the analysis of inter-functional processes and identification of Key Performance Indicators (KPIs) we referred to Balanced Scorecard (BSC) [9,10,11] appropriately adapted to Healthcare Organizations. BSC has been extended from private companies to other sectors. In particular, for the Health and Social Organizations the perspective of BSC should be adapted to the effective needs of “high” quality and economic control required by management and legal regulations. The perspectives listed below identify 4 key areas where qualitative information is connected with financial data in a holistic way. We focused on the Financial, Customer, Internal process and Human Capital perspectives.

  • Financial area: this perspective views organizational financial performance and the use of financial resources;

  • Customer area: this perspective views organizational performance from the point of view of the customer that is the patient of Nursing Homes;

  • Internal Process area: views performance of critical processes which have significant impact on quality and efficiency;

  • Human Capital area: views performance of human capital, infrastructure, technology, culture.

We decided to adapt these perspectives primary to the needs of our end users and also to the available data sources in order to obtain the right data and KPI’s in the following way:

  • Economic and Financial perspective: KPI’s as variable, critical costs and staff costs;

  • Residents and Patients perspective: KPI’s as social and healthcare information about patients collected in Electronic Health Records (or external excel files);

  • Process perspective: KPI’s for all processes within the organization which are critical also for legal aspects (e.g. the time/pro patients dedicated by doctors, nurses, therapists, professional careers, the bed rotation, the timing of waiting list, quality problems of external suppliers, …);

  • Personnel perspective: KPI’s for absence/presence, turnover, trainings, salary computation.

Those different perspectives are inter-related: it means that our Business Intelligence (BI) dashboards should provide a standardized nomenclature and support for decision maker using a shared ontology across the different perspectives/key areas (Fig. 2).

Fig. 2.
figure 2

BSC system.

Full size image

2.3 Data Warehouse Design and Development

Designing a data warehouse requires a preliminary analysis phase in which the interesting business processes are identified to evaluate the experimental sites’ performances. The identified business processes are the foundation to identify facts and dimensions to be translated into a data warehouse (DWH) schema according to the Kimball’s Enterprise Data Warehouse Bus Architecture in [8]. A typical approach to DWH schema design is the star schema approach [8] in which Key Performance Indicators (KPIs) are represented with fact tables containing mainly numeric and quantitative measures. The contextual information allowing to explore the measures along various levels of aggregation and dimensions are represented into separated dimensional tables the fact will refer to. This design approach allows to keep DWH schema clean and flexible in case new KPI or new dimensions will arrive and allows to create queries that are optimized for reporting. Dimensions represents concepts stakeholders can easily understand. This allows to interact more easily with stakeholders to define list of dimensions and level of details along which the KPIs should be explored.

2.4 User Test with Think-Aloud

The Kimball’s approach to DWH design is quite effective however if applied as-is it tends to be, in our opinion, a waterfall approach as it lacks the flexibility and agility to deal with new projects in which the interaction mechanisms with users are not yet well defined and should be discovered along the project. For that reason, we combined traditional DWH design techniques with user-centered design (UCD) techniques. Particularly we apply user test technique that is used in user-center interaction to evaluate how users interact with the product.

Different techniques are available in literature to conduct user tests, but we will refer to the think-aloud technique. This technique requires users to complete some tasks using the interface and to report aloud thoughts and expectations on the product. Think-aloud is quite simple and cheap to apply in any phase of software development and type of prototype. It may be difficult to participants to be instinctive and to talk alone in front of the screen without thinking too much [14]. The feeling to be “observed” may influence their feedback giving modified comments and consequently biased results. For that reason, it is fundamental the role of the moderator to conduct the test, make the users feel comfortable and keep them focused on the tasks to complete. Another fundamental role is represented by the observer to get as much feedback as possible from users as the moderator is busy with them. Thanks to user tests it is possible to understand if users are capable to complete the tasks autonomously and the time required. Furthermore, it is possible to evaluate the level of satisfaction of users in using the product. [15]

We introduced user experience in the project with the intent to develop an understandable and intuitive BI dashboard. Similar approach was used in [19] to develop a mobile BI dashboard. Their user test was aimed at recreating the use of the product in normal daily routines, adding a heuristic evaluation of the interface to see if the design was in accordance with heuristics. SIBI [20] employed a user-experience approach to evaluate the introduction of BI in the Wright State University, providing dynamic reports on different thematic areas. The goal was to make data accessible to everyone. In the PhD thesis of Micheline Elias [21] user tests have been applied to make dashboards information useful to people who are not experienced in reading graphics. In one user test different types of users, newbies and experts, have been included to highlight several types of approach by capturing audio, video and screen. Elias approach uses also the think-aloud protocol to better understand the intensions and feelings of users.

In [17] think-aloud technique is used to develop a framework doctors can use to design their own reports. In our case this solution is not suitable because: we are facing several types of users (not only doctors but also administrative staff), we are dealing with a less precise domain (clinical diagnosis and procedures are well defined with ICD-9 codes, but socio-assistive services and procedures are less standardized).

A similar approach is used in [18] with more emphasis on the visualization aspects of the dashboard creation. Thought the idea to let users design autonomously their own dashboards is appealing from our experience it is affordable by few highly skilled and motivated users: our typical user does not have time and motivation to create a dashboard but only to perform little tasks and customizations on existing solutions to “play” with data (e.g. add/remove filters, rearrange graph dimensions and measures).

3 Approach

We applied the approach depicted in Fig. 3 combining the Kimball DWH development lifecycle adapted with Lean-BSC-CM (Lean, Balanced Scorecard and Change Management) and user-tests (see colored boxes).

Fig. 3.
figure 3

Kimball DWH development cycle adapted with Lean-BSC-CM and user-tests. (Color figure online)

Full size image

We created a multidisciplinary and cross functional team (6/7 participants), as follows (Fig. 4).

Fig. 4.
figure 4

Multidisciplinary team and flow of steps and techniques applied.

Full size image

In addition to the organization director we involved the reference people for financial, HR, medical macro-areas with the coordination of the quality control in supervising the identification of the KPIs and particularly the processes area.

The Six Sigma DMAIC (Define, Measure, Analyse, Improve, Control) was our internal roadmap for problem solving because we had to correct plans, take new actions and initiatives for each issue we met (e.g. incomplete data entry, lack of communication within the 4 areas, fear for transparency and change). DMAIC helped us to follow the project step by step focusing on little daily improvements (Kaizen) with our teams.

During a first round of interviews (3 meetings in 3 weeks, once a week, in group or single) the working practice of each reference person was depicted and the point of contact and interactions with the IT systems identified. This allowed to perform an as-is analysis of the actual business processes, to differentiate primary from support processes and to highlight inefficiencies, waste, costs and weaknesses of the current organizational system. From the outlined information flows we identified some critical points: business procedures are different, inefficient, processing data manually outside IT systems (e.g. using office automation tools like spreadsheets or even paper documents) increasing the risk to introduce errors, incorrect data, delays and overhead on workers; data is replicated across different systems and it is difficult to have a unique and official version of the information especially when it derives from manual manipulation outside the operative IT systems. From this first analysis phase we identified the data sources (Data Sources Identification) that can feed the DWH with consistent data, the organization’s structure which shape the data hierarchy (Data hierarchy definition), the KPIs to be presented and explored. The interconnection among the various stages of business requirements definition was the Lean-BSC-CM approaches trying to cover all the thematic area [10, 11]: Economic and Financial, Residents and Patients, Processes and Personnel. In Table 1 we show an example of KPIs derived by the business requirements definition phase.

Table 1. Example of KPIs.
Full size table

From the critical points derived from this initial analysis we identified possibilities to optimize and integrate processes. Furthermore, where current IT systems were used only partially we propose some training and coaching sessions devoted to minimize the manual work and maximize the data processed automatically by the IT systems to make them readily available to the BI platform. This Continuous improvement: process optimization, integration and training phase may be triggered also by results in the experimental phase as by the growth and maintenance steps in which the BI system will evolve to comply changing requirements. The other steps of our approach are the same of the Kimball lifecycle (Technical Architecture Design, Product Selection and Installation, Dimensional Modeling, Physical Design, ETL Design & Development, BI Dashboard Application Design, Deployment) as they are unavoidable steps to develop the DWH. A further extension of the original Kimball lifecycle is the addition of user-test phases performed as soon as the first dashboard prototype is available. Based on user-test results the dashboard evolves in a develop-test cycle repeated until an acceptable level of satisfaction is achieved.

Since the beginning we were aware that our BI project was related to Change Management which is a style of management which encourage organizations and individuals to deal effectively with the changes taking place in their work.

We had to face some resistances due to the deep change we introduced with our approach with the understanding that it impacted deeply on processes, systems, organizational structure and job roles. Those organizations are hierarchical not used to cross functional work and we had to face some initial “resistances”. We involved directors and end users because the change has to start from the bottom and we reinforce our role to facilitate bottom up and top down communication about their processes and critical problems.

With regular meetings and follow up we tried to face the “human” issues which create more difficulties than the technical ones by using all methods and strategies we could manage (Lean philosophy, Balanced scorecard approach, Change Management) in order to overcome different problems and “challenges”.

4 Implementation

In this section we present the details of the implementation and the application of the approach.

4.1 The Overall Architecture

We developed a complete Business Intelligence solution entirely based on Pentaho community edition modules (i.e. Pentaho Data IntegrationFootnote 1) and Apache SupersetFootnote 2. We realized a monitoring dashboard using Apache Superset sitting on top of a set of data cubes populated with Pentaho Spoon ETL jobs/transformations.

Most of involved pilot sites used some of our IT solutions for Electronic Health Records, Human Resource Management, Warehouse and Financial Management. This allowed us to have full control and knowledge of the majority of source data schema. However, also IT systems owned by different providers are used (e.g. for externalized services). In that case the source schema is not available, and data can be integrated from excel files through ad-hoc ETL.

Furthermore, data is updated even after a long time they are inserted in the IT systems with no automatic way to identify updates (in most of the cases there is no management of history or timestamp of last update). We decided to periodically reload data deleting potentially obsolete data to load the consolidated snapshot (e.g. for economic and personnel data the entire year will be reloaded while for medical data a daily snapshot is enough as medical data are consolidated daily).

We found very useful the capability of Pentaho Kettle in managing several types of sources and data types/format (structured and unstructured) allowing us to prepare a versatile ETL layer (receiving data from input files, databases) enabling the transformation and mapping to the standardized data warehouse schema (Fig. 5).

Fig. 5.
figure 5

Business Intelligence system architecture.

Full size image

4.2 The UX Evaluation

As soon as the structure of the interface has been defined we performed periodic sessions of user tests to highlight strengths and weaknesses of the dashboard to be corrected in the ongoing development process. In the first phase of the user test we asked users to complete some tasks. In the second phase of the test we proposed a questionnaire to evaluate usability of dashboards in terms of: accessibility and clarity of the displayed KPI with different graphical layout (table vs graph views); dashboard interactivity and granularity of exploration; accessibility and exporting capabilities.

We performed the user tests on 4 users that will be the actual users of the system: 2 directors, 1 quality manager, 1 administrative manager. The first time we conducted the user test face to face with the users. The subsequent tests were performed remotely.

The test has been divided into two parts: in the first part we used the think-aloud technique proposing a list of tasks to be completed on the dashboard and asking each user if they manage or not to complete the task; in the second part, we collected through a questionnaire the usability and the dashboard’s usage experience. The user test sessions have been registered (video and audio) to perform ex-post some assessments.

Table 2 shows an extract of the evaluation questionnaire provided to users at the end of user test session. For each question (if possible) we asked an evaluation with Likert scale to measure if we managed to improve the prototype. The first question was particularly interesting because it allowed to get the first emotional impression of the user in approaching and using the dashboard for the first time. Answers were quite different going from ‘My God! I feel a little lost’ to ‘Tranquility/Curiosity’, ‘Very Positive’. This first impression was very useful to decide for which type of users simplification was more valuable that data analysis flexibility and details.

Table 2. Evaluation questionnaire example.
Full size table

After each user test session, the data analyst identified a list of development activities to be performed to improve the dashboard and to be tested in the next user test.

5 Evaluation

In a previous BI project, we did not involve end-users along all the project phases (process analysis, process re-engineering and BI dashboard design). Our previous approach was aimed at:

  • designing a standardize set of KPIs that can satisfy as many customers as possible (30–100 customers) rather than customize indicators to customer’s specificities

  • providing tools which non-technical users can use autonomously to design their own reports starting from templates, with no a priori knowledge of the underlying data warehouse schema.

That project failed because:

  • it is hard to define a common set of KPIs valid for all customers, even if we focus on a very specific set of customers (medium-large residence home) because KPIs should reflect internal organizational procedures, that are only partially standardized by law

  • it is hard for non-technical people to design reports on their own, using tools slightly more sophisticated than an excel spreadsheet due to: frequent turnover, lack of time and motivation, reporting tool complexity.

In this project we take a completely different perspective: we asked users which KPIs might help their organization to work more efficiently, instead of which KPIs they should provide by law.

We notice how this project involves both technological and organisational challenges.

On Technological Side

As for any software product, also for a Business Intelligence solution user tests are fundamental to improve the solution, avoid errors and correct critical points as soon as possible. In addition to the advantages listed in Sect. 2.4, user tests are cheap and easy to perform: it is not necessary any particular infrastructure and, as we describe through our work, they can be performed even remotely [15]. Furthermore, it is not necessary to test with many people, 5 participants will be enough to discover the majority of problems and 15 participants can discover all usability issues. The number of participants should be enlarged if other key areas are considered [16].

At the end of the experimental phase we asked users an overall evaluation of project’s experience with an on-line questionnaire (Table 3). The goals were to evaluate:

Table 3. Final project questionnaire.
Full size table
  1. 1.

    the overall project approach and especially user’s involvement in testing the solution during the first development step and in the experimental session (second step);

  2. 2.

    the final usability of the solution and its level of maturity in satisfying users’ expectations;

  3. 3.

    the tangible advantages in terms of time (efficiency) and costs saved.

The experimental phase is still on-going, and we have a preliminary evaluation of the prototype. Based on that we completely restructure the dashboard to further improve the user experience. Further feedbacks and complete estimation of the ROI (in terms of costs reduction and time saved) will be collected during an additional 6 months experimental period. The preliminary results are encouraging: the main advantage underlined by stakeholders is the KPI’s service quality monitoring. They estimate to save at least 1 day of manual work each month to extract KPIs and at least 3 days each 3 month.

We also expect further time saving and errors reduction when exporting reports, as required by external auditors (3/5 days each month required to make them manually), will no longer be done manually. Furthermore, trend analysis in a simple, intuitive and timely base allows management to perform internal auditing analysis and prevent inefficiencies and quality process issues. In this way, it will be possible to test the effectiveness of operative procedures and the impact of new strategies (e.g. outsourcing decision and ROI evaluation). It will also allow the quality check of outsourced services that are currently hard (or even impossible) to control. During a usability test session one of our end user defined our dashboard “reassuring” because it gives a new experience of interactivity and simplicity by exploring data timely and without errors.

On Organizational Side

From our company side we establish a multidisciplinary team (IT, Sales, Marketing and organizational specialists) that could manage several issues we had to face (e.g. different local regulations and health services, different staff and use of existing data base, different needs and also different culture and attitude to the change).

We had to figure out a way to analyse customer needs, their processes, their hierarchical organization and their motivation because a BI project involves deep internal changes and it can fail because of cultural and mental barriers (definitely not for IT problems).

Technical solutions don’t guarantee success, especially for long-term care organizations, because their workers (medical units and staff) are not used to change and to interact as an “inter-functional group”. Often those groups focus on medical issues excluding other functions and processes; furthermore, they often obey “unwritten rules”, which are difficult to understand and to change.

This BI project made us aware of the need of an educational and training program, because the different users have to understand what is going to change and how the project will impact their daily work.

Our real internal challenge is to motivate and develop a new “Vision”, according to Lean Philosophy, which could better help the team to check processes and no value activities in the four BSC key areas. Furthermore, the BI solution can trigger a “cascading process” in all areas for continuous improvement required by the management and also by local rules.

This BI solution linked to Lean Philosophy and Change Management can improve the overall quality of the work within those organizations. It takes into consideration key processes as health care, financial and staff processes, and improves efficiency and effectiveness of each employee, nurse and doctor. Looking at the whole approach this project can reduce waste of time and resources, while it can deeply improve the health service quality.

Usability, KPI’s Analysis and the design of the DWH are key “technical” aspects but other “soft” issues can make the difference. This BI project brings to our company a new vision no longer exclusively based on the IT software sales but on a counselling service that can really change and improve customer’s work. The “BI service” should lead a customer to analyse its organization and processes from different points of view (technological but also organizational) and to find efficient solutions to improve the life quality of their patients.

Even if service quality in healthcare domain is hard to measure we have to include these KPI’s (process perspective in the BSC) and consider them as critical for a continuous quality improvement.

6 Conclusions and Future Work

As we have underlined, future challenges will involve technological and especially organizational and educational aspects.

Technically, we have the following open points: (i) some data are still on excel files - outside our systems (frequently data are manually redacted to get a final balance report); (ii) information kept as “free text” by using notes fields instead of categorized fields (frequently for medical/social information); (iii) use of several IT systems owned by different providers.

In order to face these issues, we firstly involved end users as much as possible with training sessions and regular follow up because we had to change not only their process but also their “vision”. Secondly, we worked on the technological infrastructure harmonizing data managed by different systems to allow an overall vision of cross-organization processes.

We are not proposing a solution to a novel problem (data warehousing and Business Intelligence systems are now state of the art solutions), but we think that the particular approach, linked to different methodologies as Lean, BSC and Change Management is quite innovative. It was an effort also to involve stakeholders in all the phases of the project, from the analysis to the final solutions, while coordinating our multidisciplinary team.

Furthermore, this BI solution with multidimensional KPIs can provide comparative analysis and graphs in order to observe the dynamics of KPIs. This system can reveal potential areas (medical, financial and human resources areas) where performance is lacking and the management can examine and identify root causes determining performance gaps. BI solutions, combined with Lean Tools and Internal Benchmarking, can make the difference in order to improve health care quality standards and all the processes within those organizations.