Keywords

1 Introduction

BIM will be a milestone at the AEC programs in the near future. The AEC Industries nowadays face an inflection point, as scholarship describes it: “twenty years ago, AutoCAD pushed designers into a new era; BIM represents a new generation of virtual model already widely accepted it by the industry [1] BIM’s multi-dimensional approach allows to see how the pieces of their project fit together in real time” [2], that gives the students the opportunity to understand how all the knowledge acquired through the different courses fit together and the relation between them. The students will use BIM technologies to create a model which contains all the information in a single file to produce all the documentation that other-wise would be created in isolation and duplication [3].

The purpose of this paper is to explain the design of the methodology to obtain the data to analyze the implementation of BIM Technology in the AEC curriculum taking into consideration these factors: The motivation of the students towards computer tools technologies and the usability of BIM applications in their studies and career; the student proficiency with the tools required; the adaptability of these technologies to the course where it will be introduced; and the students future intentions towards the different software used after this experience.

The methodology of this study will be a mixed-method quantitative/qualitative, where the student interaction with the BIM tools will be analyzed. The research will be done in several courses related to the areas of design, construction and computer tools where BIM is included. The analysis will quantify the motivation, and satisfaction of the students and determine the reasons after this data.

2 Related Work

2.1 Enhanced Learning: Increasing the Motivation and Satisfaction of the Students

The use of ICTs (Information Communication Technologies) in educational methods is defined in the curricula of many undergraduate and master’s degrees, including the architecture degree, the focus of this study [4]. These descriptions indicate that the student be able to acquire both personal and collaborative competencies and skills related to active learning, as well as information management, through applications and devices that enable the adoption of Project Based Learning (PBL) approaches. These methods should allow students to work using specific and effective roles much more quickly than with traditional systems and should be able to apply them in their work environment in the future. For all of these reasons, it is necessary to propose and implement new methods through PBL and BIM systems, including the use of appropriate technologies that enable the student to more optimally dedicate him/herself to project and time management.

We must approach the world and teaching of architecture as a component of traditional teaching, which also includes law, medicine, the fine arts and politics. From this perspective, drawing, painting and photography would be precursors to the main technological innovation incorporated at the educational and professional levels: the computer and the use of CAD (Computer-Assisted Design) applications [5, 6]. The following evolution is much more recent and is, in a sense, the cause of the current changing landscape and great need for much broader training in all types of technologies: the appearance of BIM (Building Information Modelling) [7]. Traditional proposals, CAD, the more recent BIM, and other techniques, such as digital sketching (DS) and digital infographics (also known as digital graphics, DG), are all aimed at solving one of the main problems in architecture: the modelling and visualization of complex elements. The acquisition and representation of data associated with land are complex and incompletely documented aspects of modelling [8]. From an academic viewpoint, these systems are used to improve the acquisition of skills and spatial competencies to analyse the visual impact of any building or architectural project.

Numerous types of studies have linked the use of ICTs with improved student motivation and, correspondingly, academic performance [9]. While focusing on the study of user behaviour and emotions, we cannot forget its connection with the area of widely documented knowledge corresponding to user experience and usability (UX). These areas are historically related to the field of human-computer interaction (HCI). From that perspective, it would be interesting to analyse any innovation that involves the use of new computer systems or technologies [10].

2.2 Building Information Modelling (BIM), in the Curriculum of the Student

The implementation process of BIM in schools revealed that it should not be simply to create a new course in the curriculum because, as claim Taylor et al. [11]: “BIM has the potential to be introduced throughout the program”. Some schools in the AEC sector are exploring BIM applications in their disciplines and struggling to integrate them with other ones. This is what all the schools should do, in the opinion of Camps [12]. The schools can maintain the strengths of traditional education based on disciplines and become multidisciplinary [13]. A separate integrative approach, in which the subjects are divided into separate courses, but try to collaborate with each other, ensures no insulation [14].

Thus the principles of BIM can be first introduced into a subject and then between disciplines [15]. The first two years would focus on the individual skills of modelling and analysis of the model [16] (BIM Course and DGR). The subsequent years could focus more on teamwork and complexity through collaboration [16]. (BIM Course, Design Studio and Building Technology). The last year could deal with actual construction projects in collaboration with companies [16] (BIM Course and Management) [17].

As the great study cited [17] above explains, the BIM implementation at the AEC curriculum has a lack of data. Our study will provide the methodology to collect scientific data about the students’ motivation, satisfaction and performance after the implantation of BIM technology in the courses of Design, Construction and Modeling and Simulation. This data will be a perfect tool to analyze and design a solid BIM implementation.

2.3 Mixed Methods Assessment in Educational Environments

Quantitative methods are vital for rapidly assessing the key factors from the design perspective that are to be evaluated at the beginning, during, or at the end of the development process. By characterizing and providing detail to the quantitative data, however, qualitative methods provide a point of view that is more subjective and, at times, complementary to the user. Whereas quantitative studies have traditionally been linked to statistical and sociological studies, qualitative approaches have been used in the social sciences and usability research, especially due to the personal input that can characterize highly detailed responses. A combination of both approaches generates what is commonly known as mixed-methods research, which has been widely tested and continues to produce study results, filling gaps in each model and refining the obtained results [18].

To analyze the proposed methods and assess the degree to which they were accepted, we used a formula for data extraction that was previously validated in other educational studies in the same field [19]. An initial pre-test was conducted to obtain the student’s profile and starting level of motivation/knowledge about the use of selected technologies. At the end of the course, a post-test was conducted to assess the level of satisfaction and completed use of each system, also a personal interview was conducted using the Bipolar Laddering (BLA) technique [20], which allows us to identify and quantify the strong and weak points of the proposed methods. The current proposal has a clear line of continuation in future iterations because it reveals correlations between statistical results and final grades in a way that allows us to evaluate the relationships between different variables in the study and the student’s improvement. This proposal also allows us to form conclusions in the future that can be extrapolated and validated in other areas of knowledge. Likewise, the mixed approach will allow us to extract data and identify noteworthy and adaptable aspects in future applications of the proposed system.

3 The Case Study

This section presents the development of the methodology to collect the data for the analysis of the BIM technology implementation in the Architectural Engineering Curriculum. This methodology has to deal with the difficulty of being applied in different courses of the design, construction and computer tools areas of the architectural engineering program. Every course has its own different teaching methodology, criteria and objectives.

The main objectives of the study are to create a methodology to collect mixed data; to measure the students motivation and satisfaction in the courses where we apply the new proposal; to create a new method to collect samples of the student’s marks, from the course designed; and finally, to create a method to analyze the performance of the students.

3.1 Methodology

This research is based on a mixed-method research methodology. As Johnson et al. [21] define it, mixed-methods research “is an intellectual and practical synthesis based on qualitative and quantitative research; it is the third methodological or research paradigm (along with qualitative and quantitative research). It recognizes the importance of traditional quantitative and qualitative research but also offers a powerful third paradigm choice that often will provide the most informative, complete, balanced, and useful research results”. The research that we are developing aims to provide real quantitative and qualitative data as a tool for the AEC schools to decide how to introduce BIM in their curriculum.

The methodology design in this paper will be applied during three semesters. In the first semester it will be considered as preliminary, and we will use it to validate or improve the final methodology for the study.

The study will have two main elements the tests and the exercises delivered by the students. These two elements will provide us the data to evaluate the motivation, satisfaction and performance of the students.

3.2 Test Design

The design of the “test of user” is a common topic inside the scientific research and experimentation based on user’s responses that will provide us the data. The main objective of these tests is to assess the usability of the new learning environment. In this specific case, the learning environment change is due to implementation of BIM technology in the courses of design, construction and computer tools.

In the university framework there are successful studies to design the survey process that we will take into account to develop this methodology. These studies analyze the implementation of new technologies at the curriculum based on the user profile. The focus of these studies is the efficiency and effectiveness of the course, and the level of satisfaction and students preferences [22]. The most common parameters considered in this type of survey are, the degree of knowledge of the technologies, the use made of social networks, computer tools application level and theoretical knowledge of the course applying the new technology studied that will be implemented [23].

For this specific research which is based on the implementation of the BIM technology at the architecture, engineering and construction management curriculum, we will develop a two test survey; the pre-test to know the students proficiency and expectations; and the post-test to know their motivation, and satisfaction.

We will develop a cyclical process in order to validate or improve the test methodology for this study. This cyclical process is composed of three phases. The first phase is the development, validation or improvement of the test. The second phase is the extraction of the data for the main research. The third phase is the analyses of the results of the survey looking for unnecessary and lost data that we should collect for the main research.

To begin the research with a solid foundation, we developed the first test from the one published at [23]. This test is used in several upstanding studies like [24, 25], and already tested and validated. The first step was to analyze their test and check which chapters could be useful and which ones we should replace for our specific research.

There were six chapters in their initial profile test, new technologies; Internet social network and other tools; applications; computers and laptops; mobiles; and augmented reality. After the analysis of the questions we decided that the last chapter would focus on the technology they were implementing which is not the same as we were testing. All the others are used to create a user profile that will be useful to realize a better analysis of our results. For this reason we decided to change the augmented reality chapter, creating a new chapter to analyze the students’ proficiency and expectations of BIM technology applied to the courses. The chapters that we used from their test where rewritten for an easier understanding of our students.

The BIM chapter that we developed aims to extract three main items: their level on BIM before starting the course; their expectations towards BIM technology and finally their technology preferences to be applied in the construction, design and other type of courses.

The pre-test will be run during before the fourth week of the course, after the students finalized their registration. This item will provide us the student profile of each course that we are studying. It will help us to adjust and personalize the course to our students (Fig. 1).

Fig. 1.
figure 1

Pre-test example design

Full size image

The cyclical process to improve the methodology will be done for the whole process as a set of elements. This process will be right after the results analysis of both tables. This process aims to validate or modify the questionnaires to improve the data collected. To develop the second test, we review the mixed-methods research literature and choose the test from [19] as the base. We decided to implement their general assessment part of the questionnaires, and create two specific new chapters about the BIM technology.

The first chapter will test the satisfaction of the students in relation to the BIM technology contribution in their studies and future career. During this first chapter we will ask questions as: how useful will BIM be in your studies, and do you think that BIM will improve your grades.

The second chapter will test the users’ perception of the BIM tools in the different areas of the AEC (architecture, engineering and construction management) and the usability of these technologies to improve the AEC curriculum. We will ask questions as: Evaluate Revit as a tool to understand the building systems, components and joints; and evaluate Naviswork as a tool to review your projects using clashes and comments. With all these questions we will have a map of the students’ understanding and perception towards the BIM tools and the courses where it is applied.

This second questionnaire will be delivered to the students during the last two weeks of lectures before the final exams. We have developed the tests using Excel and Google forms. The second tool gives us the possibility to run the survey online.

3.3 Performance Data, Collection and Analysis Design

As part of the methodology for our research, we develop criteria to collect the samples of the exercises from the courses. This criterion is one of the key points for our research. To provide it with the proper importance, we set up several attributes as a must. The process to collect the exercise samples ruled by our criteria has to be reproducible, measurable, objective, methodological, scheduled, and organized. All these attributes of the criterion will give us a solid scientific foundation to begin our research, and creates a common framework where the researchers will be able to share, compare and validate the results.

To begin the solid foundation of our research we need samples from the courses before the implementation of the new technology we are going to test. To collect these samples, as stated before, we have created the following rules.

The first criteria to establish will be the type of samples that will be collected. The samples from the previous courses will be graded exercises, which we will document by PDF. To be methodological, standardized and organized we will add to these exercises several pieces of information: the name of the institution where the course is done (United Arab Emirates University), the country (United Arab Emirates), the semester (Spring, Fall or Summer), the year of the course (2016), the course name (Intermediate Design Studio), the number of students on each group (1, 2, 3…), the grade accomplish (A+, A…), and the grading criteria if it is available.

The second criteria will be from where we will take these samples. The samples will be collected from the official course file. This will create the possibility for any other researcher to collect the same information to validate our experiment.

The third criteria will be the number of samples to collect. We will collect three graded-samples, one from a very poor student, another one from an average student and the last one from an outstanding student. When a very poor sample is not available, we will collect only the average and good student samples.

The fourth criteria will be from when these samples can be taken. The samples will be taken from the previous three semesters before the technology that we want to try is implemented (Fig. 2).

Fig. 2.
figure 2

Samples of previous graded exercises from the course building components

Full size image

To collect the samples from the actual course, we will follow the first three criteria stated before. For the first one, we will collect graded-samples of the exercises in PDF, adding all the relevant information. For the second one, we will do it in the opposite direction, so the new graded exercises done with the new technology implementation that we will use to compare with the old ones, will be the exercises that we will add in our course file. For the third, we will collect the three graded-samples as explained before.

As this methodology is set up to be used by any course that wants to implement and test a new technology, the criteria of evaluation will be defined by elements which are stated in the syllabus of the courses. For this reason the evaluation, analysis and comparison of the graded samples already collected will be done by following the criteria of the course that is written in the syllabus. The evaluation will use the outcomes, milestones or objectives and grades given by the professor as the three different resources to test the samples. We will test all the samples three times, evaluating the level of accomplishment at each criterion. These criteria are set up by the professor in the syllabus of the course. We are going to lay out one sample for each of the outcomes and milestone criteria, understanding that it is a partial view of the real analysis. The outcome and milestone criteria used as samples for the article was developed by assistant professor Abdulaziz Banawi of the United Arab Emirates University, for the intermediate course syllabus in the Spring of 2015.

The first analysis that we are going to introduce will be the outcomes analysis. “Outcome 1: Students demonstrate proficiency in designing and developing detailed construction drawings for their proposals including design of integrated profiles, material selection and systems design [C, E, J]”. These letters makes reference to the ABET accreditation program outcomes (A-K): http://www.abet.org/wp-content/uploads/2015/05/E001-15-16-EAC-Criteria-03-10-15.pdf.

The analysis of this outcome as a criteria to analyze the graded-samples, gives us this set of rules to evaluate the exercises of intermediate design studio. We should look for detailed construction drawings, the integration of the elements in the design, applied criteria to select the materials and systems alternatives, and selection criteria. It is very clear from the outcome, so it can be analyzed and compared.

The second type of analysis that we will run, will be the milestones or objectives of the course. The sample that we brought is a milestone that states what should be target by the exercises, “Building Design Research: design issues, site factors analysis, spatial analysis, case studies, design intent”. As stated in the milestone, we will look in the samples for the site, spatial and case studies analysis, as well as the design approach. As we can see, the outcomes and the milestones give us a totally different analysis approach.

The third and last analyses will synthesis of the other two previous analyses introducing the variable of the grades given by the professor.

4 Conclusions

As a result of this paper, we have created a mixed-method methodology to analyze the motivation, satisfaction and performance of the students at a course that is implemented with a new ICT technologies. This methodology gives everyone, the opportunity to perform the same experiment and validate or discuss our data. In this paper we have presented two separate methodologies, a test-based one that provides us the motivational and satisfaction analysis, and another one based on graded sample exercises that test the performance of the students.

The test-based methodology is a mixed-method survey that gives us the “user profile”, their motivation, and satisfaction based on two tests that are performed at the beginning and at the end of the course and is specifically designed for the implementation of BIM.

We created a graded-sample methodology to analyze the performance and improvement of the students in the course which is implementing a new innovative teaching methodology. This methodology developed in this paper provides: a research common framework based on available and official data; and a reproducible, measurable, objective, methodological, scheduled, and organized system to collect and analyzed the samples.

The combination of the test-based and the graded-sample methodologies provides realistic and more accurate data about the implementation of BIM in the AEC curriculum which has not been done before. This study will provide new insights and the proper feedback to optimize the implementation of BIM in the (AEC) curriculum.