System for recognizing lecture quality based on analysis of physical parameters

https://doi.org/10.1016/j.tele.2017.06.014Get rights and content

Highlights

  • We implemented the system that can determine if students are satisfied with the lecture quality.

  • The system analyzes parameters from the physical environment obtained using smart devices.

  • The system is able to extract sound features from the ambient sound.

  • The system is implemented in Matlab and his accuracy ranges from 70.7% to 83.9%

Abstract

In this paper we have presented a smart classroom system that is able to classify students’ satisfaction with the lecture quality by examining parameters of the physical environment obtained using different smart devices. The system is based on the Random forest classifier, which showed the best accuracy among all machine learning algorithms available in Weka tool, with dataset collected during 28 lectures and evaluated using 10-fold cross validation. The system is implemented using different set of tools (such as Matlab and Weka) and can extract features from the ambient sound and analyze values obtained from different smart devices deployed in the classroom. Based on the extracted and captured data the system provides in real time information about the students’ satisfaction with the lecture quality. For the validation purposes, we recorded 13 more lectures attended by four different student groups where the number of students varied from 5 to 18. The system accuracy was evaluated by comparing system outputs with the students’ feedback and ranged from 70.7% to 83.9%.

Introduction

The service can be seen as a process that transforms inputs in order to provide customers with the added value (Holbrook, 1994). Services are ‘deeds, processes and performances’ (Zeithaml and Bitner, 2002) having the following characteristics: intangibility, inseparability of production and consumption and heterogeneity (Zeithaml et al., 1985). A lecturer presents a lecture with the aim to improve students’ knowledge that can be seen as an added value. Additionally, presenting a lecture has all the previously mentioned characteristics that are typical for services (Clewes, 2003).

Therefore, giving a lesson represents a service in education. All findings from the services literature can be applied to the context of higher education (Voss and Gruber, 2006). If lecturers know what students need, they may be able to adapt their behavior and presentation to meet students’ underlying expectations, which should have a positive impact on their perceived quality and their levels of satisfaction.

Quality in higher education is a complex concept where the single definition cannot capture all its different aspects (Harvey and Green, 1993). As quality of services can be divided into two distinct components: outcome and process quality (Gronroos, 1982) and since everything that states for the service quality can be applied to the lecture quality (Devinder and Datta, 2003), a quality of a lecture delivered to the students consists of the same components. Outcome quality can be seen as the extent of skills gained during the lecture, while the process quality can be divided into tangible (classroom condition, illumination, acoustics and quality of presentation) and intangible quality (lecturer ability to deliver the lecture, the willingness to help students, etc.) (Devinder and Datta, 2003).

When we talk about service quality we usually think of the perceived quality, which can be seen as the comparison between customer service expectations and their perceptions of actual performance (Zeithaml et al., 1990). For this reason, the classroom teaching service will be considered as a quality service when the lecturer meets or exceeds student’s expectations (Parasuraman et al., 1988).

Service quality cannot be measured objectively (Patterson and Johnson, 1993) and the best way to define and measure service quality does not exist yet (Clewes, 2003). There are many studies that confirm positive correlation (Hasan and Ilias, 2008, Ham and Hayduk, 2003) between perception of service quality and student satisfaction. Satisfaction can be defined as an emotional reaction to a product or service experience (Spreng and Singh, 1993). Satisfaction is a subjective perception (Dabholkar, 1995) of the degree to which customers’ requirements have been fulfilled (ISO, 2000). Many studies confirmed that students’ perceived service quality is an antecedent to student satisfaction (Browne et al., 1998, Guolla, 1999).

For all these reasons, in order to predict lecture quality we decided to observe students’ satisfaction with the lecture quality during lectures. Students’ overall satisfaction can be measured using different surveys collected after a lecture, but this piece of information is not useful for differentiating segments of the lecture with which the students were satisfied from the segments with which they were not satisfied. To reveal if the students are satisfied with the lecture quality at a given moment (on a particular segment), we need their real-time feedback instead of the surveys filled in after the lecture ended. For this reason, we gave detailed instruction to the students how to evaluate the ongoing lecture and collected their feedback in real-time to annotate segments of lectures as satisfactory or not satisfactory.

This system is based on the concept that is called Internet of Things (IoT). The International Telecommunication Unit (ITU) views IoT “as a global infrastructure for the information society, enabling advanced services by interconnecting (physical and virtual) things based on existing and evolving interoperable information and communication technologies” (ITU-T). Things or objects in the IoT refer to a broad range of devices such as RFID tags, sensors, actuators, smartphones, among others. A range of new applications based on this technology is broad and diverse, i.e. e-health, traffic, environmental monitoring, smart homes, smart classrooms, to mention a few. This paper focuses on using IoT in smart classrooms. Smart classrooms can be defined as intelligent environments equipped with an assembly of many different kinds of hardware and software modules such as projectors, cameras, sensors, face recognition module, and many more (Xie et al., 2001). In our case, a smart classroom is equipped with a set of sensors able to monitor parameters of the physical environment (for example CO2, CO, temperature, humidity, noise), a Bluetooth headset used to capture lecturer’s voice, as well as a smartphone with a 3-axis accelerometer sensor used for determining intensity of moving lecturer’s hand.

Authors have published few articles where they have investigated some aspects of the learning process and the parameters that influence them (Gligoric et al., 2012, Gligoric et al., 2015, Uzelac et al., 2015).

The first study (Gligoric et al., 2012) represents the basis for the further researches. The authors have specified the problem of the real-time feedback on the lecture quality using IoT. Among with problem specification, the system requirements were specified and the system experimental design was proposed. The parameters that may have influence on the lecture quality were selected based on the conducted questionnaire. The research lacks implementation part and the thorough analysis of the selected parameters.

The second paper (Uzelac et al., 2015) investigates the parameters of the physical environment and their impact on students’ focus. The primary goal of the paper was to identify parameters that significantly affect students’ focus during lecture. The parameters that were investigated include humidity, ambient temperature, noise, as well as thorough analysis of the features extracted from the lecturer’s voice. Again, this research lacks the implementation part. In contrast with the current research, the study investigates students’ focus, not the students’ satisfaction with the lecture quality.

In the third research (Gligoric et al., 2015), a smart classroom system that detects students’ overall satisfaction level with the given lecture is presented. The aim of the implemented system is to find patterns in students’ behavior through the intensity of their motions and by analyzing the sound they produce (without considering any environmental parameters) and use them to determine if the students are satisfied with the lecture at a given moment. This research contains the thorough analysis of the selected parameters as well as the implementation part. As this research does not explore the influence of the environmental parameters (such as humidity, ambient temperature, etc.), and the lecturer’ influence on the learning process, further research was needed to shed more light on that part. This led to the different recording setting; new series of recordings were conducted in order to obtain the new dataset. In the new research, instead of analyzing ambient sound, the features from the lecture’s voice were extracted and examined. Additionally, different method for parameters selection was used: instead of kappa values, in the new research we are using attributes importance obtained via Weka’s toolkit (Hall et al., 2009). Furthermore, we improved the algorithm that calculates the intensity of the lecturer’s motions. Additionally, the implementation part was changed: the algorithm used for the classification was switched – instead of Adaboost M1 algorithm, Random forest classifier was implemented; image processing part was thrown, while the new part used for calculating intensity of the lecture’s movements was introduced. The cloud platform used for storing values also needed to be altered. All of these changes influenced and changed the overall system architecture.

Due to the previous discussion, the aim of this study is to present a new smart classroom system that is able to classify the students’ satisfaction with the lecture quality by examining parameters of the physical environment obtained using different smart devices. The main contributions of this manuscript are: (1) an innovative approach to analyze the impact of different parameters in the physical environment on the students’ satisfaction with the lecture quality, (2) identification and analysis of the features extracted from the lecturer’s voice and their impact on the students’ satisfaction with the lecture quality, (3) presents a smart classroom system that is able to determine students’ satisfaction with the lecture quality.

Smart classrooms are continually being improved by applying technology innovations. The first generation of smart classrooms happened between 2001 and 2008, and “the early smart classroom implementations were primarily focused on synchronous delivery of learning content to local (i.e. students in actual physical classroom with face-to-face learning/teaching mode) and remote/online (i.e. students, in remote locations with online mode of learning/teaching) as well as synchronous teacher-students and local student-to-remote student communications” (Uskov et al., 2015), while the second (current) generation of smart classroom implementations “is mainly based on active use of mobile technology, user/student/learner mobile devices and automatic communications between them and smart classroom environment” (Uskov et al., 2015). Currently, smart classrooms are evolving with the help of IoT technology and are rapidly becoming environments able to sense and modify their physical or behavioral characteristics to adjust to the students’ needs. Different models and frameworks based on IoT are being proposed.

Many different learning systems have already been developed. One of the first applications of IoT in a smart classroom domain was in 2008 when an early infrastructure of IoT with a set of prototypes for different learning spaces was proposed (González et al., 2008). In another research a smart classroom system that is able to detect students’ attention by analyzing student’s eye movement and emotion by examining short and long-term features of speech was developed (Luo et al., 2009). A system that enables lecturer to monitor the level of students’ attention using LED lamp was demonstrated in (Gligoric et al., 2014). A service-oriented framework for the future classroom with IoT concept together with two example applications was presented in (Chang and Chen, 2015). A so-called free learning system based on IoT which architecture consists of smart classrooms and remote and virtual labs that communicate with each other autonomously was recently introduced by Said and Albagory (Said and Albagory, 2016).

IoT is being used in education with different purposes: to determine classroom attendance, optimize learning environment, enhance the learning process, to mention a few. Many smart classrooms were developed with the aim to determine students attendance. Locating students and calculating their attendances can be done using different technologies: such as RFID (Chang, 2011, Atabekov, 2016), NFC (Shen et al., 2014), or smartphones (Yang et al., 2016). Beside determining students attendance, IoT has an important role in managing the learning environment. For example, IoT can be used for conserving electrical energy in the classrooms (Gupta et al., 2015). Furthermore, learning environments can be optimized by using IoT and the techniques of learning analytics in terms of giving suggestions to the teachers to adjust their teaching strategies and activities (Cheng and Liao, 2012). A model where students could provide preferred values of environmental variables was presented in (Simic et al., 2014).

In order to reach its full potential, IoT is sometimes merged with technologies sush as Big Data. In a study published in 2016, the researches investigated the impact of these technologies on the education of the learning-disabled students (Lenz et al., 2016). Another example how different technologies can be combined with the IoT was presented in (Palma et al., 2014), where an application that collects information from the classroom with the aim to create a control tool that displays access and the status of all classrooms was presented. Additionally, IoT can be combined with mobile technologies in order to extend and enhance the traditional mobile learning (Shan et al., 2016).

Smart classrooms that belong to the current generation with their main characteristics are presented in the Table 1.

In our previously published paper, we introduced smart classroom system that is able to detect the level of students’ interest in near real-time with the accuracy of 80% (Gligoric et al., 2015). In this study we have used devices such as camera and a broadband microphone to monitor students’ behavior to find patterns that are predictors of the levels of students’ interest during the lecture. We have also tried to measure lecturer’s activity using a smartphone with built-in 3-axis accelerometer located in his/her pocket. The aim in this study was to monitor students and their activities, while in the new one we have focused on monitoring environmental parameters and made a significant progress in measuring lecturer’s motions. In addition, the new study is focused on determining students’ satisfaction with the lecture quality that, combined with previously determined level of students’ interest, will be able to give us more precise and accurate picture of student's needs during a lecture.

As the relationship between the quality of lectures and student’s achievement (Centra, 1977), as well as student's performance (Ahmed et al., 2010), were confirmed, we assume that the same parameters that have shown to affect student’s achievement or performance may influence the lecture quality as well. As no researchers investigated direct influence of different environmental parameters on the lecture quality, we decided to consider parameters that are confirmed to be positively correlated with either student’s achievement or student’s performance.

Many researchers report the negative impact of inadequate temperatures on student’s performance (Wyon, 1970, Schoer and Shaffran, 1973, Wargocki and Wyon, 2007). Other studies investigated air quality, especially the levels of CO2 on student’s achievements or performance quality (Shaughnessy et al., 2006, Wargocki and Wyon, 2007, Coley et al., 2007, Bako-Biro et al., 2012, Molhave et al., 1986, Otto et al., 1992) and confirmed that either low ventilation rate or high level of CO2 has an adverse impact on students’ performance. Also, it has been shown that concentration increases as air pressure rises (Howarth and Hoffman, 1984). Additionally, it has been confirmed that higher levels of CO affect subject’s speed to process data, have a negative effect on human performance (Teichner, 1975) and make reaction-time slower (Ramsey, 1970). Many experiments confirm the negative impact of noise on academic performance (Crook and Langdon, 1974, Johnson, 2001, Downs and Crum, 1978, Bronzaft and McCarthy, 1975, Evans and Maxwell, 2007, Kyzar, 1977).

A lecturer can also affect students’ achievements and satisfaction through his/her expressive behaviors (Murray, 1997). There are different channels for showing expressive behavior such as the face, speech, the body, and tone of the voice (Ambady and Rosenthal, 1992). Many studies confirmed that analyzing different non-linguistic vocal features such as pitch, rhythm, energy, speech rate, intonation, perceptual loudness and voice quality can give us information about speaker’s emotional state (Fairbanks and Pronovost, 1939, Williams and Stevens, 1981, Huber et al., 2000, Batliner et al., 2003, Lee and Narayanan, 2002), which may affect students’ engagement during lecture and their success in the classroom (Zembylas and Schutz, 2009). There is a great number of features that can be extracted from the speech signal, and we decided to use social signaling measures that were proposed by Pentland (2004) and refer to indicators in speech and conversations that unconsciously convey information about the speaker’s intentions (Caneel, 2000). These features have shown to be successful in various contexts, from determining if a particular couple is attracted to each other in speed dating sessions (Madan et al., 2004), to predicting the outcome of the job interview (Soman and Madan, 2009). The following sound features have proved to be valuable for measuring social signals: formant frequency, spectral entropy, autocorrelation, energy, voiced segment, speaking segment, speaking time and voicing rate (Pentland, 2004). Following the previous discussion, we assume that the voice channel is a carrier of useful information related to measuring expressive behavior, and for this reason we decided to record it.

Beside voice channel, teacher can use a different channel such as movement to show his/her expressive behavior (Murray, 1983). Since a hand is the most fluent and articulate part of the body, capable of expressing almost infinite meanings (Zhao and Badler, 1998), we decided to measure movements of lecturer’s hand.

Due to the previous discussion, we have decided to measure tangible parameters such as parameters of the physical environment: CO2, CO, temperature, air pressure, humidity, and noise. From the intangible parameters we have agreed to analyze lecturer’s voice and his/her movements.

Section snippets

Method

In this section, we have explained the experiment in more details regarding participants, the recording setup and the procedure. The list of parameters that we decided to investigate is also presented. Lectures were recorded with the aim to build the dataset for the machine learning algorithm. The dataset consists of collected parameters with their values altogether with the labels assigned by students. Each segment of a lecture was annotated as “satisfied” or “not satisfied” depending on the

Results

The null hypothesis states that there is no difference between the attribute values on the segments labeled as “satisfied” and the segments labeled as “not satisfied”. After performing t-test using significance value of 0.05 for each attribute, the null hypothesis was rejected for 12 attributes. For this reason, we segregated these attributes to investigate their impact on the lecture quality.

From all the features extracted from the lecturer’s voice, the following seven were found to have a

Discussion

In this study, we have implemented a system that is able to answer if students are satisfied with the quality of an ongoing lecture by analyzing values obtained using different IoT devices. IoT devices measure parameters of the physical environment, capture lecturer’s voice and motions and send them for further analysis. From six tangible parameters of the physical environment, only two: level of CO2 and humidex (a combination of temperature and humidity) have shown to have a significant

Conclusions

In this study we have presented the system that is able to determine continuously in real-time if students are satisfied with the lecture quality. Although prior researches agreed that certain parameters of the physical environment such as temperature, environment noise and the level of CO2 impact students’ performance and/or achievements, to the best of our knowledge, there is no study that has tried to examine their direct influence on students’ satisfaction with the lecture quality. In this

Dr. Ana Uzelac holds a Ph.D. from University of Belgrade, Serbia. She has graduated Faculty of Mathematics at Belgrade University. She spent eight years working as a programmer where she held a number of positions (junior programmer, senior programmer, team leader). She works as an Assistant Professor at Faculty of Transport and Traffic Engineering at Belgrade University where she is engaged in subjects related to programming languages and databases. Her research interests include Internet of

References (97)

  • A.L. Bronzaft et al.

    The effect of elevated train noise on reading ability

    Environ. Behav.

    (1975)
  • B. Browne et al.

    Student as customers: factors affecting satisfaction and assessments of institutional quality

    J. Market. Higher Educ.

    (1998)
  • R. Caneel

    Social signaling in decision making

    (2000)
  • J. Centra

    Student ratings of instruction and their relationship to student learning

    Am. Educ. Res. J.

    (1977)
  • Chang, C.H., 2011. Smart Classroom Roll Caller System with IOT Architecture. In: Proceedings of the 2nd International...
  • F.C. Chang et al.

    Future CLASSROOM WITH THE INTERNET OF THINGS A SERVICE-ORIENTED FRAMEWORK

    J. Inf. Hiding Multimedia Signal Process.

    (2015)
  • Cheng, H.H., Liao, W.W., 2012. Establishing an lifelong learning environment using IOT and learning analytics. In: 14th...
  • D. Clewes

    A student-centred conceptual model of service quality in higher education

    Q. Higher Educ.

    (2003)
  • D.A. Coley et al.

    The effect of low ventilation rates on the cognitive function of a primary school class

    Int. J. Vent.

    (2007)
  • P.A. Dabholkar

    Contingency framework for predicting causality between customer satisfaction and service quality

    Adv. Consum. Res.

    (1995)
  • K. Devinder et al.

    A study of the effect of perceived lecture quality on post-lecture intention

    Work Stud.

    (2003)
  • D. Downs et al.

    Processing demands during auditory learning under degraded listening conditions

    J. Speech Hear. Res.

    (1978)
  • G.W. Evans et al.

    Chronic noise exposure and reading deficits: the mediating effects of language acquisition

    Environ. Behav.

    (2007)
  • G. Fairbanks et al.

    An experimental study of the pitch characteristics of the voice during the expression of emotions

    Speech Monogr.

    (1939)
  • Freund, Y., Mason, L., 1999. The Alternating decision tree algorithm. In: Proceedings of the 16th International...
  • Freund, Y., Schapire, R., 1996. Experiments with a new boosting algorithm. In: Proceedings of the 13th International...
  • N. Gligoric et al.

    Smart classroom: real-time feedback on lecture quality

    Proceedings of IEEE International Conference on Pervasive Computing and Communication Workshops.

    (2012)
  • N. Gligoric et al.

    Internet of things enabled LED lamp controlled by satisfaction of students in a classroom

    IPSI BgD Trans Internet Res.

    (2014)
  • N. Gligoric et al.

    Smart classroom system for detecting level of interest a lecture creates in a classroom

    J. Ambient Intell. Smart Environ.

    (2015)
  • González, G.R., Organero, M.M., Kloos, C.D., 2008. Early infrastructure of an internet of things in spaces for...
  • C. Gronroos

    Strategic Management and Marketing in the Service Sector

    (1982)
  • M. Guolla

    Assessing the teaching quality to student satisfaction relationship: applied customer satisfaction research in the classroom

    J. Market. Theory Pract.

    (1999)
  • Gupta, A., Gupta, P., Chhabra, J., 2015. IoT based power efficient system design using automation for classrooms. In:...
  • M. Hall et al.

    The WEKA Data Mining Software: An Update

    ACM SIGKDD Explorations Newslett.

    (2009)
  • L. Ham et al.

    Gaining competitive advantages in higher education: analyzing the gap between expectations and perceptions of service quality

    Int. J. Value-Based Manage.

    (2003)
  • L. Harvey et al.

    Defining quality

    Assess. Eval. Higher Educ..

    (1993)
  • H.F.A. Hasan et al.

    Service quality and student satisfaction: a case study at private higher education institutions

    Int. Bus. Res.

    (2008)
  • M.B. Holbrook

    The nature of customer value: an axiology of services in the consumption experience

  • E. Howarth et al.

    A Multidimensional Approach to the Relationship between Mood and Weather

    Br. J. Psychol.

    (1984)
  • Huber, R., Batliner, A., Buckow, J., Noth, E., Warnke, V., Niemann, H., 2000. Recognition of Emotion in realistic...
  • Intel, 2014. Intel Galileo Gen 2 Development Board. Available from:...
  • ISO

    ISO 9000:2000: International Standard: Quality Management Systems – Fundamentals and Vocabulary

    (2000)
  • ITU-T, Telecommunication Standardization Sector of International Telecommunication Union 2015. Overview of the Internet...
  • John, G.H., Langley, P., 1995. Estimating Continuous Distributions in Bayesian Classifiers. In: Proceedings of the 11th...
  • E. Johnson

    Let’s hear it for learning: Improving classroom acoustics can maximize student productivity at a minimal cost

    Am. School Univ.

    (2001)
  • Kohavi, R., 1995. The power of decision tables. In: Preceedings of the 8th European Conference on Machine Learning, pp....
  • B.L. Kyzar

    Noise pollutions and schools: how much is too much?

    Counc. Educ. Facilities Plan. J.

    (1977)
  • Lee, C.M., Narayanan, S.S., 2002. Combining acoustic and language information for emotion recognition. In: Proceedings...
  • Cited by (23)

    • Understanding socio-technological challenges of smart classrooms using a systematic review

      2021, Computers and Education
      Citation Excerpt :

      Other intelligent technologies proposed for smart classrooms include image comparison algorithms (Delfin et al., 2019), streaming servers (Khan et al., 2019), task recognition based on deep learning (Khan et al., 2019) and learner personality modeling (Tlili et al., 2019). Incorporating these forms of intelligence aids in developing emotionally aware classrooms using real-time deep learning-based emotion recognition (Kim, 2018; Kim et al., 2018), context-aware intelligence applying probabilistic logic and web ontologies (Han & Youn, 2013; Qin et al., 2007), and feature extraction that provides information in real-time (Uzelac et al., 2017). Focusing on these technologies, some studies consider on intelligent automation for managing classrooms.

    • Scientometric Analysis of Smart Learning

      2024, IEEE Transactions on Engineering Management
    • Ai-based digital technologies in smart universities

      2023, Advancements in Artificial Intelligence, Blockchain Technology, and IoT in Higher Education: Mitigating the Impact of COVID-19
    • Design and validation of IoT based smart classroom

      2023, Multimedia Tools and Applications
    View all citing articles on Scopus

    Dr. Ana Uzelac holds a Ph.D. from University of Belgrade, Serbia. She has graduated Faculty of Mathematics at Belgrade University. She spent eight years working as a programmer where she held a number of positions (junior programmer, senior programmer, team leader). She works as an Assistant Professor at Faculty of Transport and Traffic Engineering at Belgrade University where she is engaged in subjects related to programming languages and databases. Her research interests include Internet of Things and its application in the Smart Classroom domain. Ana has published more than 20 journal articles and conference papers.

    Dr. Nenad Gligorić holds a Ph.D. degree from University of Belgrade, Serbia. His research interest are M2M, Internet of Things and Pervasive computing. As a research consultant in Ericsson, he was engaged for FP7 projects (HOBNET, EXALTED and SmartSantander). Currently he is working at DunavNET, Research & Development department. He works as an Assistant Professor at Alfa University. His work has been published in peer reviewed journal and academic venues including Transactions on Emerging Telecommunications Technologies, Journal of Ambient Intelligence and Smart Environments, PerCom, DCOSS, CyberC, etc.

    Dr. Srđan Krčo holds a PhD degree from the University of Novi Sad. He was with Ericsson R&D for more than 10 years where he held a number of positions (researcher, engineer, manager). Srdjan co-founded DunavNET in 2006 and is particularly responsible for creation of new products and solutions in the domain of IoT. He has participated in a number of FP7 projects. Srdjan is one of the founding members of the international IoT Forum. He has more than 10 patents. In 2007 he received the Innovation engineer of the Year Award in Ireland from the Institute of Engineers of Ireland.

    View full text