Improving University Faculty Evaluations via multi-view Knowledge Graph
Introduction
The evaluation of academic development has been a long-term focus of both educational and industrial circles. How to comprehensively assess a scholar via a systemic evaluation has become an essential question to advance intelligent educational services, such as intelligent tutor systems (ITS), scientific resource retrieval, and faculty career evaluations [1], [2], [3]. To address this question, several previous efforts have been carried out that establish qualitative and quantitative measurements to evaluate a scholar or professor not only in terms of publications but also in aspects including research projects [4], patents [5], teaching quality [6], [7], etc. Given the full integration of the Internet into the academic domain, information about scholarly activities is now preserved and disseminated on the Internet.
However, a comprehensive evaluation system must be established in which all the required information is based on multi-source and multi-modal data without a uniform representation. Especially in evaluating university faculties, the assessment criteria cover multiple fields, including research ability and teaching skills, which require evaluators to collect heterogeneous data from the Internet and internal systems [8], [9], [10].
A comprehensive indicator considering as many aspects as possible for predicting faculty development is also required. Predicting the professional potential of university faculty based on comprehensive aspects through modeling or profiling using both empirical and projected indicators provides a novel approach for assessing faculty [11], [12].
For these reasons, in this paper, we propose an innovative university faculty evaluation system based on a knowledge graph that integrates heterogeneous data for profiling university faculties. First, the relevant data is collected from the Internet and internal systems, including faculty information and related scientific research papers, patents, funds, monographs, awards, professional activities and teaching performance. The graph structure is defined according to the collected data and preprocessing. Second, we construct entity representations through knowledge graph embedding methods to retain semantic information. By applying the embedded representations on the constructed knowledge graph, the effectiveness of the proposed graph structure among faculties and other entities is vividly displayed. Third, this paper also proposes an ADF for the prediction of faculty academic development, which integrates faculty member’s academic development status and student evaluation data from previous years. A case experiment is conducted to examine the effectiveness of this factor in two aspects: the relevance between the indicator and features of the knowledge graph as well as student evaluations, and the predictive accuracy and correlation of the ADF and other widely used academic evaluation indicators.
The rest of this paper is organized as follows. We present a brief literature review in Section 2. The definition, hypothesis and initial construction of the knowledge graph of university faculties are introduced in Section 3. Section 4 proposes embedding representations of entities stored in the previously constructed graph and conducts experiments on the ADF to examine the effectiveness and practicability of our proposed system. The whole architecture of the system (UFES-KG) is presented in Section 5, and concluding remarks are given in Section 6.
Section snippets
Related work
Currently, the methods used to evaluate college faculty can be roughly divided into five categories: peer review, bibliometric methods, student evaluations of teaching, altmetrics, and scholarly knowledge graphs.
Faculty knowledge graph
This paper proposes the use of knowledge graph to evaluate university faculties. In this section we introduce the construction of the faculty knowledge graph.
Comprehensive evaluation of faculty
In this section, we introduce the comprehensive evaluation method of faculty based on the constructed multi-view knowledge graph. The method contains three aspects: graph embedding -based feature representation, entities’ representation, and academic development prediction.
System architecture
Based on the related technologies and experimental results introduced above, we construct a university faculty evaluation system based on a multi-view knowledge graph called UFES-KG. The overall process of UFES-KG is summarized in Fig. 5.
The system is divided into seven layers. At the bottom level, a large amount of heterogeneous data related to college faculty is collected from the Internet through web crawlers, including the scholars’ homepages, the Aminer academic system, the Wanfang
Conclusion and discussion
To alleviate the deficiencies of current evaluation methodologies of university faculty, which usually focus on only one aspect of their activities, this paper proposes a method of constructing a knowledge graph by integrating scholars’ related heterogeneous data to evaluate their performance. The main contents of this paper are summarized as follows. First, the paper collects relevant data of college faculty members on the Internet. Then, the knowledge graph is constructed by effectively
CRediT authorship contribution statement
Qika Lin: Conceptualization, Methodology, Software, Writing - original draft. Yifan Zhu: Data curation, Resources, Validation, Writing - review & editing. Hao Lu: Formal analysis, Investigation. Kaize Shi: Supervision, Writing - review & editing. Zhendong Niu: Supervision, Writing - review & editing, Project administration, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work is supported by the National Key R&D Program of China under Grant 2019YFB1406302, the National Natural Science Foundation of China under Grant 61370137, the Ministry of Education-China Mobile Research Foundation Project under Grant 2016/2-7, the Postgraduate Education Research Project of Beijing Institute of Technology, China under Grant 2017JYYJG-004 and supported in part by China’s National Strategic Basic Research Program (973 Program) under Grant 2012CB720700.
Qika Lin received the B.E. degree and the M.E. degree from Beijing Institute of Technology, Beijing, China in 2016 and 2019 respectively. His research interests include deep learning and e-learning.
References (66)
Ratemyprofessors is hogwash (but I care)
Comput. Hum. Behav.
(2016)- et al.
Evaluating students’ evaluations of professors
Econ. Educ. Rev.
(2014) Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime
J. Inform.
(2014)- et al.
From sociability to creditability for academics
Comput. Hum. Behav.
(2017) - et al.
A learner oriented learning recommendation approach based on mixed concept mapping and immune algorithm
Knowl. Based Syst.
(2016) AMiner: Mining deep knowledge from big scholar data
- et al.
Knowledge-based recommendation: a review of ontology-based recommender systems for e-learning
Artif. Intell. Rev.
(2018) - et al.
Detecting interest-factor influenced abnormal evaluation of teaching via multimodal embedding and priori knowledge based neural network
- et al.
Research fund evaluation based on academic publication output analysis: the case of chinese research fund evaluation
Scientometrics
(2019) - et al.
Research and innovation in higher education: empirical evidence from research and patenting in Brazil
Scientometrics
(2018)
Teaching evaluation: Past, present, and future
New Dir. Teach. Learn.
A Semantic Ontology-Based Records Management Approach for Academic Users’ Decision Support
Faculty performance evaluation system: An ontological approach
Ontology-based approach for academic evaluation system
Using machine learning techniques for rising star prediction in co-author network
Scientometrics
Identifying the academic rising stars via pairwise citation increment ranking
Peer review: Troubled from the start
Nature
Peer-review ’heroes’ do lion’s share of the work
Nature
The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise
PLoS One
Pubcreds: Fixing the peer review process by “privatizing” the reviewer commons
Bull. Ecol. Soc. Am.
No crisis in supply of peer reviewers
Nature
Let’s make peer review scientific
Nature
Let’s make peer review scientific
Nature
The history and meaning of the journal impact factor
JAMA
An index to quantify an individual’s scientific research output
Proc. Natl. Acad. Sci.
Theory and practice of the g-index
Scientometrics
Rising star forecasting based on social network analysis
IEEE Access
Academic rising star prediction via scholar’s evaluation model and machine learning techniques
Scientometrics
Student evaluations of teaching (mostly) do not measure teaching effectiveness
ScienceOpen Res.
Does professor quality matter? Evidence from random assignment of students to professors
J. Polit. Econ.
Gender biases in student evaluations of teachers
Sci. Po Publ.
Lexical based automated teaching evaluation via students’ short reviews
Comput. Appl. Eng. Educ.
Scientometrics 2.0: New metrics of scholarly impact on the social Web
First Monday
Cited by (28)
Quantitative stock portfolio optimization by multi-task learning risk and return
2024, Information FusionMulti-source aggregated classification for stock price movement prediction
2023, Information FusionFusing topology contexts and logical rules in language models for knowledge graph completion
2023, Information FusionMultiview feature augmented neural network for knowledge graph embedding
2022, Knowledge-Based SystemsCitation Excerpt :Han Xiao et al. [42] offer a multiview clustering framework, which can generate semantic representations of the entity or relational features. Qika Lin et al. [43] proposed community answer question generation based on a knowledge graph, which can automatically generate natural language answers. Bo Li et al. [20] proposed a new multiview inference framework that uses uncertainty knowledge to improve relation extraction.
Special Issue on Machine Learning and Knowledge Graphs
2022, Future Generation Computer SystemsRule-enhanced iterative complementation for knowledge graph reasoning
2021, Information Sciences
Qika Lin received the B.E. degree and the M.E. degree from Beijing Institute of Technology, Beijing, China in 2016 and 2019 respectively. His research interests include deep learning and e-learning.
Yifan Zhu received the B.E. degree in computer science from Beijing Information Science & Technology University, Beijing, China in 2016. He is currently working toward the Ph.D. degree at School of Computer Science & Technology, Beijing Institute of Technology, Beijing, China. His research interests include opinion mining, user profiling and social computing.
Hao Lu received the Ph.D. degree from the School of Computer Science & Technology, Beijing Institute of Technology, Beijing, China, in 2020. He is currently an Associate Professor with the State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, China. His research interests include intelligent systems, knowledge automation, and social computing.
Kaize Shi is currently pursuing the Ph.D. degree in School of Computer Science & Technology, Beijing Institute of Technology, Beijing, China. His current research interests include opinion mining in social network, knowledge engineering, intelligent transportation, and artificial intelligence technology.
Zhendong Niu received the Ph.D. degree in computer science from the Beijing Institute of Technology, Beijing, China, in 1995. He was a Post-Doctoral Researcher with the University of Pittsburgh, Pittsburgh, PA, USA, from 1996 to 1998, a Research/Adjunct Faculty Member with Carnegie Mellon University, Pittsburgh, from 1999 to 2004, and a Joint Research Professor with the School of Computing and Information, University of Pittsburgh,from 2006. He is a Professor with the School of Computer Science and Technology and the director of Library, Beijing Institute of Technology. His current research interests include informational retrieval, software architecture, digital libraries, and Web-based learning techniques. Prof. Niu is a recipient of the IBM Faculty Innovation Award in 2005 and the New Century Excellent Talents in University of Ministry of Education of China in 2006.
- 1
Qika Lin and Yifan Zhu contribute equally to this paper.