Assessing and predicting the quality of research master’s theses: an application of scientometrics

Abstract

The educational quality of research master’s degree can be in part reflected by the examiner score of the thesis. This study focuses on finding positive predictors of this score with the aim of developing assessment and prediction methods for the educational quality of postgraduates. This study is based on regression analysis of the characteristics extracted from publications and references involving 1038 research master’s theses written at three universities in China. The analysis indicates that for a thesis, the number and the integrated impact factor of its references in Science Citation Index Expanded (SCIE) journals are significantly positive predictors of having publications in such journals. Additionally, the number and the integrated impact factor of a thesis’ representative publications, defined as the publications authored by the master’s student as a first author or second author with tutors in lead position, in SCIE journals, are significantly positive predictors of its examiner score. Based on these predictors, a range of indicators is provided to assess thesis quality, to measure the contributions of disciplines to postgraduate education, to predict postgraduates’ research outcomes, and to provide benchmarks regarding the quality and quantity of their reading work.

Appendices

Appendix A: Minimum sample size

Assume the size of group from which a sample is taken to be infinite. Let the confidence level be $1-\alpha$. Denote the corresponding z-score of $\alpha$ by $z_{ {\alpha }/{2}}$, the expected proportion by p, the population standard deviation by $\sigma$, and the margin of error by E. If the expected proportion and population standard deviation are not known, the sample proportion and sample standard deviation can be used (Eng 2003).

The formula of the minimum sample size required for estimating the population proportion is

$$\begin{aligned} n=\frac{z_{\frac{\alpha }{2}}^2 p(1-p)}{E^2}. \end{aligned}$$

(4)

Let $\alpha =5\%$, $p=$ sample population proportion, and $E=0.15$. For regression analysis on having representative publications, $n=42,33,29, 38$ for Biological, Engineering, Information, and Physical sciences respectively.

The corresponding formula for estimating the population mean is

$$\begin{aligned} n=\frac{z_{\frac{\alpha }{2}}^2\sigma ^2}{E^2}. \end{aligned}$$

(5)

Let $\alpha =5\%$, $\sigma =$ sample standard deviation, and $E= 1.5\%$. For the regression analysis on examiner score, $n=52, 53, 51, 43$ for Biological, Engineering, Information, and Physical sciences respectively.

Appendix B: More results of regression

Figure 10 shows linear regression results between the examiner score and the indexes derived from the references of theses. The number and the integrated impact factor of SCIE references are significantly positive predictors of the examiner score in information sciences, and the number is significantly positive in engineering. There are no significant relationship in the other cases.

Fig. 10

The relationship between the examiner score and the indexes derived from references. The panels show the mean examiner score of theses with the same index value (red squares), the predicted score (solid dot lines), and confidence intervals (dashed lines). The p value is that of $\chi ^2$-test. (Color figure online)

Figure 11 shows that for each disciplinary group, the number of representative publications of a thesis follows a Gamma distribution. Therefore, Gamma regression can be utilized to analyse the relationship between the number of representative publications and the indexes derived from references. Gamma regression is a generalized linear model that assumes that the response variable follows a Gamma distribution. The negative reciprocal of the expected value of the Gamma distribution is fitted by a linear combination of predictors (Nelder and Wedderburn 1972).

Figure 12 shows Gamma regression results. Except for biological sciences, the number of SCIE references is a significantly positive predictor of the number of representative publications. And there is no significant relationship between the number of non-SCIE references and the number of representative publications. These results may be statistically meaningless due to the small sample size of theses having a given number of representative publications.

Fig. 11

The distribution of the number of representative publications. The panels show the empirical distributions (red circles) and Gamma distributions (blue squares). The KS test cannot reject the hypothesis that the number of representative publications follows a Gamma distribution, p value $>5\%$. (Color figure online)

Fig. 12

The relationship between the number of representative publications and that of SCIE/non-SCIE references. The panels show the average number of representative publications of theses with the same index value (red squares), the predicted value (solid dot lines), and confidence intervals (dashed lines). The p value is that of $\chi ^2$-test. (Color figure online)