Abstract
To better understand the added-value of the academic evaluation process, this paper studies the relationship between scores given by 105 evaluators to 1900 doctoral candidates who received a scholarship and their outcomes 10 years after the competition. I first find that a one point increase in total score is associated with a 1.4 percentage point (2.1% of a s.e.) increase in the probability of completing a Ph.D. in 5 years, with a 1.0 percentage point (2.1% of a s.e.) increase in the probability of completing a Ph.D. in 10 years, and with a 1.4 percentage point increase (3% of a s.e.) in the probability of becoming a tenure-track professor 10 years after the competition. I then use the individual evaluator-candidate scores to provide evidence that male evaluators give higher scores than do female evaluators to students who complete their doctoral program in 5 years. Since there is no difference between scores given by male and female evaluators to candidates who become tenure-track professors, male evaluators seem more focused on shorter time to degree than are female evaluators.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The National Science Foundation (NSF) spends 920 million USD per year to fund graduate students, and the Social Science and Humanities Research Council of Canada (SSHRC) gives out 750 million CAD per year for graduate scholarships.
SSHRC awarded $65,928,665 CAD for the 2004–2005 competition and $65,775,00 CAD for the 2005–2006 competition. Approximately $50 mio USD.
See Table 1 for a list of disciplines per committee.
One has to keep in mind that students who obtained a scholarship in fourth year in 2004 started their Ph.D. in 2000. When I searched for them in 2015, they had 15 years to complete their Ph.D. Students who obtained a scholarship in first year in 2005 started their Ph.D. in 2005. When I searched for them in 2015, they had 10 years to complete their Ph.D.
Since some universities do not regularly update their list of graduate students or search engines may show archived websites, I may have been able to find the person using outdated information and thus over-estimate the number of recipients found.
Some candidates are in fine arts.
Generally, in Canada, students first complete an MA before starting their Ph.D. The two programs are usually separate.
Candidates can receive the scholarship in their first, second, third or fourth year of Ph.D. Students in fifth year can receive a scholarship if they took a parental or sick leave during their first four years, but it is unusual.
Since recipients attended more than 200 universities, I only created dummies for the largest Ph.D. granting universities in Canada: Alberta, Calgary, Carleton, Concordia, Dalhousie, Laval, Manitoba, McGill, McMaster, Montreal, Ottawa, Queen’s, Simon Fraser, UBC, UQAM, Toronto, Western, York. Moreover, there are also one dummy variable for the top American schools (Yale, Princeton, Harvard, MIT, Chicago, Northwestern, Berkeley, Stanford and Columbia) and one for the most prestigious British universities: Oxford and Cambridge. The other universities are simply put in the “other” category.
CGS recipients receive $35,000 per year, while recipients of the SSHRC doctoral scholarship receive $20,000. See Sect. 2 for more detail.
From Fig. 1, 10 years seem to be a reasonable time frame to determine whether a student will ever finish a Ph.D., as the share of students who complete in more than 10 years is very low.
References
Bagues, M., Sylos-Labini, M., & Zinovyeva, N. (2017). Does the gender composition of scientific committees matter? American Economic Review, 107(4), 1207–38.
Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331.
Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45(1), 197–245.
CAUT. (2014). CAUT almanach of post-secondary education in Canada.
Chandler, V. (2018). Short and long-term impacts of an increase in graduate funding. Economics of Education Review, 62, 104–112.
Dittrich, M., & Leipold, K. (2014). Gender differences in time preferences. Economics Letters, 122(3), 413–415.
Edge, J. (2015). Inside and outside the academy: Valuing and preparing Ph.Ds. for careers. In Conference board of Canada.
Fang, F. C., Bowen, A., & Casadevall, A. (2016). NIH peer review percentile scores are poorly predictive of grant productivity. Elife, 5, e13323.
Jacob, B., & Lefgren, L. (2011). The impact of NIH research grant funding on scientific productivity. Journal of Public Economics, 45(9–10), 1168–1177.
King, D. (2008). Doctoral Graduates in Canada: Findings from the survey of earned doctorates, 2004/2005. Culture, Tourism and the Centre for Education Statistics Research papers. Statistics Canada Catalogue no. 81-595-M No. 065.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.
Li, D., & Agha, L. (2015). Big names or big ideas: Do peer review panels select the best science proposals. Science, 348(6233), 434–438.
Matsa, D., & Miller, A. (2013). A female style in corporate leadership? Evidence from Quotas, American Economic Journal: Applied Economics, 5(3), 136–169.
Moneta-Koehler, L., Brown, A. M., Petrie, K. A., Evans, B. J., & Chalkley, R. (2017). The limitations of the GRE in predicting success in biomedical graduate school. PLoS ONE, 12(1), e0166742.
SSHRC. (2016). Apply for funding: Doctoral awards. http://www.sshrc-crsh.gc.ca/funding-financement/programs-programmes/fellowships/doctoral-doctorat-eng.aspx. Accessed 1 Feb 2019.
Sternberg, R. J., & Williams, W. M. (1997). Does the Graduate Record Examination predict meaningful success in the graduate training of psychology? A case study. American Psychologist, 52(6), 630.
Author information
Authors and Affiliations
Corresponding author
Additional information
The author wishes to thank SSHRC for accepting to share information and is particularly indebted to Margaret Blakeney and Andreas Reichert for their help with the data. Financial assistance from the Ontario Graduate Scholarship and the Fonds de Recherche du Quebec - Societe et Culture is gratefully acknowledged. Comments from Steve Lehrer and Casey Warman and those from two anonymous referees were very useful to improve this paper. I would finally like to thank Debbie Liu for excellent research assistance. The data used in this study can be accessed through an Access to Information and Privacy request at the Social Science and Humanities Research Council. The usual caveat applies.
Rights and permissions
About this article
Cite this article
Chandler, V. Identifying emerging scholars: seeing through the crystal ball of scholarship selection committees. Scientometrics 120, 39–56 (2019). https://doi.org/10.1007/s11192-019-03120-0
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-019-03120-0