Abstract
This study developed the Multi-Dimensional Research Agendas Inventory to measure the key factors associated with the process of research agenda setting. Research agendas reflect the preferences, strategies, influences and goals that guide researchers’ decisions to investigate specific topics. The results of exploratory and confirmatory factor analyses indicated that the instrument has eight distinct dimensions: Scientific Ambition, Convergence, Divergence, Discovery, Conservative, Tolerance for Low Funding, Mentor Influence and Collaboration. The model underlying the instrument exhibited a very good fit [X 2/df = 1.710; CFI = 0.961; PCFI = 0.791; RMSEA = 0.035; P(rmsea ≤ 0.05) < 0.001], and the instrument itself was found to have excellent measuring properties (in terms of validity, reliability and sensitivity). Potential interpretations of the instrument and its implications for research and practice are also discussed in this article.
Similar content being viewed by others
Notes
Decision-making processes related to research focus also tend to be collective rather than individual in some fields of knowledge such as biomedicine (Verbree et al. 2015), and are substantially centralised in some fields of knowledge such as physics, particularly in the context of large experimental laboratories (Boisot 2011).
David Kenny (whose work on linear modeling is seminal) has maintained very comprehensive and up-to-date guidelines for SEM on his personal webpage, http://davidakenny.net/cm/causalm.htm, which may be useful to readers interested in learning how to operate SEM software.
More in-depth information on these lower-order factors is provided in later sections of this article.
However, higher education research is to some extent multi-disciplinary, with contributions from most of the social science fields (e.g. economics, political science, sociology and psychology). In future research, using a new set of data, the authors will carry out further validation exercises with different cohorts (in this case, academics from other fields) to maximise the robustness of the instrument.
Although the sample size at this stage allowed EFA to be conducted on all of the items simultaneously, we opted to perform EFA with separate question blocks, as in the preliminary test, to ensure consistency.
This indicator is described in detail in a later section of the article.
References
Allison, P. D., Long, J. S., & Krauze, T. K. (1982). Cumulative advantage and inequality in science. American Sociological Review, 47, 615–625.
Anderson, D., Burnham, K., & White, G. (1998). Comparison of Akaike information criterion and consistent Akaike information criterion for model selection and statistical inference from capture–recapture studies. Journal of Applied Statistics, 25(2), 263–282.
Andrews, R. N., & Johnson, E. (2016). Energy use, behavioral change, and business organizations: Reviewing recent findings and proposing a future research agenda. Energy Research and Social Science, 11, 195–208.
Arbuckle, J. (2007). AMOS 16.0 user’s guide. Chicago, IL: SPSS.
Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42(5), 815–824.
Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1–26.
Becher, T. (1994). The significance of disciplinary differences. Studies in Higher Education, 19(2), 151–161.
Becher, T., & Trowler, P. R. (2001). Academic tribes and territories: Intellectual inquiry and the culture of disciplines. Buckingham: Open University Press/SRHE.
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238.
Bentler, P. M. (2007). On tests and indices for evaluating structural models. Personality and Individual Differences, 42(5), 825–829.
Bentler, P. M., & Weeks, D. G. (1980). Linear structural equations with latent variables. Psychometrika, 45(3), 289–308.
Boisot, M. (2011). Collisions and collaboration: The organization of learning in the ATLAS experiment at the LHC. Oxford: Oxford University Press.
Bollen, K. A. (2014). Structural equations with latent variables. London: Wiley.
Bourdieu, P. (1999). The specificity of the scientific field. In M. Biagoli (Ed.), The science studies reader (pp. 31–50). New York, NY: Routledge.
Brown, T. A. (2015). Confirmatory factor analysis for applied research. New York, NY: Guilford Publications.
Cantwell, B. (2011). Transnational mobility and international academic employment: Gatekeeping in an academic competition arena. Minerva, 49(4), 425–445.
Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago, IL: University of Chicago Press.
Cummings, J. N., & Kiesler, S. (2005). Collaborative research across disciplinary and organizational boundaries. Social Studies of Science, 35(5), 703–722.
DiStefano, C., Zhu, M., & Mindrila, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research and Evaluation, 14(20), 1–11.
Ebadi, A., & Schiffauerova, A. (2015). How to receive more funding for your research? Get connected to the right people! PLoS One, 10(7), e0133061.
Ebrahimy, S., & Osareh, F. (2014). Design, validation, and reliability determination a citing conformity instrument at three levels: Normative, informational, and identification. Scientometrics, 99(2), 581–597.
Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430–457.
Ertmer, P. A., & Glazewski, K. D. (2014). Developing a research agenda: Contributing new knowledge via intent and focus. Journal of Computing in Higher Education, 26(1), 54–68.
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50.
Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2007). Multivariate data analysis. Ahmedabad: Bookman.
Harris, M. (2001). Centering in on professional choices. College Composition and Communication, 52(3), 429–440.
Hillson, D., & Murray-Webster, R. (2007). Understanding and managing risk attitude. Aldershot, Hants, UK: Gower Publishing Ltd.
Hoffman, D. M., Blasi, B., Ćulum, B., Dragšić, Ž., Ewen, A., Horta, H., et al. (2014). The methodological illumination of a blind spot: Information and communication technology and international research team dynamics in a higher education research program. Higher Education, 67(4), 473–495.
Hooper, D., Coughlan, J., & Mullen, M. R. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6(1), 53–60.
Horlings, E., & Gurney, T. (2013). Search strategies along the academic lifecycle. Scientometrics, 94(3), 1137–1160.
Horta, H., & Jung, J. (2014). Higher education research in Asia: An archipelago, two continents or merely atomization? Higher Education, 68(1), 117–134.
Horta, H., & Santos, J. M. (2016). The impact of publishing during the Ph.D. on career research publications, visibility, and collaborations. Research in Higher Education, 57(1), 28–50.
Horta, H., & Yonezawa, A. (2013). Going places: Exploring the impact of intra-sectoral mobility on research productivity and communication behaviors in Japanese academia. Asia Pacific Education Review, 14(4), 537–547.
Johnson, B., & Stevens, J. J. (2001). Exploratory and confirmatory factor analysis of the School Level Environment Questionnaire (SLEQ). Learning Environments Research, 4(3), 325–344.
Jöreskog, K. G., & Sörbom, D. (1989). LISREL 7: A guide to the program and applications. Chicago, IL: SPSS.
Jung, J. (2014). Research productivity by career stage among Korean academics. Tertiary Education and Management, 20(2), 85–105.
Jung, J., & Horta, H. (2013). Higher education research in Asia: A publication and co-publication analysis. Higher Education Quarterly, 67(4), 398–419.
Jung, J., & Horta, H. (2015). The contribution of East Asian countries to internationally published Asian higher education research: The role of system development and internationalization. Higher Education Policy, 28(4), 419–439.
Kassam, A., Papish, A., Modgill, G., & Patten, S. (2012). The development and psychometric properties of a new scale to measure mental illness related stigma by health care providers: The Opening Minds Scale for Health Care Providers (OMS-HC). BMC Psychiatry, 12(1), 1.
Katz, J. S., & Martin, B. R. (1997). What is research collaboration? Research Policy, 26(1), 1–18.
Klavans, R., Boyack, K. W., Sorensen, A. A., & Chen, C. (2013). Towards the development of an indicator of conformity. In 14th international society of scientometrics and informetrics conference. ISSI 2013.
Kline, R. B. (2011). Principles and practice of structural equation modeling. New York, NY: Guilford Press.
Kolenikov, S., & Bollen, K. A. (2012). Testing negative error variances: Is a Heywood case a symptom of misspecification? Sociological Methods and Research, 41(1), 124–167.
Kosmützky, A., & Krücken, G. (2014). Growth or steady state? A bibliometric focus on international comparative higher education research. Higher Education, 67(4), 457–472.
Latour, B., & Woolgar, S. (2013). Laboratory life: The construction of scientific facts. Princeton, NJ: Princeton University Press.
Leahey, E. (2007). Not by productivity alone: How visibility and specialization contribute to academic earnings. American Sociological Review, 72(4), 533–561.
Levitt, D. G. (2010). Careers of an elite cohort of US basic life science postdoctoral fellows and the influence of their mentor’s citation record. BMC Medical Education, 10(1), 80.
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84.
Mamun, S. A. K., & Rahman, M. M. (2015). Is there any feedback effect between academic research publication and research collaboration? Evidence from an Australian university. Scientometrics, 105(3), 2179–2196.
Maroco, J. (2003). Análise estatística: com utilização do SPSS. Lisbon, Portugal: Edições Sílabo.
Maroco, J. (2007). Consistency and efficiency of ordinary least squares, maximum likelihood, and three type II linear regression models. Methodology, 3(2), 81–88.
Marôco, J. (2010). Análise de equações estruturais: Fundamentos teóricos, software & aplicações. Pêro Pinheiro: Report Number.
Martimianakis, M. A., & Muzzin, L. (2015). Discourses of interdisciplinarity and the shifting topography of academic work: Generational perspectives on facilitating and resisting neoliberalism. Studies in Higher Education, 40(8), 1454–1470.
Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56–63.
Morley, L. (2003). Quality and power in higher education. Berkshire: Society for Research into Higher Education, Open University.
Pinheiro, D., Melkers, J., & Youtie, J. (2014). Learning to play the game: Student publishing as an indicator of future scholarly success. Technological Forecasting and Social Change, 81, 56–66.
Pump, B. (2011). Beyond metaphors: New research on agendas in the policy process. Policy Studies Journal, 39(s1), 1–12.
Rammstedt, B., & John, O. P. (2007). Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41(1), 203–212.
Rzhetsky, A., Foster, J. G., Foster, I. T., & Evans, J. A. (2015). Choosing experiments to accelerate collective discovery. Proceedings of the National Academy of Sciences, 112(47), 14569–14574.
Schut, M., van Paassen, A., Leeuwis, C., & Klerkx, L. (2014). Towards dynamic research configurations: A framework for reflection on the contribution of research to policy and innovation processes. Science and Public Policy, 41(2), 207–218.
Simon, H. A. (1992). Economics, bounded rationality, and the cognitive revolution. Aldershot: Elgar.
Slovic, P., Fischhoff, B., & Lichtenstein, S. (1982). Why study risk perception? Risk Analysis, 2(2), 83–93.
Spector, P. E., Van Katwyk, P. T., Brannick, M. T., & Chen, P. Y. (1997). When two factors don’t reflect two constructs: How item characteristics can produce artifactual factors. Journal of Management, 23(5), 659–677.
Steiger, J. H., Shapiro, A., & Browne, M. W. (1985). On the multivariate asymptotic distribution of sequential Chi square statistics. Psychometrika, 50(3), 253–263.
Uddin, S., Hossain, L., & Rasmussen, K. (2013). Network effects on scientific collaborations. PLoS One, 8(2), e57546.
Verbree, M., Horlings, E., Groenewegen, P., Van der Weijden, I., & Van den Besselaar, P. (2015). Organizational factors influencing scholarly performance: A multivariate study of biomedical research groups. Scientometrics, 102(1), 25–49.
Whitley, R. (2000). The intellectual and social organization of the sciences. Oxford: Oxford University Press.
Acknowledgments
This study was supported by doctoral Grant PD/BD/113999/2015 from the Fundação para a Ciência e Tecnologia (FCT), co-funded by the European Social Fund (ESF) and the Portuguese Ministry of Science and Education. The study was also supported by the Seed Funding Programme for Basic Research of The University of Hong Kong.
Author information
Authors and Affiliations
Corresponding author
Appendix: Multi-dimensional research agendas inventory (MDRAI)
Appendix: Multi-dimensional research agendas inventory (MDRAI)
You will be asked a series of questions regarding your motivations and goals as an academic. To respond to this questionnaire, read each statement carefully and decide how much do you agree with each of them. For each statement, check one of the 7 boxes next to the corresponding item. If you don’t know or a particular sentence does not apply to you, check the N/A box.
There are no right or wrong answers. Please read each statement and check the box which best applies to you.
How much do you agree with the following statements?
Completely disagree | Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree | Completely agree | N/A | ||
---|---|---|---|---|---|---|---|---|---|
A1 | I aim to one day be one of the most respected experts in my field. | ||||||||
A2 | Being a highly regarded expert is one of my career goals. | ||||||||
A3 | I aim to be recognized by my peers. | ||||||||
A4 | Standing out from the rest of my peers is one of my goals. | ||||||||
A5 | I feel the need to constantly publish new and interesting papers. | ||||||||
A6 | I am constantly striving to publish new papers. | ||||||||
C1 | My expertise is focused on a single scientific area. | ||||||||
C2 | I believe that specialization in one area is preferable to diversification. | ||||||||
C3 | Shifting towards another field of science is not a part of my plans. | ||||||||
C4 | Studying subjects outside of my main field of work is pointless. | ||||||||
C5 | I have invested far too much in my current field to consider branching out into another. | ||||||||
DI1 | I find “cutting-edge” scientific areas more appealing than well-established ones. | ||||||||
DI2 | I would rather conduct revolutionary research with little chance of success than replicate research with a high chance of success. | ||||||||
DI3 | I prefer “cutting-edge” research to “safe” research, even when the odds of success are much lower. | ||||||||
CN1 | I prefer “safe” or “stable” fields of study. | ||||||||
CN2 | I prefer fields of study that are considered “safe” or “stable.” | ||||||||
TL1 | Limited funding does not constrain my choice of field. | ||||||||
TL2 | Highly limited funding does not constrain my choice of field. | ||||||||
TL3 | The availability of research funding for a certain topic does not influence me doing research on that topic. | ||||||||
CO1 | I enjoy collaborating with other authors in my scientific articles. | ||||||||
CO2 | My scientific articles are enhanced by collaboration with other authors. | ||||||||
CO3 | I see myself as a team player when it comes to research collaboration. | ||||||||
CO4 | I often seek peers with whom I can collaborate on scientific articles. | ||||||||
CO5 | My peers often seek my collaboration in their scientific articles. | ||||||||
CO6 | I am often invited to do collaborative work with my peers. | ||||||||
M1 | My Ph.D. mentor’s opinion carries much weight in my research choices. | ||||||||
M2 | A part of my work is largely due to my Ph.D. mentor. | ||||||||
M3 | My research choices are highly influenced by my Ph.D. mentor’s opinion. | ||||||||
M4 | My Ph.D. mentor is responsible for a large part of my work. | ||||||||
M5 | My Ph.D. mentor still often works alongside me. | ||||||||
M6 | My Ph.D. mentor largely determines my venues of research. | ||||||||
D1 | I look forward to diversifying into other areas. | ||||||||
D2 | I would be interested in pursuing research in other fields. | ||||||||
D3 | I enjoy multi-disciplinary research more than single-discipline research. | ||||||||
D4 | For me, multi-disciplinary research is more interesting than single-discipline research. |
Rights and permissions
About this article
Cite this article
Horta, H., Santos, J.M. An instrument to measure individuals’ research agenda setting: the multi-dimensional research agendas inventory. Scientometrics 108, 1243–1265 (2016). https://doi.org/10.1007/s11192-016-2012-4
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-016-2012-4