Skip to main content
Log in

An instrument to measure individuals’ research agenda setting: the multi-dimensional research agendas inventory

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This study developed the Multi-Dimensional Research Agendas Inventory to measure the key factors associated with the process of research agenda setting. Research agendas reflect the preferences, strategies, influences and goals that guide researchers’ decisions to investigate specific topics. The results of exploratory and confirmatory factor analyses indicated that the instrument has eight distinct dimensions: Scientific Ambition, Convergence, Divergence, Discovery, Conservative, Tolerance for Low Funding, Mentor Influence and Collaboration. The model underlying the instrument exhibited a very good fit [X 2/df = 1.710; CFI = 0.961; PCFI = 0.791; RMSEA = 0.035; P(rmsea ≤ 0.05) < 0.001], and the instrument itself was found to have excellent measuring properties (in terms of validity, reliability and sensitivity). Potential interpretations of the instrument and its implications for research and practice are also discussed in this article.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Decision-making processes related to research focus also tend to be collective rather than individual in some fields of knowledge such as biomedicine (Verbree et al. 2015), and are substantially centralised in some fields of knowledge such as physics, particularly in the context of large experimental laboratories (Boisot 2011).

  2. David Kenny (whose work on linear modeling is seminal) has maintained very comprehensive and up-to-date guidelines for SEM on his personal webpage, http://davidakenny.net/cm/causalm.htm, which may be useful to readers interested in learning how to operate SEM software.

  3. More in-depth information on these lower-order factors is provided in later sections of this article.

  4. However, higher education research is to some extent multi-disciplinary, with contributions from most of the social science fields (e.g. economics, political science, sociology and psychology). In future research, using a new set of data, the authors will carry out further validation exercises with different cohorts (in this case, academics from other fields) to maximise the robustness of the instrument.

  5. Although the sample size at this stage allowed EFA to be conducted on all of the items simultaneously, we opted to perform EFA with separate question blocks, as in the preliminary test, to ensure consistency.

  6. This indicator is described in detail in a later section of the article.

References

  • Allison, P. D., Long, J. S., & Krauze, T. K. (1982). Cumulative advantage and inequality in science. American Sociological Review, 47, 615–625.

    Article  Google Scholar 

  • Anderson, D., Burnham, K., & White, G. (1998). Comparison of Akaike information criterion and consistent Akaike information criterion for model selection and statistical inference from capture–recapture studies. Journal of Applied Statistics, 25(2), 263–282.

    Article  MATH  Google Scholar 

  • Andrews, R. N., & Johnson, E. (2016). Energy use, behavioral change, and business organizations: Reviewing recent findings and proposing a future research agenda. Energy Research and Social Science, 11, 195–208.

    Article  Google Scholar 

  • Arbuckle, J. (2007). AMOS 16.0 user’s guide. Chicago, IL: SPSS.

    Google Scholar 

  • Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42(5), 815–824.

    Article  Google Scholar 

  • Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1–26.

    Article  Google Scholar 

  • Becher, T. (1994). The significance of disciplinary differences. Studies in Higher Education, 19(2), 151–161.

    Article  MathSciNet  Google Scholar 

  • Becher, T., & Trowler, P. R. (2001). Academic tribes and territories: Intellectual inquiry and the culture of disciplines. Buckingham: Open University Press/SRHE.

    Google Scholar 

  • Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238.

    Article  Google Scholar 

  • Bentler, P. M. (2007). On tests and indices for evaluating structural models. Personality and Individual Differences, 42(5), 825–829.

    Article  Google Scholar 

  • Bentler, P. M., & Weeks, D. G. (1980). Linear structural equations with latent variables. Psychometrika, 45(3), 289–308.

    Article  MathSciNet  MATH  Google Scholar 

  • Boisot, M. (2011). Collisions and collaboration: The organization of learning in the ATLAS experiment at the LHC. Oxford: Oxford University Press.

    Book  Google Scholar 

  • Bollen, K. A. (2014). Structural equations with latent variables. London: Wiley.

    MATH  Google Scholar 

  • Bourdieu, P. (1999). The specificity of the scientific field. In M. Biagoli (Ed.), The science studies reader (pp. 31–50). New York, NY: Routledge.

    Google Scholar 

  • Brown, T. A. (2015). Confirmatory factor analysis for applied research. New York, NY: Guilford Publications.

    Google Scholar 

  • Cantwell, B. (2011). Transnational mobility and international academic employment: Gatekeeping in an academic competition arena. Minerva, 49(4), 425–445.

    Article  Google Scholar 

  • Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago, IL: University of Chicago Press.

    Google Scholar 

  • Cummings, J. N., & Kiesler, S. (2005). Collaborative research across disciplinary and organizational boundaries. Social Studies of Science, 35(5), 703–722.

    Article  Google Scholar 

  • DiStefano, C., Zhu, M., & Mindrila, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research and Evaluation, 14(20), 1–11.

    Google Scholar 

  • Ebadi, A., & Schiffauerova, A. (2015). How to receive more funding for your research? Get connected to the right people! PLoS One, 10(7), e0133061.

    Article  Google Scholar 

  • Ebrahimy, S., & Osareh, F. (2014). Design, validation, and reliability determination a citing conformity instrument at three levels: Normative, informational, and identification. Scientometrics, 99(2), 581–597.

    Article  Google Scholar 

  • Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430–457.

    Article  MathSciNet  Google Scholar 

  • Ertmer, P. A., & Glazewski, K. D. (2014). Developing a research agenda: Contributing new knowledge via intent and focus. Journal of Computing in Higher Education, 26(1), 54–68.

    Article  Google Scholar 

  • Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50.

    Article  Google Scholar 

  • Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2007). Multivariate data analysis. Ahmedabad: Bookman.

    Google Scholar 

  • Harris, M. (2001). Centering in on professional choices. College Composition and Communication, 52(3), 429–440.

    Article  Google Scholar 

  • Hillson, D., & Murray-Webster, R. (2007). Understanding and managing risk attitude. Aldershot, Hants, UK: Gower Publishing Ltd.

    Google Scholar 

  • Hoffman, D. M., Blasi, B., Ćulum, B., Dragšić, Ž., Ewen, A., Horta, H., et al. (2014). The methodological illumination of a blind spot: Information and communication technology and international research team dynamics in a higher education research program. Higher Education, 67(4), 473–495.

    Article  Google Scholar 

  • Hooper, D., Coughlan, J., & Mullen, M. R. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6(1), 53–60.

    Google Scholar 

  • Horlings, E., & Gurney, T. (2013). Search strategies along the academic lifecycle. Scientometrics, 94(3), 1137–1160.

    Article  Google Scholar 

  • Horta, H., & Jung, J. (2014). Higher education research in Asia: An archipelago, two continents or merely atomization? Higher Education, 68(1), 117–134.

    Article  Google Scholar 

  • Horta, H., & Santos, J. M. (2016). The impact of publishing during the Ph.D. on career research publications, visibility, and collaborations. Research in Higher Education, 57(1), 28–50.

    Article  Google Scholar 

  • Horta, H., & Yonezawa, A. (2013). Going places: Exploring the impact of intra-sectoral mobility on research productivity and communication behaviors in Japanese academia. Asia Pacific Education Review, 14(4), 537–547.

    Article  Google Scholar 

  • Johnson, B., & Stevens, J. J. (2001). Exploratory and confirmatory factor analysis of the School Level Environment Questionnaire (SLEQ). Learning Environments Research, 4(3), 325–344.

    Article  Google Scholar 

  • Jöreskog, K. G., & Sörbom, D. (1989). LISREL 7: A guide to the program and applications. Chicago, IL: SPSS.

    Google Scholar 

  • Jung, J. (2014). Research productivity by career stage among Korean academics. Tertiary Education and Management, 20(2), 85–105.

    Article  Google Scholar 

  • Jung, J., & Horta, H. (2013). Higher education research in Asia: A publication and co-publication analysis. Higher Education Quarterly, 67(4), 398–419.

    Article  Google Scholar 

  • Jung, J., & Horta, H. (2015). The contribution of East Asian countries to internationally published Asian higher education research: The role of system development and internationalization. Higher Education Policy, 28(4), 419–439.

    Article  Google Scholar 

  • Kassam, A., Papish, A., Modgill, G., & Patten, S. (2012). The development and psychometric properties of a new scale to measure mental illness related stigma by health care providers: The Opening Minds Scale for Health Care Providers (OMS-HC). BMC Psychiatry, 12(1), 1.

    Article  Google Scholar 

  • Katz, J. S., & Martin, B. R. (1997). What is research collaboration? Research Policy, 26(1), 1–18.

    Article  Google Scholar 

  • Klavans, R., Boyack, K. W., Sorensen, A. A., & Chen, C. (2013). Towards the development of an indicator of conformity. In 14th international society of scientometrics and informetrics conference. ISSI 2013.

  • Kline, R. B. (2011). Principles and practice of structural equation modeling. New York, NY: Guilford Press.

    MATH  Google Scholar 

  • Kolenikov, S., & Bollen, K. A. (2012). Testing negative error variances: Is a Heywood case a symptom of misspecification? Sociological Methods and Research, 41(1), 124–167.

    Article  MathSciNet  Google Scholar 

  • Kosmützky, A., & Krücken, G. (2014). Growth or steady state? A bibliometric focus on international comparative higher education research. Higher Education, 67(4), 457–472.

    Article  Google Scholar 

  • Latour, B., & Woolgar, S. (2013). Laboratory life: The construction of scientific facts. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Leahey, E. (2007). Not by productivity alone: How visibility and specialization contribute to academic earnings. American Sociological Review, 72(4), 533–561.

    Article  Google Scholar 

  • Levitt, D. G. (2010). Careers of an elite cohort of US basic life science postdoctoral fellows and the influence of their mentor’s citation record. BMC Medical Education, 10(1), 80.

    Article  Google Scholar 

  • MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84.

    Article  Google Scholar 

  • Mamun, S. A. K., & Rahman, M. M. (2015). Is there any feedback effect between academic research publication and research collaboration? Evidence from an Australian university. Scientometrics, 105(3), 2179–2196.

    Article  Google Scholar 

  • Maroco, J. (2003). Análise estatística: com utilização do SPSS. Lisbon, Portugal: Edições Sílabo.

    Google Scholar 

  • Maroco, J. (2007). Consistency and efficiency of ordinary least squares, maximum likelihood, and three type II linear regression models. Methodology, 3(2), 81–88.

    Article  Google Scholar 

  • Marôco, J. (2010). Análise de equações estruturais: Fundamentos teóricos, software & aplicações. Pêro Pinheiro: Report Number.

  • Martimianakis, M. A., & Muzzin, L. (2015). Discourses of interdisciplinarity and the shifting topography of academic work: Generational perspectives on facilitating and resisting neoliberalism. Studies in Higher Education, 40(8), 1454–1470.

    Article  Google Scholar 

  • Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56–63.

    Article  Google Scholar 

  • Morley, L. (2003). Quality and power in higher education. Berkshire: Society for Research into Higher Education, Open University.

    Google Scholar 

  • Pinheiro, D., Melkers, J., & Youtie, J. (2014). Learning to play the game: Student publishing as an indicator of future scholarly success. Technological Forecasting and Social Change, 81, 56–66.

    Article  Google Scholar 

  • Pump, B. (2011). Beyond metaphors: New research on agendas in the policy process. Policy Studies Journal, 39(s1), 1–12.

    Article  Google Scholar 

  • Rammstedt, B., & John, O. P. (2007). Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41(1), 203–212.

    Article  Google Scholar 

  • Rzhetsky, A., Foster, J. G., Foster, I. T., & Evans, J. A. (2015). Choosing experiments to accelerate collective discovery. Proceedings of the National Academy of Sciences, 112(47), 14569–14574.

    Article  Google Scholar 

  • Schut, M., van Paassen, A., Leeuwis, C., & Klerkx, L. (2014). Towards dynamic research configurations: A framework for reflection on the contribution of research to policy and innovation processes. Science and Public Policy, 41(2), 207–218.

    Article  Google Scholar 

  • Simon, H. A. (1992). Economics, bounded rationality, and the cognitive revolution. Aldershot: Elgar.

  • Slovic, P., Fischhoff, B., & Lichtenstein, S. (1982). Why study risk perception? Risk Analysis, 2(2), 83–93.

    Article  Google Scholar 

  • Spector, P. E., Van Katwyk, P. T., Brannick, M. T., & Chen, P. Y. (1997). When two factors don’t reflect two constructs: How item characteristics can produce artifactual factors. Journal of Management, 23(5), 659–677.

    Article  Google Scholar 

  • Steiger, J. H., Shapiro, A., & Browne, M. W. (1985). On the multivariate asymptotic distribution of sequential Chi square statistics. Psychometrika, 50(3), 253–263.

    Article  MathSciNet  MATH  Google Scholar 

  • Uddin, S., Hossain, L., & Rasmussen, K. (2013). Network effects on scientific collaborations. PLoS One, 8(2), e57546.

    Article  Google Scholar 

  • Verbree, M., Horlings, E., Groenewegen, P., Van der Weijden, I., & Van den Besselaar, P. (2015). Organizational factors influencing scholarly performance: A multivariate study of biomedical research groups. Scientometrics, 102(1), 25–49.

    Article  Google Scholar 

  • Whitley, R. (2000). The intellectual and social organization of the sciences. Oxford: Oxford University Press.

    Google Scholar 

Download references

Acknowledgments

This study was supported by doctoral Grant PD/BD/113999/2015 from the Fundação para a Ciência e Tecnologia (FCT), co-funded by the European Social Fund (ESF) and the Portuguese Ministry of Science and Education. The study was also supported by the Seed Funding Programme for Basic Research of The University of Hong Kong.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hugo Horta.

Appendix: Multi-dimensional research agendas inventory (MDRAI)

Appendix: Multi-dimensional research agendas inventory (MDRAI)

You will be asked a series of questions regarding your motivations and goals as an academic. To respond to this questionnaire, read each statement carefully and decide how much do you agree with each of them. For each statement, check one of the 7 boxes next to the corresponding item. If you don’t know or a particular sentence does not apply to you, check the N/A box.

There are no right or wrong answers. Please read each statement and check the box which best applies to you.

How much do you agree with the following statements?

  

Completely disagree

Strongly disagree

Disagree

Neither agree nor disagree

Agree

Strongly agree

Completely agree

N/A

A1

I aim to one day be one of the most respected experts in my field.

        

A2

Being a highly regarded expert is one of my career goals.

        

A3

I aim to be recognized by my peers.

        

A4

Standing out from the rest of my peers is one of my goals.

        

A5

I feel the need to constantly publish new and interesting papers.

        

A6

I am constantly striving to publish new papers.

        

C1

My expertise is focused on a single scientific area.

        

C2

I believe that specialization in one area is preferable to diversification.

        

C3

Shifting towards another field of science is not a part of my plans.

        

C4

Studying subjects outside of my main field of work is pointless.

        

C5

I have invested far too much in my current field to consider branching out into another.

        

DI1

I find “cutting-edge” scientific areas more appealing than well-established ones.

        

DI2

I would rather conduct revolutionary research with little chance of success than replicate research with a high chance of success.

        

DI3

I prefer “cutting-edge” research to “safe” research, even when the odds of success are much lower.

        

CN1

I prefer “safe” or “stable” fields of study.

        

CN2

I prefer fields of study that are considered “safe” or “stable.”

        

TL1

Limited funding does not constrain my choice of field.

        

TL2

Highly limited funding does not constrain my choice of field.

        

TL3

The availability of research funding for a certain topic does not influence me doing research on that topic.

        

CO1

I enjoy collaborating with other authors in my scientific articles.

        

CO2

My scientific articles are enhanced by collaboration with other authors.

        

CO3

I see myself as a team player when it comes to research collaboration.

        

CO4

I often seek peers with whom I can collaborate on scientific articles.

        

CO5

My peers often seek my collaboration in their scientific articles.

        

CO6

I am often invited to do collaborative work with my peers.

        

M1

My Ph.D. mentor’s opinion carries much weight in my research choices.

        

M2

A part of my work is largely due to my Ph.D. mentor.

        

M3

My research choices are highly influenced by my Ph.D. mentor’s opinion.

        

M4

My Ph.D. mentor is responsible for a large part of my work.

        

M5

My Ph.D. mentor still often works alongside me.

        

M6

My Ph.D. mentor largely determines my venues of research.

        

D1

I look forward to diversifying into other areas.

        

D2

I would be interested in pursuing research in other fields.

        

D3

I enjoy multi-disciplinary research more than single-discipline research.

        

D4

For me, multi-disciplinary research is more interesting than single-discipline research.

        

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Horta, H., Santos, J.M. An instrument to measure individuals’ research agenda setting: the multi-dimensional research agendas inventory. Scientometrics 108, 1243–1265 (2016). https://doi.org/10.1007/s11192-016-2012-4

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-2012-4

Keywords

Navigation