Skip to main content
Springer Nature Link
Log in

Societal impact of research: a text mining study of impact types

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In addition to academic impact, researchers are increasingly concerned with understanding and demonstrating the practical impact of research outside academia. Several frameworks capturing key impact types have been developed based on project experiences, expert opinions, and surveys. This empirical study seeks to contribute to this development by identifying impact types documented in 6,882 case studies submitted to impact evaluation groups in Australia (Engagement and Impact Assessment) and the United Kingdom (Research Excellence Framework). The results of text mining indicate three emerging impact types that extend existing frameworks in terms of the recognition of new opportunities, the length of use, and experience improvement, thereby allowing a variety of researchers, not just those who address popular, short-term, and instrumental issues, to understand and demonstrate their practice impact.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Asveld, L., & van Dam-Mieras, R. (2017). Introduction: Responsible research innovation for sustainability. In L. Asveld, R. van Dam-Mieras, T. Swierstra, S. Lavrijssen, K. Linse, & J. van den Hoven (Eds.), Responsible innovation 3: A european agenda? (pp. 1–6). Springer International Publishing.

    Chapter  Google Scholar 

  • Australian Research Council. (2018). Engagement and Impact (Ei) 2018 Assessment Handbook. Retrieved March 1, 2019, from https://www.arc.gov.au/file/10666/download?token=62kvwQjS

  • Blei, D. M. (2012). Topic modeling and digital humanities. Journal of Digital Humanities, 2(1), 8–11.

    Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022s.

    MATH  Google Scholar 

  • Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233.

    Article  Google Scholar 

  • Bornmann, L., & Marx, W. (2014). How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics, 98(1), 211–219.

    Article  Google Scholar 

  • Chi, P.-S., & Glänzel, W. (2018). Comparison of citation and usage indicators in research assessment in scientific disciplines and journals. Scientometrics, 116(1), 537–554.

    Article  Google Scholar 

  • Chuang, J., Gupta, S., Manning, C., & Heer, J. (2013). Topic Model Diagnostics: Assessing Domain Relevance Via Topical Alignment. In: Paper presented at the International conference on machine learning.

  • Given, L. M., Kelly, W., & Willson, R. (2015). Bracing for Impact: The Role of Information Science in Supporting Societal Research Impact. In: Paper presented at the Proceedings of the 78th ASIS&T Annual Meeting: Information Science with Impact: Research in and for the Community.

  • Glänzel, W., & Chi, P.-S. (2020). The big challenge of scientometrics 2.0: Exploring the broader impact of scientific research in public health. Scientometrics, 125, 1–21.

    Article  Google Scholar 

  • Grant, J., & Hinrichs, S. (2015). The Nature, scale and beneficiaries of research impact: An initial analysis of research excellence framework (REF) 2014 Impact Case Studies. HEFCE-Higher Education Funding Council for England.

  • Greenhalgh, T., & Fahy, N. (2015). Research impact in the community-based health sciences: An analysis of 162 case studies from the 2014 UK research excellence framework. BMC Medicine, 13(1), 232.

    Article  Google Scholar 

  • Hughes, T., Webber, D., & O’Regan, N. (2019). Achieving wider impact in business and management: Analysing the case studies from REF 2014. Studies in Higher Education, 44(4), 628–642.

    Article  Google Scholar 

  • Jacobi, C., Van Atteveldt, W., & Welbers, K. (2016). Quantitative analysis of large amounts of journalistic texts using topic modelling. Digital Journalism, 4(1), 89–106.

    Article  Google Scholar 

  • Jarman, N., & Bryan, D. (2015). Beyond the academy: applying anthropological research, a case study of demonstrating impact in the UK 2014 REF. Anthropology in Action, 22(2), 36–41.

    Article  Google Scholar 

  • Kelly, D., Kent, B., McMahon, A., Taylor, J., & Traynor, M. (2016). Impact case studies submitted to REF 2014: The hidden impact of nursing research. Journal of Research in Nursing, 21(4), 256–268.

    Article  Google Scholar 

  • Landry, R., Amara, N., & Lamari, M. (2001). Utilization of Social science research knowledge in Canada. Research Policy, 30(2), 333–349.

    Article  Google Scholar 

  • Marcella, R., Lockerbie, H., & Bloice, L. (2016). Beyond REF 2014: The impact of impact assessment on the future of information research. Journal of Information Science, 42(3), 369–385.

    Article  Google Scholar 

  • McCallum, A. K. (2002). MALLET: A Machine Learning for Language Toolkit. http://mallet.cs.umass.edu.

  • Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science and Technology, 66(10), 1988–2002.

    Article  Google Scholar 

  • Morrow, E. M., Goreham, H., & Ross, F. (2017). Exploring research impact in the assessment of leadership governance and management research. Evaluation, 23(4), 407–431.

    Article  Google Scholar 

  • Morton, S. (2015). Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, 24(4), 405–419.

    Article  Google Scholar 

  • Ozanne, J. L., Davis, B., Murray, J. B., Grier, S., Benmecheddal, A., Downey, H., & Gall-Ely, M. L. (2017). Assessing the societal impact of research: The relational engagement approach. Journal of Public Policy & Marketing, 36(1), 1–14.

    Article  Google Scholar 

  • Pan, S. L., & Pee, L. G. (2020). Usable, in-use, and useful research: A 3u framework for demonstrating practice impact. Information Systems Journal, 30(2), 403–426.

    Article  Google Scholar 

  • Pee, L., & Chua, A. (2016). Duration, frequency, and diversity of knowledge contribution: differential effects of job characteristics. Information and Management, 53(4), 435–446.

    Article  Google Scholar 

  • Pee, L. G., & Kankanhalli, A. (2009). Knowledge Management Capability: A Resource-Based Comparison of Public and Private Organizations. In: Paper presented at the 30th International Conference on Information Systems, Phoenix, Arizona, USA.

  • Pee, L. G., Tham, Z.-C., Kankanhalli, A., & Tan, G. W. (2008). Turnover in information systems development projects-managing forgetting.

  • Piepenbrink, A., & Gaur, A. S. (2017). Topic models as a novel approach to identify themes in content analysis. In: Paper presented at the Academy of Management Proceedings.

  • Research Excellence Framework. (2012). Panel criteria and working methods. Retrieved February 1, 2019, from https://www.ref.ac.uk/2014/pubs/2012-01/

  • Sousa, S. B., & Brennan, J. L. (2014). The UK research excellence framework and the transformation of research production. In C. Musselin & P. N. Teixeira (Eds.), Reforming higher education: Public policy design and implementation (pp. 65–80). Netherlands: Springer.

    Chapter  Google Scholar 

  • Terämä, E., Smallman, M., Lock, S. J., Johnson, C., & Austwick, M. Z. (2016). Beyond academia–Interrogating research impact in the research excellence framework. PLoS ONE, 11(12): e0168533

    Article  Google Scholar 

  • University of Leicester. (2014). Impact on Continuing Professional Development for Science Educators. from https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=37267

  • University of Manchester. (2014). Experiments to Stimulate Civic Behaviour. from https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=28085

  • University of Nottingham. (2014). Delivering Public Health Services through Community Pharmacy. Retrieved February 1, 2020, from https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=28016

  • University of Oxford. (2014). Uoa02–05: Hormone Replacement Therapy and Cancer Risk: The Million Women Study. Retrieved February 1, 2020, from https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=9539

  • University of Southampton. (2014). REF impact case study: schopenhauer and nietzsche on the value of existence. Retrieved February 1, 2020, from https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=44195

  • University of Ulster. (2014). Fairy Magic: Enabling Cinematic Experiences on Mobile Devices in Real-Time. Retrieved February 1, 2020, from https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=575

  • Wiek, A., Talwar, S., O’Shea, M., & Robinson, J. (2014). Toward a methodological scheme for capturing societal effects of participatory sustainability research. Research Evaluation, 23(2), 117–132.

    Article  Google Scholar 

  • Zhang, L., Zhao, W., Sun, B., Huang, Y., & Glänzel, W. (2020). How Scientific research reacts to international public health emergencies: A global analysis of response patterns. Scientometrics, 124:1–27.

Download references

Acknowledgements

This study was funded by the Singapore Ministry of Education Academic Research Fund Tier 1 (Grant number: 2017-T1-001-095-06).

Author information

Authors and Affiliations

  1. Wee Kim Wee School of Communication and Information, Nanyang Technological University, Singapore, 637718, Singapore

    Han Zheng & L. G. Pee

  2. Department of Information Resources Management, Business School, Nankai University, 94 Weijin Road, Nankai District, Tianjin, 300071, China

    Dan Zhang

Authors
  1. Han Zheng
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. L. G. Pee
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Dan Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Dan Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, H., Pee, L.G. & Zhang, D. Societal impact of research: a text mining study of impact types. Scientometrics 126, 7397–7417 (2021). https://doi.org/10.1007/s11192-021-04096-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-021-04096-6

Keywords