Skip to main content
Springer Nature Link
Log in

The Essential Competencies of Data Scientists: A Framework for Hiring and Training

  • Conference paper
  • First Online:
Human Interface and the Management of Information (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14691))

Included in the following conference series:

  • 347 Accesses

Abstract

Data science has emerged as a critical field for organizations seeking to harness the power of big data to inform strategic decisions and gain a competitive edge. However, the demand for data scientists far exceeds the currently available pool of qualified candidates, making it a significant challenge for organizations to hire and train the right talent. The discipline of data science is inherently multi-faceted, requiring a diverse set of technical and non-technical skills that can be rare to find in individuals or teams. In response to this challenge, our study has developed a comprehensive framework, drawing insights from extensive literature, identifying and underscoring the enduring relevance of 130 distinct competencies for the future data scientist. This framework stands out for its depth and breadth, offering a more holistic perspective than existing models found in the literature. By embracing this framework, organizations can craft more effective recruitment strategies, enhance the professional growth of their data science teams, and ultimately strengthen their capacity to leverage data for making informed and strategic decisions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Haneke, U., Trahasch, S., Zimmer, M., Felden, C.: Data Science: Grundlagen, Architekturen und Anwendungen. dpunkt. verlag (2021)

    Google Scholar 

  2. Hattingh, M., Marshall, L., Holmner, M., Naidoo, R.: Data science competency in organisations: a systematic review and unified model. In: Proceedings of the South African Institute of Computer Scientists and Information Technologists 2019, pp. 1–8 (2019). https://doi.org/10.1145/3351108.3351110

  3. Lovaglio, P.G., Cesarini, M., Mercorio, F., Mezzanzanica, M.: Skills in demand for ICT and statistical occupations: evidence from web-based job vacancies. Stat. Anal. Data Mining ASA Data Sci. J. 11(2), 78–91 (2018). https://doi.org/10.1002/sam.11372

    Article  MathSciNet  Google Scholar 

  4. da Silveira, C.C., Marcolin, C.B., da Silva, M., Domingos, J.C.: What is a data scientist? analysis of core soft and technical competencies in job postings. Revista Inovação, Projetos e Tecnologias 8(1), 25–39 (2020). https://doi.org/10.5585/iptec.v8i1.17263

    Article  Google Scholar 

  5. Wu, D., Lv, S., Xu, H.: An analysis on competency of human-centered data science employment. Proc. Assoc. Inf. Sci. Technol. 57(1), e219 (2020). https://doi.org/10.1002/pra2.219

    Article  Google Scholar 

  6. Prifti, L., Knigge, M., Kienegger, H., Krcmar, H.: A competency model for Industrie 4.0 employees. In: Proceedings der 13. Internationalen Tagung Wirtschaftsinformatik (WI 2017), pp. 46–60 (2017)

    Google Scholar 

  7. McCartney, S., Murphy, C., Mccarthy, J.: 21st century HR: a competency model for the emerging role of HR Analysts. Pers. Rev. 50(6), 1495–1513 (2021). https://doi.org/10.1108/PR-12-2019-0670

    Article  Google Scholar 

  8. Provost, F., Fawcett, T.: Data science and its relationship to big data and data-driven decision making. Big Data 1(1), 51–59 (2013). https://doi.org/10.1089/big.2013.1508

    Article  Google Scholar 

  9. De Mauro, A., Greco, M., Grimaldi, M., Nobili, G., et al.: Beyond data scientists: a review of big data skills and job families. In: Proceedings of IFKAD, pp. 1844–1857 (2016). https://hdl.handle.net/11580/55712

  10. Lyon, L.: Dealing with data: roles, rights, responsibilities, and relationships consultancy report (2007)

    Google Scholar 

  11. Nosarka, N.B.: Data scientist: using a competency-based approach to explore an emerging role. PhD thesis (2018). https://hdl.handle.net/10539/27155

  12. Stadelmann, T., Stockinger, K., Bürki, G.H., Braschler, M.: Data scientists. In: Applied Data Science: Lessons Learned for the Data-Driven Business, pp. 31–45 (2019). https://doi.org/10.1007/978-3-030-11821-1_3

  13. Smaldone, F., Ippolito, A., Lagger, J., Pellicano, M.: Employability skills: profiling data scientists in the digital labour market. Eur. Manag. J. 40(5), 671–684 (2022). https://doi.org/10.1016/j.emj.2022.05.005

    Article  Google Scholar 

  14. Saltz, J.S., Grady, N.W.: The ambiguity of data science team roles and the need for a data science workforce framework. In: 2017 IEEE International Conference on Big Data (Big Data), pp. 2355–2361. IEEE (2017). https://doi.org/10.1109/BigData.2017.8258190

  15. Jerina Jean Ecleo and Adrian Galido: Surveying linkedin profiles of data scientists: the case of the Philippines. Procedia Comput. Sci. 124, 53–60 (2017). https://doi.org/10.1016/j.procs.2017.12.129

    Article  Google Scholar 

  16. Song, I.Y., Zhu, Y.: Big data and data science: what should we teach? Exp. Syst. 33(4), 364–373 (2016). https://doi.org/10.1111/exsy.12130

    Article  Google Scholar 

  17. Della Volpe, M., Esposito, F.: How universities fill the talent gap: the data scientist in the Italian case. Afr. J. Bus. Manag. 14(2), 53–64 (2020). https://doi.org/10.5897/AJBM2019.8885

    Article  Google Scholar 

  18. Schwab-McCoy, A., Baker, C.M., Gasper, R.E.: Data science in 2020: computing, curricula, and challenges for the next 10 years. J. Stat. Data Sci. Educ. 29(sup1), S40–S50 (2021). https://doi.org/10.1080/10691898.2020.1851159

  19. Stanton, W.W., Stanton, A.D.: Helping business Entry-level Requirements needed for a career in analytics: a comprehensive industry assessment of entry-level requirements. Decis. Sci. J. Innov. Educ. 18(1), 138–165 (2020). https://doi.org/10.1111/dsji.12199

    Article  Google Scholar 

  20. Kim, J.Y., Lee, C.K.: An empirical analysis of requirements for data scientists using online job postings. Int. J. Softw. Eng. Appl. 10(4), 161–172 (2016)

    Google Scholar 

  21. Kim, M., Zimmermann, T., DeLine, R., Begel, A.: The emerging role of data scientists on software development teams. In: Proceedings of the 38th International Conference on Software Engineering, pp. 96–107 (2016). https://doi.org/10.1145/2884781.2884783

  22. Ehlers, U.D.: Future skills: the future of learning and higher education. BoD–Books on Demand (2020). https://www.learntechlib.org/p/208249/

  23. Pedro, F., Subosa, M., Rivas, A., Valverde, P.: Artificial intelligence in education: challenges and opportunities for sustainable development (2019). https://hdl.handle.net/20.500.12799/6533

  24. Wang, D., et al.: Human-AI collaboration in data science: Exploring data scientists’ perceptions of automated AI. In: Proceedings of the ACM on Human-Computer Interaction CSCW , vol. 3, pp. 1–24(2019). https://doi.org/10.1145/3359313

  25. McClelland, D.C.: Testing for competence rather than for “intelligence’’. Am. Psychol. 28(1), 1 (1973)

    Article  Google Scholar 

  26. Nascimbeni, F., et al.: The Opengame competencies framework: an attempt to map open education attitudes, knowledge, and skills. In: EDEN Conference Proceedings, vol. 1, pp. 105–112 (2020)

    Google Scholar 

  27. Staškeviča, A., et al.: The importance of competency model development. Acta Oeconomica Pragensia 27(2), 62–71 (2019). https://doi.org/10.18267/j.aop.622

  28. Skhvediani, A., Sosnovskikh, S., Rudskaia, I., Kudryavtseva, T.: Identification and comparative analysis of the skills structure of the data analyst profession in Russia. J. Educ. Bus. 97(5), 295–304 (2022). https://doi.org/10.1080/08832323.2021.1937018

    Article  Google Scholar 

  29. Binkley, M., et al.: Defining twenty-first century skills. In: Assessment and Teaching of 21st Century Skills, pp. 17–66 (2012). https://doi.org/10.1007/978-94-007-2324-5

  30. Boyatzis, R.E.: The Competent Manager: A Model for Effective Performance. John Wiley & Sons, Hoboken (1991)

    Google Scholar 

  31. Mike, K., Hazzan, O.: What is data science? Commun. ACM 66(2), 12–13 (2023). https://doi.org/10.1145/3575663

    Article  Google Scholar 

  32. Sanders, N.: A balanced perspective on prediction and inference for data science in industry. Harvard Data Sci. Rev. 1(1), 1–28 (2019). https://doi.org/10.1162/99608f92.644ef4a4

    Article  Google Scholar 

  33. Meyer, M.A.: Healthcare data scientist qualifications, skills, and job focus: a content analysis of job postings. J. Am. Med. Inf. Assoc. 26(5), 383–391 (2019). https://doi.org/10.1093/jamia/ocy181

    Article  Google Scholar 

  34. Shirani, A.: Identifying data science and analytics competencies based on industry demands. Issues Inf. Syst. 17(4), 137–144 (2016). https://doi.org/10.48009/4_iis_2016_137-144

  35. Ghasemaghaei, M., Ebrahimi, S., Hassanein, K.: Data analytics competency for improving firm decision making performance. J. Strat. Inf. Syst. 27(1), 101–113 (2018). https://doi.org/10.1016/j.jsis.2017.10.001

    Article  Google Scholar 

  36. Kansal, J., Singhal, S.: Development of a competency model for enhancing the organisational effectiveness in a knowledge-based organisation. Int. J. Indian Cult. Bus. Manag. 16(3), 287–301 (2018). https://doi.org/10.1504/IJICBM.2018.090909

    Article  Google Scholar 

  37. Murawski, M., Bick, M.: Digital competences of the workforce-a research topic? Bus. Process Manag. J. 23(3), 721–734 (2017). https://doi.org/10.1108/BPMJ-06-2016-0126

    Article  Google Scholar 

  38. Persaud, A.: Key competencies for big data analytics professions: a multimethod study. Inf. Technol. People 34(1), 178–203 (2021). https://doi.org/10.1108/ITP-06-2019-0290

    Article  Google Scholar 

  39. Bonesso, S., Gerli, F., Bruni, E.: The emotional and social side of analytics professionals: an exploratory study of the behavioral profile of data scientists and data analysts. Int. J. Manpower 43(9), 19–41 (2022). https://doi.org/10.1108/IJM-07-2020-0342

    Article  Google Scholar 

  40. Erpenbeck, J., Heyse, V.: Kompetenzmodelle und personalentwicklung. In: Jahrbuch Personalentwicklung, pp. 71–80 (2008)

    Google Scholar 

  41. Suhairom, N., Musta’amal, A.H., Amin, N.F.M., Johari, N.K.A.: The development of competency model and instrument for competency measurement: the research methods. Procedia-Soc. Behav. Sci. 152, 1300–1308 (2014). https://doi.org/10.1016/j.sbspro.2014.09.367

    Article  Google Scholar 

  42. Mina, M.A.E., Barzola, D.D.P.G.: Data scientist: a systematic review of the literature. In: International Conference on Technology Trends, pp. 476–487. Springer, Heidelberg (2018). https://doi.org/10.1007/978-3-030-05532-5_35

  43. Surbakti, F.P.S., Wang, W., Indulska, M., Sadiq, S.: Factors influencing effective use of big data: a research framework. Inf. Manag. 57(1), 103146 (2020). https://doi.org/10.1016/j.im.2019.02.001

    Article  Google Scholar 

  44. Joshua, E., Zarefard, M., Marsden, N.: Investigating skill requirements and gender bias in job openings for HCI professionals across the USA, Australia, Germany, India, and South Africa. In: 2023 9th International HCI and UX Conference in Indonesia (CHIuXiD), pp. 1–6. IEEEXplore (2023). In press

    Google Scholar 

  45. Vijayarani, S., Janani, R.: Text mining: open source tokenization tools-an analysis. Adv. Comput. Intell. Int. J. (ACII) 3(1), 37–47 (2016). https://doi.org/10.5121/acii.2016.3104

    Article  Google Scholar 

  46. De Mauro, A., Greco, M., Grimaldi, M., Ritala, P.: Human resources for big data professions: a systematic classification of job roles and required skill sets. Inf. Process. Manag. 54(5), 807–817 (2018). https://doi.org/10.1016/j.ipm.2017.05.004

    Article  Google Scholar 

  47. Chen, T., Liu, Y.X., Huang, L.: ImageGP: an easy-to-use data visualization web server for scientific researchers. Imeta 1(1), e5 (2022). https://doi.org/10.1002/imt2.5

    Article  Google Scholar 

  48. Le Deist, F.D., Winterton, J.: What is competence? Human Res. Dev. Int. 8(1), 27–46 (2005). https://doi.org/10.1080/1367886042000338227

    Article  Google Scholar 

  49. Brown, T., De Neve, G.: Skills, training and development: an introduction to the social life of skills in the global south (2023). https://doi.org/10.1080/01436597.2023.2219615

  50. Chen, C., Jiang, H.: Important skills for data scientists in china: two Delphi studies. J. Comput. Inf. Syst. (2018). https://doi.org/10.1080/08874417.2018.1472047

  51. Fatih Gurcan and Nergiz Ercil Cagiltay: Big data software engineering: analysis of knowledge domains and skill sets using LDA-based topic modeling. IEEE Access 7, 82541–82552 (2019). https://doi.org/10.1109/ACCESS.2019.2924075

    Article  Google Scholar 

  52. Davies, A., Mueller, J., Moulton, G.: Core competencies for clinical informaticians: a systematic review. Int. J. Med. Inf. 141, 104237 (2020). https://doi.org/10.1016/j.ijmedinf.2020.104237

    Article  Google Scholar 

  53. Bukhari, D.: Data science curriculum: current scenario. Int. J. Data Min. Knowl. Manag. Process (IJDKP) 10 (2020). https://doi.org/10.5121/ijdkp.2020.10301

  54. Sabaityte, J., Davidaviciene, V., Karpoviciute, R.: Learning skills for enhancing the use of big data. World J. Educ. Technol. Curr. Issues 12(1), 23–36 (2020). https://doi.org/10.18844/wjet.v12i1.4438

  55. Lnenicka, M., Kopackova, H., Machova, R., Komarkova, J.: Big and open linked data analytics: a study on changing roles and skills in the higher educational process. Int. J. Educ. Technol. High. Educ. 17, 1–30 (2020). https://doi.org/10.1186/s41239-020-00208-z

    Article  Google Scholar 

  56. Li, G., Yuan, C., Kamarthi, S., Moghaddam, M., Jin, X.: Data science skills and domain knowledge requirements in the manufacturing industry: a gap analysis. J. Manuf. Syst. 60, 692–706 (2021). https://doi.org/10.1016/j.jmsy.2021.07.007

    Article  Google Scholar 

  57. Dinh, L.T.N., Karmakar, G., Kamruzzaman, J.: A survey on context awareness in big data analytics for business applications. Knowl. Inf. Syst. 62, 3387–3415 (2020)

    Article  Google Scholar 

  58. Zarefard, M., Jeong, D.Y.: The Effect of Entrepreneurial Leadership Competencies in Iranian ICTs, p. 128. LAP Lambert Academic Publishing (2019). ISBN: 978-6134981484

    Google Scholar 

  59. Finegold, D., Notabartolo, A.S.: 21st century competencies and their impact: an interdisciplinary literature review. Transform. US Workforce Dev. Syst. 19, 19–56 (2010)

    Google Scholar 

  60. Škrinjarić, B.: Competence-based approaches in organizational and individual context. Human. Soc. Sci. Commun. 9(1), 1–12 (2022). https://doi.org/10.1057/s41599-022-01047-1

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work has been partially funded by the German Federal Institute for Vocational Education and Training (Bundesinstitut für Berufsbildung BIBB) under Grant Number 21INVI1802 as part of the project ‘KI-gestütztes Matching individueller und arbeitsmarktbezogener Anforderungen für die berufliche Weiterbildung. Teilvorhaben: Nutzer*innenzentrierte Anforderungsanalyse, Konzeptualisierung und Modellierung des Lern- und Matching-Angebots unter Berücksichtigung von Gender- und Diversity-Aspekten’. The responsibility for all content supplied lies with the authors.

Author information

Authors and Affiliations

  1. Heilbronn University, Heilbronn, Germany

    Motahareh Zarefard & Nicola Marsden

Authors
  1. Motahareh Zarefard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicola Marsden
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Motahareh Zarefard .

Editor information

Editors and Affiliations

  1. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  2. Tokyo University of Science, Tokyo, Japan

    Yumi Asahi

Appendix

Appendix

Table 2. Areas and Competence Dimensions; Functional and Ethics
Full size table
Table 3. Areas and Competence Dimensions ; Cognitive and Awareness
Full size table
Table 4. Areas and Competence Dimensions; Social and Organizational
Full size table
Table 5. Areas and Competence Dimensions; Behavioral
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zarefard, M., Marsden, N. (2024). The Essential Competencies of Data Scientists: A Framework for Hiring and Training. In: Mori, H., Asahi, Y. (eds) Human Interface and the Management of Information. HCII 2024. Lecture Notes in Computer Science, vol 14691. Springer, Cham. https://doi.org/10.1007/978-3-031-60125-5_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-60125-5_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-60124-8

  • Online ISBN: 978-3-031-60125-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics