Abstract
This paper serves several purposes. First and foremost, it is devoted to developing a better understanding of the effectiveness of blended learning (BL) in higher education. This is achieved through a meta-analysis of a sub-collection of comparative studies of BL and classroom instruction (CI) from a larger systematic review of technology integration (Schmid et al. in Comput Educ 72:271–291, 2014). In addition, the methodology of meta-analysis is described and illustrated by examples from the current study. The paper begins with a summary of the experimental research on distance education (DE) and online learning (OL), encapsulated in meta-analyses that have been conducted since 1990. Then it introduces the Bernard et al. (Rev Educ Res 74(3):379–439, 2009) meta-analysis, which attempted to alter the DE research culture of always comparing DE/OL with CI by examining three forms of interaction treatments (i.e., student–student, student–teacher, student–content) within DE, using the theoretical framework of Moore (Am J Distance Educ 3(2):1–6, 1989) and Anderson (Rev Res Open Distance Learn 4(2):9–14, 2003). The rest of the paper revolves around the general steps and procedures (Cooper in Research synthesis and meta-analysis: a step-by-step approach, 4th edn, SAGE, Los Angeles, CA, 2010) involved in conducting a meta-analysis. This section is included to provide researchers with an overview of precisely how meta-analyses can be used to respond to more nuanced questions that speak to underlying theory and inform practice—in other words, not just answers to the “big questions.” In this instance, we know that technology has an overall positive impact on learning (g + = +0.35, p < .01, Tamim et al. in Rev Educ Res 81(3):4–28, 2011), but the sub-questions addressed here concern BL interacting with technology in higher education. The results indicate that, in terms of achievement outcomes, BL conditions exceed CI conditions by about one-third of a standard deviation (g + = 0.334, k = 117, p < .001) and that the kind of computer support used (i.e., cognitive support vs. content/presentational support) and the presence of one or more interaction treatments (e.g., student–student/–teacher/–content interaction) serve to enhance student achievement. We examine the empirical studies that yielded these outcomes, work through the methodology that enables evidence-based decision-making, and explore how this line of research can improve pedagogy and student achievement.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12528-013-9077-3/MediaObjects/12528_2013_9077_Fig1_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12528-013-9077-3/MediaObjects/12528_2013_9077_Fig2_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12528-013-9077-3/MediaObjects/12528_2013_9077_Fig3_HTML.gif)
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Means et al. (2013), Cook et al. (2008), Jahng et al. (2007), Lou et al. (2006), Allen et al. (2006), Sitzmann et al. (2006), Williams (2006), Zhao et al. (2005), Allen et al. (2004), Bernard et al. (2004), Cavanaugh et al. (2004), Shachar and Neumann (2003), Ungerleider and Burns (2003), Cavanaugh (2001), Allen et al. (2002), Machtmes and Asher (2000).
More information about specific topics concerning systematic review and meta-analysis is available in the following publications: (1) history (Hunt, 1997; Glass, 1976); (2) general methodology (e.g., Cooper et al. 2009; Cooper 2010; Pettigrew and Roberts 2006; Lipsey and Wilson 2001; Cooper et al. 2009); and (3) statistics (e.g., Piggot 2012; Borenstein et al. 2009; Hunter and Schmidt 1990; Hedges and Olkin 1985).
References
Note: A list of studies included in the meta-analysis is available upon request from the authors.
Abrami, P. C. (2010). On the nature of support in computer supported collaborative learning using gStudy. Computers in Human Behavior, 26(5), 835–839. doi:10.1016/j.chb.2009.04.007.
Abrami, P. C., & Bernard, R. M. (2006). Research on distance education: In defense of field experiments. Distance Education, 27(1), 5–26.
Abrami, P. C., & Bernard, R. M. (2012). Statistical control versus classification of study quality in meta-analysis. Effective Education, 4(1), 43–72. doi:10.1080/19415532.2012.761889.
Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M., Tamim, R. M., et al. (2008). Instructional interventions affecting critical thinking skills and dispositions: A stage one meta-analysis. Review of Educational Research, 78(4), 1102–1134. doi:10.3102/0034654308326084.
Abrami, P. C., Bernard, R. M., Bures, E. M., Borokhovski, E., & Tamim, R. (2011). Interaction in distance education and online learning: Using evidence and theory to improve practice. Journal of Computing in Higher Education, 23(2/3), 82–103. doi:10.1007/s12528-011-9043-x.
Albrecht, B. (2006). Enriching student experience through blended learning. ECAR Research Bulletin, 12.
Allen, M., Bourhis, J., Burrell, N., & Mabry, E. (2002). Comparing student satisfaction with distance education to traditional classrooms in higher education: A meta-analysis. American Journal of Distance Education, 16(2), 83–97. doi:10.1207/S15389286AJDE1602_3.
Allen, M., Bourhis, J., Mabry, E., Burrell, N., Timmerman, E., & Titsworth, S. (2006). Comparing distance education to face-to-face methods of education. In B. Gayle, R. Preiss, N. Burrell, & M. Allen (Eds.), Classroom and communication education research: Advances through meta-analysis (pp. 229–241). Hillsdale, NJ: Erlbaum.
Allen, M., Mabry, E., Mattrey, M., Bourhis, J., Titsworth, S., & Burrell, N. (2004). Evaluating the effectiveness of distance learning: A comparison using meta-analysis. Journal of Communication, 54(3), 402–420. doi:10.1111/j.1460-2466.2004.tb02636.x.
Anderson, T. (2003). Getting the mix right again: An updated and theoretical rationale for interaction. International Review of Research in Open and Distance Learning, 4(2), 9–14. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/149.
Arabasz, P., & Baker, M. (2003). Respondent Summary: Evolving campus support models for e-learning courses. EDUCAUSE Center for Applied Research. http://www.educause.edu/ir/library/pdf/EKF/ekf0303.pdf.
Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13(2), 111–127. doi:10.2190/9LMD-3U28-3A0G-FTQT.
Bele, J. L., & Rugelj, J. (2007). Blended learning—an opportunity to take the best of both worlds. International Journal of Emerging Technologies in Learning, 2(3). doi:10.3991%2Fijet.v2i3.133.
Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., et al. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243–1289. doi:10.3102/0034654309333844.
Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., et al. (2004). How does distance education compare to classroom instruction? A Meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379–439. doi:10.3102/00346543074003379.
Bernard, R. M., Rojo de Rubalcava, B., & St-Pierre, D. (2000). Collaborative online distance education: Issues for future practice and research. Distance Education, 21(2), 260–277. doi:10.1080/0158791000210205.
Bernard, R. M., Zhang, D., Abrami, P. C., Sicoly, F., Borokhovski, E., & Surkes, M. (2008). Exploring the structure of the Watson-Glaser critical thinking appraisal: One scale or many subscales? Thinking Skills and Creativity, 3, 15–22. doi:10.1016/j.tsc.2007.11.001.
Bliuc, A. M., Goodyear, P., & Ellis, R. A. (2007). Research focus and methodological choices in studies into students’ experiences of blended learning in higher education. The Internet and Higher Education, 10(4), 231–244. doi:10.1016/j.iheduc.2007.08.001.
Bonk, C. J., & Graham, C. R. (Eds.). (2006). The handbook of blended learning: Global perspectives, local designs. San Francisco, CA: Pfeiffer.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. (2005). Comprehensive meta-analysis version 2 .2.048. Englewood, NJ: Biostat.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. (2009). Introduction to meta-analysis. Chichester, UK: Wiley.
Borokhovski, E., Tamim, R. M., Bernard, R. M., Abrami, P. C., & Sokolovskaya, A. (2012). Are contextual and design student–student interaction treatments equally effective in distance education? A follow-up meta-analysis of comparative empirical studies. Distance Education, 33(3), 311–329. doi:10.1080/01587919.2012.723162.
Campbell, D., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
Cavanaugh, C. S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7(1), 73–88. Norfolk, VA: AACE. Retrieved March 13, 2007 from http://www.editlib.org/p/8461.
Cavanaugh, C., Gillan, K. J., Kromey, J., Hess, M., & Blomeyer, R. (2004). The effects of distance education on K-12 student outcomes: A meta-analysis. Naperville, IL: Learning Point Associates.
Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459. doi:10.3102/00346543053004445.
Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21–29. doi:10.1007/BF02299088.
Clark, R. E., Yates, K., Early, S., & Moulton, K. (2009). An analysis of the failure of electronic media and discovery-based learning: Evidence for the performance benefits of guided training methods. In K. H. Silber & R. Foshay (Eds.), Handbook of training and improving workplace performance, Volume I: Instructional design and training delivery. Washington, DC: ISPI.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.
Cook, D. A. (2009). The failure of e-learning research to inform educational practice and what we can do about it. Medical Teacher, 31(2), 158–162. doi:10.1080/01421590802691393.
Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2008). Internet-based learning in the health professions: A meta-analysis. Journal of the American Medical Association, 300(10), 1181–1196. doi:10.1001/jama.300.10.1181.
Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles, CA: SAGE Publications.
Cooper, H. M., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.
Driscoll, M., & Carliner, S. (2005). Advanced web-based training strategies. Blended learning as a curriculum design strategy (pp. 87–116). New York, NY: ASTD Press.
Duval, S., & Tweedie, R. (2004). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. doi:10.1111/j.0006-341X.2000.00455.x.
Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. San Francisco, CA: Jossey-Bass.
Glass, G. V. (1976). Primary, secondary and meta-analysis. Educational Researcher, 5(10), 3–8.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
Graham, C. R. (2005). Blended learning systems. In C. J. Bonk, & C. R. Graham (Eds.), The handbook of blended learning: Global perspectives, local designs. Chichester: Wiley (originally Pfeiffer).
Hammerstrøm, K., Wade, C. A. & Jørggensen, A.-M. K. (2010). Searching the literature: A guide to information retrieval for campbell systematic reviews 2010: Supplement 1. Oslo, Norway: The Campbell Collaboration. doi:10.4073/csrs.2010.1.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
Hedges, L. V., Shymansky, J. A. & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. (Stock Number PB-52). Washington, DC: National Science Teachers Association. (ERIC Document Reproduction Service No. ED309952).
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analysis. British Medical Journal, 327, 557–560. doi:10.1136/bmj.327.7414.557.
Hunt, M. (1997). How science takes stock: The story of meta-analysis. New York, NY: Russell Sage Foundation.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: SAGE Publications.
Jahng, N., Krug, D. & Zhang, Z. (2007). Student achievement in online education compared to face-to-face education. European Journal of Open, Distance and E-Learning. Retrieved from http://www.eurodl.org/materials/contrib/2007/Jahng_Krug_Zhang.htm.
Johnson, D. W., & Johnson, R. T. (2009). An educational psychology success story: Social interdependence theory and cooperative learning. Educational Researcher, 368(5), 365–379. doi:10.3102/0013189X09339057.
Keegan, D. (1996). Foundations of distance education (3rd ed.). London: Routledge.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. New York, NY: Sage Publications.
Lou, Y., Abrami, P. C., & D’Appollonia, S. (2001). Small group and individual learning with technology: A meta-analysis. Review of Educational Research, 71, 449–521. doi:10.3102/00346543071003449.
Lou, Y., Bernard, R. M., & Abrami, P. C. (2006). Media and pedagogy in undergraduate distance education: A theory-based meta-analysis of empirical literature. Educational Technology Research and Development, 54, 141–176. doi:10.1007/s11423-006-8252-x.
Machtmes, K., & Asher, J. W. (2000). A meta-analysis of the effectiveness of telecourses in distance education. American Journal of Distance Education, 14(1), 27–46. doi:10.1080/08923640009527043.
Marquis, C. (2004). WebCT survey discovers a blend of online learning and classroom-based teaching is the most effective form of learning today. WebCT.com. Retrieved from http://www.webct.com/service/ViewContent?contentID=19295938.
Means, B, Toyama, Y., Murphy, R. F., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature, Teachers College Record, 115(3), 1–47. Retrieved from http://www.tcrecord.org/library/content.asp?contentid=16882.
Means, B., Toyama, Y., Murphy, R., Bakia, M,. & Jones, K. (2010). Evaluation of evidence-based practices in online Learning: A meta-analysis and review of online learning studies. Technical Report. U. S. Department of Education, Washington, DC.
Moore, M. G. (1989). Editorial: Three types of interaction. The American Journal of Distance Education, 3(2), 1–6. doi:10.1080/08923648909526659.
Pettigrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Oxford, UK: Blackwell Publishing.
Piggot, T. D. (2012). Advances in meta-analysis. New York, NY: Springer.
Pintrich, P. R. (2003). A Motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667–686. doi:10.1037/0022-0663.95.4.667.
Rosen, Y., & Salomon, G. (2007). The differential learning achievements of constructivist technology-intensive learning environments as compared with traditional ones: A meta-analysis. Journal of Educational Computing Research, 36(1), 1–14. doi:10.2190/R8M4-7762-282U-554J.
Ross, S. M., & Morrison, G. R. (1989). In search of a happy medium in instructional technology research: Issues concerning external validity, media replications, and learner control. Educational Technology Research and Development, 37(1), 19–33. doi:10.1007/BF02299043.
Ross, S. M., Morrison, G. R., & Lowther, D. L. (2010). Educational technology research past and present: Balancing rigor and relevance to impact school learning. Contemporary Educational Technology, 1(1), 17–35. Retrieved from http://www.cedtech.net/articles/112.pdf.
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester, UK: Wiley.
Scammacca, N., Roberts, G., & Stuebing, K. K. (2013). Meta-analysis with complex research designs: Dealing with dependence from multiple measures and multiple group comparisons. Review of Educational Research. Published online 13 September 2013. doi:10.3102/0034654313500826.
Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Surkes, M. A., et al. (2014). The effects of technology use in postsecondary education: A meta-analysis of classroom applications. Computers & Education, 72, 271–291. doi:10.1016/j.compedu.2013.11.002.
Shachar, M., & Neumann, Y. (2003). Differences between traditional and distance education academic performances: A meta-analytical approach. International Review of Research in Open and Distance Education, 4(2). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/153/704.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59(3), 623–664. doi:10.1111/j.1744-6570.2006.00049.x.
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research., 81(3), 4–28. doi:10.3102/0034654310393361.
Ungerleider, C. S., & Burns, T. C. (2003). Information and communication technologies in elementary and secondary education: A state of the art review. International Journal of Educational Policy, Research & Practice, 3(4), 27–54.
Valentine, J. C., & Cooper, H. (2003). Effect size substantive interpretation guidelines: Issues in the interpretation of effect sizes. Washington, DC: What Works Clearinghouse.
Valentine, J. C., & Cooper, H. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The Study Design and Implementation Assessment Device (Study DIAD). Psychological Methods, 13(2), 130–149. doi:10.1037/1082-989X.13.2.130.
Watson, G., & Glaser, E. M. (1980). Watson–Glaser critical thinking appraisal: Forms A and B. San Antonio, TX: PsychCorp.
Williams, S. L. (2006). The effectiveness of distance education in allied health science programs: A meta-analysis of outcomes. American Journal of Distance Education, 20(3), 127–141. doi:10.1207/s15389286ajde2003_2.
Zhao, Y., Lei, J., Yan, B., & Tan, S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Retrieved from http://ott.educ.msu.edu/literature/report.pdf.
Zhoa, Y., & Breslow, L. (2013). Literature review on hybrid/blended learning. Unpublished manuscript. Retrieved from http://tll.mit.edu/sites/default/files/library/Blended_Learning_Lit_Reveiw.pdf.
Zimmerman, B. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaertzs & P. R. Pintrich (Eds.), Handbook of self-regulation (pp. 13–39). New York, NY: Academic Press.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bernard, R.M., Borokhovski, E., Schmid, R.F. et al. A meta-analysis of blended learning and technology use in higher education: from the general to the applied. J Comput High Educ 26, 87–122 (2014). https://doi.org/10.1007/s12528-013-9077-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12528-013-9077-3