Skip to main content
SpringerLink
Log in

Does collaborative learning design align with enactment? An innovative method of evaluating the alignment in the CSCL context

  • Published:
International Journal of Computer-Supported Collaborative Learning Aims and scope Submit manuscript

Abstract

This study reports on a novel design methodology, namely, design-centered research (DCR), developed to analyze and evaluate the alignment between an online collaborative learning design and its enactment. The approach is illustrated in a study involving 40 groups in total. Twenty different online collaborative learning activities were designed and enacted by 20 groups of three students in each of two iterations. The collaborative learning design plans from the first round were adjusted after reflecting on misalignments observed through the method during the enactment, and then enacted and tested again by another 20 groups in the second round. The proposed method involves an interaction path graph as well as three proposed indicators of group functioning. These three indicators include: (a) the range of activated knowledge, (b) the degree of knowledge building, and (c) an interactivity of the approach. This approach to quantification of alignment between a collaborative learning design and its enactment was successful in revealing areas for improvement of the design. The results of the two round study indicate that the alignment significantly improved after the optimization of the collaborative learning design based on the analysis of the first round. The findings also suggest that optimizing a collaborative learning design using this method is associated with improvements in group performance. Building on these findings, the collaborative learning design framework is discussed in detail in this article, and resulting implications for practitioners are discussed in depth.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Alharbi, N. M., Athauda, R. I., & Chiong, R. (2018). Empowering collaboration in project-based learning using a scripted environment: Lessons learned from analysing instructors’ needs. Technology, Pedagogy and Education, 27(3), 381–397. https://doi.org/10.1080/1475939X.2018.1473289.

    Article  Google Scholar 

  • Altebarmakian, M., & Alterman, R. (2019). Cohesion in online environments. International Journal of Computer-Supported Collaborative Learning, 14(4), 443–465. https://doi.org/10.1007/s11412-019-09309-y.

    Article  Google Scholar 

  • Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16–25. https://doi.org/10.3102/0013189x11428813.

    Article  Google Scholar 

  • Asensio-Pérez, J. I., Dimitriadis, Y., Pozzi, F., Hernández-Leo, D., Prieto, L. P., Persico, D., & Villagrá-Sobrino, S. L. (2017). Towards teaching as design: Exploring the interplay between full-lifecycle learning design tooling and teacher professional development. Computers & Education, 114, 92–116. https://doi.org/10.1016/j.compedu.2017.06.011.

    Article  Google Scholar 

  • Avcı, Ü. (2020). Examining the role of sentence openers, role assignment scaffolds and self-determination in collaborative knowledge building. Educational Technology Research and Development, 68(1), 109–135. https://doi.org/10.1007/s11423-019-09672-5.

    Article  Google Scholar 

  • Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1–14. https://doi.org/10.1207/s15327809jls1301_1.

    Article  Google Scholar 

  • Barron, B. (2003). When smart groups fail. The Journal of the Learning Sciences, 12(3), 307–359. https://doi.org/10.1207/S15327809JLS1203_1.

    Article  Google Scholar 

  • Bober, M. J., Sullivan, H. J., & Harrison, L. P. (1998). Instructional practices of teachers enrolled in educational technology and general education programs. Educational Technology Research and Development, 46(3), 81–97. https://doi.org/10.2307/30221066.

    Article  Google Scholar 

  • Burroughs, N., Gardner, J., Lee, Y., Guo, S., Touitou, I., Jansen, K., & Schmidt, W. (2019). Relationships between instructional alignment, time, instructional quality, teacher quality, and student mathematics achievement. In N. Burroughs et al. (Eds.), Teaching for excellence and equity (pp. 63–100). Cham: Springer.

    Chapter  Google Scholar 

  • Carter, L. (2008). Five big ideas: Leading total instructional alignment. Bloomington: Solution Tree Press.

    Google Scholar 

  • Chen, B., Scardamalia, M., & Bereiter, C. (2015). Advancing knowledge-building discourse through judgments of promising ideas. International Journal of Computer-Supported Collaborative Learning, 10(4), 345–366. https://doi.org/10.1007/s11412-015-9225-z.

    Article  Google Scholar 

  • Cohen, S. (1987). Instructional alignment: Searching for a magic bullet. Educational Researcher, 16(8), 16–20. https://doi.org/10.2307/1175370.

    Article  Google Scholar 

  • Carvalho, L., Martinez-Maldonado, R., & Goodyear, P. (2019). Instrumental genesis in the design studio. International Journal of Computer-Supported Collaborative Learning, 14(1), 77–107. https://doi.org/10.1007/s11412-019-09294-2.

    Article  Google Scholar 

  • Demetriadis, S. N., Papadopoulos, P. M., Stamelos, I. G., & Fischer, F. (2008). The effect of scaffolding students’ context-generating cognitive activity in technology-enhanced case-based learning. Computers & Education, 51(2), 939–954. https://doi.org/10.1016/j.compedu.2007.09.012.

    Article  Google Scholar 

  • Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. https://doi.org/10.3102/0013189X032001005.

    Article  Google Scholar 

  • Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C. (1996). The evolution of research on collaborative learning. In E. Spada & P. Reiman (Eds.), Learning in humans and machine: Towards an interdisciplinary learning science (pp. 189–211). Oxford: Elsevier.

    Google Scholar 

  • Dillenbourg, P. (1999). What do you mean by collaborative learning? In P. Dillenbourg (Ed.), Collaborative-learning: Cognitive and computational approaches (pp. 1–19). Oxford: Elsevier.

    Google Scholar 

  • Dillenbourg, P., & Tchounikine, P. (2007). Flexibility in macro-scripts for computer-supported collaborative learning. Journal of Computer Assisted Learning, 23(1), 1–13. https://doi.org/10.1111/j.1365-2729.2007.00191.x.

    Article  Google Scholar 

  • Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning. In N. Balacheff, S. Ludvigsen, T. D. Jong, A. Lazonder, & S. Barnes (Eds.), Technology-enhanced learning (pp. 3–19). Dordrecht: Springer.

    Chapter  Google Scholar 

  • De Wever, B., Schellens, T., Van Keer, H., & Valcke, M. (2008). Structuring asynchronous discussion groups by introducing roles: Do students act in line with assigned roles? Small Group Research, 39(6), 770–794. https://doi.org/10.1177/1046496408323227.

    Article  Google Scholar 

  • Easterday, M., Rees Lewis, D., & Gerber, E. (2014). Design-based research process: Problems, phases, and applications. In Proceeding of International Conference of Learning Sciences (pp. 1–8).

  • Engeström, Y. (1999). Activity theory and individual and social transformation. In Y. Engeström, R. Miettinen, & R.-L. Punamäki (Eds.), Perspectives on activity theory (pp. 19–38). New York, NY: Cambridge University Press.

    Chapter  Google Scholar 

  • Eryilmaz, E., van der Pol, J., Ryan, T., Clark, P. M., & Mary, J. (2013). Enhancing student knowledge acquisition from online learning conversations. International Journal of Computer-Supported Collaborative Learning, 8(1), 113–144. https://doi.org/10.1007/s11412-012-9163-y.

    Article  Google Scholar 

  • Eshuis, E. H., ter Vrugte, J., Anjewierden, A., Bollen, L., Sikken, J., & de Jong, T. (2019). Improving the quality of vocational students’ collaboration and knowledge acquisition through instruction and joint reflection. International Journal of Computer-Supported Collaborative Learning, 14(1), 53–76. https://doi.org/10.1007/s11412-019-09296-0.

    Article  Google Scholar 

  • Gerard, L., Kidron, A., & Linn, M. C. (2019). Guiding collaborative revision of science explanations. International Journal of Computer-Supported Collaborative Learning, 14(3), 291–324. https://doi.org/10.1007/s11412-019-09298-y.

    Article  Google Scholar 

  • Heimbuch, S., Ollesch, L., & Bodemer, D. (2018). Comparing effects of two collaboration scripts on learning activities for wiki-based environments. International Journal of Computer-Supported Collaborative Learning, 13(3), 331–357. https://doi.org/10.1007/s11412-018-9283-0.

    Article  Google Scholar 

  • Ingulfsen, L., Furberg, A., & Strømme, T. A. (2018). Students’ engagement with real-time graphs in CSCL settings: Scrutinizing the role of teacher support. International Journal of Computer-Supported Collaborative Learning, 13(4), 365–390. https://doi.org/10.1007/s11412-018-9290-1.

    Article  Google Scholar 

  • Johnson, C., Hill, L., Lock, J., Altowairiki, N., Ostrowski, C., dos Santos, L. D. R., & Liu, Y. (2017). Using design-based research to develop meaningful online discussions in undergraduate field experience courses. The International Review of Research in Open and Distributed Learning, 18(6), 36–53. https://doi.org/10.19173/irrodl.v18i6.2901.

    Article  Google Scholar 

  • Kapur, M., Voiklis, J., & Kinzer, C. K. (2011). A complexity-grounded model for the emergence of convergence in CSCL groups. In S. Puntambekar, G. Erkens, & C. Hmelo-Silver (Eds.), Analyzing interactions in CSCL (pp. 3–23). Boston, MA: Springer US.

    Chapter  Google Scholar 

  • Kramarski, B., & Dudai, V. (2009). Group-metacognitive support for online inquiry in mathematics with differential self-questioning. Journal of Educational Computing Research, 40(4), 377–404. https://doi.org/10.2190/EC.40.4.a.

    Article  Google Scholar 

  • Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19(3), 335–353.

    Article  Google Scholar 

  • Leinonen, T., Keune, A., Veermans, M., & Toikkanen, T. (2016). Mobile apps for reflection in learning: A design research in K-12 education. British Journal of Educational Technology, 47(1), 184–202. https://doi.org/10.1111/bjet.12224.

    Article  Google Scholar 

  • Lin, C. Y., & Reigeluth, C. M. (2016). Scaffolding wiki-supported collaborative learning for small-group projects and whole-class collaborative knowledge building. Journal of Computer Assisted Learning, 32(6), 529–547. https://doi.org/10.1111/jcal.12140.

    Article  Google Scholar 

  • Ludvigsen, S. (2016). CSCL towards the future: The second decade of ijCSCL. International Journal of Computer-Supported Collaborative Learning, 11(1), 1–7. https://doi.org/10.1007/s11412-016-9230-x.

    Article  Google Scholar 

  • Macphail, A., Tannehill, D., & Goc Karp, G. (2013). Preparing physical education preservice teachers to design instructionally aligned lessons through constructivist pedagogical practices. Teaching & Teacher Education, 33(2), 100–112. https://doi.org/10.1016/j.tate.2013.02.008.

    Article  Google Scholar 

  • Martin, F. (2011). Instructional design and the importance of instructional alignment. Community College Journal of Research and Practice, 35(12), 955–972. https://doi.org/10.1080/10668920802466483.

    Article  Google Scholar 

  • Martinez-Maldonado, R. (2019). A handheld classroom dashboard: Teachers’ perspectives on the use of real-time collaborative learning analytics. International Journal of Computer-Supported Collaborative Learning, 14(3), 383–411. https://doi.org/10.1007/s11412-019-09308-z.

    Article  Google Scholar 

  • Martone, A., & Sireci, S. G. (2009). Evaluating alignment between curriculum, assessment, and instruction. Review of Educational Research, 79(4), 1332–1361. https://doi.org/10.2307/40469099.

    Article  Google Scholar 

  • Meijer, H., Hoekstra, R., Brouwer, J., & Strijbos, J. W. (2020). Unfolding collaborative learning assessment literacy: A reflection on current assessment methods in higher education. Assessment & Evaluation in Higher Education, 1–19. https://doi.org/10.1080/02602938.2020.1729696.

  • Mende, S., Proske, A., Körndle, H., & Narciss, S. (2017). Who benefits from a low versus high guidance CSCL script and why? Instructional Science, 45(4), 439–468. https://doi.org/10.1007/s11251-017-9411-7.

    Article  Google Scholar 

  • Näykki, P., Isohätälä, J., Järvelä, S., Pöysä-Tarhonen, J., & Häkkinen, P. (2017). Facilitating socio-cognitive and socio-emotional monitoring in collaborative learning with a regulation macro script–an exploratory study. International Journal of Computer-Supported Collaborative Learning, 12(3), 251–279. https://doi.org/10.1007/s11412-017-9259-5.

    Article  Google Scholar 

  • Njenga, S. T., Oboko, R. O., Omwenga, E. I., & Muuro, E. M. (2017). Regulating group cognitive conflicts using intelligent agents in collaborative M-learning. In Proceedings of 2017 IEEE AFRICON (pp. 38–43). https://doi.org/10.1109/AFRCON.2017.8095452.

  • Pattalitan, J. A. (2016). The implications of learning theories to assessment and instructional scaffolding techniques. American Journal of Educational Research, 4(9), 695–700. https://doi.org/10.12691/education-4-9-9.

    Article  Google Scholar 

  • Popper, K. R. (1963). Science as falsification. Conjectures and refutations, 1, 33–39.

    Google Scholar 

  • Raes, A., Schellens, T., De Wever, B., & Vanderhoven, E. (2012). Scaffolding information problem solving in web-based collaborative inquiry learning. Computers & Education, 59(1), 82–94. https://doi.org/10.1016/j.compedu.2011.11.010.

    Article  Google Scholar 

  • Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture (s), 33, 134–154.

    Google Scholar 

  • Roach, A. T., Niebling, B. C., & Kurz, A. (2008). Evaluating the alignment among curriculum, instruction, and assessment: Implications and applications for research and practice. Psychology in the Schools, 45(2), 158–176. https://doi.org/10.1002/pits.20282.

    Article  Google Scholar 

  • Rodríguez-Triana, M. J., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343. https://doi.org/10.1111/bjet.12198.

    Article  Google Scholar 

  • Shin, Y., Kim, D., & Jung, J. (2018). The effects of representation tool (visible-annotation) types to support knowledge building in computer-supported collaborative learning. Journal of Educational Technology & Society, 21(2), 98–110.

    Google Scholar 

  • Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 409–426). Cambridge: Cambridge University Press.

    Google Scholar 

  • Stahl, G. (2017). Group practices: A new way of viewing CSCL. International Journal of Computer-Supported Collaborative Learning, 12(1), 113–126. https://doi.org/10.1007/s11412-017-9251-0.

    Article  Google Scholar 

  • Stahl, G., & Hakkarainen, K. (2019). Theories of CSCL. In U. Cress, C. Rosé, A. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning. New York: Springer http://GerryStahl.net/pub/cscltheories.pdf.

    Google Scholar 

  • Tissenbaum, M., & Slotta, J. (2019). Supporting classroom orchestration with real-time feedback: A role for teacher dashboards and real-time agents. International Journal of Computer-Supported Collaborative Learning, 14(3), 325–351. https://doi.org/10.1007/s11412-019-09306-1.

    Article  Google Scholar 

  • Tversky, A. (1977). Features of similarity. Psychological Review, 84(4), 327–352. https://doi.org/10.1037/0033-295X.84.4.327.

    Article  Google Scholar 

  • Vogel, F., Wecker, C., Kollar, I., & Fischer, F. (2017). Socio-cognitive scaffolding with computer-supported collaboration scripts: A meta-analysis. Educational Psychology Review, 29, 477–511. https://doi.org/10.1007/s10648-016-9361-7.

    Article  Google Scholar 

  • Vygotsky, L. S. (1978). Mind in society. Cambridge: Harvard University Press.

    Google Scholar 

  • Wang, X., & Mu, J. (2017). Flexible scripting to facilitate knowledge construction in computer-supported collaborative learning. Singapore: Springer.

    Book  Google Scholar 

  • Wake, J. D., Guribye, F., & Wasson, B. (2018). Learning through collaborative design of location-based games. International Journal of Computer-Supported Collaborative Learning, 13(2), 167–187. https://doi.org/10.1007/s11412-018-9278-x.

    Article  Google Scholar 

  • Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education. Academic Achievement, 1(11), 1–46. https://doi.org/10.1111/j.1467-9892.1990.tb00038.x.

    Article  Google Scholar 

  • Yang, K. (2013). DBR and DCR, which can bridge the gap between educational theory and practice? E-Education Research, 12, 11–15. https://doi.org/10.13811/j.cnki.eer.2013.12.010.

    Article  Google Scholar 

  • Yang, K., & Liu, H. (2018). A case study on design of problem-solving collaborative learning from DCR perspective. E-Education Research, 11, 5–12.

    Google Scholar 

  • Yao, Q., Yang, K., Zhao, G., & Huang, R. (2006). A concept map scoring algorithm based on proposition chains for concept mapping. In Proceeding of the second international conference on concept mapping (pp. 8–15). San Jose: Universidad de Costa Rica.

    Google Scholar 

  • You, Y. (1993). What can we learn from chaos theory? An alternative approach to instructional systems design. Educational Technology Research and Development, 41(3), 17–32.

    Article  Google Scholar 

  • Zheng, B., Niiya, M., & Warschauer, M. (2015). Wikis and collaborative learning in higher education. Technology, Pedagogy and Education, 24(3), 357–374. https://doi.org/10.1080/1475939X.2014.948041.

    Article  Google Scholar 

  • Zheng, L. (2015a). A systematic literature review of design-based research from 2004 to 2013. Journal of Computers in Education, 2(4), 399–420. https://doi.org/10.1007/s40692-015-0036-z.

    Article  Google Scholar 

  • Zheng, L. (2015b). A case study on the consistence between the instructional design and the implementation of teaching. Modern Distance Education Research, 3, 95–103.

    Google Scholar 

  • Zheng, L. (2017). Knowledge building and regulation in computer-supported collaborative learning. Singapore: Springer.

    Book  Google Scholar 

  • Zheng, L., Li, X., Zhang, X., & Sun, W. (2019). The effects of group metacognitive scaffolding on group metacognitive behaviors, group performance, and cognitive load in computer-supported collaborative learning. The Internet and Higher Education, 42, 13–24. https://doi.org/10.1016/j.iheduc.2019.03.002.

    Article  Google Scholar 

  • Zheng, L., & Yang, K. (2014). Why we should research on the consistency rather than the effectiveness? Chinese Educational Technology, 9, 20–23.

    Google Scholar 

  • Zheng, L., Yang, K., & Huang, R. (2012). Analyzing interactions by an IIS-map-based method in face-to-face collaborative learning: An empirical study. Journal of Educational Technology & Society, 15(3), 116–132.

    Google Scholar 

Download references

Acknowledgements

This study is funded by the youth project of Humanities and Social Science Research in the Ministry Education (19YJC880141) and National Natural Science Foundation of China (61907003).

Author information

Authors and Affiliations

  1. School of Educational Technology, Faculty of Education, Beijing Normal University, No. 19, XinJieKouWai St., HaiDian District, Beijing, 100875, People’s Republic of China

    Lanqin Zheng, Panpan Cui & Xuan Zhang

Authors
  1. Lanqin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Panpan Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lanqin Zheng.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, L., Cui, P. & Zhang, X. Does collaborative learning design align with enactment? An innovative method of evaluating the alignment in the CSCL context. Intern. J. Comput.-Support. Collab. Learn 15, 193–226 (2020). https://doi.org/10.1007/s11412-020-09320-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11412-020-09320-8

Keywords

Search

Navigation