Skip to main content

Advertisement

SpringerLink
Log in

A design science research methodology for developing a computer-aided assessment approach using method marking concept

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Assessment has been defined as an authentic method that plays an important role in evaluating students’ learning attitude in acquiring lifelong knowledge. Traditional methods of assessment including the Computer-Aided Assessment (CAA) for mathematics show limited ability to assess students’ full work unless multi-step questions are sub-divided into sub questions. This issue persisted significant drawback especially within the notion of method marking approach. To address this issue, the aim of the study is to develop a methodological framework that will create an information and communications technology (ICT) artefact prototype. The prototype (termed as method marking assessment (MMA) artefact) implements a method-marking assessment concept to assess through multi-step questions. Extensive literature reviews have revealed that there are features in common between complex-problem solution characteristics and multi-steps questions assessment using ICT; therefore complex problems paradigm is used in the study for developing the MMA prototype.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Angeli, C. (2010). Diagnostic expert systems: from expert’s knowledge to real—time systems. Advanced Knowledge Based Systems: Model, Applications & Search, 1, 50–73.

    Google Scholar 

  • Ashton, H. S., Beevers, C. E., Korabinski, A. A., & Youngson, M. A. (2006). Incorporating partial credit in computer-aided assessment of mathematics in secondary education. British Journal of Educational Technology, 37(1), 93–119. doi:10.1111/j.1467-8535.2005.00512.x.

    Article  Google Scholar 

  • Balcombe, A., Brennan, M. & Everiss, L. (2011). Formative Learning: A Description of an Assessment Framework in Initial Teacher Education. In: M. Hodis & S. Kaiser (Eds.), Proceedings of the Symposium on Assessment and Learner Outcomes, Victoria University, (pp. 25–40). Wellington, New Zealand.

  • Bayazit, N. (2004). Investigating design: a review of forty years of design research. Design Issues, 20(1), 16–29.

    Article  Google Scholar 

  • Beevers, C., Wild, D., McGuire, G., Fiddes, D., & Youngson, M. (1999). Issues of partial credit in mathematical assessment by computer. Research in Learning Technology, 7(1), 26–32.

  • Black, P., & Wiliam, D. (2005). Lessons from around the world: how policies, politics and cultures constrain and afford assessment practices. Curriculum Journal, 16(2), 249–261. doi:10.1080/09585170500136218.

    Article  Google Scholar 

  • Bloxham, S., & Boyd, P. (2007). Developing Effective Assessment In Higher Education: A Practical Guide: A Practical Guide, McGraw-Hill Education, UK.

  • Borgman, C. L. (2012). The conundrum of sharing research data. Journal of the American Society for Information Science and Technology, 63(6), 1059–1078.

    Article  Google Scholar 

  • Boyd, N. S. (2010). Domain vocabulary. Retrieved 2013/12/19, from http://educery.com/educe/patterns/domain-vocabulary.html.

  • Chatterjee, S. (2010). Design research in information systems: theory and practice (Vol. 22). Springer.

  • Creswell, J. (2003). Research design: Qualitative, quantitative, and mixed methods approaches. Boston: SAGE Publications.

    Google Scholar 

  • Cross, N. (1984). Developments in design methodology. Chichester: John Wiley & Sons Ltd.

    Google Scholar 

  • Davis, D. (2007). Complex software problem solving by means of abstractive techniques. Paper presented at the Proceedings of the 11th international conference on Computer aided systems theory, Las Palmas de Gran Canaria, Spain.

  • de Jong, T. (2014). Emerging representation technologies for problem solving. In: Spector, J. M., Merrill, M. D., Elen, J. & Bishop, M. J. (Eds.), Handbook of Research on Educational Communications and Technology (pp. 809–816). New York: Springer.

  • Faste, T., & Faste, H. (2012). Demystifying “design research”: design is not research, research is design, IDSA Education Symposium.

  • Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. The Journal of Problem Solving, 4(1), 19–42. doi:10.7771/1932-6246.1118.

  • Funke, J. (2010). Complex problem solving: a case for complex cognition? Gov’t review. Cognitive Processing, 11(2), 133–142.

  • Gears, C. (2005). Classroom assessment: minute by minute, day by day. Assessment, 63(3), 19–24.

  • Grosan, C., & Abraham, A. (2011). Rule-based expert systems. Intelligent Systems. Springer.

  • Hevner, A. R. (2007). A three cycle view of design science research. Scandinavian journal of information systems, 19(2), 87–92.

  • Hevner, A., & Chatterjee, S. (2010). Design research in information systems: Theory and practice. New York: Springer Science & Business Media.

    Book  Google Scholar 

  • Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.

    Google Scholar 

  • Hwang, G.-J., Chen, C.-Y., Tsai, P.-S., & Tsai, C.-C. (2011). An expert system for improving web-based problem-solving ability of students. Expert Systems with Applications, 38, 8664–8672.

    Article  Google Scholar 

  • Jaques, P. A., Seffrin, H., Rubi, G., Morais, F. D., Guilardi, C., Bittencourt, I. I., & Isotani, S. (2013). Rule-based expert systems to support step-by-step guidance in algebraic problem solving: the case of the tutor PAT2Math. Expert Systems with Applications, 40(14), 5456–5465.

    Article  Google Scholar 

  • Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: a research paradigm whose time has come. Educational Researcher, 33(7), 14–26.

    Article  Google Scholar 

  • Jones, I. S. (2008). Computer-aided assessment questions in engineering mathematics usingMapleTA®. International Journal of Mathematical Education in Science and Technology, 39(3), 341–356. doi:10.1080/00207390701734523.

    Article  Google Scholar 

  • Kaidar, S. M., Hussain, R. I., Bohani, F. A., Sahran, S., Binti Zainuddin, N., Ismail, F., Thanabalan, J., Kalimuthu, G., & Abdullah, S. N. H. S. (2013). Brain tumor treatment advisory system. Soft Computing Applications and Intelligent Systems. Springer.

  • Kamel Boulos, M. N. (2012). Expert system shells for rapid clinical decision support module development: an ESTA demonstration of a simple rule-based system for the diagnosis of vaginal discharge. Healthcare Information Research, 18, 252–258.

    Article  Google Scholar 

  • Khan, F. S., Razzaq, S., Irfan, K., Maqbool, F., Farid, A., Illahi, I., & Ul Amin, T. (2008). Wheat: A Web-based expert system for diagnosis of diseases and pests in Pakistani wheat. Proceedings of the World Congress on Engineering, Oct 2008, 2–4.

  • Kirsh, D. (2010). Thinking with external representations. AI & SOCIETY, 25(4), 441–454. doi:10.1007/s00146-010-0272-8.

    Article  Google Scholar 

  • Lawson, D. (2012). Computer-aided assessment in mathematics: panacea or propaganda? International Journal of Innovation in Science and Mathematics Education (formerly CAL-laborate International), 9(1), 6–12.

  • Livne, N. L., Livne, O. E., & Wight, C. A. (2007). Can automated scoring surpass hand grading of students’ constructed responses and error patterns in mathematics. MERLOT Journal of Online Learning and Teaching, 3(3), 295–306.

    Google Scholar 

  • March, S. T., & Smith, G. F. (1995). Design and natural science research on information technology. Decision Support Systems, 15(4), 251–266.

    Article  Google Scholar 

  • Miah, S. J., Kerr, D., & Gammack, J. (2009). A methodology to allow rural extension professionals to build target specific expert systems for Australian rural business operators. Expert Systems with Applications, 36(1), 735–744.

    Article  Google Scholar 

  • Miah, S. J., Kerr, D., & Von-Hellens, L. (2014). A collective artefact design of decision support systems: design science research perspective. Information Technology & People, 27(3), 259–279.

    Article  Google Scholar 

  • Offermann, P., Blom, S., Schönherr, M., & Bub, U. (2010). Artifact types in information systems design science–a literature review. In: Winter, R., Zhao, J. L. & Aier, S. (Eds.), Global perspectives on design science research (pp. 77–92). Springer Berlin Heidelberg.

  • Österle, H., Becker, J., Frank, U., Hess, T., Karagiannis, D., Krcmar, H., et al. (2011). Memorandum on design-oriented information systems research. European Journal of Information Systems, 20(1), 7–10.

    Article  Google Scholar 

  • Palm, T. (2008). Performance assessment and authentic assessment: a conceptual analysis of the literature. Practical Assessment, Research & Evaluation, 13(4), 1–11.

    Google Scholar 

  • Passmore, T., Brookshaw, L., & Butler, H. (2011). A flexible, extensible online testing system for mathematics. Australasian Journal of Educational Technology, 27(6), 896–906.

    Article  Google Scholar 

  • Pereira, R., Almeida, R., & da Silva, M. M. (2013). How to generalize an information technology case study. In: Vom Brocke, J., Hekkala, R., Ram, S. & Rossi, M. (Eds.), Design science at the intersection of physical and virtual design (pp. 150–164). Springer Berlin Heidelberg.

  • Price, M., Carroll, J., O’Donovan, B., & Rust, C. (2011). If I was going there I wouldn’t start from here: a critical commentary on current assessment practice. Assessment & Evaluation in Higher Education, 36(4), 479–492.

    Article  Google Scholar 

  • Sangwin, C. (2012). Computer aided assessment of mathematics using STACK. Paper presented at the Proceedings of ICME.

  • Sangwin, C., Cazes, C., Lee, A., & Wong, K. L. (2010). Micro-level automatic assessment supported by digital technologies. Hoyles, C. & Lagrange, J.-B. (Eds.), Mathematics Education and Technology-Rethinking the Terrain, (pp. 227-250). Springer USA. 13, 227–250.

  • Shepard, L. A. (2005). Linking formative assessment to scaffolding. Educational Leadership, 63(3), 66–70.

    Google Scholar 

  • Sonnleitner, P., Keller, U., Martin, R., & Brunner, M. (2013). Students’ complex problem-solving abilities: their structure and relations to reasoning ability and educational success. Intelligence, 41(5), 289–305. doi:10.1016/j.intell.2013.05.002.

    Article  Google Scholar 

  • Vaishnavi, V. K., & Kuechler Jr, W. (2007). Design science research methods and patterns: innovating information and communication technology. CRC Press.

  • Van den Broeck, J., Brestoff, J. R., & Chhagan, M. (2013). Maintaining data integrity. In: Van den Broeck, J. & Brestoff, J. R. (Eds.), Epidemiology: principles and practical guidelines (pp. 379–388). UK: Springer.

  • Wiliam, D. (2011). What is assessment for learning? Studies in Educational Evaluation, 37(1), 3–14. doi:10.1016/j.stueduc.2011.03.001.

    Article  Google Scholar 

  • Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving—more than reasoning? Intelligence, 40(1), 1–14. doi:10.1016/j.intell.2011.11.003.

    Article  Google Scholar 

  • Yeo, R. K., & Marquardt, M. J. (2012). Complex problem solving through action learning: implications for human resource development. International Journal of Human Resources Development and Management, 12(4), 258–273.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Engineering and Sciences, Victoria University, Footscray Park Campus, Melbourne, Australia

    Hussein Genemo & Alasdair McAndrew

  2. College of Business, Victoria University, Footscray Park Campus, Melbourne, Australia

    Shah Jahan Miah

Authors
  1. Hussein Genemo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shah Jahan Miah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alasdair McAndrew
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shah Jahan Miah.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Genemo, H., Miah, S.J. & McAndrew, A. A design science research methodology for developing a computer-aided assessment approach using method marking concept. Educ Inf Technol 21, 1769–1784 (2016). https://doi.org/10.1007/s10639-015-9417-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-015-9417-1

Keywords

Search

Navigation