Abstract
In this work, we explore weakly supervised machine learning for classifying questions into distinct Bloom’s Taxonomy levels. Bloom’s levels provide important information that guides teachers and adaptive learning algorithms in selecting appropriate questions for their students. However, manually providing Bloom labels is expensive and labor-intensive, which motivates a machine learning approach. Current automated Bloom’s level classification methods employ supervised learning that relies on large labeled datasets that are difficult and costly to construct. In this paper, we propose a weakly supervised learning method that performs binary Bloom’s level labeling without any a priori known Bloom’s taxonomy labels. The key idea behind BLACBOARD (for Bloom’s Level clAssifiCation Based On weAkly supeRviseD learning) is to appropriately incorporate human domain knowledge into the modeling process to produce a weakly labeled dataset on which discriminative models can then be trained. We compare BLACBOARD to fully supervised learning methods and show that it achieves little to no performance compromise while using entirely unlabeled data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A demonstration and associated code of BLACBOARD are available at https://github.com/manningkyle304/edu-research-demo.
- 2.
- 3.
We also experimented with other featurization methods, but the results were similar to TF-IDF. We thus use TF-IDF for all experiments in this paper.
References
Swart, A.J.: Evaluation of final examination papers in engineering: a case study using Bloom’s taxonomy. IEEE Trans. Educ. 53(2), 257–264 (2010)
Blumberg, P.: Maximizing learning through course alignment and experience with different types of knowledge. Innovative High. Educ. 34(2), 93–103 (2009)
Bloom, B., Engelhart, M., Furst, E., Hill, W., Krathwohl, D.: Taxonomy of educational objectives: the classification of educational goals. Handbook 1: cognitive domain. David McKay, New York (1956)
Krathwohl, D.R.: A revision of bloom’s taxonomy : an overview. Theory Pract. 41(4), 212–218 (2002)
Abduljabbar, D.A., Omar, N.: Exam questions classification based on Bloom’s taxonomy cognitive level using classifiers combination. J. Theor. Appl. Inf. Technol. 78(3), 447 (2015)
Osadi, K., Fernando, M., Welgama, W., et al.: Ensemble classifier based approach for classification of examination questions into Bloom’s taxonomy cognitive levels. Int. J. Comput. Appl. 162(4), 1–6 (2017)
Yahya, A.A., Osman, A.: Automatic classification of questions into Bloom’s cognitive levels using support vector machines. In: International Arab Conference on Information Technology (2011)
Jayakodi, K., Bandara, M., Perera, I.: An automatic classifier for exam questions in engineering: a process for Bloom’s taxonomy. In: IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 195–202 (2015)
Mohamed, O.J., Zakar, N.A., Alshaikhdeeb, B.: A combination method of syntactic and semantic approaches for classifying examination questions into bloom’s taxonomy cognitive. J. Eng. Sci. Technol. 14(2), 935–950 (2019)
Mohammed, M., Omar, N.: Question classification based on Bloom’s taxonomy using enhanced TF-IDF. Int. J. Adv. Sci. Eng. Inf. Technol. 8, 1679–1685 (2018)
Osman, A., Yahya, A.A.: Classifications of exam questions using natural language syntatic features: a case study based on Bloom’s taxonomy. In: Proceedings of the International Arab Conference on Quality Assurance in Higher Education (2016)
Sangodiah, A., Ahmad, R., Ahmad, W.F.W.: A review in feature extraction approach in question classification using support vector machine. In: IEEE International Conference on Control System, Computing and Engineering, pp. 536–541 (2014)
Pota, M., Esposito, M., De Pietro, G.: A forward-selection algorithm for SVM-based question classification in cognitive systems. In: De Pietro, G., Gallo, L., Howlett, R.J., Jain, L.C. (eds.) Intelligent Interactive Multimedia Systems and Services 2016. SIST, vol. 55, pp. 587–598. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39345-2_52
Supriyanto, C., Yusof, N., Nurhadiono, B., et al.: Two-level feature selection for Naive Bayes with kernel density estimation in question classification based on Bloom’s cognitive levels. In: International Conference on Information Technology and Electrical Engineering, pp. 237–241 (2013)
Diab, S., Sartawi, B.: Classification of questions and learning outcome statements (LOS) into Blooms taxonomy (BT) by similarity measurements towards extracting of learning outcome from learning material. arXiv preprint (2017)
Zoller, U.: Are lecture and learning compatible? Maybe for LOCS: unlikely for HOCS. J. Chem. Edu. 70(3), 195 (1993)
Karamustafaoğlu, S., Sevim, S., Karamustafaoğlu, O., Cepni, S.: Analysis of Turkish high-school chemistry-examination questions according to bloom’s taxonomy. Chem. Educ. Res. Pract. 4(1), 25–30 (2003)
Crowe, A., Dirks, C., Wenderoth, M.P.: Biology in Bloom: implementing bloom’s taxonomy to enhance student learning in biology. CBE-Life Sci. Educ. 7(4), 368–381 (2008)
Baker, E.L.: Developing comprehensive assessments of higher order thinking. Assess. High. Order Think. Math. 7, 20 (1990)
Herrington, J., Oliver, R.: Using situated learning and multimedia to investigate higher-order thinking. J. Interact. Learn. Res. 10(1), 3–24 (1999)
Ratner, A., et al.: Snorkel: rapid training data creation with weak supervision. VLDB J. 29(2), 709–730 (2019). https://doi.org/10.1007/s00778-019-00552-1
Varma, P., Ré, C.: Snuba: automating weak supervision to label training data. Proc. VLDB Endow. 12(3), 223–236 (2018)
Varma, P., He, B.D., Bajaj, P., Khandwala, N., Banerjee, I., Rubin, D., Ré, C.: Inferring generative model structure with static analysis. Adv. Neural. Inf. Process. Syst. 30, 240–250 (2017)
Bach, S.H., He, B., Ratner, A., Ré, C.: Learning the structure of generative models without labeled data. In: Proceedings of the International Conference on Machine Learning, pp. 273–282 (2017)
Fries, J.A., et al.: Trove: ontology-driven weak supervision for medical entity classification. arXiv e-prints, August 2020
Fries, J.A., et al.: Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences. Nature Commun. 10(1), 3111 (2019)
Callahan, A., Fries, J.A., Ré, C., Huddleston, J.I., Giori, N.J., Delp, S., Shah, N.H.: Medical device surveillance with electronic health records. NPJ Digit. Med. 2(1), 94 (2019)
Kuleshov, V., et al.: A machine-compiled database of genome-wide association studies. Nature Commun. 10(1), 3341 (2019)
Datta, D., Phillips, M., Chiu, J., Watson, G.S., Bywater, J.P., Barnes, L., Brown, D.: Improving classification through Weak supervision in context-specific conversational agent development for teacher education. arXiv e-prints, October 2020
Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill Inc, New York, USA (1986)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer New York Inc., New York, USA (2001)
Bishop, C.M.: Pattern Recognition and Machine Learning (information science and statistics). Springer-Verlag, Berlin, Heidelberg (2006)
Wang, Y., Zheng, J., Liu, Q., Zhao, Z., Xiao, J., Zhuang, Y.: Weak supervision enhanced generative network for question generation. arXiv preprint (2019)
Datta, D., et al.: Improving classification through weak supervision in context-specific conversational agent development for teacher education. arXiv preprint (2020)
Ramesh, A., Kumar, S.H., Foulds, J., Getoor, L.: Weakly supervised models of aspect-sentiment for online course discussion forums. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 74–83 (2015)
Qiu, Y., Wang, Y., Jin, X., Zhang, K.: Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 474–482 (2020)
Acknowledgements
This work was supported by NSF grants 1842378 and 1937134 and by ONR grant N0014-20-1-2534. We thank Prof. Colleen Countryman (Ithaca College), Prof. Lauren Rast (The University of Alabama at Birmingham), Joyce Spangler (Six Red Marbles), and Andrew Giannakakis (OpenStax) for helpful discussions on the Labeling Functions. Thanks to Fred Sala for insights on WSL. Thanks to anonymous reviewers and CJ Barberan for suggestions on the manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Z., Manning, K., Mallick, D.B., Baraniuk, R.G. (2021). Towards Blooms Taxonomy Classification Without Labels. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2021. Lecture Notes in Computer Science(), vol 12748. Springer, Cham. https://doi.org/10.1007/978-3-030-78292-4_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-78292-4_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78291-7
Online ISBN: 978-3-030-78292-4
eBook Packages: Computer ScienceComputer Science (R0)