Abstract
A modeling system may be required to predict an agent's future actions under constraints of inadequate or contradictory relevant historical evidence. This can result in low prediction accuracy, or otherwise, low prediction rates, leaving a set of cases for which no predictions are made. A previous study that explored techniques for improving prediction rates in the context of modeling students' subtraction skills using Feature Based Modeling showed a tradeoff between prediction rate and predication accuracy. This paper presents research that aims to improve prediction rates without affecting prediction accuracy. The FBM-C4.5 agent modeling system was used in this research. However, the techniques explored are applicable to any Feature Based Modeling system, and the most effective technique developed is applicable to most agent modeling systems. The default FBM-C4.5 system models agents' competencies with a set of decision trees, trained on all historical data. Each tree predicts one particular aspect of the agent's action. Predictions from multiple trees are compared for consensus. FBM-C4.5 makes no prediction when predictions from different trees contradict one another. This strategy trades off reduced prediction rates for increased accuracy. To make predictions in the absence of consensus, three techniques have been evaluated. They include using voting, using a tree quality measure and using a leaf quality measure. An alternative technique that merges multiple decision trees into a single tree provides an advantage of producing models that are more comprehensible. However, all of these techniques demonstrated the previous encountered trade-off between rate of prediction and accuracy of prediction, albeit less pronounced. It was hypothesized that models built on more current observations would outperform models built on earlier observations. Experimental results support this hypothesis. A Dual-model system, which takes this temporal factor into account, has been evaluated. This fifth approach achieved a significant improvement in prediction rate without significantly affecting prediction accuracy.
Similar content being viewed by others
References
Ali, K., Brunk, C., and Pazzani, M.: 1994, On learning multiple descriptions of a concept. In Proceedings of the Sixth International Conference on Tools with Artificial Intelligence, New Orleans, LA: IEEE Press, 476-483.
Amato, N. H. and Tsang, C. P.: 1990, Student modeling in a keyboard scale tutoring system. In C. J. Barter & M. J. Brooks (Eds.) Proceedings of the Second Australian Joint Conference on Artificial Intelligence, Berlin: Springer-Verlag, 225-239.
Anderson, J. R., Boyle, C. F., and Reiser, B. J.: 1985, Intelligent tutoring systems. Science, 228 456-462.
Anderson, J. R., Boyle, C. F., Corbett, A. T., and Lewis, M. W.: 1990, Cognitive modeling and intelligent tutoring. Artificial Intelligence, 42 7-49.
Baffes, P., and Mooney, R.: 1996, Refinement-based student modeling and automated bug library construction. Journal of Artificial Intelligence in Education, 7(1) 75-117.
Balabanovic, M.: 1998, Exploring versus exploiting when learning user models for text recommendation. In this volume.
Breiman, L.: 1996, Bagging predictors. Machine Learning, 24 123-140.
Brown, J. S., and VanLehn, K.: 1980, Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4 379-426.
Brown, J. S., and Burton, R. R.: 1978, Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 2 155-192.
Burton, R. R, and Brown, J. S.: 1976, A tutoring and student modeling paradigm for gaming environments. Computer Science and Education. ACM SIGCSE Bulletin, 8(1) 236-246.
Carbonell, J. R.: 1970, AI in CAI: An artificial intelligence approach to computer-assisted instruction. IEEE Transactions on Man-Machine Systems, 11(4) 190-202.
Chan, P. K., and Stolfo, S. J.: 1995, A comparative evaluation of voting and meta-learning on Partitioned data. In Prieditis, A., and Russell, S., eds., Proceedings of the 12th International Conference on Machine Learning, 90-98.
Chiu, B. C., Webb, G. I., and Kuzmycz, M.: 1997, A comparison of first-order and zeroth-order induction for input-output agent modeling. In Jameson A., Paris C., and Tasso C., eds., Proceedings of the Sixth International Conference on User Modeling, UM97, 347-358.
Corbett, A. T., and Anderson, J. R.: 1992, Student modeling and mastery learning in a computer-based programming tutor. In Frasson, C., Gauthier, G., and McCalla, G. I., eds., Intelligent Tutoring Systems. Berlin: Springer-Verlag, 413-420.
Desmoulins, C., and Van Labeke, N.: 1996, Towards student modeling in geometry with inductive logic programming. In Brna, P., Paiva, A., and Self, J., eds., Proceedings of the European Conference on Artificial Intelligence in Education, 94-100.
Dietterich, T., and Bakiri, G.: 1994, Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2263-286.
Giangrandi, P., and Tasso, C.:1995, Truth maintenance techniques for modeling students' behavior. Journal of Artificial Intelligence in Education, 6(2/3) 153-202.
Giangrandi, P., and Tasso, C.: 1996, Modeling the temporal evolution of student's knowledge. In Brna, P., Paiva, A., and Self, J., eds., Proceedings of the European Conference on Artificial Intelligence in Education, 184-190.
Gilmore, D., and Self, J.: 1988, The application of machine learning to intelligent tutoring systems. In Self, J., ed., Artificial Intelligence and Human Learning: Intelligent Computer-aided Instruction. London: Chapman and Hall, 179-196.
Goldstein, I. P.: 1979, The genetic graph: A representation for the evolution of procedural knowledge. International Journal of Man-machine Studies, 11(1) 51-77.
Hoppe, H. U.: 1994, Deductive error diagnosis and inductive error generalization for intelligent tutoring systems. Journal of Artificial Intelligence in Education, 5(1) 27-49.
Heath D., Kasif, S. and Salzberg, S.: 1996, Committees of decision trees. In Gorayska, B., and Mey, J., eds., Cognitive Technology: In Search of a Human Interface. Amsterdam: Elsevier Science B.V., 305-317.
Ikeda, M., Kono, Y., and Mizoguchi, R.: 1993, Nonmonotonic model inference: A formalization of student modeling. Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence: IJCAI'93, 467-473.
Kohavi, R.: 1995, A study of cross-validation and bootstrap for accuracy estimation and model selection. In Mellish, C. S., ed., Proceedings of the 14th International Joint Conference on Artificial Intelligence. Morgan Kaufmann, 1137-1145.
Kuzmycz, M.: 1994, A dynamic vocabulary for student modeling. Proceedings of the Fourth International Conference on User Modeling, Hyannis, MA, 185-190.
Kuzmycz, M.: 1997, Resolving conflicting knowledge in student models. Proceedings of the Eighth World Conference on Artificial Intelligence in Education. Amsterdam: IOS Press, 522-529.
Kuzmycz, M., and Webb, G. I.: 1992, Evaluation of feature based modelling in subtraction. In Frasson, C., Gauthier, G., and McCalla, G. I., eds., Intelligent Tutoring Systems. Berlin: Springer-Verlag, 269-276.
Kwok, S., and Carter, C.: 1990, Multiple decision trees. Uncertainty in Artificial Intelligence, 4 327-335.
Langley, P., and Ohlsson, S.: 1984, Automated cognitive modeling. Proceedings of the National Conference on Artificial Intelligence, Austin, Texas, 193-197.
Langley, P., Wogulis, J., and Ohlsson, S.: 1990, Rules and principles in cognitive diagnosis. Diagnostic Monitoring of Skill and Knowledge Acquisition. Hillsdale, NJ: Erlbaum, 217-250.
Martin, J., and VanLehn, K.: 1995, Student assessment using Bayesian nets. International Journal of Human-Computer Studies, 42(6) 575-591.
Nock, R., and Cascuel, O.: 1995, On learning decision committees. Proceedings of the Twelfth International Conference on Machine Learning, San Mateo, CA: Morgan Kaufmann. 413-420.
Ohlsson, S., and Langley, P.: 1985, Identifying solution paths in cognitive diagnosis. Technical Report CMU-RI-TR-85-2, Carnegie-Mellon University, Pittsburgh, PA.
Oliver, J. J., and Hand, D. J.: 1995, On pruning and averaging decision trees. Proceedings of the Twelfth International Conference on Machine Learning, San Mateo, CA: Morgan Kaufmann, 430-437.
Quinlan, J. R.: 1986, The effect of noise on concept learning. In Michalski, R. S., Carbonell. G. and Mitchell T.M. eds., Machine Learning: An Artificial Intelligence Approach, Vol. 2. San Mateo, CA: Morgan Kaufmann, 149-166.
Quinlan, J. R.: 1993, C4.5: Programs for Machine Learning San Mateo, CA: Morgan Kaufmann.
Salvia, A. A.: 1990, Introduction to Statistics. Saunders College Pub., Philadelphia.
Schapire, R. E.: 1990, The strength of weak learnability. Machine Learning, 5 197-227.
Sleeman, D.: 1982, Assessing aspects of competence in basic algebra. In Sleeman, D. H., and Brown, J. S., eds., Intelligent Tutoring Systems. London: Academic Press, 185-199.
Sleeman, D., Ward, R. D., Kelly, E., Martinak, R., and Moore, J.: 1991, An overview of recent studies with Pixie. In Goodyear, P., ed., Teaching Knowledge and Intelligent Tutoring. Norwood, NJ: Ablex, 173-185.
Ting, K. M., and Low, B. T.: 1997, Model combination in the multiple-data-batches scenario. Proceedings of the Ninth European Conference on Machine Learning, Lecture Notes in Artificial Intelligence, Vol. 1224, Berlin: Springer-Verlag, 250-265.
Webb, G. I.: 1989, A machine learning approach to student modeling. Proceedings of the Third Australian Joint Conference on Artificial Intelligence, Melbourne, 195-205.
Webb, G. I.: 1991, Inside the Unification Tutor: The architecture of an intelligent educational system. In Proceedings of the Fourth Australian Society for Computers in Learning in Tertiary Education Conference, Launceston, 677-684.
Webb, G. I., Chiu, B. C., and Kuzmycz, M.: 1997, Comparative evaluation of alternative induction engines for Feature Based Modelling. International Journal of Artificial Intelligence in Education, 8, to be printed.
Webb, G. I., and Kuzmycz, M.: 1996, Feature based modeling: A methodology for producing coherent, dynamically changing models of agent's competencies. User Modeling and User-Adapted Interaction, 5(2) 117-150.
Wolpert, D. H.: 1992, Stacked generalisation. Neural Networks, 5 241-259.
Young, R., and O'Shea, T.: 1981, Errors in children's subtraction. Cognitive Science, 5(1) 153-177.
Zheng, Z., and Webb, G. I.: 1997, Lazy Bayesian Tree. Technical Report TC97/07, School of computing and mathematics, Deakin University.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Cheung Chiu, B., Webb, G.I. Using Decision Trees for Agent Modeling: Improving Prediction Performance. User Modeling and User-Adapted Interaction 8, 131–152 (1998). https://doi.org/10.1023/A:1008296930163
Issue Date:
DOI: https://doi.org/10.1023/A:1008296930163