Abstract
Question Classification (QC) is critical in many natural language applications, especially factoid question answering (QA). Although a substantial number of studies have addressed QC, most of them have focused on statistical QC. The consensus in the literature is that rule-based QC is high cost and unportable, however, the fact is that rule-based QC is still the primary method in most top-performing factoid QA systems.
Just as statistical QC needs proper feature engineering, we argue that rule-based QC should be based on proper knowledge engineering guidelines, an aspect that has been overlooked in QC works thus far. To address this gap in the literature, we conducted a statistical case study of rule-based and statistical QC. We performed paired t-tests of the classifiers on several heterogeneous unseen datasets, which showed that rule-based QC significantly outperformed statistical QC in terms of fine-grained accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Forner, P., Peñas, A., Agirre, E., Alegria, I., Forăscu, C., Moreau, N., Osenova, P., Prokopidis, P., Rocha, P., Sacaleanu, B., Sutcliffe, R., Sang, E.T.K.: Overview Of The CLEF 2008 Multilingual Question Answering Track. In: Cross-Language Evaluation Forum (CLEF) Workshop (2008)
Dang, H.T., Kelly, D., Lin, J.: Overview of the TREC 2007 Question Answering Track. In: Proceedings of The Sixteenth Text REtrieval Conference, TREC (2007)
Mitamura, T., Nyberg, E., Shima, H., Kato, T., Mori, T., Lin, C.-Y., Song, R., Lin, C.-J., Sakai, T., Ji, D., Kando, N.: Overview of the NTCIR-7 ACLIA Tasks: Advanced Cross-Lingual Information Access. In: The 7th NTCIR Workshop (2008)
Li, B., Liu, Y., Ram, A., Garcia, E.V., Agichtein, E.: Exploring Question Subjectivity Prediction in Community QA. In: Proceedings of 31st Annual International ACM SIGIR Conference, pp. 735–736 (2008)
Moldovan, D., Paşca, M., Harabagiu, S., Surdeanu, M.: Performance Issues and Error Analysis in an Open-Domain Question Answering System. ACM Transactions on Information Systems 21, 133–154 (2003)
Lee, C.-W., Day, M.-Y., Sung, C.-L., Lee, Y.-H., Jiang, T.-J., Wu, C.-W., Shih, C.-W., Chen, Y.-R., Hsu, W.-L.: Chinese-Chinese and English-Chinese Question Answering with ASQA at NTCIR-6 CLQA. In: Proceedings of NTCIR-6 Workshop, Tokyo, Japan, pp. 175–181 (2007)
Laurent, D., Séguéla, P., Nègre, S.: Cross Lingual Question Answering using QRISTAL for CLEF 2006. In: CLEF (2006)
Moldovan, D., Clark, C., Bowden, M.: Lymba’s PowerAnswer 4 in TREC 2007. In: The Sixteenth Text REtrieval Conference, TREC 2007 (2007)
Prager, J., Brown, E., Coden, A., Radev, D.: Question-Answering by Predictive Annotation. In: Proceedings of the 23rd Annual International ACM SIGIR Conference, pp. 184–191 (2000)
Pasca, M.A., Harabagiu, S.M.: High Performance Question/Answering. In: Proceedings of the 24th Annual International ACM SIGIR Conference, pp. 366–374 (2001)
Hull, D.A.: Xerox TREC-8 Question Answering Track Report. In: Proceedings of the 8th Text Retrieval Conference (TREC-8), vol. 8, pp. 743–752 (1999)
Li, X., Roth, D.: Learning Question Classifiers. In: International Conference on Computational Linguistics, Taipei, Taiwan, pp. 1–7 (2002)
Zhang, D., Lee, W.S.: Question Classification Using Support Vector Machines. In: Proceedings of the 26th Annual International ACM SIGIR Conference, pp. 26–32 (2003)
Suzuki, J., Taira, H., Sasaki, Y., Maeda, E.: Question Classification Using HDAG Kernel. In: Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering, pp. 61–68 (2003)
Radev, D., Fan, W., Qi, H., Wu, H., Grewal, A.: Probabilistic Question Answering on the Web. In: Proceedings of the 11th International Conference on World Wide Web, pp. 408–419 (2002)
Day, M.-Y., Lee, C.-W., Wu, S.-H., Ong, C.-S., Hsu, W.-L.: An Integrated Knowledge-based and Machine Learning Approach for Chinese Question Classification. In: IEEE International Conference on Natural Language Processing and Knowledge Engineering, pp. 620–625 (2005)
Hsu, W.-L., Wu, S.-H., Chen, Y.-S.: Event identification based on the information map-INFOMAP. In: IEEE International Conference on Systems, Man, and Cybernetics, Tucson, AZ, USA, vol. 3, pp. 1661–1666 (2001)
Wu, S.-H., Day, M.-Y., Hsu, W.-L.: FAQ-centered Qrganizational Memory. In: Knowledge Management and Organizational Momery Workshop on the Seventeenth International Joint Conference on Artificial Intelligence, pp. 112–120 (2001)
Beck, K.: Test-Driven Development by Example. Addison-Wesley Professional, Reading (2002)
Moschitti, A., Quarteroni, S., Basili, R., Manandhar, S.: Exploiting Syntactic and Shallow Semantic Kernels for Question Answer Classification. In: ACL, pp. 776–783 (2007)
Metzler, D., Croft, W.B.: Analysis of Statistical Question Classification for Fact-Based Questions. Information Retrieval 8, 481–504 (2005)
Joachims, T.: Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In: Proceedings of the European Conference on Machine Learning, pp. 137–142 (1998)
CKIP: Autotag. Academia Sinica (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lee, CW., Day, MY., Hsu, WL. (2010). An Empirical Comparative Study of Manual Rule-Based and Statistical Question Classifiers on Heterogeneous Unseen Data. In: Cheng, PJ., Kan, MY., Lam, W., Nakov, P. (eds) Information Retrieval Technology. AIRS 2010. Lecture Notes in Computer Science, vol 6458. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17187-1_34
Download citation
DOI: https://doi.org/10.1007/978-3-642-17187-1_34
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17186-4
Online ISBN: 978-3-642-17187-1
eBook Packages: Computer ScienceComputer Science (R0)