Skip to main content

Advertisement

SpringerLink
Log in

Multiobjective optimization for classifier ensemble and feature selection: an application to named entity recognition

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

In this paper, the concept of finding an appropriate classifier ensemble for named entity recognition is posed as a multiobjective optimization (MOO) problem. Our underlying assumption is that instead of searching for the best-fitting feature set for a particular classifier, ensembling of several classifiers those are trained using different feature representations could be a more fruitful approach, but it is crucial to determine the appropriate subset of classifiers that are most suitable for the ensemble. We use three heterogenous classifiers namely maximum entropy, conditional random field, and support vector machine in order to build a number of models depending upon the various representations of the available features. The proposed MOO-based ensemble technique is evaluated for three resource-constrained languages, namely Bengali, Hindi, and Telugu. Evaluation results yield the recall, precision, and F-measure values of 92.21, 92.72, and 92.46%, respectively, for Bengali; 97.07, 89.63, and 93.20%, respectively, for Hindi; and 80.79, 93.18, and 86.54%, respectively, for Telugu. We also evaluate our proposed technique with the CoNLL-2003 shared task English data sets that yield the recall, precision, and F-measure values of 89.72, 89.84, and 89.78%, respectively. Experimental results show that the classifier ensemble identified by our proposed MOO-based approach outperforms all the individual classifiers, two different conventional baseline ensembles, and the classifier ensemble identified by a single objective–based approach. In a part of the paper, we formulate the problem of feature selection in any classifier under the MOO framework and show that our proposed classifier ensemble attains superior performance to it.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cunningham H.: GATE, a general architecture for text engineering. Comput. Human. 36, 223–254 (2002)

    Article  Google Scholar 

  2. Babych, B., Hartley, A.: Improving machine translation quality with automatic named entity recognition. In: Proceedings of EAMT/EACL 2003 Workshop on MT and other Language Technology Tools, pp. 1–8 (2003)

  3. Moldovan, D., Harabagiu, S., Girju, R., Morarescu, P., Lacatusu, F., Novischi, A., Badulescu, A., Bolohan, O.: LCC tools for question answering. In: Text REtrieval Conference (TREC) (2002)

  4. Nobata, C., Sekine, S., Isahara, H., Grishman, R.: Summarization system integrated with named entity tagging and IE pattern discovery. In: Proceedings of Third International Conference on Language Resources and Evaluation (LREC 2002), Spain (2002)

  5. Humphreys, K., Gaizauskas, R., Azzam, S., Huyck, C., Mitchell, B., Cunnigham, H., Wilks, Y.: University of sheffield: description of the LaSIE-II system as used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)

  6. Aone, C., Halverson, L., Hampton, T., Ramos-Santacruz, M.: SRA: description of the IE2 system used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)

  7. Mikheev, A., Grover, C., Moens, M.: Description of the LTG system used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)

  8. Mikheev, A., Grover, C., Moens, M.: Named entity recognition without gazeteers. In: Proceedings of EACL, Bergen, Norway, pp. 1–8 (1999)

  9. Miller, S., Crystal, M., Fox, H., Ramshaw, L., Schawartz, R., Stone, R., Weischedel, R., The Annotation~Group: BBN: Description of the SIFT system as used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)

  10. Bikel D.M., Schwartz R.L., Weischedel R.M.: An algorithm that learns what’s in a name. Mach. Learn. 34(1–3), 211–231 (1999)

    Article  MATH  Google Scholar 

  11. Borthwick, A.: Maximum entropy approach to named entity recognition. PhD thesis, New York University (1999)

  12. Borthwick, A., Sterling, J., Agichtein, E., Grishman, R.: NYU: description of the MENE named entity system as used in MUC-7. In: MUC-7, Fairfax (1998)

  13. Sekine, S.: Description of the Japanese NE system used for MET-2. In: MUC-7, Fairfax, Virginia (1998)

  14. McCallum, A., Li, W.: Early results for named entity recognition with conditional random fields, Feature Induction and Web-enhanced Lexicons. In: Proceedings of CoNLL, Canada, pp. 188–191 (2003)

  15. Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: ICML, pp. 282–289 (2001)

  16. Collins, M., Singer, Y.: Unsupervised models for named entity classification. In: Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (1999)

  17. Yangarber, R., Lin, W., Grishman, R.: Unsupervised learning of generalized names. In: Proceedings of the 19th International Conference on Computational Linguistics (COLING-2002), pp. 1–7 (2002)

  18. Miller, S., Guinness, J., Zamanian, A.: Name tagging with word clusters and discriminative training. In: Proceedings of HLT-NAACL, USA (2004)

  19. Ji, H., Grishman, R.: Data selection in semi-supervised learning for name tagging. In: Proceedings of the Workshop on Information Extraction Beyond The Document, COLING/ACL, Sydney (2006)

  20. Alfonseca, E., Manandhar, S.: An unsupervised method for general named entity recognition and automated concept discovery. In: Proceedings AAAI ’99/IAAI ’99: Proceedings of the Sixteenth National Conference on Artificial Intelligence and the Eleventh Conference on Innovative Applications of Artificial Intelligence, pp. 474–479 (1999)

  21. Shinyama, Y., Sekine, S.: Named entity discovery using comparable news articles. In: Proceedings of the International Conference on Computational Linguistics (COLING), Switzerland, pp. 848–855 (2004)

  22. Etzioni, O., Cafarrella, M., Downey, D., Popescu, A.M., Shaked, T., Soderland, S., Weld, D.S., Yates, A.: Unsupervised named entity extraction from the web: an experimental study. Artif. Intell. 165:91–134 (2005)

    Article  Google Scholar 

  23. Yu, X.: Chinese named entity recognition with cascaded hybrid model. In: Proceedings of NAACL HLT 2007, Prague, pp. 197–200 (2007)

  24. Srihari, R., Niu, C., Li, W.: A hybrid approach for named entity and sub-type tagging. In: Proceedings of Sixth Conference on Applied Natural Language Processing (ANLP), pp. 247–254 (2002)

  25. Ekbal, A., Bandyopadhyay, S.: Lexical pattern learning from corpus data for named entity recognition. In: Proceedings of the 5th International Conference on Natural Language Processing (ICON), India, pp. 123–128 (2002)

  26. Ekbal A., Naskar S., Bandyopadhyay S.: Named entity recognition and transliteration in Bengali (Special Issue of Lingvisticae Investigationes Journal). Named Entities Recogn. Classif. Use 30(1), 95–114 (2007)

    Google Scholar 

  27. Ekbal, A., Haque, R., Bandyopadhyay, S.: Named entity recognition in Bengali: a conditional random field approach. In: Proceedings of the 3rd International Joint Conference on Natural Language Processing (IJCNLP 2008), pp. 589–594 (2008)

  28. Ekbal, A., Bandyopadhyay, S.: Bengali named entity recognition using support vector machine. In: Proceedings of Workshop on NER for South and South East Asian Languages, 3rd International Joint Conference on Natural Language Processing (IJCNLP), India, pp. 51–58 (2008)

  29. Ekbal, A., Bandyopadhyay, S.: Voted NER system using appropriate unlabeled data. In: Proceedings of the 2009 Named Entities Workshop: Shared Task on Transliteration (NEWS 2009), ACL-IJCNLP 2009, pp. 202–210 (2009)

  30. Li W., McCallum A.: Rapid development of Hindi named entity recognition using conditional random fields and feature induction. ACM Trans. Asian Lang. Inf. Process. 2(3), 290–294 (2004)

    Article  Google Scholar 

  31. Patel, A., Ramakrishnan, G., Bhattacharya, P.: Relational Learning Assisted Construction of Rule Base for Indian Language NER. In: Proceedings of ICON 2009: 7th International Conference on Natural Language Processing, India (2009)

  32. Saha, S., Sarkar, S., Mitra, P.: A hybrid feature set based maximum entropy Hindi named entity recognition. In: Proceedings of the 3rd International Joint Conference in Natural Langauge Processing (IJCNLP 2008), pp. 343–350 (2008)

  33. Gali, K., Sharma, H., Vaidya, A., Shisthla, P., Sharma, D.M.: Aggregrating machine learning and rule-based heuristics for named entity recognition. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, pp. 25–32 (2008)

  34. Srikanth, P., Murthy, K.N.: Named entity recognition for Telugu. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, pp. 41–50 (2008)

  35. Shishtla, P.M., Pingali, P., Varma, V.: A character n-gram based approach for improved recall in Indian language NER. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, pp. 101–108 (2008)

  36. Goldberg D.E.: Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley, New York (1989)

    MATH  Google Scholar 

  37. Deb K.: Multi-objective Optimization Using Evolutionary Algorithms. Wiley, England (2001)

    MATH  Google Scholar 

  38. Coello~Coello C.A.: A comprehensive survey of evolutionary-based multiobjective optimization techniques. Knowl. Inf. Syst. 1(3), 269–308 (1999)

    Google Scholar 

  39. Veldhuizen D.V., Lamont G.: Multiobjective evolutionary algorithms: analyzing the state-of-the-artn. Evol. Comput. 2, 125–1473 (2000)

    Article  Google Scholar 

  40. Florian, R., Ittycheriah, A., Jing, H., Zhang, T.: Named entity recognition through classifier combination. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 (2003)

  41. Tjong Kim~Sang, E.F., De~Meulder, F.: Introduction to the Conll-2003 shared task: language independent named entity recognition. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142–147 (2003)

  42. Deb K., Pratap A., Agarwal S., Meyarivan T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 181–197 (2002)

    Article  Google Scholar 

  43. Ekbal A., Bandyopadhyay S.: A web-based Bengali news corpus for named entity recognition. Lang. Resour. Evaluat. J. 42(2), 173–182 (2008)

    Article  Google Scholar 

  44. Asif Ekbal., Sriparna Saha.: Classifier Ensemble Selection Using genetic algorithm for named entity Recognition. Res Lang and Comput. 8(1), 73–99 (2010)

    Article  Google Scholar 

  45. Darroch J., Ratcliff D.: Generalized iterative scaling for log-linear models. Ann. Math. Stat. 43, 1470–1480 (1972)

    Article  MathSciNet  MATH  Google Scholar 

  46. Sha, F., Pereira, F.: Shallow parsing with conditional random fields. In: Proceedings of NAACL ’03, Canada, pp. 134–141 (2003)

  47. Pinto, D., McCallum, A., Wei, X., Croft, W.B.: Table extraction using conditional random fields. In: Proceedings of SIGIR’03 Conference (2003)

  48. Vapnik V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995)

    MATH  Google Scholar 

  49. Joachims, T.: Making Large Scale SVM Learning Practical. MIT Press, Cambridge, pp. 169–184 (1999)

  50. Taira, H., Haruno, M.: Feature selection in SVM text categorization. In: Proceedings of AAAI-99 (1999)

  51. Bandyopadhyay S., Pal S.K., Aruna B.: Multi-objective GAs, quantitative indices and pattern classification. IEEE Trans. Syst. Man Cybern. B 34(5), 2088–2099 (2004)

    Article  Google Scholar 

  52. Anderson T.W., Scolve S.: Introduction to the Statistical Analysis of Data. Houghton Mifflin, Boston (1978)

    MATH  Google Scholar 

  53. Lin, D., Wu, X.: Phrase clustering for discriminative learning. In: Proceedings of 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pp. 1030–1038 (2009)

  54. Suzuki, J., Isozaki, H.: Semi-supervised sequential labeling and segmentation using Gigaword scale unlabeled data. In: Proceedings of ACL/HLT-08, pp. 665–673 (2008)

  55. Chieu, H.L., Ng, H.T.: Named entity recognition with a maximum entropy approach. In: Proceedings of CoNLL-2003, HLT-NAACL 2003, pp. 160–163 (2003)

  56. Dekai~Wu, G.N., Carput, M.: A stacked, voted, stacked model for named entity recognition. In: Proceedings of the CoNLL-2003, HLT-NAACL (2003)

  57. Klein, D., Smarr, J., Nguyen, H.N., Manning, C.D.: Named entity recognition with character-level models. In: Proceedings of CoNLL-2003, HLT-NAACL 2003, pp. 188–191 (2003)

  58. Singh, A.K.: Named entity recognition for South and South East Asian languages: taking stock. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, IJCNLP-08, India (2008)

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology, Patna, India

    Asif Ekbal & Sriparna Saha

Authors
  1. Asif Ekbal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sriparna Saha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Asif Ekbal.

Additional information

Both authors equally contributed for this work.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ekbal, A., Saha, S. Multiobjective optimization for classifier ensemble and feature selection: an application to named entity recognition. IJDAR 15, 143–166 (2012). https://doi.org/10.1007/s10032-011-0155-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-011-0155-7

Keywords

Search

Navigation