Skip to main content
Springer Nature Link
Log in

New Methods for Text Categorization Based on a New Feature Selection Method and a New Similarity Measure Between Documents

  • Conference paper
Advances in Applied Artificial Intelligence (IEA/AIE 2006)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4031))

  • 1690 Accesses

Abstract

In this paper, we present a new feature selection method based on document frequencies and statistical values. We also present a new similarity measure to calculate the degree of similarity between documents. Based on the proposed feature selection method and the proposed similarity measure between documents, we present three methods for dealing with the Reuters-21578 top 10 categories text categorization. The proposed methods get higher performance for dealing with the Reuters-21578 top 10 categories text categorization than that of the method presented in [4].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Caropreso, M.F., Matwin, S., Sebastiani, F.: A Learner-Independent Evaluation of the Usefulness of Statistical Phrases for Automated Text Categorization. In: Chin, A.G. (ed.) Text Databases and Document Management: Theory and Practice, pp. 78–102. Idea Group Publishing, Hershey, PA (2001)

    Google Scholar 

  2. Chakrabarti, S.: Mining the Web, pp. 137–144. Morgan Kaufmann, New York (2003)

    Google Scholar 

  3. Chua, S.K.N.: Semantic Feature Selection Using WordNet. In: Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence (2004)

    Google Scholar 

  4. Doan, S.: An Efficient Feature Selection Using Multi-Criteria in Text Categorization. In: Proceedings of the IEEE Fourth International Conference on Hybrid Intelligent Systems (2004)

    Google Scholar 

  5. Dumais, S.T., Plant, J., Heckerman, D., Sahami, M.: Inductive Learning Algorithms and Representations for Text Categorization. In: Proceedings of the 7th ACM International Conference on Information and Knowledge Management, pp. 148–155 (1998)

    Google Scholar 

  6. Galavotti, L., Sebastiani, F., Simi, M.: Experiments on the Use of Feature Selection and Negative Evidence in Automated Text Categorization. In: Borbinha, J.L., Baker, T. (eds.) ECDL 2000. LNCS, vol. 1923, pp. 59–68. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  7. Lam, W., Ho, C.Y.: Using a Generalized Instance Set for Automatic Text Categorization. In: Proceedings of SIGIR 1998 the 21st ACM International Conference on Research and Development in Information Retrieval, pp. 195–202 (1998)

    Google Scholar 

  8. Larkey, L.S., Croft, W.B.: Combining Classifiers in Text Categorization. In: Proceedings of the 19th ACM International Conference on Research and Development in Information Retrieval, pp. 289–297 (1996)

    Google Scholar 

  9. Larkey, L.S.: Automatic Essay Grading Using Text Categorization Techniques. In: Proceedings of the 21st ACM International Conference on Research and Development in Information Retrieval, pp. 90–95 (1998)

    Google Scholar 

  10. Lewis, D.D.: An evaluation of phrasal and clustered representations on a text categorization task. In: Proceedings of the 15th ACM International Conference on Research and Development in Information Retrieval, pp. 37–50 (1992)

    Google Scholar 

  11. Lewis, D.D.: Representation and Learning in Information Retrieval. Ph.D. Dissertation, Department of Computer Science, University of Massachusetts, Amherst, MA (1992)

    Google Scholar 

  12. Lewis, D.D., Ringuette, M.: A Comparison of Two Learning Algorithms for Text Categorization. In: Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval, pp. 81–93 (1994)

    Google Scholar 

  13. Li, Y.H., Jain, A.K.: Classification of Text Documents. Computer Journal 41(8), 537–546 (1998)

    Article  MATH  Google Scholar 

  14. Li, H., Yamanishi, K.: Text Classification Using ESC-Based Stochastic Decision Lists. In: Proceedings of the 8th ACM International Conference on Information and Knowledge Management, pp. 122–130 (1999)

    Google Scholar 

  15. Mladenic, D.: Feature Subset Selection in Text Learning. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 95–100. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  16. Ng, H.T., Goh, W.B., Low, K.L.: Feature Selection, Perceptron Learning, and a Usability Case Study for Text Categorization. In: Proceedings of the 20th ACM International Conference on Research and Development in Information Retrieval, pp. 67–73 (1997)

    Google Scholar 

  17. Porter, M.F.: An Algorithm for Suffic Stripping Program 14(3), 130–137 (1980)

    Google Scholar 

  18. Salton, G., Wong, A., Yang, C.: A Vector Space Model for Automatic Indexing. Communications of the ACM 18(11), 613–620 (1975)

    Article  MATH  Google Scholar 

  19. Sebastiani, F.: Machine Learning in Automated Text Categorization. ACM Computing Survey 34(1), 1–47 (2002)

    Article  Google Scholar 

  20. Sebastiani, F., Sperduti, A., Valdambrini, N.: An Improved Boosting Algorithm and its Application to Automated Text Categorization. In: Proceedings of the 9th ACM International Conference on Information and Knowledge Management, pp. 78–85 (2000)

    Google Scholar 

  21. Shima, K., Todoriki, M., Suzuki, A.: SVM-Based Feature Selection of Latent Semantic Features. Pattern Recognition Letters 25, 1051–1057 (2004)

    Article  Google Scholar 

  22. Yang, Y.: An Evaluation of Statistical Approaches to Text Categorization. Information Retrieval Journal 1(1–2), 69–90 (1999)

    Article  Google Scholar 

  23. Yang, Y., Pedersen, J.: A Comparative Study on Feature Selection in Text Categorization. In: Proceedings of the 14th International Conference on Machine Learning, pp. 412–420 (1997)

    Google Scholar 

  24. Yang, Y., Liu, X.: A Re-examination of Text Categorization Methods. In: Proceedings of the SIGIR 1999. 22nd ACM International Conference on Research and Development in Information Retrieval, Berkeley, CA, pp. 42–49 (1999)

    Google Scholar 

  25. Reuter-21578 Apte Split Data Set, http://kdd.ics.uci.edu/data-bases/reuter21578/reuter221578.html

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, R.O.C.

    Li-Wei Lee & Shyi-Ming Chen

Authors
  1. Li-Wei Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shyi-Ming Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, Texas State University-San Marcos, Nueces 247, 601 University Drive, 78666-4616, San Marcos, TX, USA

    Moonis Ali

  2. ESIA Laboratoire d’Informatique, Sytèmes, Traitement de l’Information et de la Connaissance, Université de Savoie, B.P. 806, F-74016, ANNECY Cedex, France

    Richard Dapoigny

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lee, LW., Chen, SM. (2006). New Methods for Text Categorization Based on a New Feature Selection Method and a New Similarity Measure Between Documents. In: Ali, M., Dapoigny, R. (eds) Advances in Applied Artificial Intelligence. IEA/AIE 2006. Lecture Notes in Computer Science(), vol 4031. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11779568_135

Download citation

  • DOI: https://doi.org/10.1007/11779568_135

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-35453-6

  • Online ISBN: 978-3-540-35454-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics