TF–IDF

TF–IDF (term frequency–inverse document frequency) is a term weighting scheme commonly used to represent textual documents as vectors (for purposes of classification, clustering, visualization, retrieval, etc.). Let T = { t₁, …, t _n } be the set of all terms occurring in the document corpus under consideration. Then a document d _i is represented by a n-dimensional real-valued vector x$_{i} = (x_{i_{1}},\ldots,x_{in})$ with one component for each possible term from T.

The weight x _ij corresponding to term t _j in document d _i is usually a product of three parts: one which depends on the presence or frequency of t _j in d _i , one which depends on t _j ’s presence in the corpus as a whole, and a normalization part which depends on d _j . The most common TF–IDF weighting is defined by $x_{ij} =\mathrm{ TF}_{i} \cdot \mathrm{ IDF}_{j} \cdot (\sum _{j}(\mathrm{TF}_{ij}\mathrm{IDF}_{j})^{2})^{-1/2}$, where TF _ij is the term frequency (i.e., number of occurrences) of t _j in d _i , and IDFjis...