Tf-Idf Vectorizer

Term frequency

$$tf(t,d)=\frac{f_{t,d}}{\sum_{t'\in d}{f_{t',d}}}$$

f_t,d là tần suất xuất hiện của một từ (term) trong 1 tài liệu (document)

Mẫu số là tổng số từ (term) trong 1 tài liệu

Inverse document frequency

$$idf(t, D) = 1 + ln\frac{N}{df(t)}$$

N: là tổng số tài liệu document trong corpus

df(t): số tài liệu chứa từ (term)

Smooth Inverse document frequency

$$idf(t, D) = 1 + ln\frac{N + 1}{df(t) + 1}$$

Example

d1: the cats are in the house
d2: the dogs are in the house and outside
d3: the cats and dogs are friends

tf - idf

	and	are	cats	dogs	friends	house	in	outside	the
d1	0	1	1	0	0	1	1	0	2
d2	1	1	0	1	0	1	1	1	2
d3	1	1	1	1	1	0	0	0	1
df(t)	2	3	2	2	1	2	2	1	3
$$idf(t, D) = 1 + ln\frac{N}{df(t)}$$	$$1+ln\frac{3}{2}=1.4055$$	$$1+ln\frac{3}{3}=1$$	$$1+ln\frac{3}{2}=1.4055$$	$$1+ln\frac{3}{2}=1.4055$$	$$1+ln\frac{3}{1}=2.0986$$	$$1+ln\frac{3}{2}=1.4055$$	$$1+ln\frac{3}{2}=1.4055$$	$$1+ln\frac{3}{1}=2.0986$$	$$1+ln\frac{3}{3}=1$$

tf - idf Non Normal

and	are	cats	dogs	friends	house	in	outside	the
0.0	1.0	1.4055	0.0	0.0	1.4055	1.4055	0.0	2.0
1.4055	1.0	0.0	1.4055	0.0	1.4055	1.4055	2.0986	2.0
1.4055	1.0	1.4055	1.4055	2.0986	0.0	0.0	0.0	1.0

tf - idf l1-norm

and	are	cats	dogs	friends	house	in	outside	the
0.0	0.1386	0.1948	0.0	0.0	0.1948	0.1948	0.0	0.2771
0.1311	0.0933	0.0	0.1311	0.0	0.1311	0.1311	0.1958	0.1866
0.169	0.1203	0.169	0.169	0.2524	0.0	0.0	0.0	0.1203

tf - idf l2-norm

and	are	cats	dogs	friends	house	in	outside	the
0.0	0.3025	0.4252	0.0	0.0	0.4252	0.4252	0.0	0.6051
0.3379	0.2404	0.0	0.3379	0.0	0.3379	0.3379	0.5045	0.4808
0.4003	0.2848	0.4003	0.4003	0.5977	0.0	0.0	0.0	0.2848

Python Code

Prepare

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
    "the cats are in the house",
    "the dogs are in the house and outside",
    "the cats and dogs are friends",
]

Non Normalize

vectorizer = TfidfVectorizer(use_idf=True, norm=None, smooth_idf=False)
vectorizer.fit_transform(corpus)
vectorizer.get_feature_names_out()

l1-norm

vectorizer = TfidfVectorizer(use_idf=True, norm='l1', smooth_idf=False)

l2-norm

vectorizer = TfidfVectorizer(use_idf=True, norm='l2', smooth_idf=False)

l2-norm smooth-idf

vectorizer = TfidfVectorizer(use_idf=True, norm='l2', smooth_idf=True)