Similarity Elasticsearch

Cosine

$$cosine(X,Y) = \frac{X.Y}{||X||.||Y||}=\frac{\sum_{i=1}^n(X_i.Y_i)}{\sqrt{\sum_{i=1}^n{X_i^2}\sum_{i=1}^n{Y_i^2}}}$$

Score Cosine in Elasticsearch

$$score = \frac{1+cosine(query, vector)}{2}$$

Example

x = (3, 5, 2)
y = (2, 3, 1)

$$cosine(x, y) = \frac{2 * 3 + 5 * 3 + 2 * 1}{\sqrt{(3 ^ 2 + 5 ^ 2 + 2 ^ 2)(2 ^ 2 + 3 ^ 2 + 1 ^2)}}=0.9972$$

$$score = \frac{1+cosine(x,y)}{2}=\frac{1+0.9972}{2}=0.9986$$

Code Python

from sklearn.metrics.pairwise import cosine_similarity
from math import sqrt
X = [
    (3, 5, 2),
    (2, 3, 1)
]
cos = cosine_similarity(X)

L2 norm

$$\text{l2_norm}(X,Y) = \sqrt{\sum_{i=1}^n{(X_i - Y_i)^2}}$$

Score l2 norm in Elasticsearch

$$score = \frac{1}{1 + \text{l2_norm}(query, vector)^2}$$

Example

x = (3, 5, 2)
y = (2, 3, 1)

$$l2\_norm(x, y) = \sqrt{(2 - 3)^2 + (5 - 3)^2 + (2 - 1)^2} = 2.4495$$

$$score = \frac{1}{1 + l2\_nom(x,y)}=\frac{1}{1+2.4495}=0.2899$$

Dot product

$$\text{dot_product}(X,Y) = X.Y = \sum_{i=1}^n{X_i . Y_i}$$

Score dot product in Elasticsearch

$$score = \frac{1 + \text{dot_product}(query, vector)}{2}$$

Example

x = (3, 5, 2)
y = (2, 3, 1)

$$dot\_product(x, y) = 2 * 3 + 5 * 3 + 2 * 1 = 23$$

$$score = \frac{1 + dot\_product(x, y)}{2} = \frac{1+23}{2} = 12$$

Mặc định trong Elasticsearch sử dụng Cosine Similarity

Create Schema

PUT product{
    "mappings": {
        "properties": {
            "vector": {
                "type": "dense_vector",
                "similarity": "l2_norm",
                "dims": 3
            },
            "title":{
                "type": "keyword"
            }
        }
    }
}

Insert Data

POST /product/_bulk?refresh=true
{"index": {"_id": 1}}
{"vector": [3, 2, 9], "title": "Document 1"}
{"index": {"_id": 2}}
{"vector": [4, 5, 7], "title": "Document 2"}
{"index": {"_id": 3}}
{"vector": [6, 1, 5], "title": "Document 3"}
{"index": {"_id": 4}}
{"vector": [8, 2, 1], "title": "Document 4"}

Query

GET /product/_search?filter_path=hits.hits{
    "query": {
        "knn": {
            "query_vector": [3, 5, 2],
            "field": "vector",
            "k": 2
        }
    }
}