evaluators¶

This module implements the evaluation indicators, such as F1-score, recall@k, precision@k and etc using faiss-gpu package.

calculator.py: A evaluation manager class.
metrics.py: All evaluation functions are implemented here.

Class¶

Calculator¶

class gedml.core.evaluators.calculator.Calculator(k_list, include=(), exclude=(), k=None)[source]¶

Bases: object

A evaluation manager class. Given query and referecne vectors, this module will return a evaluation dictionary including all evaluation results.

Parameters

k_list (list) – K-index list for recall at k and precision at k.
include (tuple) – Evaluation functions must be included.
exclude (tuple) – Evaluation functions should be excluded.
k (int) – Parameter for knn search.

Example

>>> calculator = Calculator(k_list=[1,2,4,8])
>>> x = np.random.randn(100, 128)
>>> labels = np.random.randint(0, 10, size=(100,))
>>> output_dict = calculator.get_accuracy(x, x, labels, labels, True)

get_accuracy(query: numpy.ndarray, reference: numpy.ndarray, query_labels: numpy.ndarray, reference_labels: numpy.ndarray, embeds_same_source: bool, include=(), exclude=(), device_ids=None)[source]¶

Compute all evaluation indicators.

Parameters

query (np.ndarray) – Samples to be tested (or without labels). size: \(B_1 \times dim\).
reference (np.ndarray) – Samples working as reference. size: \(B_2 \times dim\).
query_labels (np.ndarray) – Queris’ labels. size: \(B_1\).
reference_labels (np.ndarray) – References’ labels. size: \(B_2\).
embeds_same_source (bool) – Whether query and reference data are from the same source.
include (tuple) – Evaluation functions to include.
exclude (tuple) – Evaluation functions to exclude.
device_ids (list) – Device indices to call by Faiss package. default: None.

Returns

A dictionary which contains all results.

Return type

dict

CalculatorFromMat¶

class gedml.core.evaluators.calculator_mat.CalculatorFromMat(k_list, include=(), exclude=(), k=None, is_min=True)[source]¶

Bases: object

A evaluation manager class. Given query and referecne vectors, this module will return a evaluation dictionary including all evaluation results (Given metric matrix)

Parameters

k_list (list) – K-index list for recall at k and precision at k.
include (tuple) – Evaluation functions must be included.
exclude (tuple) – Evaluation functions should be excluded.
k (int) – Parameter for knn search.
is_min (bool) – Whether to choose the min (or max) value as the “nearest” sample.

Example

>>> calculator = CalculatorFromMat(k_list=[1,2,4,8])
>>> x = np.random.randn(100, 128)
>>> labels = np.random.randint(0, 10, size=(100,))
>>> output_dict = calculator.get_accuracy(x, x, labels, labels, True)

get_accuracy(metric_mat: numpy.ndarray, query_labels: numpy.ndarray, reference_labels: numpy.ndarray, embeds_same_source: bool, include=(), exclude=(), device_ids=None)[source]¶

Compute all evaluation indicators.

Parameters

metric_mat (np.ndarray) – Distance matrix to be tested. size: \(B_1 \times B_2\).
query_labels (np.ndarray) – Queris’ labels. size: \(B_1\).
reference_labels (np.ndarray) – References’ labels. size: \(B_2\).
embeds_same_source (bool) – Whether query and reference data are from the same source.
include (tuple) – Evaluation functions to include.
exclude (tuple) – Evaluation functions to exclude.
device_ids (list) – Device indices to call by Faiss package. default: None.

Returns

A dictionary which contains all results.

Return type

dict