Evaluation Metrics¶

Searching¶

radbm.metrics.searching.user_cost_at_k_original(candidates, relevants)¶

Parameters:	candidates (iterable of set) – relevants (set) –
Returns:	user_cost_at_k – user_cost_at_k[k] is the \(\mathrm{UC}_k\)
Return type:	torch.tensor (dtype: float, ndim=1)

radbm.metrics.searching.user_cost_at_k_from_counts(candidates_size, relevant_counts, relevant_candidates_indices=None)¶

Parameters:	candidates_size (torch.tensor (dtype: int, ndim: 1)) – Need not to contain all set of candidates. Only the firsts that contain all the relevant documents. relevant_counts (torch.tensor (dtype: int, ndim: 1)) – relevant_counts[i] is the number of relevant documents found in the the kth set of candidates where k = relevant_candidates_indices[i]. If relevant_candidates_indices is None (default) then k = i. relevant_candidates_indices (torch.tensor (dtype: int, ndim: 1) or None) – The indices of set of candidates at which we can found at least one relevant documents. Setting to None is equivalent to setting to torch.arange(len(candidates_size)). (default: None)
Returns:	user_cost_at_k – user_cost_at_k[k] is the \(\mathrm{UC}_k\)
Return type:	torch.tensor (dtype: float32, ndim: 1)
Raises:	`ValueError` – If len(relevants_counts) != len(candidates_size) and relevant_candidates_indices is None.

Notes

The zeros of relevant_counts can be safely removed. However, if so, it is mandatory to provide relevant_candidates_indices.

radbm.metrics.searching.user_cost_at_k(candidates, relevants, device='cpu')¶

Parameters:	candidates (iterable of set) – relevants (set) – device (str (optional)) – Can be ‘cpu’, ‘cuda’ or any valid torch.device. (default: ‘cpu’)
Returns:	user_cost_at_k – user_cost_at_k[k] is the \(\mathrm{UC}_k\)
Return type:	torch.tensor (dtype: float, ndim=1)

Notes

This function is equivalent to user_cost_at_k_original but it uses vectorization to speedup the computation.

radbm.metrics.searching.user_cost_at_k_from_scores(scores, relevants)¶

Compute the user cost at k from a scoring of the documents. The pre-order defined by the scoring gives the sequences of candidates. This function avoid the overhead of computing the sequences of candidates and pass it to user_cost_at_k which would in the end only do counting.

Parameters:	scores (torch.tensor (ndim: 1)) – The score of each documents. relevants (torch.tensor (ndim: 1, torch.int64)) – The dtype should be torch.int64 or torch.bool because scores[relevants] needs to give the scores of the relevant documents. If relevants’dtype is torch.int64 the duplicates are removed.
Returns:	uck – uck[i] is the user cost at i.
Return type:	torch.tensor (dtype: float32, shape: (k,))
Raises:	`ValueError` – If relevants.dtype is torch.bool and len(relevants) != len(scores) `TypeError` – If relevants.dtype is not torch.bool or torch.int64

radbm.metrics.searching.costs_at_k(candidates, relevants, N=None, device='cpu')¶

Computes the user cost at k and the engine cost at k.

Parameters:

candidates (iterable of set) – Iterable of candidates (sets). This object must have a cost attribute (i.e., candidates.cost must work).
relevants (set) – The set of relevant documents.
N (int (optional)) – The size of the database. If the candidates does not contain all the relevant documents (halting), N must be provided. (default: None)
device (str (optional)) – Can be ‘cpu’, ‘cuda’ or any valid torch.device. (default: ‘cpu’)

Returns:

user_cost_at_k (torch.tensor (dtype: float, ndim=1)) – user_cost_at_k[k] is the \(\mathrm{UC}_k\)
engine_cost_at_k (torch.tensor (dtype: float, ndim=1)) – engine_cost_at_k[k] is the \(\mathrm{EC}_k\)

Raises:

ValueError – If the candidates does not contain all relevants documents and N (the database size) is not given.
RuntimeError – If the candidates does not contain all relevants documents and the candidates are larger or equal to N.

radbm.metrics.searching.total_cost_at_k(candidates, relevants, N=None, device='cpu')¶

Computes the total cost at k, i.e., the sum of the user cost at k and the engine cost at k.

Parameters:	candidates (iterable of tuple (<set>, <float>)) – Iterable of candidates (set) and total engine cost (float). relevants (set) – The set of relevant documents. N (int (optional)) – The size of the database. If the candidates does not contain all the relevant documents (halting), N must be provided. (default: None) device (str (optional)) – Can be ‘cpu’, ‘cuda’ or any valid torch.device. (default: ‘cpu’)
Returns:	total_cost_at_k – user_cost_at_k[k] is the \(\mathrm{UC}_k\)
Return type:	torch.tensor (dtype: float, ndim=1)
Raises:	`ValueError` – If the candidates does not contain all relevants documents and N (the database size) is not given. `RuntimeError` – If the candidates does not contain all relevants documents and the candidates are larger or equal to N.

radbm.metrics.searching.total_cost_ratio_from_costs(uck, eck, N)¶

Computes the total cost ratio from the user cost at k \(\mathrm{UC}_k\) and the engine cost at k \(\mathrm{EC}_k\) using this equation:

\[\mathrm{TCR} = \frac{2}{N + 1} \frac{1}{|r|}\sum_{k=1}^{|r|} \mathrm{UC}_k + \mathrm{EC}_k\]

where r is the set of relevant documents, thus \(|r|\) is the number of relevant documents.

Parameters:	uck (torch.tensor (dtype: float, ndim=1)) – user_cost_at_k[k] is the \(\mathrm{UC}_k\) eck (torch.tensor (dtype: float, ndim=1)) – engine_cost_at_k[k] is the \(\mathrm{EC}_k\) N (int) – The database size.
Returns:	tcr – The total cost ratio.
Return type:	torch.tensor (ndim: 0, dtype: float32)

radbm.metrics.searching.total_cost_ratio(candidates, relevants, N, device='cpu')¶

Computes the total cost ratio.

Parameters:	candidates (iterable of tuple (<set>, <float>)) – Iterable of candidates (sets) and total engine cost (float). relevants (set) – The set of relevant documents. N (int) – The size of the database. device (str (optional)) – Can be ‘cpu’, ‘cuda’ or any valid torch.device. (default: ‘cpu’)
Returns:	tcr – The total cost ratio.
Return type:	torch.tensor (ndim: 0, dtype: float32)

Ranking¶

radbm.metrics.ranking.pre_average_precision(scores, relevants)¶

Compute the pre average precision from a scoring of each documents Whenever two (or more) documents share the same score, the ordering defined by this scoring is no longer antisymmetric. Thus order defined by this scoring becomes a total pre-order. This methods uses the user cost at k to compute the pre-AP, a generalization of the AP for pre-ranking (i.e. ranking defined with a total pre-order). If each score is unique then this is equivalent to the original AP.

Parameters:	scores (torch.tensor (ndim: 1)) – The score of each documents. relevants (torch.tensor (ndim: 1, torch.int64)) – The dtype should be torch.int64 or torch.bool because scores[relevants] needs to give the scores of the relevant documents. If relevants’dtype is torch.int64 the duplicates are removed.
Returns:	pre_ap – The pre average precision.
Return type:	torch.tensor (ndim: 0, dtype: float32)
Raises:	`ValueError` – If relevants.dtype is torch.bool and len(relevants) != len(scores) `TypeError` – If relevants.dtype is not torch.bool or torch.int64

radbm.metrics.ranking.pre_mean_average_precision(scores, relevants)¶

Parameters:	scores (torch.tensor (ndim: 2)) – scores[i,j] is the scores between the ith query and the jth documents. relevants (torch.tensor (ndim: 2, dtype: bool) or list of torch.tensors (ndim: 1, dtype: int)) – The adjacency matrix or adjacency list of relevance. If adjacency matrix, relevants[i,j] is a boolean indicating if the jth document is relevant to the ith query. Otherwise, if adjacency list, relevants[i] are the indices of the relevant documents.
Returns:	pre_map
Return type:	torch.tensor (dtype: float, ndim: 0)

radbm.metrics.ranking.pre_average_precision_from_user_cost(uck)¶

Compute the pre average precision from the user cost at k (\(\mathrm{UC}_k\)) using the following equation:

\[\sum_{k=1}^{|r|} \frac{k}{\mathrm{UC}_k}\]

Parameters:	uck (torch.tensor (shape: (k,))) – The user cost at k, uck[k] must be \(\mathrm{UC}_k\).
Returns:	pre_ap – The pre-average precision.
Return type:	torch.tensor (ndim: 0, dtype: float32)

radbm.metrics.ranking.batch_pre_average_precision(queries, documents, relevants, scoring_function, batch_size)¶

Computes pre_average_precision in batches to make the most out of a GPU.

Parameters:	queries (torch.tensor) – documents (torch.tensor) – relevants (iterator of torch.tensor (dtype: int64 or bool)) – scoring_function (callable) – scoring_function takes a batch of query and a batch of documents and computes the score if each pairs. The function should be broadcastable friendly, i.e. scoring_function(queries[:,None], documents[None]) should make sense. batch_size (int) – The size that will be used to split the queries. This is only used for memory purposes. In other words, this number should be as high as the memory allows it.
Returns:	pre_aps – The pre average precision of each query.
Return type:	torch.tensor (dtype: torch.float32, shape: (len(queries),))
Raises:	`ValueError` – If len(queries) != len(relevants).

Spatial¶

radbm.metrics.spatial.hamming_distance(x, y, dim=-1, keepdim=False)¶

Compute the Hamming distance.

Parameters:	x (torch.Tensor (dtype=torch.bool)) – y (torch.Tensor (dtype=torch.bool)) – dim (int (optional)) – The dimension along which to compute the hamming distance. (default: -1) keepdim (bool (optional)) – Whether to keep the reduced dimension. (default: False)
Returns:	z – The Hamming distance between x and y
Return type:	torch.Tensor (dtype=torch.int64)

radbm.metrics.spatial.membership_hamming_cost(x, y)¶

Compute the membership Hamming cost, i.e., the minimum Hamming distance between x’s code and the l y’s codes.

Parameters:	x (torch.Tensor (dtype=torch.bool, shape=(.., n))) – y (torch.Tensor (dtype=torch.bool, shape=(.., l, n))) – x.unsqueeze(-2) and y should be broadcastable.
Returns:	z – The membership Hamming cost between x and y.
Return type:	torch.Tensor (dtype=torch.int64)

radbm.metrics.spatial.intersection_hamming_cost(x, y)¶

Compute the intersection Hamming cost, i.e., the minimum Hamming distance between the k x’s code and the l y’s codes.

Parameters:	x (torch.Tensor (dtype=torch.bool, shape=(.., k, n))) – y (torch.Tensor (dtype=torch.bool, shape=(.., l, n))) – x[…, :, None, :] and y[…, None, :, :] should be broadcastable.
Returns:	z – The intersection Hamming cost between x and y.
Return type:	torch.Tensor (dtype=torch.int64)

radbm.metrics.spatial.superset_hamming_cost(x, y)¶

Compute the superset Hamming cost.

Parameters:	x (torch.Tensor (dtype=torch.bool, shape=(.., k, n))) – y (torch.Tensor (dtype=torch.bool, shape=(.., l, n))) – x[…, :, None, :] and y[…, None, :, :] should be broadcastable.
Returns:	z – The superset Hamming cost between x and y.
Return type:	torch.Tensor (dtype=torch.int64)

radbm.metrics.spatial.conditional_distance_counts(documents, queries, relevances, distance_function, max_distance, batch_size=100)¶

Compute the count of relevant and not relevant match w.r.t all distance.

Parameters:

documents (torch.Tensor) – The code of a batch of documents (database). documents[i] is the ith document.
queries (torch.Tensor (2D, dtype=torch.bool)) – The code of a batch of queries. queries[i] is the ith query.
relevances (list of set of int) – len(relevances) must be len(queries). For each corresponding query, it give the set of relevant documents (given by its index). Explicitly, j in relevances[i] iff query[i] matches with documents[j].
distance_function (function (torch.Tensor, torch.Tensor) -> torch.Tensor (dtype: int)) – The distance should be integer only.
max_distance (int) – The maximum distance to expect.
batch_size (int (optional)) – The number of query for which we compute the distance at a time. if it is to big the results might not fit in RAM (or on the GPU). (default 100)

Returns:

relevant_counts (torch.Tensor (1D, dtype=torch.float)) – len(relevant_counts) = maximum_distance + 1 and relevant_dcounts[i] is the number of relevant documents at distance i.
irrelevant_counts (torch.Tensor (1D, dtype=torch.float)) – len(irrelevant_countst) = maximum_distance + 1 and irrelevant_counts[i] is the number of irrelevant documents at distance i.

Notes

This assume that each sets in relevances is small compared to len(documents) and should be used on a GPU otherwise it is quite slow.

radbm.metrics.spatial.conditional_hamming_counts(documents, queries, relevances, batch_size=100)¶

Compute the count of relevant and not relevant match w.r.t all Hamming distance.

Parameters:

documents (torch.Tensor (2D, dtype=torch.bool)) – The binary reprentation of a batch of documents (database). documents[i] is the ith document. documents.shape[1] must be equal to queries.shape[1]. (should also be on the same device as queries)
queries (torch.Tensor (2D, dtype=torch.bool)) – The binary reprentation of a batch of queries. queries[i] is the ith query. queries.shape[1] must be equal to documents.shape[1]. (should also be on the same device as documents)
relevances (list of set of int) – len(relevances) must be len(queries). For each corresponding query, it give the set of relevant documents (given by its index). Explicitly, j in relevances[i] iff query[i] matches with documents[j].
batch_size (int (optional)) – The number of query for which we compute the Hamming distance at a time. if it is to big the results might not fit in RAM (or on the GPU). (default 100)

Returns:

relevant_counts (torch.Tensor (1D, dtype=torch.float)) – len(relevant_counts) = queries.shape[1] + 1 (also equal to documents.shape[1] + 1) and relevant_dcounts[i] is the number of relevant documents at Hamming distance i.
irrelevant_counts (torch.Tensor (1D, dtype=torch.float)) – len(irrelevant_countst) = queries.shape[1] + 1 (also equal to documents.shape[1] + 1) and irrelevant_counts[i] is the number of irrelevant documents be at Hamming distance i.

Notes

This assume that each sets in relevances is small compared to len(documents) and should be used on a GPU otherwise it is quite slow.

radbm.metrics.spatial.hamming_pr_curve(documents, queries, relevances, batch_size=100, return_valid_dists=False)¶

Compute the precision-recall curve w.r.t the Hamming distance. I.e. it computes the precision and recall for each Hamming distance decision thresholds.

Parameters:

documents (torch.Tensor (2D, dtype=torch.bool)) – The binary reprentation of a batch of documents (database). documents[i] is the ith document. documents.shape[1] must be equal to queries.shape[1]. (should be on the same device as queries)
queries (torch.Tensor (2D, dtype=torch.bool)) – The binary reprentation of a batch of queries. queries[i] is the ith query. queries.shape[1] must be equal to documents.shape[1]. (should be on the same device as documents)
relevances (list of set of int) – len(relevances) must be len(queries). For each corresponding query, it give the set of relevant documents (given by its index). Explicitly, j in relevances[i] iff query[i] matches with documents[j].
batch_size (int (optional)) – The number of query for which we compute the Hamming distance at a time. if it is to big the results might not fit in RAM (or on the GPU). (default 100)
return_valid_dists (bool (optional)) – Some dists might have an undefined precision. In those case the returned value will be nan by default. If return_valid_dists is True those value won’t be there and dists will be returned with precision and recall. See the returns section for more info. (default False)

Returns:

dists (torch.Tensor (1D, dtype=torch.int64) if return_valid_dists is True) – Only present if return_valid_dists is True. It correspond to the valid distances where the precision is define.
precisions (torch.Tensor (1D, dtype=torch.float)) – len(precisions) = queries.shape[1] + 1 (also equal to documents.shape[1] + 1) and precision[i] is the precison w.r.t. a Hamming distance of i if return_valid_dists is False otherwise, len(precisions) = len(dists) and precisions[i] is the precision w.r.t a Hamming distance of dists[i].
recalls (torch.Tensor (1D, dtype=torch.float)) – len(recalls) = queries.shape[1] + 1 (also equal to documents.shape[1] + 1) and recalls[i] is the recall w.r.t. to a Hamming distance of i if return_valid_dists is False otherwise, len(recalls) = len(dists) and recalls[i] is the recall w.r.t a Hamming distance of dists[i].

Notes

This assume that each sets in relevances is small compared to len(documents) and should be used on a GPU otherwise it is quite slow.

radbm.metrics.spatial.superset_cost(x, y, dim=-1, keepdim=False)¶

Compute the number of position True in x which are False in y. e.g. x = torch.tensor([True, True, False, False, True]) and y = torch.tensor([True, False, True, False, False]) we have for i in [1, 4], x[i] is True and y[i] is False thus superset_cost(x, y) = len([1, 4]) = 2.

Parameters:	x (torch.Tensor or numpy.ndarray (dtype=torch.bool)) – Broacastable with y. y (torch.tensor or numpy.ndarray (dtype=torch.bool)) – Broacastable with x. dim (int (optional)) – The dimension along which to compute the hamming distance. (default: -1) keepdim (bool (optional)) – Whether to keep the reduced dimension. (default: False)
Returns:	out – The shape of out is the broadcasted shape between x and y.
Return type:	torch.Tensor or numpy.ndarray (dtype=int)