Metrics¶

The costcla.metrics module includes metrics to assess performance on cost-sensitive classification tasks given class prediction and cost-matrix

Functions named as *_score return a scalar value to maximize: the higher the better

Function named as *_error or *_loss return a scalar value to minimize: the lower the better

costcla.metrics.cost_loss(y_true, y_pred, cost_mat)[source]¶

Cost classification loss.

This function calculates the cost of using y_pred on y_true with cost-matrix cost-mat. It differ from traditional classification evaluation measures since measures such as accuracy asing the same cost to different errors, but that is not the real case in several real-world classification problems as they are example-dependent cost-sensitive in nature, where the costs due to misclassification vary between examples.

Parameters:	y_true : array-like or label indicator matrix Ground truth (correct) labels. y_pred : array-like or label indicator matrix Predicted labels, as returned by a classifier. cost_mat : array-like of shape = [n_samples, 4] Cost matrix of the classification problem Where the columns represents the costs of: false positives, false negatives, true positives and true negatives, for each example.
Returns:	loss : float Cost of a using y_pred on y_true with cost-matrix cost-mat

See also

savings_score

References

[R11]

C. Elkan, “The foundations of Cost-Sensitive Learning”, in Seventeenth International Joint Conference on Artificial Intelligence, 973-978, 2001.

[R12]

A. Correa Bahnsen, A. Stojanovic, D.Aouada, B, Ottersten, “Improving Credit Card Fraud Detection with Calibrated Probabilities”, in Proceedings of the fourteenth SIAM International Conference on Data Mining, 677-685, 2014.

Examples

>>> import numpy as np
>>> from costcla.metrics import cost_loss
>>> y_pred = [0, 1, 0, 0]
>>> y_true = [0, 1, 1, 0]
>>> cost_mat = np.array([[4, 1, 0, 0], [1, 3, 0, 0], [2, 3, 0, 0], [2, 1, 0, 0]])
>>> cost_loss(y_true, y_pred, cost_mat)
3

costcla.metrics.savings_score(y_true, y_pred, cost_mat)[source]¶

Savings score.

This function calculates the savings cost of using y_pred on y_true with cost-matrix cost-mat, as the difference of y_pred and the cost_loss of a naive classification model.

Parameters:	y_true : array-like or label indicator matrix Ground truth (correct) labels. y_pred : array-like or label indicator matrix Predicted labels, as returned by a classifier. cost_mat : array-like of shape = [n_samples, 4] Cost matrix of the classification problem Where the columns represents the costs of: false positives, false negatives, true positives and true negatives, for each example.
Returns:	score : float Savings of a using y_pred on y_true with cost-matrix cost-mat The best performance is 1.

See also

cost_loss

References

[R13]

A. Correa Bahnsen, A. Stojanovic, D.Aouada, B, Ottersten, “Improving Credit Card Fraud Detection with Calibrated Probabilities”, in Proceedings of the fourteenth SIAM International Conference on Data Mining, 677-685, 2014.

Examples

>>> import numpy as np
>>> from costcla.metrics import savings_score, cost_loss
>>> y_pred = [0, 1, 0, 0]
>>> y_true = [0, 1, 1, 0]
>>> cost_mat = np.array([[4, 1, 0, 0], [1, 3, 0, 0], [2, 3, 0, 0], [2, 1, 0, 0]])
>>> savings_score(y_true, y_pred, cost_mat)
0.5

costcla.metrics.brier_score_loss(y_true, y_prob)[source]¶

Compute the Brier score

The smaller the Brier score, the better, hence the naming with “loss”.

Across all items in a set N predictions, the Brier score measures the mean squared difference between (1) the predicted probability assigned to the possible outcomes for item i, and (2) the actual outcome. Therefore, the lower the Brier score is for a set of predictions, the better the predictions are calibrated. Note that the Brier score always takes on a value between zero and one, since this is the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 and 1).

The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but is inappropriate for ordinal variables which can take on three or more values (this is because the Brier score assumes that all possible outcomes are equivalently “distant” from one another).

Parameters:	y_true : array, shape (n_samples,) True targets. y_prob : array, shape (n_samples,) Probabilities of the positive class.
Returns:	score : float Brier score

References

http://en.wikipedia.org/wiki/Brier_score

Examples

>>> import numpy as np
>>> from costcla.metrics import brier_score_loss
>>> y_true = [0, 1, 1, 0]
>>> y_prob = [0.1, 0.9, 0.8, 0.3]
>>> brier_score_loss(y_true, y_prob) 
0.037...
>>> brier_score_loss(y_true, np.array(y_prob) > 0.5)
0.0

costcla.metrics.binary_classification_metrics(y_true, y_pred, y_prob)[source]¶

classification_metrics.

This function cal...

Parameters:	y_true : array-like Ground truth (correct) labels. y_pred : array-like Predicted labels, as returned by a classifier. y_prob : array-like Predicted probabilities, as returned by a classifier.
Returns:	dict(tp, fp, fn, tn, accuracy, recall, precision, f1score, auc, brier_loss)

Examples

>>> from costcla.metrics import binary_classification_metrics
>>> y_pred = [0, 1, 0, 0]
>>> y_true = [0, 1, 1, 0]
>>> y_prob = [0.2, 0.8, 0.4, 0.3]
>>> binary_classification_metrics(y_true, y_pred, y_prob)
{'accuracy': 0.75,
 'auc': 0.75,
 'brier_loss': 0.13249999999999998,
 'f1score': 0.6666666666666666,
 'fn': 1.0,
 'fp': 0.0,
 'precision': 1.0,
 'recall': 0.5,
 'tn': 2.0,
 'tp': 1.0}