Probcal

The costcla.probcal module includes methods for probability calibration

class costcla.probcal.ROCConvexHull

Implementation the the calibration method ROCConvexHull

See also

sklearn.IsotonicRegression

References

[R14]

J. Hernandez-Orallo, P. Flach, C. Ferri, ‘A Unified View of Performance Metrics : Translating Threshold Choice into Expected Classification Loss’, Journal of Machine Learning Research, 13, 2813–2869, 2012.

Examples

>>> from costcla.probcal import ROCConvexHull
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.cross_validation import train_test_split
>>> from costcla.datasets import load_creditscoring1
>>> from costcla.metrics import brier_score_loss
>>> data = load_creditscoring1()
>>> sets = train_test_split(data.data, data.target, data.cost_mat, test_size=0.33, random_state=0)
>>> X_train, X_test, y_train, y_test, cost_mat_train, cost_mat_test = sets
>>> f = RandomForestClassifier()
>>> f.fit(X_train, y_train)
>>> y_prob_test = f.predict_proba(X_test)
>>> f_cal = ROCConvexHull()
>>> f_cal.fit(y_test, y_prob_test)
>>> y_prob_test_cal = f_cal.predict_proba(y_prob_test)
>>> # Brier score using only RandomForest
>>> print(brier_score_loss(y_test, y_prob_test[:, 1]))
0.0577615264881
>>> # Brier score using calibrated RandomForest
>>> print(brier_score_loss(y_test, y_prob_test_cal))
0.0553677407894

Attributes

calibration_map

(array-like) calibration map for maping the raw probabilities to the calibrated probabilities.

Methods

fit(y, p)	Fit the calibration map
predict_proba(p)	Calculate the calibrated probabilities

fit(y, p)

Fit the calibration map

Parameters:

y_true : array-like of shape = [n_samples]

True class to be used for calibrating the probabilities

y_prob : array-like of shape = [n_samples, 2]

Predicted probabilities to be used for calibrating the probabilities

Returns:

self : object

Returns self.

predict_proba(p)

Calculate the calibrated probabilities

Parameters:

y_prob : array-like of shape = [n_samples, 2]

Predicted probabilities to be calibrated using calibration map

Returns:

y_prob_cal : array-like of shape = [n_samples, 1]

Predicted calibrated probabilities