Combined Metrics

The skfb.metrics module includes score functions with a reject option and plotting utilities.

Combined Classification Metrics

skfb.metrics.predict_reject_accuracy_score(y_true, y_pred)[source]

Calculates the ratio of true acceptance and rejection to all predictions.

Parameters:

  • y_true (array-like) – True labels.

  • y_pred (FBNDarray) – Base estimator predictions w/ fallback mask.

Returns:

score – (TA + TR) / (TA + TR + FA + FR)

Return type:

float

Examples

>>> import numpy as np
>>> from skfb.core import array as ska
>>> from skfb.metrics import predict_reject_accuracy_score
>>> y_true =    np.array([0, 1, 0, 0, 1, 1, 0, 1, 0, 1])
>>> y_pred = ska.fbarray([0, 1, 0, 1, 0, 1, 1, 1, 0, 1],
...                      [1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
>>> predict_reject_accuracy_score(y_true, y_pred)
np.float64(0.5)
skfb.metrics.predict_reject_recall_score(y_true, y_pred, beta=0.5)[source]

Calculates weighted average of prediction and rejection recalls.

Parameters:

  • y_true (array-like) – True labels.

  • y_pred (FBNDarray) – Base estimator predictions w/ fallback mask.

  • beta (float, default=0.5) – The weight of prediction recall.

Returns:

score – TA / (TA + FR) * beta + TR / (TR + FA) * (1 - beta)

Return type:

float

Examples

>>> import numpy as np
>>> from skfb.core import array as ska
>>> from skfb.metrics import predict_reject_recall_score
>>> y_true =    np.array([0, 1, 0, 0, 1, 1, 0, 1, 0, 1])
>>> y_pred = ska.fbarray([0, 1, 0, 1, 0, 1, 1, 1, 0, 1],
...                      [1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
>>> # TR = 1, FA = 2, TA = 4, FR = 3
>>> predict_reject_recall_score(y_true, y_pred, beta=0.75)
0.5119...
skfb.metrics.predict_accept_confusion_matrix(y_true, y_pred, labels=None, sample_weight=None, normalize=None)[source]

Computes confusion matrix w/ rows as accuracy and columns as acceptance.

Parameters:

  • y_true (array-like of shape (n_samples,)) – Ground truth (correct) target values.

  • y_pred (FBNDArray of shape (n_samples,)) – Estimated targets as returned by both a rejector and a classifier.

  • labels (array-like of shape (2,), default=None) – List of labels to index the matrix. This may be used to reorder or select a subset of labels. If None is given, those that appear at least once in y_true or y_pred are used in sorted order.

  • sample_weight (array-like of shape (n_samples,), default=None) – Sample weights.

  • normalize ({'true', 'pred', 'all'}, default=None) – Normalizes confusion matrix over the true (rows), predicted (columns) conditions or all the population. If None, confusion matrix will not be normalized.

Returns:

C – TR (true-reject) FA (false-accept) FR (false-reject) TA (true-accept)

Return type:

ndarray of shape (2, 2)

See also

sklearn.metrics.confusion_matrix

True vs Predicted confusion matrix.

Examples

>>> import numpy as np
>>> from skfb.core import array as ska
>>> from skfb.metrics import predict_accept_confusion_matrix
>>> y_true =    np.array([0, 1, 0, 0, 1, 1, 0, 1, 0, 1])
>>> y_pred = ska.fbarray([0, 1, 0, 1, 0, 1, 1, 1, 0, 1],
...                      [1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
>>> cm = predict_accept_confusion_matrix(y_true=y_true, y_pred=y_pred)
>>> cm
array([[1, 2],
       [3, 4]])
skfb.metrics.fallback_quality_auc_score(y_true, y_score, score_func=<function accuracy_score>, predict_method='predict', min_fallback_rate=0.0, max_fallback_rate=0.95, raise_warning=True)[source]

Returns area under prediction quality-fallback rate curve.

First, determines unique thresholds on input samples, then for every threshold, calculates the fallback rate and prediction quality.

skfb.metrics.fallback_quality_curve(y_true, y_score, score_func=<function accuracy_score>, predict_method='predict', min_fallback_rate=0.0, max_fallback_rate=0.95, raise_warning=True)[source]

Constructs prediction quality vs fallback rate curve.

First, determines unique thresholds on input samples, then for every threshold, calculates the fallback rate and prediction quality.

Common Metrics

skfb.metrics.prediction_quality(y_true, y_pred, score_func, fallback_label=None, raise_warning=True, **kwargs)[source]

Runs score_func on accepted samples.

Parameters:

  • y_true (array-like, shape (n_samples,)) – True labels.

  • y_pred (array-like or FBNDArray, shape (n_samples,) or (n_samples, n_classes)) – Either array of combined predictions or estimator predictions w/ fallback mask. If combined, then fallback_label should be provided.

  • score_func (callable) – Scoring function to call on accepted samples.

  • fallback_label (any, default=None) – If predictions are combined, indicates the label of fallback.

  • raise_warning (bool, default=True) – If all samples were rejected, raises a warning.

Examples

>>> import numpy as np
>>> from sklearn.metrics import accuracy_score
>>> from skfb.metrics import prediction_quality_score
>>> y_true = np.array([1, 0, 0, 1, 0, 1])
>>> y_pred = np.array([0, 0, -1, -1, 0, 1])
>>> prediction_quality_score(y_true, y_pred, accuracy_score)
0.75
>>> y_pred = np.array([-1, -1, -1, -1, -1, -1])
>>> prediction_quality_score(y_true, y_pred, accuracy_score,
...                          raise_warning=False)
nan

Plotting Utilities

class skfb.metrics.PAConfusionMatrixDisplay(confusion_matrix, *, display_labels=('No', 'Yes'), fallback_rate=None)[source]

Predict-Accept Confusion Matrix visualization.

It is recommend to use from_estimator() or from_predictions() to create a PAConfusionMatrixDisplay. All parameters are stored as attributes.

Parameters:

  • confusion_matrix (ndarray of shape (2, 2)) – Predict-accept confusion matrix.

  • display_labels (ndarray of shape (n_classes,), default=("No", "Yes")) – Display labels for plot. If None, display labels are set from 0 to n_classes - 1.

  • fallback_rate (float, default=None) – Ratio of rejected samples to all samples.

See also

skfb.metrics.predict_accept_confusion_matrix

Compute Confusion Matrix to evaluate the quality of predictions vs fallbacks.

skfb.metrics.PAConfusionMatrixDisplay.from_estimator

Plot the confusion matrix given an estimator, the data, and the label.

skfb.metrics.PAConfusionMatrixDisplay.from_predictions

Plot the confusion matrix given the true and predicted labels.

sklearn.metrics.ConfusionMatrixDisplay

We inherit this class and adapt its methods to rejections.

Notes

Adapted from ConfusionMatrixDisplay.

Methods

from_estimator(rejector, X, y, *[, labels, ...])

Plots PA Confusion Matrix given a rejector and some data.

from_predictions(y_true, y_pred, *[, ...])

Plots PA Confusion Matrix given true and predicted labels.

plot(*[, include_values, cmap, ...])

Plots predict-accept confusion matrix.
classmethod from_estimator(rejector, X, y, *, labels=None, sample_weight=None, normalize=None, display_labels=None, include_values=True, xticks_rotation='horizontal', values_format=None, cmap='viridis', ax=None, colorbar=True, im_kw=None)[source]

Plots PA Confusion Matrix given a rejector and some data.

Parameters:

  • estimator (rejector instance) – Fitted classifier or a fitted Pipeline in which the last rejector is a classifier.

  • X ({array-like, sparse matrix} of shape (n_samples, n_features)) – Input values.

  • y (FBNDArray of shape (n_samples,)) – Target values.

  • labels (array-like of shape (n_classes,), default=None) – List of labels to index the confusion matrix. This may be used to reorder or select a subset of labels. If None is given, those that appear at least once in y_true or y_pred are used in sorted order.

  • sample_weight (array-like of shape (n_samples,), default=None) – Sample weights.

  • normalize ({'true', 'pred', 'all'}, default=None) –

    Either to normalize the counts display in the matrix: - if ‘true’, the confusion matrix is normalized over the true

    conditions (e.g. rows);

    • if ‘pred’, the confusion matrix is normalized over the predicted conditions (e.g. columns);

    • if ‘all’, the confusion matrix is normalized by the total number of samples;

    • if None (default), the confusion matrix will not be normalized.

  • display_labels (array-like of shape (n_classes,), default=None) – Target names used for plotting. By default, labels will be used if it is defined, otherwise the unique labels of y_true and y_pred will be used.

  • include_values (bool, default=True) – Includes values in confusion matrix.

  • xticks_rotation ({'vertical', 'horizontal'} or float, default='horizontal') – Rotation of xtick labels.

  • values_format (str, default=None) – Format specification for values in confusion matrix. If None, the format specification is ‘d’ or ‘.2g’ whichever is shorter.

  • cmap (str or matplotlib Colormap, default='viridis') – Colormap recognized by matplotlib.

  • ax (matplotlib Axes, default=None) – Axes object to plot on. If None, a new figure and axes is created.

  • colorbar (bool, default=True) – Whether or not to add a colorbar to the plot.

  • im_kw (dict, default=None) – Dict with keywords passed to matplotlib.pyplot.imshow call.

Returns:

display

Return type:

PAConfusionMatrixDisplay

See also

PAConfusionMatrixDisplay.from_predictions

Examples

>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> from sklearn.linear_model import LogisticRegression
>>> from skfb.estimators import ThresholdFallbackClassifier
>>> from skfb.metrics import PAConfusionMatrixDisplay
>>> X = np.array([[0, 0], [4, 4], [1, 1], [3, 3], [2.5, 2], [2., 2.5]])
>>> y = np.array([0, 1, 0, 1, 0, 1])
>>> estimator = LogisticRegression(random_state=0)
>>> rejector = ThresholdFallbackClassifier(estimator, threshold=0.6).fit(X, y)
>>> PAConfusionMatrixDisplay.from_estimator(rejector, X, y)
<...>
>>> plt.show()
classmethod from_predictions(y_true, y_pred, *, labels=None, sample_weight=None, normalize=None, display_labels=('No', 'Yes'), include_values=True, xticks_rotation='horizontal', values_format=None, cmap='viridis', ax=None, colorbar=True, im_kw=None)[source]

Plots PA Confusion Matrix given true and predicted labels.

Parameters:

  • y_true (array-like of shape (n_samples,)) – True labels.

  • y_pred (FBNDArray (n_samples,)) – The predicted labels w/ the fallback mask given by the method predict of a rejector

  • labels (array-like of shape (n_classes,), default=None) – List of labels to index the confusion matrix. This may be used to reorder or select a subset of labels. If None is given, those that appear at least once in y_true or y_pred are used in sorted order.

  • sample_weight (array-like of shape (n_samples,), default=None) – Sample weights.

  • normalize ({'true', 'pred', 'all'}, default=None) –

    Either to normalize the counts display in the matrix: - if ‘true’, the confusion matrix is normalized over the true

    conditions (e.g. rows);

    • if ‘pred’, the confusion matrix is normalized over the predicted conditions (e.g. columns);

    • if ‘all’, the confusion matrix is normalized by the total number of samples;

    • if None (default), the confusion matrix will not be normalized.

  • display_labels (array-like of shape (n_classes,), default=None) – Target names used for plotting. By default, labels will be used if it is defined, otherwise the unique labels of y_true and y_pred will be used.

  • include_values (bool, default=True) – Includes values in confusion matrix.

  • xticks_rotation ({'vertical', 'horizontal'} or float, default='horizontal') – Rotation of xtick labels.

  • values_format (str, default=None) – Format specification for values in confusion matrix. If None, the format specification is ‘d’ or ‘.2g’ whichever is shorter.

  • cmap (str or matplotlib Colormap, default='viridis') – Colormap recognized by matplotlib.

  • ax (matplotlib Axes, default=None) – Axes object to plot on. If None, a new figure and axes is created.

  • colorbar (bool, default=True) – Whether or not to add a colorbar to the plot.

  • im_kw (dict, default=None) – Dict with keywords passed to matplotlib.pyplot.imshow call.

Returns:

display

Return type:

PAConfusionMatrixDisplay

See also

PAConfusionMatrixDisplay.from_estimator

Examples

>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> from sklearn.linear_model import LogisticRegression
>>> from skfb.estimators import ThresholdFallbackClassifier
>>> from skfb.metrics import PAConfusionMatrixDisplay
>>> X = np.array([[0, 0], [4, 4], [1, 1], [3, 3], [2.5, 2], [2., 2.5]])
>>> y = np.array([0, 1, 0, 1, 0, 1])
>>> estimator = LogisticRegression(random_state=0)
>>> rejector = ThresholdFallbackClassifier(estimator, threshold=0.6).fit(X, y)
>>> y_pred = rejector.predict(X)
>>> PAConfusionMatrixDisplay.from_predictions(y, y_pred)
<...>
>>> plt.show()
plot(*, include_values=True, cmap='viridis', xticks_rotation='horizontal', values_format=None, ax=None, colorbar=True, im_kw=None)[source]

Plots predict-accept confusion matrix.

Same as plot except that changes the label names.

Parameters:

  • include_values (bool, default=True) – Includes values in confusion matrix.

  • cmap (str or matplotlib Colormap, default='viridis') – Colormap recognized by matplotlib.

  • xticks_rotation ({'vertical', 'horizontal'} or float, default='horizontal') – Rotation of xtick labels.

  • values_format (str, default=None) – Format specification for values in confusion matrix. If None, the format specification is ‘d’ or ‘.2g’ whichever is shorter.

  • ax (matplotlib axes, default=None) – Axes object to plot on. If None, a new figure and axes is created.

  • colorbar (bool, default=True) – Whether or not to add a colorbar to the plot.

  • im_kw (dict, default=None) – Dict with keywords passed to matplotlib.pyplot.imshow call.

Returns:

display – Returns a PAConfusionMatrixDisplay instance that contains all the information to plot the confusion matrix.

Return type:

PAConfusionMatrixDisplay

Examples

>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> from skfb.core import array as ska
>>> from skfb.metrics import predict_accept_confusion_matrix
>>> from skfb.metrics import PAConfusionMatrixDisplay
>>> y_true =    np.array([0, 1, 0, 0, 1, 1, 0, 1, 0, 1])
>>> y_pred = ska.fbarray([0, 1, 0, 1, 0, 1, 1, 1, 0, 1],
...                      [1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
>>> cm = predict_accept_confusion_matrix(y_true=y_true, y_pred=y_pred)
>>> PAConfusionMatrixDisplay(cm).plot()
<...>
>>> plt.show()
class skfb.metrics.FQCurveDisplay(*, fallback_rates, scores, fq_auc=None, estimator_name=None, metric_name=None)[source]

Fallback-Quality Curve visualization.

It is recommend to use from_estimator() or from_predictions() to create a FQCurveDisplay. All parameters are stored as attributes.

Parameters:

  • fallback_rates (array-like) – Rates of rejected samples.

  • scores (array-like) – Evaluation scores for every fallback rate.

  • fq_auc (float, default=None) – Area under fallback-quality curve.

  • estimator_name (str, default=None) – Name of the estimator predicted the scores.

  • metric_name (str, default=None) – Name of the scoring method.

line_

FQ Curve.

Type:

matplotlib Artist

ax_

Axes with FQ Curve.

Type:

matplotlib Axes

figure_

Figure containing the curve.

Type:

matplotlib Figure

See also

sfkb.metrics.fallback_quality_curve, skfb.metrics.fallback_quality_auc_score

Methods

from_estimator(estimator, X, y, *[, ...])

Plots visualization given an estimator and some data.

from_predictions(y_true, y_pred[, ...])

Plots visualization given true labels and certainty predictions.

plot([ax, line_kwargs, ax_kwargs])

Plots visualization.
classmethod from_estimator(estimator, X, y, *, score_func=<function accuracy_score>, predict_method='predict', min_fallback_rate=0.0, max_fallback_rate=0.95, raise_warning=True, sample_weight=None, estimator_name=None, metric_name=None, ax=None, line_kwargs=None, ax_kwargs=None)[source]

Plots visualization given an estimator and some data.

classmethod from_predictions(y_true, y_pred, score_func=<function accuracy_score>, predict_method='predict', min_fallback_rate=0.0, max_fallback_rate=0.95, raise_warning=True, sample_weight=None, estimator_name=None, metric_name=None, ax=None, line_kwargs=None, ax_kwargs=None)[source]

Plots visualization given true labels and certainty predictions.

plot(ax=None, *, line_kwargs=None, ax_kwargs=None)[source]

Plots visualization.

class skfb.metrics.PairedHistogramDisplay(score_true, score_false)[source]

Plots histograms of probabilities of true and false predictions.

Methods

from_estimator(estimator, X, y, *[, ax, ...])

Plots visualization given an estimator and some data.

from_predictions(y_true, y_score[, ax, ...])

Plots visualization given true labels and certainty predictions.

plot(*[, ax, cumulative])

Plots visualization.
classmethod from_estimator(estimator, X, y, *, ax=None, cumulative=True)[source]

Plots visualization given an estimator and some data.

classmethod from_predictions(y_true, y_score, ax=None, cumulative=True)[source]

Plots visualization given true labels and certainty predictions.

plot(*, ax=None, cumulative=True)[source]

Plots visualization.