Classification Metrics

Classification metrics and plots for model evaluation.

class openadmet.models.eval.classification.ClassificationMetrics(*, n_resamples: int = 9999, bootstrap_confidence_level: float = 0.95, use_wandb: bool = False, **extra_data: Any)[source]

Bases: EvalBase

Compute and report classification metrics such as accuracy, precision, recall, F1, ROC AUC, and PR AUC.

Variables:

  • bootstrap_confidence_level (float) – Confidence level for the bootstrap.

  • use_wandb (bool) – Whether to log metrics to Weights & Biases.

  • _evaluated (bool) – Whether the evaluation has been performed.

  • _metrics (dict) – Dictionary of metrics to compute, with metric functions and properties.

bootstrap_confidence_level: float
evaluate(y_true=None, y_pred=None, use_wandb=False, tag=None, **kwargs)[source]

Evaluate the classification model and compute metrics.

Parameters:

  • y_true (array-like) – True labels.

  • y_pred (array-like) – Predicted probabilities or class labels.

  • use_wandb (bool, optional) – Whether to log metrics to Weights & Biases.

  • tag (str, optional) – Tag for the evaluation run.

  • kwargs (Dict) – Additional keyword arguments

Returns:

Dictionary of computed metrics and confidence intervals.

Return type:

dict

property metric_names

Return the metric names.

Returns:

List of metric names.

Return type:

list of str

model_config: ClassVar[ConfigDict] = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context: Any, /) → None

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Args:

self: The BaseModel instance. context: The context.

report(write=False, output_dir=None)[source]

Report the evaluation results, optionally writing to disk.

Parameters:

  • write (bool, optional) – Whether to write the report to disk.

  • output_dir (str, optional) – Output directory for the report.

Returns:

Dictionary of computed metrics.

Return type:

dict

use_wandb: bool
write_report(output_dir)[source]

Write the evaluation report to a JSON file and optionally log to wandb.

Parameters:

output_dir (str) – Output directory for the report.

class openadmet.models.eval.classification.ClassificationPlots(*, n_resamples: int = 9999, plots: dict = {}, use_wandb: bool = False, dpi: int = 300, **extra_data: Any)[source]

Bases: EvalBase

Generate and save classification plots such as ROC and PR curves.

Variables:

  • plots (dict) – Dictionary of plot functions.

  • use_wandb (bool) – Whether to log plots to Weights & Biases.

  • dpi (int) – DPI for the plots.

dpi: int
evaluate(y_true=None, y_pred=None, use_wandb=False, **kwargs)[source]

Generate classification plots.

Parameters:

  • y_true (array-like) – True labels.

  • y_pred (array-like) – Predicted probabilities or class labels.

  • use_wandb (bool, optional) – Whether to log plots to Weights & Biases.

  • kwargs (Dict) – Additional keyword arguments.

model_config: ClassVar[ConfigDict] = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

plots: dict
pr_curve(y_true, y_pred, xlabel='Recall', ylabel='Precision', title='Precision-Recall Curve')[source]

Plot the precision-recall curve.

Parameters:

  • y_true (array-like) – True labels.

  • y_pred (array-like) – Predicted probabilities or class labels.

  • xlabel (str, optional) – Label for the x-axis.

  • ylabel (str, optional) – Label for the y-axis.

  • title (str, optional) – Title for the plot.

Returns:

The precision-recall curve figure.

Return type:

matplotlib.figure.Figure

report(write=False, output_dir=None)[source]

Report the generated plots, optionally writing to disk.

Parameters:

  • write (bool, optional) – Whether to write the plots to disk.

  • output_dir (str, optional) – Output directory for the plots.

Returns:

Dictionary of plot figures.

Return type:

dict

roc_curve(y_true, y_pred, xlabel='False Positive Rate', ylabel='True Positive Rate', title='Receiver Operating Characteristic Curve')[source]

Plot the ROC curve.

Parameters:

  • y_true (array-like) – True labels.

  • y_pred (array-like) – Predicted probabilities or class labels.

  • xlabel (str, optional) – Label for the x-axis.

  • ylabel (str, optional) – Label for the y-axis.

  • title (str, optional) – Title for the plot.

Returns:

The ROC curve figure.

Return type:

matplotlib.figure.Figure

use_wandb: bool
write_report(output_dir)[source]

Write the generated plots to PNG files and optionally log to wandb.

Parameters:

output_dir (str) – Output directory for the plots.

openadmet.models.eval.classification.pr_auc_score(y_true, y_pred)[source]

Calculate the area under the precision-recall curve.

Parameters:

  • y_true (array-like) – True binary labels or binary label indicators.

  • y_pred (array-like) – Target scores, probability estimates of the positive class.

Returns:

Area under the precision-recall curve.

Return type:

float