Uncertainty Metrics

Evaluators for uncertainty quantification metrics and plots.

class openadmet.models.eval.uncertainty.UncertaintyMetrics(*, n_resamples: int = 9999, use_wandb: bool = False, **extra_data: Any)[source]

Bases: EvalBase

Evaluator for uncertainty metrics using uncertainty_toolbox.

Variables:

  • use_wandb (bool) – Whether to use wandb for logging.

  • _data (dict) – Stores computed metrics for each task.

  • _metrics (dict) – Mapping of metric keys to human-readable names.

evaluate(y_true, y_pred, y_std, target_labels=None, bins=100, resolution=99, scaled=True, **kwargs)[source]

Evaluate uncertainty metrics for each task.

Parameters:

  • y_true (array-like) – Ground truth values.

  • y_pred (array-like) – Predicted mean values.

  • y_std (array-like) – Predicted standard deviations.

  • target_labels (list of str, optional) – List of target labels for each task.

  • bins (int, default=100) – Number of bins for calibration metrics.

  • resolution (int, default=99) – Resolution for scoring rule metrics.

  • scaled (bool, default=True) – Whether to scale scoring rule metrics.

  • **kwargs – Additional keyword arguments.

Raises:

ValueError – If required inputs are missing or shapes are inconsistent.

property metric_names

Get the list of metric keys.

Returns:

List of metric keys.

Return type:

list of str

model_config: ClassVar[ConfigDict] = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context: Any, /) → None

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Args:

self: The BaseModel instance. context: The context.

report(write=False, output_dir=None)[source]

Report the evaluation results.

Parameters:

  • write (bool, default=False) – Whether to write the report to disk.

  • output_dir (Path or str, optional) – Directory to write the report to.

Returns:

Dictionary of computed metrics.

Return type:

dict

property task_names

Get the list of evaluated task names.

Returns:

List of task names.

Return type:

list of str

use_wandb: bool
write_report(output_dir)[source]

Write the evaluation report to disk and optionally log to wandb.

Parameters:

output_dir (Path or str) – Directory to write the report to.

class openadmet.models.eval.uncertainty.UncertaintyPlots(*, n_resamples: int = 9999, use_wandb: bool = False, dpi: int = 300, **extra_data: Any)[source]

Bases: EvalBase

Evaluator for generating uncertainty plots.

Variables:

  • use_wandb (bool) – Whether to use wandb for logging.

  • dpi (int) – DPI for the generated plots.

  • _plots (dict) – Mapping of plot tags to plotting functions.

  • _plot_data (dict) – Stores generated plot figures.

static calibration_plot(y_true, y_pred, y_std, title='', dpi=300)[source]

Create a calibration plot for uncertainty estimates.

Parameters:

  • y_true (array-like) – Ground truth values.

  • y_pred (array-like) – Predicted mean values.

  • y_std (array-like) – Predicted standard deviations.

  • title (str, default=””) – Title for the plot.

  • dpi (int, default=300) – DPI for the plot.

Returns:

The generated calibration plot figure.

Return type:

matplotlib.figure.Figure

dpi: int
evaluate(y_true, y_pred, y_std, target_labels=None, **kwargs)[source]

Generate uncertainty plots for each task.

Parameters:

  • y_true (array-like) – Ground truth values.

  • y_pred (array-like) – Predicted mean values.

  • y_std (array-like) – Predicted standard deviations.

  • target_labels (list of str, optional) – List of target labels for each task.

  • **kwargs – Additional keyword arguments.

Returns:

Dictionary of generated plot figures.

Return type:

dict

Raises:

ValueError – If required inputs are missing or shapes are inconsistent.

model_config: ClassVar[ConfigDict] = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(_UncertaintyPlots__context)[source]

Post-initialization hook to set plot types.

Parameters:

__context (Any) – Pydantic context (unused).

report(write=False, output_dir=None)[source]

Report the generated plots.

Parameters:

  • write (bool, default=False) – Whether to write the plots to disk.

  • output_dir (Path or str, optional) – Directory to write the plots to.

Returns:

Dictionary of generated plot figures.

Return type:

dict

use_wandb: bool
write_report(output_dir)[source]

Write the generated plots to disk and optionally log to wandb.

Parameters:

output_dir (Path or str) – Directory to write the plots to.