phenotypic.tune.GeneralizationReport#

class phenotypic.tune.GeneralizationReport(kind: Literal['group', 'within_group', 'none'], calibration_score: float, heldout_score: float | None, relative_drop: float | None, absolute_drop: float | None, gap: float | None, flagged: bool, estimate: Literal['held_out', 'calibration_stability'], cv_deferred: bool, within_group_caveat: bool, dataset_changed: bool, warning: str | None, gap_margin_relative: float, gap_margin_absolute: float, calibration_stability: float | None = None)[source]#

Bases: object

The winner’s held-out generalization verdict — a frozen deliverable.

Serialized to deliverables/generalization.json (via to_dict()). It is report-only: the winner is never changed by the held-out pass, and the immutable trial journal is untouched (the per-trial Trial.gap stays the calibration dispersion; the TRUE held-out gap lives only here).

Parameters:

  • kind (Literal['group', 'within_group', 'none']) – The split tier this verdict was produced under — "group" (whole-group hold-out, the strongest cross-batch test), "within_group" (a weaker, within-group hold-out), or "none" (data-poor — no real held-out gap, calibration-stability fallback).

  • calibration_score (float) – The winner’s calibration (in-search) score.

  • heldout_score (float | None) – The winner’s held-out score, or None for the data-poor fallback (no untouched held-out set was evaluated).

  • relative_drop (float | None) – The relative calibration→held-out drop, or None for the data-poor fallback.

  • absolute_drop (float | None) – The absolute calibration→held-out drop, or None for the data-poor fallback.

  • gap (float | None) – The true held-out generalization gap (= absolute_drop), or None for the data-poor fallback.

  • flagged (bool) – True when the overfit gate fired (both margins exceeded); always False for the data-poor fallback. Report-only — it never changes the winner.

  • estimate (Literal['held_out', 'calibration_stability']) – "held_out" when a real held-out pass ran, else "calibration_stability" (the data-poor proxy).

  • cv_deferred (bool) – True for the data-poor fallback — the §8 cross-validation estimate is deferred; the report substitutes a calibration-stability proxy (see DEFERRED-WORK.md).

  • within_group_caveat (bool) – True for kind="within_group" — the held-out plates share a group with calibration, so the guarantee is weaker.

  • dataset_changed (bool) – True when the loaded plates no longer match the persisted split’s dataset_identity (the verdict still reflects the original split membership; resume reuses the persisted split).

  • warning (str | None) – A human-readable caveat (within-group / data-poor / dataset-drift), or None when the verdict carries the strongest guarantee.

  • gap_margin_relative (float) – The relative margin the overfit gate used.

  • gap_margin_absolute (float) – The absolute margin the overfit gate used.

  • calibration_stability (float | None) – The winner’s per-trial calibration dispersion (Trial.gap) — the data-poor proxy for a held-out gap; None when a real held-out pass ran (or the winner had no gap signal).

Methods

__init__

to_dict

The JSON-serializable mapping written to generalization.json.

Attributes

calibration_stability

kind

calibration_score

heldout_score

relative_drop

absolute_drop

gap

flagged

estimate

cv_deferred

within_group_caveat

dataset_changed

warning

gap_margin_relative

gap_margin_absolute

kind: Literal['group', 'within_group', 'none']#
calibration_score: float#
heldout_score: float | None#
relative_drop: float | None#
absolute_drop: float | None#
gap: float | None#
flagged: bool#
estimate: Literal['held_out', 'calibration_stability']#
cv_deferred: bool#
within_group_caveat: bool#
dataset_changed: bool#
warning: str | None#
gap_margin_relative: float#
gap_margin_absolute: float#
calibration_stability: float | None = None#
to_dict() → dict[str, Any][source]#

The JSON-serializable mapping written to generalization.json.

Returns:

A plain dict of the report fields (all JSON-native scalars), ready for json.dumps(..., indent=2).

Return type:

dict[str, Any]