phenotypic.tune.run_held_out

phenotypic.tune.run_held_out(spec: Any, winner: Any, split: Any, images_by_name: dict[str, Any], *, current_identity: str | None = None) → GeneralizationReport

Re-evaluate the winner on the held-out plates → a GeneralizationReport.

The report-only generalization pass (RESOLVED design — held-out orchestration lives in the run layer, never the engine). It re-runs the winner’s parameters through the spec’s own Evaluator over the held-out plates only, then builds a 3-tier verdict by split.kind:

• "group": a real held-out gap (the strongest cross-batch test);

• "within_group": a real held-out gap plus a weaker-guarantee caveat;

• "none": data-poor — no untouched held-out set, so no real gap (gap=None, flagged=False); the report falls back to a calibration-stability estimate carrying the winner’s Trial.gap (the per-trial calibration dispersion) with cv_deferred=True.

The winner is never changed; Trial.gap is not mutated (Option A). When current_identity differs from split.dataset_identity the report’s dataset_changed is set with a drift warning (the split is reused verbatim on resume; this only annotates the verdict).

Parameters:

• spec (Any) – The resolved tuning spec — only spec.evaluator, spec.pipeline, spec.scorer, and spec.held_out (the gap margins) are read.

• winner (Any) – The winning trial — winner.params, winner.score, and winner.gap are read.

• split (Any) – The resolved split (its kind / held_out / group_key / within_group_caveat / dataset_identity drive the verdict).

• images_by_name (dict[str, Any]) – {image.name: image} of the loaded plates.

• current_identity (str | None) – The current dataset identity (a mismatch vs split.dataset_identity sets dataset_changed); None skips the drift check.

Returns:

The GeneralizationReport to write to generalization.json.

Return type:

GeneralizationReport