phenotypic.tune.compute_generalization_gap#

phenotypic.tune.compute_generalization_gap(cal_score: float, heldout_score: float, *, rel_margin: float, abs_margin: float) → tuple[float, float, bool][source]#

The BOTH-thresholds overfit gate on a calibration→held-out score drop.

Computes the relative and absolute drops and flags overfit only when both margins are exceeded (the HeldOutConfig policy):

  • relative_drop = (cal_score - heldout_score) / max(|cal_score|, eps);

  • absolute_drop = cal_score - heldout_score;

  • flagged = (relative_drop > rel_margin) and (absolute_drop > abs_margin).

Requiring both guards two false positives: a large relative drop that is absolutely negligible (e.g. 0.04 → 0.03 — 25% relative, 0.01 absolute), and a large absolute drop on an objective whose calibration score was so high the relative slack already covers it.

Parameters:

  • cal_score (float) – The winner’s calibration (in-search) score (higher = better; under the cost convention the caller passes the goodness-equivalent 1 − cal_cost — see Note).

  • heldout_score (float) – The winner’s held-out score (higher = better; under the cost convention the caller passes 1 − heldout_cost — see Note).

  • rel_margin (float) – The relative-drop margin (HeldOutConfig.gap_margin_relative).

  • abs_margin (float) – The absolute-drop margin (HeldOutConfig.gap_margin_absolute).

Returns:

A (relative_drop, absolute_drop, flagged) triple. The drops are signed (negative when the held-out score improved); flagged is True only when both exceed their margins.

Return type:

tuple[float, float, bool]

Note

This function is direction-agnostic. Under the cost convention the caller passes goodness-equivalents (1 − cost) so the unchanged formula is the standard loss-space gap (heldout_cost − cal_cost).

Examples

>>> rel, absolute, flagged = compute_generalization_gap(
...     0.9, 0.5, rel_margin=0.15, abs_margin=0.05
... )
>>> round(rel, 3), round(absolute, 3), flagged
(0.444, 0.4, True)
>>> # A tiny absolute drop is never flagged, even at 25% relative.
>>> compute_generalization_gap(0.04, 0.03, rel_margin=0.15, abs_margin=0.05)[2]
False
On this page

This Page