Hyperparameter Tuning#

The tune engine searches over an ImagePipeline’s parameters and returns the combination that scores best on your plates. It is sample-efficient — grid and seeded-random for small spaces, Optuna samplers (TPE, CMA-ES, GP, NSGA-II) for larger ones — and pairs the search with pluggable scoring objectives and a robust held-out generalization check, so a winning pipeline has to prove it holds up on plates the search never optimized against.

The engine replaces the old sweep module: instead of enumerating a fixed grid of pipeline variants, you author (or auto-infer) a search space, pick a scoring objective, and let the optimizer propose candidates.

```{admonition} Migrating from a legacy sweep manifest :class: tip

If you have an existing sweep manifest, convert it to a tuning_spec.json — a Categorical grid over the manifest’s varying parameters (plus __enabled__ presence knobs for optional ops) — with the migration script:

python scripts/migrate_sweep_manifest.py old_manifest.json \
    -o tuning_spec.json --metadata layout.csv

--metadata is the per-image layout CSV/Parquet that feeds the count scorer (grouped by --groupby, default Metadata_ImageName). Manifests that swept nested operations raise NotImplementedError — author those spaces by hand against the new SearchSpace API.


## Python interface

### Running a study from the CLI

The basic run takes a `tuning_spec.json`, an image directory, and an output
directory:

```bash
python -m phenotypic.tune run spec.json -i ./plates -o ./out

Override the spec’s strategy and trial budget, and turn on the two-round screening freeze, from the command line:

python -m phenotypic.tune run spec.json -i ./plates -o ./out \
    --strategy tpe \
    --n-trials 200 \
    --screen

--strategy grid and --strategy random use the built-in configs; tpe, cmaes, gp, and nsga2 build an OptunaConfig and require the tune extra (uv sync --extras tune). --screen enables the two-round screening freeze (it is off by default; --no-screen is the explicit default). --n-trials overrides the spec’s Budget.n_trials.

A run writes its deliverables under <output>/deliverables/:

  • best_pipeline.json — the winning pipeline, ready to run with python -m phenotypic.

  • tuning_spec.json — the fully resolved spec (pipeline + space + scorer + strategy + budget), so the run round-trips.

  • param_importance.json — per-parameter importance (random-forest / fANOVA) showing which knobs moved the objective.

  • pareto/ — the non-dominated front, for a multi-objective (CompositeScorer(multi_objective=True)) study.

  • generalization.json — the held-out verdict: how the winner scored on the reserved plates versus the search set.

Alongside the deliverables, trials.parquet is written at the output root. It is dual-purpose: the Optuna resume store and the trial journal.

Resume is automatic. Re-running run with -o pointed at a prior run directory resumes the study rather than restarting it — grid/random pick up from trials.parquet, while the Optuna samplers resume from the study store (the local .pht-tune-cache/study.db or the --storage-url study). A killed or extended run picks up its prior trials.

Authoring a TuningSpec in Python#

A TuningSpec is one self-contained, round-trippable recipe. Its fields are:

  • pipeline — the base ImagePipeline being tuned (embedded in the spec).

  • search_space — the SearchSpace of Knobs to search over.

  • scorer — the tuning objective (any Scorer subclass; see below).

  • evaluator — the candidate-evaluation policy (Evaluator).

  • strategy — the optimizer config (GridConfig, RandomConfig, or OptunaConfig).

  • budget — the stopping criteria (Budget).

  • held_out — the held-out / generalization policy (defaults to a conservative HeldOutConfig).

All of these are keyword-only pydantic models. A minimal spec — tune a Gaussian-blur sigma ahead of an Otsu detector, scored against an expected 96-well colony count:

>>> import pandas as pd
>>> from phenotypic import ImagePipeline
>>> from phenotypic.enhance import GaussianBlur
>>> from phenotypic.detect import OtsuDetector
>>> from phenotypic.analysis import ExpectedVsDetectedCount
>>> from phenotypic.tune import (
...     TuningSpec, Budget, Evaluator, SearchSpace, Knob, FloatRange,
... )
>>> from phenotypic.tune.score import QCScorer
>>> from phenotypic.tune.strategy import GridConfig
>>> pipe = ImagePipeline(ops=[GaussianBlur(sigma=2.0), OtsuDetector()])
>>> space = SearchSpace(knobs=(
...     Knob(key="0.sigma", domain=FloatRange(low=0.5, high=8.0)),
... ))
>>> layout = pd.DataFrame(
...     {"Metadata_ImageName": ["plate1"] * 96, "Object_Label": list(range(96))}
... )
>>> scorer = QCScorer(
...     check=ExpectedVsDetectedCount(
...         metadata=layout, groupby=["Metadata_ImageName"]
...     )
... )
>>> spec = TuningSpec(
...     pipeline=pipe,
...     search_space=space,
...     scorer=scorer,
...     evaluator=Evaluator(),
...     strategy=GridConfig(),
...     budget=Budget(n_trials=8),
... )
>>> spec.search_space.keys()
['0.sigma']

A knob’s key is a position-indexed path: "0.sigma" targets the sigma field of op 0 (the GaussianBlur). Save the spec with spec.model_dump_json() and feed it to python -m phenotypic.tune run. For a spec that must round-trip through JSON, configure the count check from a metadata path (ExpectedVsDetectedCount(metadata="layout.csv", ...)) — a check built from an in-memory frame cannot be rebuilt from JSON.

Inferring a search space with auto-space#

When you don’t want to hand-author every knob, auto-space mines a configured pipeline’s pydantic fields into a reviewable candidate space — no engine runs:

python -m phenotypic.tune auto-space pipeline.json -o ./out

The same inference is available in Python as infer_search_space, which returns an InferredSearchSpace carrying:

  • .knobs — the inferred Knobs (each with a source provenance tag and a needs_review flag).

  • .excluded — Excluded records for fields inference declined to tune, each with a reason.

  • .needs_review — True when any knob is flagged for review (or a field was excluded blindly).

Flagged knobs (needs_review=True) keep the autonomy gate conservative: an unattended run stays cautious until a human confirms the shaky guesses, so a generously-inferred bound never silently drives a fully-automatic study.

The four scoring objectives#

Every scorer is a Scorer subclass with a higher-is-better objective in [0, 1]. Pick the one that matches the ground truth you have.

QCScorer#

Use it when you have no ground-truth masks but you know the expected colony count per plate (e.g. a 96-well layout). It is a purely statistical check: it wraps phenotypic.analysis.ExpectedVsDetectedCount and scores each groupby unit on |detected − expected| / expected, folded to a higher-is-better term.

It requires a configured count check on its check field. Configure that check from a metadata path so the scorer round-trips through tuning_spec.json:

from phenotypic.analysis import ExpectedVsDetectedCount
from phenotypic.tune.score import QCScorer

scorer = QCScorer(
    check=ExpectedVsDetectedCount(
        metadata="layout.csv", groupby=["Metadata_ImageName"]
    )
)

ReferenceFreeScorer#

Use it when you have no ground truth at all — not even expected counts — and want to score segmentation quality from the result itself. It computes fixed-normalized proxy terms: ShapeRegularity (mean shape-prior plausibility from the Shape_* columns), Contrast (Otsu between-class separation of foreground vs. background), and SizeCV (within-replicate size uniformity). Supply an optional count_check to add the Count term.

It requires nothing to construct, but it is gated behind meta-validation: availability() returns False (so the engine fails safe and degrades to QCScorer) until the proxy is correlated against ground truth and clears the enable threshold (Spearman ρ ≥ 0.7; ρ ≥ 0.8 for fully unattended tuning). Point gt_masks_source at a ground-truth source to let the gate validate; without it the gate abstains and the scorer stays unavailable.

from phenotypic.tune.score import ReferenceFreeScorer

scorer = ReferenceFreeScorer(replicate_groupby=["Metadata_ImageName"])

SupervisedScorer#

Use it when you have ground-truth annotations. It is modality-tiered: a directory of per-image masks runs the mask tier (per-object Dice or IoU, macro-averaged into the Region term); a .csv/.parquet count table runs the count tier (count divergence into the CountMAE term); nothing resolvable makes it abstain.

It requires a GroundTruthMasks loader on its gt field, whose gt_masks_source path selects the modality. The mask tier carries exactly one region metric (region_metric="dice" or "iou" — they rank identically); the count tier additionally needs a configured count_check:

from phenotypic.analysis import ExpectedVsDetectedCount
from phenotypic.tune.score import GroundTruthMasks, SupervisedScorer

# Mask tier: a directory of per-image masks.
masks = SupervisedScorer(
    gt=GroundTruthMasks(gt_masks_source="./gt_masks/"),
    region_metric="dice",
)

# Count tier: a count table plus the reused count check.
counts = SupervisedScorer(
    gt=GroundTruthMasks(gt_masks_source="./counts.csv"),
    count_check=ExpectedVsDetectedCount(
        metadata="counts.csv", groupby=["Metadata_ImageName"]
    ),
)

CompositeScorer#

Use it to combine the above into one objective — the small complementary panel no single metric covers. It nests a list[Scorer] on its scorers field and composes them two ways:

  • single-objective (default): a weights-weighted arithmetic blend, or (without weights) the geometric mean — so one weak axis cannot be masked by a strong one.

  • multi-objective (multi_objective=True): per-child objectives kept separate, driving a true Pareto study (requires an Optuna nsga2 strategy; pairing a multi-objective scorer with grid/random is rejected at spec construction). The non-dominated front lands in deliverables/pareto/.

from phenotypic.tune.score import CompositeScorer

scorer = CompositeScorer(
    scorers=[qc_scorer, reference_free_scorer],
    weights={"s0": 2.0, "s1": 1.0},
)

GUI interface#

phenotypic-gui mounts a /tune/ Dash co-pilot as a tab alongside Home · Builder · Tune · Run · Viewer · Analysis. The co-pilot is a full author → run → monitor workflow organised as three top-level destinations, selected from the destination row at the top of the page:

Setup — author a tuning spec#

The landing destination. Point it at a base pipeline (or an existing tuning spec) and a metadata layout CSV/Parquet for the default QCScorer; the search-space and scorer sections stay locked until a pipeline is present, and Continue stays blocked until the required scorer metadata is supplied. Continue writes a typed, path-backed authored spec under <root>/.phenotypic-gui/presets/tune/ and advances to Run. (Review and edit the inferred search space itself on the Space sub-tab under Monitor — see below.)

Run — configure and launch#

The Run destination (unlocked once a pipeline is chosen) is a launch form whose live command card mirrors your choices into the real python -m phenotypic.tune run invocation — the same subcommand and flag names you would type by hand. It exposes:

  • Inputs — an in-sandbox image-source override (blank falls back to the shared source root) and the output directory.

  • Strategy — strategy dropdown (grid / random / tpe / cmaes / gp / nsga2), trial budget, and storage URL.

  • Compute — Local vs. SLURM, the two-round screening toggle, and the SLURM fleet sizing fields (workers, partition, memory, time).

  • Evaluation — held-out fraction and CV-group overrides.

A pre-flight gate blocks Deploy when the configuration cannot run (for example a grid search over a continuous FloatRange). Unlike the earlier read-only co-pilot, Deploy actually launches the run: it registers the run in the shared run registry and starts it through the local runner, or submits the SLURM fleet. You no longer have to copy the command out and run it in a terminal (though the live command card still lets you, if you prefer).

Monitor — watch, curate, and export#

Once a run is deployed (or you Bind run to an existing python -m phenotypic.tune output directory via the sandbox-bounded run picker), the Monitor destination opens the live read over the study. It carries a run switcher across registered Local/SLURM runs, a Local-only Cancel (SLURM runs are not killed from the GUI in v1), and an Export best pipeline button — plus the four classic sub-tabs:

  • Monitor — the live study read: objective curve (raw scores + running best), a parameter-importance bar chart, a winner-stability (generalization-gap) badge, and a trials table, polled every 3 s while a run is in flight. If the live study store is unreachable it degrades to the finished trials.parquet journal. A Pareto-front card appears for multi-objective runs only.

  • Curate — a shortlist of the best (and gap-flagged) trials; pin two into A / B slots and compare their colony overlays side-by-side (or as a single difference image) on any plate, with linked pan/zoom. “Set as winner” writes best_pipeline.json.

  • Space — the inferred search space as editable knob-rows; export the edited space back to tuning_spec.json.

  • Launch — a strategy / budget / storage-URL form whose live command card renders a copy-paste python -m phenotypic.tune run invocation.

The co-pilot keeps its import surface optuna-free — the live study is opened lazily inside the Monitor poll only — and binding an existing run never writes to its directory.

For the full walkthrough, see the Tune co-pilot tutorial.

Distributed runs#

For running a single tune study across many SLURM compute nodes that share one Optuna study (via --slurm and a Postgres --storage-url), see Distributed Tuning on HPCC Clusters.