# Hyperparameter Tuning The tune engine searches over an `ImagePipeline`'s parameters and returns the combination that scores best on your plates. It is sample-efficient — grid and seeded-random for small spaces, Optuna samplers (TPE, CMA-ES, GP, NSGA-II) for larger ones — and pairs the search with pluggable scoring objectives and a robust held-out generalization check, so a winning pipeline has to prove it holds up on plates the search never optimized against. The engine replaces the old `sweep` module: instead of enumerating a fixed grid of pipeline variants, you author (or auto-infer) a search space, pick a scoring objective, and let the optimizer propose candidates. :::{admonition} Migrating from a legacy `sweep` manifest :class: tip If you have an existing sweep manifest, convert it to a `tuning_spec.json` — a `Categorical` grid over the manifest's varying parameters (plus `__enabled__` presence knobs for optional ops) — with the migration script: python scripts/migrate_sweep_manifest.py old_manifest.json \ -o tuning_spec.json --metadata layout.csv `--metadata` is the per-image layout CSV/Parquet that feeds the count scorer (grouped by `--groupby`, default `Metadata_ImageName`). Manifests that swept **nested** operations raise `NotImplementedError` — author those spaces by hand against the new `SearchSpace` API. ::: ## Python interface ### Running a study from the CLI The basic run takes a `tuning_spec.json`, an image directory, and an output directory: ```bash uv run phenotypic-tune run spec.json -i ./plates -o ./out ``` Override the spec's strategy and trial budget, and turn on the two-round screening freeze, from the command line: ```bash uv run phenotypic-tune run spec.json -i ./plates -o ./out \ --strategy tpe \ --n-trials 200 \ --screen ``` `--strategy grid` and `--strategy random` use the built-in configs; `tpe`, `cmaes`, `gp`, and `nsga2` build an `OptunaConfig` and require the `tune` extra (`uv sync --extra tune`). `--screen` enables the two-round screening freeze (it is off by default; `--no-screen` is the explicit default). `--n-trials` overrides the spec's `Budget.n_trials`. A run writes its deliverables under `/deliverables/`: - `best_pipeline.json` — the winning pipeline, ready to run with `python -m phenotypic`. - `tuning_spec.json` — the fully resolved spec (pipeline + space + scorer + strategy + budget), so the run round-trips. - `param_importance.json` — per-parameter importance (random-forest / fANOVA) showing which knobs moved the objective. - `pareto/` — the non-dominated front, for a multi-objective (`CompositeScorer(multi_objective=True)`) study. - `generalization.json` — the held-out verdict: how the winner scored on the reserved plates versus the search set. Alongside the deliverables, `trials.parquet` is written at the **output root**. It is the published trial archive, including PRUNED partial evaluations. PRUNED trials consume terminal budget but never participate in best, per-axis, Pareto-front, or knee-point selection; only finite COMPLETE trials are eligible for those published choices. Optuna keeps its live local store in `.pht-tune-cache/study.db`; a SLURM run instead records one shared `.pht-tune-cache/journal.log` and publishes `trials.parquet` only after the terminal finalizer verifies the budget and a COMPLETE winner. **Resume** is automatic for a local run: re-running `run` with `-o` pointed at a prior directory reopens its local `.pht-tune-cache/study.db`. A distributed run records its shared journal in the run marker and is published only by its terminal lifecycle finalizer. Do not relaunch an output with an active lifecycle; after scheduler quiescence, use `uv run phenotypic-tune finalize OUTPUT` to recover publication from the recorded backend. ### Authoring a `TuningSpec` in Python A `TuningSpec` is one self-contained, round-trippable recipe. Its fields are: - `pipeline` — the base `ImagePipeline` being tuned (embedded in the spec). - `search_space` — the `SearchSpace` of `Knob`s to search over. - `scorer` — the tuning objective (any `Scorer` subclass; see below). - `evaluator` — the candidate-evaluation policy (`Evaluator`). - `strategy` — the optimizer config (`GridConfig`, `RandomConfig`, or `OptunaConfig`). - `budget` — the stopping criteria (`Budget`). - `held_out` — the held-out / generalization policy (defaults to a conservative `HeldOutConfig`). All of these are keyword-only pydantic models. A minimal spec — tune a Gaussian-blur `sigma` ahead of an Otsu detector, scored against an expected 96-well colony count: ```python >>> import pandas as pd >>> from phenotypic import ImagePipeline >>> from phenotypic.enhance import BlurGauss >>> from phenotypic.detect import OtsuDetector >>> from phenotypic.analysis import ExpectedVsDetectedCount >>> from phenotypic.tune import ( ... TuningSpec, Budget, Evaluator, SearchSpace, Knob, FloatRange, ... ) >>> from phenotypic.tune.score import QCScorer >>> from phenotypic.tune.strategy import GridConfig >>> pipe = ImagePipeline(ops=[BlurGauss(sigma=2.0), OtsuDetector()]) >>> space = SearchSpace(knobs=( ... Knob(key="0.sigma", domain=FloatRange(low=0.5, high=8.0)), ... )) >>> layout = pd.DataFrame( ... {"Metadata_ImageName": ["plate1"] * 96, "Object_Label": list(range(96))} ... ) >>> scorer = QCScorer( ... check=ExpectedVsDetectedCount( ... metadata=layout, groupby=["Metadata_ImageName"] ... ) ... ) >>> spec = TuningSpec( ... pipeline=pipe, ... search_space=space, ... scorer=scorer, ... evaluator=Evaluator(), ... strategy=GridConfig(), ... budget=Budget(n_trials=8), ... ) >>> spec.search_space.keys() ['0.sigma'] ``` A knob's `key` is a position-indexed path: `"0.sigma"` targets the `sigma` field of op `0` (the `BlurGauss`). Save the spec with `spec.model_dump_json()` and feed it to `uv run phenotypic-tune run`. For a spec that must round-trip through JSON, configure the count check from a metadata *path* (`ExpectedVsDetectedCount(metadata="layout.csv", ...)`) — a check built from an in-memory frame cannot be rebuilt from JSON. ### Inferring a search space with `auto-space` When you don't want to hand-author every knob, `auto-space` mines a configured pipeline's pydantic fields into a reviewable candidate space — no engine runs: ```bash uv run phenotypic-tune auto-space pipeline.json -o ./out ``` The same inference is available in Python as `infer_search_space`, which returns an `InferredSearchSpace` carrying: - `.knobs` — the inferred `Knob`s (each with a `source` provenance tag and a `needs_review` flag). - `.excluded` — `Excluded` records for fields inference declined to tune, each with a reason. - `.needs_review` — `True` when any knob is flagged for review (or a field was excluded blindly). Flagged knobs (`needs_review=True`) keep the autonomy gate conservative: an unattended run stays cautious until a human confirms the shaky guesses, so a generously-inferred bound never silently drives a fully-automatic study. ## The four scoring objectives Every scorer is a `Scorer` subclass that emits a bounded cost in `[0, 1]`: `0` is perfect, `1` is worst, and every objective is minimized. ### `QCScorer` Use it when you have **no ground-truth masks** but you know the expected colony **count** per plate (e.g. a 96-well layout). It is a purely statistical check: it wraps `phenotypic.analysis.ExpectedVsDetectedCount` and converts each `groupby` unit's `|detected − expected| / expected` discrepancy into a bounded lower-is-better cost. It requires a configured count check on its `check` field. Configure that check from a metadata **path** so the scorer round-trips through `tuning_spec.json`: ```python from phenotypic.analysis import ExpectedVsDetectedCount from phenotypic.tune.score import QCScorer scorer = QCScorer( check=ExpectedVsDetectedCount( metadata="layout.csv", groupby=["Metadata_ImageName"] ) ) ``` ### `ReferenceFreeScorer` Use it when you have **no ground truth at all** — not even expected counts — and want to score segmentation quality from the result itself. It computes fixed-normalized proxy terms: `ShapeRegularity` (mean shape-prior plausibility from the `Shape_*` columns), `Contrast` (Otsu between-class separation of foreground vs. background), and `SizeCV` (within-replicate size uniformity). Supply an optional `count_check` to add the `Count` term. It requires nothing to construct, but it is **gated behind meta-validation**: `availability()` returns `False` (so the engine fails safe and degrades to `QCScorer`) until the proxy is correlated against ground truth and clears the enable threshold (Spearman ρ ≥ 0.7; ρ ≥ 0.8 for fully unattended tuning). Point `gt_masks_source` at a ground-truth source to let the gate validate; without it the gate abstains and the scorer stays unavailable. ```python from phenotypic.tune.score import ReferenceFreeScorer scorer = ReferenceFreeScorer(replicate_groupby=["Metadata_ImageName"]) ``` ### `SupervisedScorer` Use it when you **have ground-truth annotations**. It is modality-tiered: a directory of per-image masks runs the **mask tier** (per-object Dice or IoU, macro-averaged into the `Region` term); a `.csv`/`.parquet` count table runs the **count tier** (count divergence into the `CountMAE` term); nothing resolvable makes it abstain. It requires a `GroundTruthMasks` loader on its `gt` field, whose `gt_masks_source` path selects the modality. The mask tier carries exactly one region metric (`region_metric="dice"` or `"iou"` — they rank identically); the count tier additionally needs a configured `count_check`: ```python from phenotypic.analysis import ExpectedVsDetectedCount from phenotypic.tune.score import GroundTruthMasks, SupervisedScorer # Mask tier: a directory of per-image masks. masks = SupervisedScorer( gt=GroundTruthMasks(gt_masks_source="./gt_masks/"), region_metric="dice", ) # Count tier: a count table plus the reused count check. counts = SupervisedScorer( gt=GroundTruthMasks(gt_masks_source="./counts.csv"), count_check=ExpectedVsDetectedCount( metadata="counts.csv", groupby=["Metadata_ImageName"] ), ) ``` ### `CompositeScorer` Use it to **combine** the above into one objective — the small complementary panel no single metric covers. It nests a `list[Scorer]` on its `scorers` field and composes them two ways: - **single-objective** (default): an augmented Tchebycheff blend that is worst-axis-dominant; set `blend="weighted_mean"` for the compensatory option. A geometric mean is intentionally unavailable because one zero-cost axis would annihilate the product and mask weak axes. - **multi-objective** (`multi_objective=True`): per-child objectives kept separate, driving a true Pareto study (requires an Optuna `nsga2` strategy; pairing a multi-objective scorer with grid/random is rejected at spec construction). The non-dominated front lands in `deliverables/pareto/`. ```python from phenotypic.tune.score import CompositeScorer scorer = CompositeScorer( scorers=[qc_scorer, reference_free_scorer], weights={"s0": 2.0, "s1": 1.0}, ) ``` ## GUI interface ```{admonition} Currently unmounted :class: warning The `/tune/` co-pilot described below is unmounted from the GUI hub (see `docs/superpowers/specs/2026-08-26-gui-simplification-removals`, §2): its code is retained on disk and unit-tested, but it is not reachable from `phenotypic-gui`'s top-bar navigation. The section below documents its design for when it is re-mounted; use the CLI interface above for tuning in the meantime. ``` `phenotypic-gui` mounts a `/tune/` Dash co-pilot as a tab alongside **Home · Builder · Tune · Run · Viewer · Analysis**. The co-pilot is a full **author → run → monitor** workflow organised as three top-level destinations, selected from the destination row at the top of the page: ### Setup — author a tuning spec The landing destination. Point it at a base pipeline (or an existing tuning spec) and a **metadata layout CSV/Parquet** for the default `QCScorer`; the search-space and scorer sections stay locked until a pipeline is present, and **Continue** stays blocked until the required scorer metadata is supplied. Continue writes a typed, path-backed authored spec under `/.phenotypic-gui/presets/tune/` and advances to **Run**. (Review and edit the inferred search space itself on the **Space** sub-tab under Monitor — see below.) ### Run — configure and launch The Run destination (unlocked once a pipeline is chosen) is a launch form whose live command card mirrors your choices into the real `uv run phenotypic-tune run` invocation — the same subcommand and flag names you would type by hand. It exposes: - **Inputs** — an in-sandbox image-source override (blank falls back to the shared source root) and the output directory. - **Strategy** — strategy dropdown (`grid` / `random` / `tpe` / `cmaes` / `gp` / `nsga2`), trial budget, and storage URL. - **Compute** — Local vs. SLURM, the two-round screening toggle, and the SLURM fleet sizing fields (workers, partition, memory, time). - **Evaluation** — held-out fraction and CV-group overrides. A pre-flight gate blocks **Deploy** when the configuration cannot run (for example a `grid` search over a continuous `FloatRange`). Unlike the earlier read-only co-pilot, **Deploy actually launches the run**: it registers the run in the shared run registry and starts it through the local runner, or submits the SLURM fleet. You no longer have to copy the command out and run it in a terminal (though the live command card still lets you, if you prefer). ### Monitor — watch, curate, and export Once a run is deployed (or you **Bind run** to an existing `phenotypic-tune` output directory via the sandbox-bounded run picker), the Monitor destination opens the live read over the study. It carries a run switcher across registered Local/SLURM runs, a Local-only **Cancel** (SLURM runs are not killed from the GUI in v1), and an **Export best pipeline** button — plus the four classic sub-tabs: - **Monitor** — the live study read: objective curve (raw scores + running best), a parameter-importance bar chart, a winner-stability (generalization-gap) badge, and a trials table, polled every 3 s while a run is in flight. If the live study store is unreachable it degrades to the finished `trials.parquet` journal. A Pareto-front card appears for multi-objective runs only. - **Curate** — a shortlist of the best (and gap-flagged) trials; pin two into **A** / **B** slots and compare their colony overlays side-by-side (or as a single difference image) on any plate, with linked pan/zoom. "Set as winner" writes `best_pipeline.json`. - **Space** — the inferred search space as editable knob-rows; export the edited space back to `tuning_spec.json`. - **Launch** — a strategy / budget / storage-URL form whose live command card renders a copy-paste `uv run phenotypic-tune run` invocation. The co-pilot keeps its import surface optuna-free — the live study is opened lazily inside the Monitor poll only — and binding an existing run never writes to its directory. ## Distributed runs For running a single tune study across many SLURM compute nodes that share one Optuna journal (via `--slurm`), see [Distributed Tuning on HPCC Clusters](tune_distributed_hpcc.md).