CLI Execution Modes#

The PhenoTypic CLI has one entry point and one switch that decides what a run actually does:

python -m phenotypic --mode {full,measure,recompile,process,migrate} ...

A colony-phenotyping run is expensive in the detection step and cheap in everything downstream. Segmenting a plate costs seconds to minutes per image; re-reading a segmentation to compute Size_Area costs milliseconds. The four modes exist so that when you change your mind about a measurement, a dashboard, or a threshold, you only pay for the part that actually changed. A fifth, migrate, is a one-off format conversion rather than a processing mode.

Mode	Re-runs detection?	Needs `--input`	Needs `--pipeline`	Produces
`full` (default)	Yes	Yes	Yes	Everything: per-image stores, measurements, deliverables, QC, dashboard
`measure`	No	No	Yes	New measurements + deliverables from existing stores
`recompile`	No	No	No	Refreshed aggregate deliverables only
`process`	Yes	Yes	Yes	One exported image layer per input, mirroring the input tree
`migrate`	No	No	No	A legacy `.h5` output tree converted to OME-Zarr stores, in place

--output is required in every mode.

Setting up#

Every command below is runnable. Create a pipeline and a small dataset first:

from pathlib import Path

import tifffile

from phenotypic import ImagePipeline
from phenotypic.data import load_synth_yeast_plate
from phenotypic.detect import OtsuDetector
from phenotypic.measure import MeasureSize

plates = Path("plates/set_a")
plates.mkdir(parents=True, exist_ok=True)

plate = load_synth_yeast_plate()
for i in range(2):
    tifffile.imwrite(plates / f"plate_{i:02d}.tiff", plate.rgb[:])

pipeline = ImagePipeline(ops=[OtsuDetector()], meas=[MeasureSize()])
Path("pipe.json").write_text(pipeline.to_json())

The synthetic plate is not a grid layout, so pass --image-type Image to the commands that follow. The CLI defaults to GridImage (8 × 12), which is the right choice for real arrayed plates but would impose a grid on this image.

`full` — the complete run#

full applies the pipeline to every image, measures the detected objects, and writes every deliverable. It is the default, so --mode full can be omitted.

python -m phenotypic --mode full \
    --pipeline pipe.json \
    --input ./plates \
    --output ./out \
    --image-type Image

Before committing to a long run, preview the plan. --dry-run resolves the datasets, validates the pipeline, and prints the estimated output size without processing anything:

python -m phenotypic --pipeline pipe.json --input ./plates --output ./out \
    --image-type Image --dry-run

A finished full run leaves three things under --output:

out/
├── deliverables/            # what you read and share
│   ├── master_measurements.parquet         # clean, un-joined, pre-post archive
│   ├── measurements.{csv,parquet}          # post-applied mirror (the GUI reads this)
│   ├── measurements_by_feature/            # one file per measurer
│   ├── overlays/<dataset>/                 # detection overlay PNGs
│   ├── dashboard.html, processing_report.html
│   └── README.md                           # generated column documentation
├── results/<dataset>/
│   ├── zarr/               # one <stem>.ome.zarr store per image — the
│   │                       #   reusable segmentation
│   └── measurements/       # one parquet per image
└── .phenotypic/            # machine state: progress manifest, event log, pipeline copy

The zarr/ directory is what makes the other modes cheap: each store holds that image’s object map, so measure never has to detect again. The stores are conformant OME-Zarr (NGFF 0.5 / Zarr v3), so napari, Vizarr, and Fiji open them directly.

A tree written by a pre-store release has hdf/ instead, with one .h5 per image. Convert it once with --mode migrate; every mode that writes or reprocesses refuses an unconverted tree and points you at that command.

dashboard.html is the run-progress and failure surface. Local runs show progress directly; SLURM runs add a Download tab. Explore measurements in the Results Viewer or the GUI /analysis/ app.

Scoping a run while you iterate#

Trying a new detector on a thousand plates to discover it splits every colony is a bad way to spend an afternoon. Process a random subset first:

python -m phenotypic --pipeline pipe.json --input ./plates --output ./out \
    --image-type Image --sample 1 --random-seed 42

--sample N takes N random images per dataset (each first-level subdirectory of --input is one dataset). Pair it with --random-seed so the same subset is chosen on the next invocation — otherwise you are comparing two pipelines on two different images.

Surviving interruptions#

Re-running the same compatible command skips completed work automatically. Staged GPU pipelines infer the earliest required stage from their store, Stage-2 signal, and Stage-3 marker. If everything is already done, the CLI says so and exits:

python -m phenotypic --pipeline pipe.json --input ./plates --output ./out \
    --image-type Image
# ✓ All images already processed!

Three flags control exceptional handling of prior state:

--retry-failures — also re-process exact terminal failures recorded for the current computation.
--restart — clear current machine state and start a new lifecycle.
--overwrite — delete the output directory contents first.

Use --retry-failures after fixing a terminal scientific-processing failure to reprocess only the matching failed computations. Infrastructure failures, such as OOM or timeout, remain pending and are selected by an ordinary repeat call.

`measure` — new numbers, same segmentation#

You ran a detector over 500 plates, then realized you also want texture measurements. measure re-runs the measurement stage against the stores already sitting in results/<dataset>/zarr/, using a pipeline you supply:

python -m phenotypic --mode measure --pipeline pipe.json --output ./out

There is no --input: the images are discovered from the output root. Passing one is an error.

Because measure re-reads a finished segmentation rather than producing one, it refuses flags that imply a fresh detection pass or mutate run state. This applies to --restart, --retry-failures, --overwrite, --sample, and --dry-run. measure is all-or-nothing over every store it finds. Each replacement table is validated inside its image store, the per-image completion marker is refreshed last, and the run aggregates are rebuilt only from those marker-authorized embedded tables.

Note

Swap the measurers freely, but keep the detector consistent with the one that wrote the stores. measure reads the stored object map; changing OtsuDetector to LiDetector in the pipeline you pass will not re-segment anything.

`recompile` — rebuild the deliverables#

recompile takes neither --input nor --pipeline; both are reloaded from the output root. It re-aggregates marker-authorized embedded tables into master_measurements.parquet, regenerates any missing overlay PNGs from their stores, rebuilds the progress manifest, and regenerates the progress dashboard.

uv run python -m phenotypic --mode recompile --output ./out

Note

recompile writes no image store. It reads each store’s embedded measurement table, projects it onto the store’s own recorded measurement_columns, concatenates the projections into master_measurements.parquet, and joins deliverables/metadata.csv once at finalization. No store’s bytes, root, or per-image record are touched, so a run built with --metadata recompiles exactly like one built without.

This was not always so. Until 2026-09-11 recompile re-joined the metadata snapshot into every tables/measurements/table.parquet — the shape that predates the split into a measurements table plus its own tables/metadata/pht-metadata.parquet — and so had to refuse a --metadata tree rather than un-invert it. The rewrite changed nothing the master or the mirror could see once the join moved to finalization, so it was removed rather than repointed, and the refusal went with it.

Recompile performs neither metadata migration preflight nor metadata migration. Convert legacy storage or external measurement authority first with --mode migrate; recompile then rebuilds the current artifacts. Exact legacy header spellings that remain in readable provenance inputs are normalized in memory. An external file passed via --metadata is copied byte-for-byte to deliverables/metadata.csv before work, then normalized in memory. Neither the external file nor that provenance snapshot is rewritten.

Reach for it when the numbers are right but the presentation is not:

You deleted or corrupted dashboard.html.
A SLURM array finished its images but died before the aggregation step.
You want to attach a plate-layout metadata CSV to a run that was finished without one:

python -m phenotypic --mode recompile --output ./out --metadata plate_layout.csv

A run that already had --metadata is fine too: the CSV you pass here replaces the snapshot the join reads, and no store is rewritten.

--metadata left-joins the CSV onto the measurements mirror on shared columns. Every CSV row survives — one that matches no measured object is kept with null measurements and QC_MetadataOnly set to true, so a strain that was never detected is visible rather than silently dropped.

The output directory must already exist, and passing --pipeline is an error — recompile deliberately uses the pipeline the run was executed with, not whatever is on disk now.

Warning

--study is silently ignored in recompile mode — the REMBI study fields are folded into deliverables/rembi.yaml during full and measure runs only. To attach a study.yaml to a finished run, use --mode measure --study study.yaml, which rebuilds rembi.yaml without re-detecting.

`process` — export a layer, skip the analysis#

process is an apply-only export. It runs pipeline.apply() and writes one image layer per input, mirroring the input tree under --output. No measurement, no deliverables, no QC, no dashboard.

python -m phenotypic --mode process --pipeline pipe.json \
    --input ./plates --output ./proc --image-type Image --layer objmap

proc/
├── set_a/
│   ├── plate_00.png        # mirrors plates/set_a/plate_00.tiff
│   └── plate_01.png
└── .phenotypic/

--layer is required in process mode and rejected in every other mode. Each layer is written at its native dtype through the accessor’s imsave, with PhenoTypic metadata embedded:

`--layer`	File	Dtype
`rgb`	`.tiff`	integer, at the source bit depth
`gray`	`.tiff`	float, full precision
`detect_mat`	`.tiff`	float, full precision
`objmap`	`.png`	16-bit raw labels

This is the mode for feeding another tool. Export detect_mat to inspect what your enhancement chain actually hands the detector; export objmap to import segmentations into CellProfiler, Fiji, or a training set. Note that objmap holds raw label values, not a rendered image — opening it in a viewer shows a near-black frame, because label 3 is the pixel value 3.

Flags that only make sense for measurement output are ignored with a warning rather than an error:

$ python -m phenotypic --mode process ... --layer gray --no-qc
Warning: --no-qc is ignored in --mode process (no measurement/aggregation output).

The same applies to --metadata and --no-dataset-column.

Parameters worth knowing#

Loading the images#

--image-type defaults to GridImage, which imposes an arrayed layout. Use --nrows / --ncols to override the pipeline’s grid preset (the fallback is 8 × 12); pass --image-type Image for non-arrayed plates.

--detect-mode selects the channel the detector thresholds. It defaults to gray, but colony contrast often lives elsewhere — LabA separates red-pigmented colonies from agar far better than luminance does. The full set is gray, red, green, blue, MinRGB, HsvS, HsvV, InvS, LabL, LabA, LabB.

Warning

--ext no longer selects the output format. Forward runs write a single OME-Zarr store per image; the flag now only affects overlay PNG rendering and is deprecated.

Scaling out#

--njobs sets local parallelism and defaults to -1 (all cores). Images are large and operations copy — on a memory-constrained machine, capping --njobs is often faster than swapping.

--slurm submits to a cluster instead. It takes repeated KEY=VALUE pairs:

python -m phenotypic --pipeline pipe.json --input ./plates --output ./out \
    --slurm slurm_partition=compute \
    --slurm slurm_account=myproj \
    --slurm mem_gb=16 \
    --slurm time=120

Use the slurm_ prefix for standard SBATCH directives, or the convenience keys mem_gb and time. time is in minutes, as an integer — time=120, not time=02:00:00. The CLI returns as soon as the jobs are submitted; add --wait to block and monitor them. A staged GPU run without --wait prints PROCESSING SUBMITTED and leaves aggregation, reports, and README publication to its dependent finalizer. It does not print a premature completion summary. With --wait, the CLI follows the active orchestration epoch through the finalizer completion marker. Ctrl+C detaches monitoring while the jobs continue. --force-local overrides SLURM detection, which is what you want when testing on a login node.

--checkpoint-interval N inserts checkpoint tasks every N images in a SLURM array so a walltime kill loses at most N images of progress.

GPU detectors#

A pipeline containing a GpuDetector automatically splits into three stages — CPU preprocess, resident-model GPU detect, CPU measure — reusing the per-image OME-Zarr store. You do not opt in; you only tune it:

--gpu-slurm KEY=VALUE — SBATCH profile for the GPU stage. It inherits and deltas over --slurm, so put the GPU partition and account here and leave the CPU profile in --slurm. slurm_gpus_per_node=1 is added automatically.
--gpu-shards N — parallel whole-GPU tasks (SLURM only; ignored locally). Set it to your concurrent-GPU count.
--gpu-workers-per-gpu W — reserved for future per-GPU replica packing. The current staged worker runs one resident model per GPU shard.

python -m phenotypic --pipeline gpu_pipe.json --input ./plates --output ./out \
    --slurm slurm_partition=cpu --slurm mem_gb=32 \
    --gpu-slurm slurm_partition=gpu --gpu-slurm slurm_account=myproj \
    --gpu-shards 4

GPU distribution uses --gpu-shards to spread work across whole GPUs and nodes. --gpu-workers-per-gpu is currently reserved and does not add replicas. There is no per-forward batching knob — the segmentation models PhenoTypic targets do not broadly support batched inference.

Stage 2 survives walltime through dependent continuation rounds. After a GPU array terminates, a controller checks which images still lack a complete Stage-2 signal and submits another array only when work remains. Workers do not catch walltime signals and do not call scontrol requeue. Cancellation fences the run epoch before cancelling every active job in its ledger.

Shaping the output#

--overlay-alpha (default 0.3) — opacity of the label overlay in the saved overlay PNGs.
--no-dataset-column — drop the Metadata_Dataset column, which is included by default and is what makes multi-dataset analysis possible.
--no-qc — skip the QC compute step. QC otherwise runs whenever the pipeline has a non-empty qc section, and re-running it resets GUI review progress.
--skip-validation — bypass pipeline validation. Only worth it for pipelines you have already run.

A typical iteration loop#

# 1. Preview the plan.
python -m phenotypic -p pipe.json -i ./plates -o ./out --dry-run

# 2. Tune the detector against a fixed random subset.
python -m phenotypic -p pipe.json -i ./plates -o ./trial --sample 5 --random-seed 42

# 3. Commit to the full dataset on the cluster.
python -m phenotypic -p pipe.json -i ./plates -o ./out \
    --slurm slurm_partition=compute --slurm mem_gb=16 --slurm time=240

# 4. Retry exact terminal scientific failures after fixing their cause.
python -m phenotypic -p pipe.json -i ./plates -o ./out --retry-failures

# 5. Add a texture measurer. No re-detection.
python -m phenotypic -m measure -p pipe_with_texture.json -o ./out

# 6. Attach the plate layout and rebuild the dashboard.
python -m phenotypic -m recompile -o ./out --metadata plate_layout.csv

Steps 5 and 6 are the payoff: neither one re-segments a single colony.

Where to go next#

CLI Batch Processing — the condensed recipe.
SLURM Pipelines — cluster execution in depth.
GPU Detection Setup — staging model weights on offline compute nodes.
CLI Reference — every flag.