CLI Batch Processing#
Process an entire directory of plate images using the PhenoTypic command-line interface.
This is the condensed recipe for the default full mode. For what the other
three modes (measure, recompile, process) produce and which flags each
one accepts, see CLI Execution Modes.
Basic Usage#
python -m phenotypic --mode full --pipeline pipeline.json --input /path/to/plates/ --output /path/to/output/
Required path options:
--pipeline pipeline.json— Pipeline configuration (created withpipeline.to_json())--input /path/to/plates/— Folder containing plate images--output /path/to/output/— Where results are saved
Grid Plates#
--image-type already defaults to GridImage; pass --nrows / --ncols to
override the pipeline’s grid preset (which itself falls back to 8 × 12).
python -m phenotypic --mode full --pipeline pipeline.json --input /plates/ --output /output/ \
--image-type GridImage --nrows 8 --ncols 12
Parallelism#
python -m phenotypic --mode full --pipeline pipeline.json --input /plates/ --output /output/ --njobs 4
Omit --njobs to use all available CPU cores.
Continue After Interruption#
python -m phenotypic --mode full --pipeline pipeline.json --input /plates/ --output /output/
Run the same command again to continue compatible unfinished work. Add
--retry-failures to also re-process images that previously failed, instead of
skipping them.
Testing#
# Dry run: validate pipeline and list images without processing
python -m phenotypic --mode full --pipeline pipeline.json --input /plates/ --output /output/ --dry-run
# Process 5 random images per dataset as a test
python -m phenotypic --mode full --pipeline pipeline.json --input /plates/ --output /output/ \
--sample 5 --random-seed 42
--sample draws N images from each dataset (each first-level subdirectory of
--input). Pass --random-seed to draw the same subset every time.
Run Preflight Checks#
Before any image is processed, and before anything under --output is created,
cleared, or submitted, the CLI runs a run preflight: read-only checks of
the pipeline, the environment, the cluster profile, the input files’ headers,
the metadata CSV, and the output location. The preflight never decodes a
pixel, so a single bad image cannot make it fail.
Each problem is reported as a finding with a code, the affected paths, and a remedy:
✗ Error [PF-GRID-IMAGE]: grid operation(s) meas:spread require a GridImage, but the run uses --image-type Image; each such image would fail with GridImageInputError
→ Run with --image-type GridImage (and --nrows/--ncols for your plate layout), or remove the grid operations listed above.
An error refuses the run. The CLI exits with status 1 and says that nothing under
--outputwas changed.A warning is printed and the run continues.
A finding about input files is an error when it affects every input, and a warning listing the affected files when it affects only some of them: the other images can still be processed.
--dry-run runs the same preflight, so it is the way to check a
configuration without starting a run. --skip-validation skips validation:
the execution-configuration check, the pipeline load check, and the run
preflight. It does not skip the checks that protect the output folder or that
the run cannot proceed without: the GPU placement refusal (a GPU detector in
a position the staged GPU engine cannot run), the refusal of a --restart or
--overwrite that would delete the run’s own inputs, and the parsing of
--metadata, --bit-depth, and --gpu-slurm values.
Code |
Reported when |
Severity |
Remedy |
|---|---|---|---|
|
The pipeline file cannot be read: invalid JSON, a duplicated key, or invalid parameters |
error |
Fix the pipeline file |
|
The pipeline names a class that is not in the |
error |
List a self-registering module in |
|
A grid operation or grid measurer runs on images read as plain |
by reach |
|
|
The pipeline’s |
by reach |
|
|
A |
error |
Add a detector, or use |
|
An operation needs a package that is not installed |
error |
Install the extra the message names |
|
A detector needs gated weights whose license the run has not accepted |
error |
|
|
A detector’s weights are not in the local cache |
warning |
Pre-download them on a node with network access |
|
|
error |
Fix the option |
|
A profile could not be confirmed: no |
warning |
Read |
|
The requested time exceeds the partition’s |
warning |
Lower the time, or choose another partition |
|
The staged GPU engine cannot fit the cluster’s |
error |
Reduce |
|
|
error |
|
|
An input file’s header cannot be read |
by reach |
Replace or remove the files listed |
|
An input has 2, or 5 or more, channels, which |
by reach |
Convert the files to grayscale or RGB(A) |
|
A color |
by reach |
|
|
A color operation or measurer runs on grayscale inputs |
by reach |
Supply RGB images, or remove the color operations |
|
|
by reach |
Drop |
|
RAW inputs, but |
by reach |
Install |
|
Two inputs in one dataset share a name without extension (e.g. |
error |
Rename or move one file of each pair |
|
The |
error |
Fix the CSV |
|
The CSV has a legacy and a current spelling of one column, with conflicting values |
error |
Keep one spelling |
|
The CSV shares no column that identifies an image |
error; warning when its keys may be measurement columns such as |
Add an |
|
A join key repeats in the CSV |
as |
Make each key unique |
|
Some images have no metadata row |
warning |
Add rows, or match |
|
Some metadata rows match no image |
warning |
Expected for wells that grew nothing |
|
The CSV joins on columns only measurements carry |
warning |
None needed; the join is checked at finalization |
|
|
error |
Choose another |
|
A |
warning |
Free space, or choose another |
|
On a SLURM run, a path workers read or write is on node-local storage |
warning |
Move it to shared storage |
|
A check itself failed |
warning |
Report it; the run is not blocked |
“By reach” means an error when every input is affected, and a warning listing the affected inputs otherwise.
Changes that can refuse a run that used to start#
These came with the run preflight. Each refusal replaces a run that would have failed, or silently produced the wrong result, later:
SAM3 needs an explicit license acceptance. Set
PHENOTYPIC_ACCEPT_MODEL_LICENSE=sam3(comma-separate several models, e.g.sam3,dinov3). No GPU detector prompts for a license during a run any more; an unaccepted license is reported asPF-LICENSEbefore the run starts.Camera RAW files are decoded as sensor data through
rawpy. Whererawpyis not installed, which includes Windows, RAW inputs are refused (PF-RAW-NO-RAWPY) instead of being read as the embedded preview image. Convert them to TIFF first on such a machine. A resumed run re-processes its RAW images, because they now decode to different pixels; other images are reused as before.A pipeline file with a duplicated JSON key no longer loads, in every mode including
--mode recompile, and inImagePipeline.from_json. Such a file never ran as written: the last duplicate silently replaced the first and ran in the first one’s position. Remove the duplicate.Custom operations defined outside PhenoTypic resolve in every process that loads the pipeline, including local parallel workers, once a module that registers them is listed in
PHENOTYPIC_PRELOAD_MODULES(PF-CUSTOM-OPotherwise). A module name that cannot be imported stops the run at startup.