# GPU-Accelerated Colony Detection

Set up and use deep-learning-based colony detectors (SAM2, micro-sam, SAM3,
DinoSam2) with GPU acceleration.

## Installation

The GPU detectors have different packaging constraints:

| Detector            | Package(s) needed          | Available via              | CUDA-capable?          |
|---------------------|----------------------------|----------------------------|------------------------|
| `Sam2Detector`      | `torch`, `torchvision`, `sam2` | **PyPI** (ships in `phenotypic[torch]`) | Yes — Linux + CUDA |
| `MicroSamDetector`  | `micro_sam` (+ `torch`)    | **conda-forge only**, not on PyPI | CPU by default; user-managed CUDA possible |
| `Sam3Detector`      | `transformers`, `huggingface_hub` (+ `torch`) | **PyPI** (ships in `phenotypic[foundation]`); weights **gated** | Yes — Linux + CUDA |
| `DinoSam2Detector`  | `transformers`, `sam2` (+ `torch`) | **PyPI** (ships in `phenotypic[foundation]`); DINOv2 weights **ungated** | Yes — Linux + CUDA |

### Per-model license posture

| Model         | Code license          | Weights license            | Gated? |
|---------------|-----------------------|----------------------------|--------|
| SAM2          | Apache-2.0            | Apache-2.0                 | No     |
| SAM3          | Apache-2.0 (`transformers`) | **SAM License** (commercial-OK) | **Yes** — accept on Hugging Face |
| DINOv2        | Apache-2.0            | Apache-2.0                 | No     |
| DINOv3 (opt-in) | Apache-2.0          | **DINOv3 License** (custom Meta) | **Yes** — accept on Hugging Face |
| INSID3 method (`Insid3Detector`) | Apache-2.0 (clean-room, no code vendored) | DINOv3-native backbone (gated) | **Yes** (via DINOv3) |
| FSSDINO method (`FssDinoDetector`) | paper **CC BY-NC-SA** (clean-room, no code vendored) | DINOv2 default (ungated) / DINOv3 opt-in | No (DINOv2) / Yes (DINOv3) |

PhenoTypic **does not redistribute model weights** — each weight is downloaded
by you from the upstream source under that model's license, which you accept
(see `NOTICE` and `licenses/`). The two semantic detectors
(`Insid3Detector`, `FssDinoDetector`) carry no vendored upstream code: INSID3 is
clean-room-reimplemented from its Apache-2.0 method (attributed), and FSSDINO is
clean-room-reimplemented from the paper only (the reference repo is
all-rights-reserved). When `dino_version=3` is selected, PhenoTypic displays
"Built with DINOv3" per the DINOv3 License.

PhenoTypic itself is distributed via PyPI and managed with `uv`. `micro_sam`
is not published on PyPI, so it is **not** included in any `phenotypic`
extra. Users who need `MicroSamDetector` must install `micro_sam`
themselves; the tutorial recipe below uses `pixi` for that.

> **Installing the detector packages and downloading their weights is covered in
> the {ref}`Deep Learning Detectors`
> section of the Getting Started tutorial.** In short: `uv add "phenotypic[torch]"`
> for SAM2, `"phenotypic[foundation]"` for the SAM3/DINO detectors, the `[gpu]`
> umbrella for all of them, and conda-forge `micro_sam` for `MicroSamDetector`.
> In a source checkout, use `uv sync --extra foundation` to install the same
> foundation-model dependencies. Gated SAM3/DINOv3 weights need a one-time
> Hugging Face license handshake (`hf auth login` after accepting the model
> license on Hugging Face).

`MicroSamDetector` is importable from `phenotypic.detect.nn` even when
`micro_sam` is missing; the `ImportError` is deferred to the first
`apply()` call and points back at the installation instructions.

## Downloading Model Checkpoints

Every detector downloads its checkpoints automatically on first use, and the
`phenotypic.detect.nn` CLI (`download` / `list` / `clear`) pre-fetches and
inspects them — essential before submitting SLURM jobs, since compute nodes
often lack internet access. The full command set, cache-location environment
variables (`TORCH_HOME`, `MICROSAM_CACHEDIR`, `HF_HOME`,
`PHENOTYPIC_ACCEPT_MODEL_LICENSE`, …), and the login-node pre-staging workflow
for gated foundation weights are documented in the
{ref}`Deep Learning Detectors`
tutorial section. The SLURM-specific staging is expanded under
[SLURM Deployment](#slurm-deployment) below.

## Using Sam2Detector

`Sam2Detector` wraps Meta's SAM2 automatic mask generator. It lays a grid of
prompt points over the RGB image, predicts masks at each point, filters by
quality, and assembles a labelled object map.

```python
from phenotypic.detect.nn import Sam2Detector

# Basic usage with default parameters
detector = Sam2Detector()

# Tuned for dense plates with small colonies
detector = Sam2Detector(
    model_size="small",
    points_per_side=48,
    pred_iou_thresh=0.6,
    min_mask_region_area=200,
)

# Apply to an image (downloads checkpoint on first use)
result = detector.apply(image)
print(result.num_objects)
```

### Parameter tuning for colony detection

- **`points_per_side`** (default 32): Controls the density of the prompt grid.
  Use 16 for large, well-separated colonies. Increase to 48--64 for dense
  plates with many small colonies. Higher values increase inference time
  quadratically.
- **`pred_iou_thresh`** (default 0.7): Minimum predicted IoU for keeping a
  mask. Raise to 0.85--0.95 for conservative detection (fewer false
  positives); lower to 0.5 to catch faint or ambiguous colonies.
- **`stability_score_thresh`** (default 0.92): Filters masks by boundary
  stability. Higher values keep only masks with crisp edges.
- **`min_mask_region_area`** (default 100): Minimum mask area in pixels.
  Increase to suppress agar texture, dust, and other small artefacts that
  SAM2 segments as objects. Typical range: 50--500 depending on image
  resolution.
- **`model_size`** (default `"tiny"`): `"tiny"` is fastest and sufficient
  for most colony plates. Use `"large"` for maximum mask quality on
  publication figures.

## Using MicroSamDetector

`MicroSamDetector` uses SAM models finetuned on large-scale microscopy
datasets. It is particularly effective for brightfield and darkfield
microscopy images of agar plates.

```python
from phenotypic.detect.nn import MicroSamDetector

# Default: ViT-Base light microscopy model
detector = MicroSamDetector()

# Use the larger model for higher accuracy
detector = MicroSamDetector(model_type="vit_l_lm")

result = detector.apply(image)
```

### Model selection

Light microscopy models (recommended for agar plate imaging):

- `"vit_t_lm"` -- ViT-Tiny, fastest, good for rapid screening
- `"vit_b_lm"` -- ViT-Base (default), best speed/accuracy trade-off
- `"vit_l_lm"` -- ViT-Large, highest accuracy, most VRAM

Electron microscopy models (for organelle segmentation):

- `"vit_b_em_organelles"` -- ViT-Base
- `"vit_l_em_organelles"` -- ViT-Large

Base SAM checkpoints (without microscopy finetuning):

- `"vit_t"`, `"vit_b"`, `"vit_l"`, `"vit_h"`

## Using Sam3Detector

`Sam3Detector` wraps Meta's **text-prompted** SAM3 foundation model. Unlike
SAM2's dense point grid, SAM3 segments everything matching a short text
`prompt` (default `"colony"`) in one true batched forward pass, then assembles
the predicted instance masks into a labelled `objmap` (`output_kind="instance"`).

```python
from phenotypic.detect.nn import Sam3Detector

# Override the prompt per run — SAM3 has no prompt-free "segment everything" mode.
det = Sam3Detector(prompt="yeast colony", score_thresh=0.5)
```

SAM3 weights are **gated** (SAM License). Accept the gate and authenticate
once (see {ref}`Deep Learning Detectors`)
before the first `apply()`.

**Dense plates.** SAM3 caps at 200 instances per forward and runs at 1008 px
internally, so `Sam3Detector` tiles large plates into fixed `tile_px` crops
with `tile_overlap`, infers each tile, offsets the masks back to full
coordinates, and merges cross-tile duplicates by IoU-NMS — all automatically.
Images that fit one tile run un-tiled.

Key parameters:

- `prompt` — free text describing the target (`"colony"`, `"bacterial colony"`).
- `score_thresh` / `mask_threshold` — instance-confidence and mask-probability
  cutoffs.
- `min_mask_region_area` — drop masks smaller than this (default 100).
- `tile_px` / `tile_overlap` — dense-plate tiling controls (defaults 1008 / 0.15).

## Using DinoSam2Detector

`DinoSam2Detector` is a **training-free** instance detector that composes two
ungated foundation models: SAM2's automatic mask generator produces
class-agnostic proposals, and a **DINOv2** backbone (Apache-2.0, ungated by
default) supplies dense patch features. Each proposal is scored by cosine
similarity of its pooled DINO feature to a foreground prototype; background-like
proposals are dropped, near-duplicates merged by IoU, and survivors painted into
a labelled `objmap`.

```python
from phenotypic.detect.nn import DinoSam2Detector

# Recommended config is fully ungated (SAM2 Apache + DINOv2 Apache).
det = DinoSam2Detector(dino_size="base", similarity_thresh=0.5)
```

`dino_version` selects the backbone generation: `2` = DINOv2 (default, ungated),
`3` = DINOv3 (gated opt-in). Selecting `dino_version=3` routes the snapshot pull
through the DINOv3-License acceptance gate (pre-stage it with the
`download --model-type dinov3 --accept-license` command from the
{ref}`Deep Learning Detectors`
tutorial section). The default DINOv3 checkpoint id is
`dinov3-vitb16-pretrain-lvd1689m`.

Key parameters:

- `dino_version` / `dino_size` — backbone generation (2/3) and size.
- `sam2_model_size` — SAM2 variant for the proposal generator.
- `similarity_thresh` — minimum cosine-to-prototype score to keep a proposal.
- `merge_iou_thresh` — IoU above which two survivors are merged.

## Semantic few-shot detectors (`Insid3Detector`, `FssDinoDetector`)

`Insid3Detector` (one-shot, in-context) and `FssDinoDetector` (few-shot) are
**semantic** detectors: they emit a binary `objmask` (`output_kind="semantic"`),
not their own instance labels. The mask auto-labels into the shared `objmap`
backend exactly like a threshold detector, so the repo's downstream watershed
(`SeparateObjects`) turns it into instances — pair them with a separation step
in your pipeline, just as you would `OtsuDetector`. Both run on a frozen DINO
backbone and write only `objmask`; `objmap[:] > 0` then equals `objmask[:]`.

Both ship a **curated colony exemplar** (a reference colony RGB + its mask,
rendered once from `load_synth_yeast_plate()`) as the **default** reference /
support set, so they work out of the box. Supply your own annotated exemplar to
transfer to a new colony appearance.

### `Insid3Detector` — one-shot in-context (DINOv3-native, gated)

A faithful clean-room reimplementation of INSID3 (Apache-2.0). Given a single
**reference image + reference mask**, it pools an in-context prototype and
cosine-matches every query patch — but first removes DINOv3's **positional bias**
(estimated by SVD and projected out, INSID3's defining step) so patches match on
appearance, not position. It is DINOv3-native (gated, `dino_version=3` default);
a `dino_version=2` opt-in runs gate-free (the debias is then a near-no-op).

```python
from phenotypic.detect.nn import Insid3Detector

# Default: bundled exemplar + gated DINOv3 (accept the DINOv3 License first).
det = Insid3Detector(similarity_thresh=0.5)

# Override the in-context reference with your own annotated colony pair:
det = Insid3Detector(
    reference_image="ref_plate.tiff",
    reference_mask="ref_plate_mask.png",
    similarity_thresh=0.6,
)

# Gate-free DINOv2 variant (no Hugging Face token needed):
det = Insid3Detector(dino_version=2, dino_size="small")
```

Key parameters:

- `reference_image` / `reference_mask` — the in-context exemplar (defaults to
  the bundled colony exemplar).
- `dino_version` / `dino_size` — backbone (3 = DINOv3 default/gated, 2 = DINOv2).
- `similarity_thresh` — cosine cutoff binarising the match map.
- `svd_components` — INSID3's positional-debias strength (leading SVD directions
  removed; default 4 ≈ DINOv3's register-token count; `0` disables the debias).
- `tile_px` / `tile_overlap` — large-plate tiling (defaults 1024 / 0.15).

### `FssDinoDetector` — few-shot (DINOv2 default, ungated)

A clean-room reimplementation **from the paper only** of FSSDINO
(arXiv:2602.07550, CC BY-NC-SA; the reference repo is all-rights-reserved and is
not vendored). From a **support set** it builds `n_clusters` class-specific
prototypes (k-means) plus a Gram matrix (channel co-occurrence), scores each
query patch by cosine to the prototypes and a Gram-refined energy, combines the
maps (mean ⊙ max) and assigns foreground vs background by `argmax`. It defaults
to **DINOv2** (ungated), so it runs gate-free.

```python
from phenotypic.detect.nn import FssDinoDetector

# Default: bundled one-shot exemplar + ungated DINOv2.
det = FssDinoDetector(n_clusters=5)

# A true few-shot support set:
det = FssDinoDetector(
    support_images=["s1.tiff", "s2.tiff", "s3.tiff"],
    support_masks=["s1_mask.png", "s2_mask.png", "s3_mask.png"],
    n_clusters=5,
)
```

Key parameters:

- `support_images` / `support_masks` — the few-shot support set (defaults to the
  bundled one-shot colony exemplar).
- `n_clusters` — class-specific prototypes per class (paper default 5).
- `feature_layer` — transformer hidden-state layer for the dense features.
  FSSDINO's "Semantic Selection Gap" finding is that intermediate layers often
  beat the last, but cannot be reliably selected unsupervised — so the default
  is `-1` (the last layer, the paper's safe default).
- `dino_version` / `dino_size` — backbone (2 = DINOv2 default/ungated, 3 = DINOv3
  gated opt-in).
- `similarity_thresh` — foreground-score floor on top of the fg-vs-bg argmax.
- `tile_px` / `tile_overlap` — large-plate tiling (defaults 512 / 0.15).

## Pipeline Integration

GPU detectors work like any other PhenoTypic operation in a pipeline:

```python
import phenotypic as pht
from phenotypic.detect.nn import Sam2Detector
from phenotypic.measure import SizeMeasurer

pipeline = pht.ImagePipeline(
    ops=[Sam2Detector(model_size="tiny", points_per_side=32)],
    measurer=SizeMeasurer(),
    name="sam2_colony_pipeline",
)

# Run the pipeline
results = pipeline.operate([image])
df = pipeline.measure([image])
```

### JSON serialization

Pipelines containing GPU detectors can be saved and loaded just like any
other pipeline. The detector parameters are serialized; the model weights
are not (they are re-downloaded or loaded from cache when needed):

```python
# Save
pipeline.to_json("sam2_pipeline.json")

# Load -- works without torch installed (model loads lazily on apply)
restored = pht.ImagePipeline.from_json("sam2_pipeline.json")
```

Internal state (attributes prefixed with `_`, such as the loaded model) is
excluded from serialization. The model is rebuilt transparently on the next
call to `apply`.

## Local Staged GPU Detection (CLI)

When you run a pipeline through the CLI (`python -m phenotypic`) and it contains
a `GpuDetector`, detection runs as **three internal stages** rather than invoking
the GPU model once per image. The segmentation model is built **once** and every
image is streamed through it — far more efficient than the notebook per-image
path when processing a directory:

1. **Stage 1 — CPU preprocess.** Every prior `ImageOperation` (enhancers,
   corrections) is applied per image and the result is saved to the normal
   per-image HDF (`results/<dataset>/hdf/<stem>.h5`).
2. **Stage 2 — resident-model GPU detect.** The detector's model is built once
   and kept resident while each staged HDF is streamed through
   `preprocess → infer_batch`. The labelled object map is written to a per-image
   `.npy` **sidecar** at `results/<dataset>/objmap/<stem>.npy`; the HDF is opened
   read-only here, so an interrupted run never corrupts it.
3. **Stage 3 — CPU merge + measure.** The sidecar is merged back into the image
   through the object-map accessor, the post-detector refiners and the
   measurement queue run, the HDF is re-saved atomically, and the sidecar is
   **deleted**.

The output folder is identical to a single-pass run — staging is an internal
optimization, not a different output contract.

**Resume is content-defined.** Re-running the same command skips any image whose
work already exists: Stage 1 skips when the HDF exists, Stage 2 skips when the
sidecar *or* the measurement parquet exists (Stage 3 deletes the sidecar, so the
parquet is the durable "done" marker), and Stage 3 skips when the parquet exists.
Progress is **stage-tagged** in the event log, so the run dashboard can show how
far each image has moved through the three stages. If Stage 1 fails for an image
(e.g. an unreadable file), Stages 2 and 3 skip it and record a structured failure
instead of aborting the batch.

```bash
# Forward run: detection stages automatically because the pipeline has a GpuDetector
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/

# Export just the object maps (runs Stages 1-2, then writes one objmap PNG per image)
python -m phenotypic --mode process --layer objmap \
    --pipeline sam2_pipeline.json --input /plates/ -o /output/
```

## SLURM Deployment

When a pipeline contains a `GpuDetector` operation (either `Sam2Detector` or
`MicroSamDetector`), the CLI automatically adapts:

**Local execution:** Forward GPU runs use the staged engine above (the model
loads once and streams every image); see "Local Staged GPU Detection". The
legacy per-image path (measure-only and non-objmap layer exports) still forces
sequential processing (`n_jobs=1`) to avoid multiple workers competing for the
same GPU.

**SLURM execution:** Automatically adds `--gpus-per-node=1` to the SLURM
job if GPU resources were not explicitly requested.

```bash
# GPU resources are auto-requested when the pipeline contains a GpuDetector
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/

# Override with explicit SLURM GPU arguments
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/ \
    --slurm slurm_gpus_per_node=2 \
    --slurm slurm_partition=gpu
```

Pre-cache checkpoints on the login node before submitting (see
"Downloading Model Checkpoints" above).

### SLURM Staged GPU Detection

A forward GPU run on SLURM (`--slurm ...` with a `GpuDetector` pipeline) runs as
the staged engine above, submitted as a **3-link `afterany` dependency chain**
with **per-stage resources**:

- **Stage 1** — a CPU array over images (preprocess → staged HDF), on the
  `--slurm` profile.
- **Stage 2** — a GPU array over **shards** (`--gpu-shards N`): each task is one
  whole GPU running a resident model that streams its shard of HDFs to objmap
  sidecars, on the `--gpu-slurm` profile.
- **Stage 3** — a CPU array over images (merge sidecar → measure), on the
  `--slurm` profile.

`afterany` between stages means a handful of per-image failures never block the
next stage. Because Stage 2 writes a `.npy` sidecar with the HDF opened
read-only, there is **no HDF5 write-locking on the GPU nodes**; Stages 1 & 3
write the HDF atomically (temp + rename) on CPU nodes.

```bash
# CPU partition for Stages 1 & 3 (--slurm); GPU partition + 2 concurrent GPUs for Stage 2
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/ \
    --slurm slurm_partition=batch --slurm slurm_time=02:00:00 \
    --gpu-slurm slurm_partition=gpu --gpu-shards 2
```

The resource nesting is three levels: `--gpu-shards` (whole GPUs, across nodes)
→ `--gpu-workers-per-gpu` (replicas packed per GPU for small models) →
`--gpu-batch-size` (images per forward pass; batchable models, `auto` in
Spec 2). `--gpu-slurm` inherits/deltas over `--slurm`, so shared keys
(account, qos, time) carry over and only the GPU partition/account need
restating; one GPU is requested automatically.

**Walltime survival.** The Stage-2 script carries `#SBATCH --signal=B:TERM@120`,
so SLURM sends `SIGTERM` shortly before the walltime. The shard-worker catches
it and `sbatch`-resubmits its shard (`afterany` on itself); content-defined skip
means the continuation processes only the remaining sidecar-less images, and it
repeats until the shard is complete — so a `TIMEOUT` never loses work and never
needs a manual restart.

**Pre-staging gated weights.** For offline compute nodes, download checkpoints on
the login node first and export `HF_HUB_OFFLINE=1` (and `hf auth login` for gated
Hugging Face models). Gated foundation-model weights are never bundled; accept
their license once via `PHENOTYPIC_ACCEPT_MODEL_LICENSE=<model>` (see the
`require_license_acceptance` hook).

**Custom detectors on SLURM.** Each SLURM stage worker is a fresh process that
deserializes the pipeline with `ImagePipeline.from_json`, which resolves operation
classes from the `phenotypic` namespace. If your pipeline uses a detector defined
**outside** the package, set
`PHENOTYPIC_PRELOAD_MODULES=your.module[,another.module]` — the worker imports
each before deserializing, so a self-registering module can make its class
resolvable on the compute node. `sbatch --export=ALL` (the default) propagates the
variable.

## Device Selection

Both detectors accept a `device` parameter that controls where inference runs.

### Automatic detection (default)

With `device="auto"` (the default), PhenoTypic probes accelerators in priority
order:

1. **CUDA** -- NVIDIA GPUs
2. **MPS** -- Apple Silicon (macOS)
3. **XPU** -- Intel GPUs
4. **HPU** -- Habana Gaudi accelerators

If none is found, a `RuntimeError` is raised.

### Explicit device

```python
# Force a specific device
Sam2Detector(device="cuda")   # NVIDIA GPU
Sam2Detector(device="mps")    # Apple Silicon
Sam2Detector(device="xpu")    # Intel GPU
Sam2Detector(device="cpu")    # CPU (very slow, but always available)
```

When an explicit accelerator is requested but unavailable, a `RuntimeError`
is raised with a descriptive message.

### `resolve_device()` utility

The device resolution logic is available as a standalone function for custom
workflows:

```python
from phenotypic.detect.nn._checkpoint_manager import resolve_device

device = resolve_device("auto")           # raises if no accelerator
device = resolve_device("auto", allow_cpu=True)  # falls back to CPU with warning
```

## Listing and Clearing Models

### List cached checkpoints

```bash
python -m phenotypic.detect.nn list
```

This prints a table showing all cached SAM2 and micro-sam checkpoints with
their file sizes and paths.

### Clear cached checkpoints

```bash
# Clear all cached checkpoints (prompts for confirmation)
python -m phenotypic.detect.nn clear

# Clear only SAM2 checkpoints
python -m phenotypic.detect.nn clear --model-type sam2

# Clear only micro-sam checkpoints
python -m phenotypic.detect.nn clear --model-type microsam
```

## Troubleshooting

### `ImportError: Sam2Detector requires the sam2 package`

PyTorch and the model packages are not installed. Install the `torch` extra:

```bash
uv add "phenotypic[torch]"
```

(Linux/macOS only — `sam2` is not packaged for Windows.)

### `ImportError: MicroSamDetector requires the micro_sam package`

`micro_sam` is conda-only and must be installed separately. See the
{ref}`Deep Learning Detectors`
tutorial section for the conda-forge / `pixi` recipe.

### `RuntimeError: No accelerator available`

No GPU was detected. Options:

- Ensure your GPU drivers and CUDA toolkit are installed correctly.
- On macOS with Apple Silicon, ensure PyTorch >= 2.0 with MPS support.
- Pass `device="cpu"` to force CPU inference (very slow):

```python
Sam2Detector(device="cpu")
```

### `RuntimeError: device='cuda' requested but CUDA is not available`

CUDA was explicitly requested but is not available. Check:

- `nvidia-smi` shows your GPU.
- PyTorch was installed with CUDA support (`torch.cuda.is_available()`
  returns `True`).
- On SLURM, the job was submitted to a GPU partition.

### Out of memory (OOM) errors

SAM models require significant GPU memory. To reduce VRAM usage:

- Use a smaller model: `Sam2Detector(model_size="tiny")` instead of `"large"`.
- Use `MicroSamDetector(model_type="vit_t_lm")` for the smallest micro-sam
  model.
- Reduce `points_per_side` (e.g., 16 instead of 32) to generate fewer
  candidate masks.
- Process smaller images or downscale before detection.

### Checkpoint not found on SLURM compute nodes

Compute nodes often lack internet access. Pre-download checkpoints on the
login node:

```bash
python -m phenotypic.detect.nn download --model-type sam2 --model-size tiny
python -m phenotypic.detect.nn download --model-type microsam --model-name vit_b_lm
python -m phenotypic.detect.nn list  # verify
```

Ensure `TORCH_HOME` and `MICROSAM_CACHEDIR` (if customised) point to a
shared filesystem accessible from compute nodes.

### `Illegal instruction (core dumped)` on a SLURM compute node

A stage worker exits with code 132 and the SLURM `.err` log shows `Illegal
instruction (core dumped)`. The node's CPU is too old for the installed
numpy/scipy/torch wheels (a pre-AVX node on a heterogeneous partition). This
affects **any** PhenoTypic SLURM run, not just the staged GPU engine. Pin jobs to
modern nodes — use a homogeneous modern partition, or add a SLURM feature
constraint that excludes the old CPUs, e.g.:

```bash
python -m phenotypic --pipeline p.json --input /plates/ -o /out/ \
    --slurm slurm_partition=<modern-partition> \
    --gpu-slurm slurm_partition=<gpu-partition>
```

(Stage 2's GPU work runs on GPU nodes, which are typically consistent; the CPU
Stages 1 & 3 are the ones exposed to a heterogeneous CPU partition.)