GPU-Accelerated Colony Detection#

Set up and use deep-learning-based colony detectors (SAM2, micro-sam, SAM3, DinoSam2) with GPU acceleration.

Installation#

The GPU detectors have different packaging constraints:

Detector

Package(s) needed

Available via

CUDA-capable?

Sam2Detector

torch, torchvision, sam2

PyPI (ships in phenotypic[torch])

Yes — Linux + CUDA

MicroSamDetector

micro_sam (+ torch)

conda-forge only, not on PyPI

CPU by default; user-managed CUDA possible

Sam3Detector

transformers, huggingface_hub (+ torch)

PyPI (ships in phenotypic[foundation]); weights gated

Yes — Linux + CUDA

DinoSam2Detector

transformers, sam2 (+ torch)

PyPI (ships in phenotypic[foundation]); DINOv2 weights ungated

Yes — Linux + CUDA

Per-model license posture#

Model

Code license

Weights license

Gated?

SAM2

Apache-2.0

Apache-2.0

No

SAM3

Apache-2.0 (transformers)

SAM License (commercial-OK)

Yes — accept on Hugging Face

DINOv2

Apache-2.0

Apache-2.0

No

DINOv3 (opt-in)

Apache-2.0

DINOv3 License (custom Meta)

Yes — accept on Hugging Face

INSID3 method (Insid3Detector)

Apache-2.0 (clean-room, no code vendored)

DINOv3-native backbone (gated)

Yes (via DINOv3)

FSSDINO method (FssDinoDetector)

paper CC BY-NC-SA (clean-room, no code vendored)

DINOv2 default (ungated) / DINOv3 opt-in

No (DINOv2) / Yes (DINOv3)

PhenoTypic does not redistribute model weights — each weight is downloaded by you from the upstream source under that model’s license, which you accept (see NOTICE and licenses/). The two semantic detectors (Insid3Detector, FssDinoDetector) carry no vendored upstream code: INSID3 is clean-room-reimplemented from its Apache-2.0 method (attributed), and FSSDINO is clean-room-reimplemented from the paper only (the reference repo is all-rights-reserved). When dino_version=3 is selected, PhenoTypic displays “Built with DINOv3” per the DINOv3 License.

PhenoTypic itself is distributed via PyPI and managed with uv. micro_sam is not published on PyPI, so it is not included in any phenotypic extra. Users who need MicroSamDetector must install micro_sam themselves; the recipe below uses pixi for that.

Installing Sam2Detector (PyPI-only)#

On Linux or macOS:

uv add "phenotypic[torch]"          # torch + torchvision + sam2
# or, inside a uv-managed project:
uv sync --extra torch

The torch extra is not available on Windows — sam2 requires CUDA nvcc and has no pre-built Windows wheels. Use WSL2 (Ubuntu) instead.

Installing the foundation models (Sam3Detector, DinoSam2Detector)#

The foundation detectors collapse to two pure-Python, permissive libraries — transformers (>=5.2.0, the first release shipping Sam3Model/Sam3Processor) and huggingface_hub. Both ship in the foundation extra (which pulls torch); the gpu umbrella extra installs every GPU detector at once.

uv sync --extra foundation           # SAM3 + DINO-based detectors
uv sync --extra gpu                  # umbrella: every GPU detector (dev env)
# as a dependency of a downstream project:
uv add 'phenotypic[foundation]'

Installing any extra never pulls encumbered material — only the gated weights (SAM3, DINOv3) require a license handshake, and they are fetched at runtime, never at install time (see Gated weights below).

Enabling micro_sam (optional, self-service)#

micro_sam is only published on conda-forge. Because PhenoTypic does not own your environment, we recommend managing the combined stack in your own project with pixi, which speaks both conda-forge and PyPI in a single lockfile. Create a pixi.toml in your project (not in PhenoTypic):

[project]
name = "my-phenotyping-project"
channels = ["conda-forge"]
platforms = ["osx-arm64", "linux-64", "win-64"]

[pypi-dependencies]
phenotypic = "*"
# Or, while developing against a local checkout:
# phenotypic = { path = "../PhenoTypic", editable = true }

[dependencies]
micro_sam = "*"

Then:

pixi install
pixi run python -m phenotypic --pipeline pipeline.json --input /plates/ -o /output/

Because conda’s micro_sam pulls in CPU-only conda PyTorch, combining it with Sam2Detector’s CUDA wheels in the same environment requires extra care (the conda torch will typically win). Keep SAM2 and micro-sam work in separate environments if you need both with GPU acceleration.

MicroSamDetector is importable from phenotypic.detect.nn even when micro_sam is missing; the ImportError is deferred to the first apply() call and points back at these instructions.

Alternative: pip + conda#

If you already manage your environment with conda:

pip install phenotypic                        # base (or phenotypic[torch] on non-Windows)
conda install -c conda-forge micro_sam        # adds MicroSamDetector support

Downloading Model Checkpoints#

Both SAM2 and micro-sam download checkpoints automatically on first use. However, on SLURM clusters the compute nodes often lack internet access, so you should pre-download checkpoints on a login node before submitting jobs.

SAM2 checkpoints#

# Download the default (tiny) SAM2 checkpoint
python -m phenotypic.detect.nn download

# Download a specific size
python -m phenotypic.detect.nn download --model-type sam2 --model-size large

# Download all SAM2 sizes at once
python -m phenotypic.detect.nn download --model-type sam2 --all

# Force re-download even if cached
python -m phenotypic.detect.nn download --model-type sam2 --model-size tiny --force

SAM2 checkpoints are stored in the torch.hub cache directory (~/.cache/torch/hub/checkpoints/ by default). Set the TORCH_HOME environment variable to change this location.

Available SAM2 sizes: tiny (~39 MB), small, base_plus, large (~900 MB).

micro-sam checkpoints#

# Download the default (vit_b_lm) micro-sam model
python -m phenotypic.detect.nn download --model-type microsam

# Download a specific model
python -m phenotypic.detect.nn download --model-type microsam --model-name vit_l_lm

# Download all micro-sam models
python -m phenotypic.detect.nn download --model-type microsam --all

micro-sam stores checkpoints via platformdirs. Set MICROSAM_CACHEDIR to override the cache location.

Foundation-model weights (SAM3 + DINOv3 gated, DINOv2 ungated)#

SAM3 and the DINO backbones live on the Hugging Face Hub and are downloaded via huggingface_hub.snapshot_download into the HF cache. SAM3 weights (~3.45 GB) and DINOv3 weights are gated; DINOv2 is ungated.

One-time human handshake for a gated model (per user, per model):

  1. Have a Hugging Face account.

  2. Accept the gate on the model page — https://huggingface.co/facebook/sam3 for SAM3, https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m for DINOv3 (this is the license acceptance).

  3. Authenticate locally once: uv run hf auth login (stores a token), or export HF_TOKEN.

Then download with the extended phenotypic.detect.nn CLI:

# SAM3 (gated): --accept-license acknowledges the SAM License non-interactively
uv run python -m phenotypic.detect.nn download --model-type sam3 --accept-license

# DINOv3 (gated): --accept-license acknowledges the DINOv3 License
#   (required for Insid3Detector's native backbone + the dino_version=3 opt-in)
uv run python -m phenotypic.detect.nn download --model-type dinov3 --dino-size base --accept-license

# DINOv2 (ungated): no acceptance, no token
uv run python -m phenotypic.detect.nn download --model-type dinov2 --dino-size base

If the gate is not accepted or no token is present, the download fails with an actionable message (“Request access at , then run uv run hf auth login”). The informational acceptance gate is also satisfied non-interactively by PHENOTYPIC_ACCEPT_MODEL_LICENSE=sam3,dinov3 (a comma list for several models) — this is the layer the --accept-license flag sets for you, on top of the binding Hugging Face gate.

Foundation-model environment variables#

Variable

Purpose

HF_TOKEN / HUGGING_FACE_HUB_TOKEN

auth for gated download (alternative to hf auth login)

HF_HOME (or HF_HUB_CACHE)

cache location → point at shared HPCC storage

HF_HUB_OFFLINE=1 / TRANSFORMERS_OFFLINE=1

force load-from-cache on offline compute nodes

PHENOTYPIC_ACCEPT_MODEL_LICENSE

non-interactive acknowledgement for batch/SLURM jobs

SLURM pre-caching workflow#

On a cluster, download models on the login node first:

# On the login node (has internet access)
python -m phenotypic.detect.nn download --model-type sam2 --model-size tiny
python -m phenotypic.detect.nn download --model-type microsam --model-name vit_b_lm

# Verify they are cached
python -m phenotypic.detect.nn list

# Now submit SLURM jobs -- compute nodes will use the cached checkpoints
python -m phenotypic --pipeline pipeline.json --input /plates/ -o /output/

Foundation models on HPCC (offline compute nodes). Never download inside a job — accept the gate and pre-stage weights to shared storage on the login node, then run the compute job offline:

# On the login node (has internet): cache to shared storage, accept once
export HF_HOME=/bigdata/exfab/<...>/hf_cache
uv run hf auth login
uv run python -m phenotypic.detect.nn download --model-type sam3 --accept-license
uv run python -m phenotypic.detect.nn download --model-type dinov2

# In the SLURM job: same HF_HOME, force offline, carry the accepted acknowledgement
export HF_HOME=/bigdata/exfab/<...>/hf_cache
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1
export PHENOTYPIC_ACCEPT_MODEL_LICENSE=sam3
python -m phenotypic --pipeline pipeline.json --input /plates/ -o /output/

The binding acceptance (HF gate + token) happens once on the login node; the compute job carries only the already-accepted token plus PHENOTYPIC_ACCEPT_MODEL_LICENSE and reads the pre-staged cache. Acceptance never happens silently inside a batch job.

Using Sam2Detector#

Sam2Detector wraps Meta’s SAM2 automatic mask generator. It lays a grid of prompt points over the RGB image, predicts masks at each point, filters by quality, and assembles a labelled object map.

from phenotypic.detect.nn import Sam2Detector

# Basic usage with default parameters
detector = Sam2Detector()

# Tuned for dense plates with small colonies
detector = Sam2Detector(
    model_size="small",
    points_per_side=48,
    pred_iou_thresh=0.6,
    min_mask_region_area=200,
)

# Apply to an image (downloads checkpoint on first use)
result = detector.apply(image)
print(result.num_objects)

Parameter tuning for colony detection#

  • points_per_side (default 32): Controls the density of the prompt grid. Use 16 for large, well-separated colonies. Increase to 48–64 for dense plates with many small colonies. Higher values increase inference time quadratically.

  • pred_iou_thresh (default 0.7): Minimum predicted IoU for keeping a mask. Raise to 0.85–0.95 for conservative detection (fewer false positives); lower to 0.5 to catch faint or ambiguous colonies.

  • stability_score_thresh (default 0.92): Filters masks by boundary stability. Higher values keep only masks with crisp edges.

  • min_mask_region_area (default 100): Minimum mask area in pixels. Increase to suppress agar texture, dust, and other small artefacts that SAM2 segments as objects. Typical range: 50–500 depending on image resolution.

  • model_size (default "tiny"): "tiny" is fastest and sufficient for most colony plates. Use "large" for maximum mask quality on publication figures.

Using MicroSamDetector#

MicroSamDetector uses SAM models finetuned on large-scale microscopy datasets. It is particularly effective for brightfield and darkfield microscopy images of agar plates.

from phenotypic.detect.nn import MicroSamDetector

# Default: ViT-Base light microscopy model
detector = MicroSamDetector()

# Use the larger model for higher accuracy
detector = MicroSamDetector(model_type="vit_l_lm")

result = detector.apply(image)

Model selection#

Light microscopy models (recommended for agar plate imaging):

  • "vit_t_lm" – ViT-Tiny, fastest, good for rapid screening

  • "vit_b_lm" – ViT-Base (default), best speed/accuracy trade-off

  • "vit_l_lm" – ViT-Large, highest accuracy, most VRAM

Electron microscopy models (for organelle segmentation):

  • "vit_b_em_organelles" – ViT-Base

  • "vit_l_em_organelles" – ViT-Large

Base SAM checkpoints (without microscopy finetuning):

  • "vit_t", "vit_b", "vit_l", "vit_h"

Using Sam3Detector#

Sam3Detector wraps Meta’s text-prompted SAM3 foundation model. Unlike SAM2’s dense point grid, SAM3 segments everything matching a short text prompt (default "colony") in one true batched forward pass, then assembles the predicted instance masks into a labelled objmap (output_kind="instance").

from phenotypic.detect.nn import Sam3Detector

# Override the prompt per run — SAM3 has no prompt-free "segment everything" mode.
det = Sam3Detector(prompt="yeast colony", score_thresh=0.5)

SAM3 weights are gated (SAM License). Accept the gate and authenticate once (see Foundation-model weights above) before the first apply().

Dense plates. SAM3 caps at 200 instances per forward and runs at 1008 px internally, so Sam3Detector tiles large plates into fixed tile_px crops with tile_overlap, infers each tile, offsets the masks back to full coordinates, and merges cross-tile duplicates by IoU-NMS — all automatically. Images that fit one tile run un-tiled.

Key parameters:

  • prompt — free text describing the target ("colony", "bacterial colony").

  • score_thresh / mask_threshold — instance-confidence and mask-probability cutoffs.

  • min_mask_region_area — drop masks smaller than this (default 100).

  • tile_px / tile_overlap — dense-plate tiling controls (defaults 1008 / 0.15).

Using DinoSam2Detector#

DinoSam2Detector is a training-free instance detector that composes two ungated foundation models: SAM2’s automatic mask generator produces class-agnostic proposals, and a DINOv2 backbone (Apache-2.0, ungated by default) supplies dense patch features. Each proposal is scored by cosine similarity of its pooled DINO feature to a foreground prototype; background-like proposals are dropped, near-duplicates merged by IoU, and survivors painted into a labelled objmap.

from phenotypic.detect.nn import DinoSam2Detector

# Recommended config is fully ungated (SAM2 Apache + DINOv2 Apache).
det = DinoSam2Detector(dino_size="base", similarity_thresh=0.5)

dino_version selects the backbone generation: 2 = DINOv2 (default, ungated), 3 = DINOv3 (gated opt-in). Selecting dino_version=3 routes the snapshot pull through the DINOv3-License acceptance gate (pre-stage it with the download --model-type dinov3 --accept-license command above).

Key parameters:

  • dino_version / dino_size — backbone generation (2/3) and size.

  • sam2_model_size — SAM2 variant for the proposal generator.

  • similarity_thresh — minimum cosine-to-prototype score to keep a proposal.

  • merge_iou_thresh — IoU above which two survivors are merged.

Semantic few-shot detectors (Insid3Detector, FssDinoDetector)#

Insid3Detector (one-shot, in-context) and FssDinoDetector (few-shot) are semantic detectors: they emit a binary objmask (output_kind="semantic"), not their own instance labels. The mask auto-labels into the shared objmap backend exactly like a threshold detector, so the repo’s downstream watershed (SeparateObjects) turns it into instances — pair them with a separation step in your pipeline, just as you would OtsuDetector. Both run on a frozen DINO backbone and write only objmask; objmap[:] > 0 then equals objmask[:].

Both ship a curated colony exemplar (a reference colony RGB + its mask, rendered once from load_synth_yeast_plate()) as the default reference / support set, so they work out of the box. Supply your own annotated exemplar to transfer to a new colony appearance.

Insid3Detector — one-shot in-context (DINOv3-native, gated)#

A faithful clean-room reimplementation of INSID3 (Apache-2.0). Given a single reference image + reference mask, it pools an in-context prototype and cosine-matches every query patch — but first removes DINOv3’s positional bias (estimated by SVD and projected out, INSID3’s defining step) so patches match on appearance, not position. It is DINOv3-native (gated, dino_version=3 default); a dino_version=2 opt-in runs gate-free (the debias is then a near-no-op).

from phenotypic.detect.nn import Insid3Detector

# Default: bundled exemplar + gated DINOv3 (accept the DINOv3 License first).
det = Insid3Detector(similarity_thresh=0.5)

# Override the in-context reference with your own annotated colony pair:
det = Insid3Detector(
    reference_image="ref_plate.tiff",
    reference_mask="ref_plate_mask.png",
    similarity_thresh=0.6,
)

# Gate-free DINOv2 variant (no Hugging Face token needed):
det = Insid3Detector(dino_version=2, dino_size="small")

Key parameters:

  • reference_image / reference_mask — the in-context exemplar (defaults to the bundled colony exemplar).

  • dino_version / dino_size — backbone (3 = DINOv3 default/gated, 2 = DINOv2).

  • similarity_thresh — cosine cutoff binarising the match map.

  • svd_components — INSID3’s positional-debias strength (leading SVD directions removed; default 4 ≈ DINOv3’s register-token count; 0 disables the debias).

  • tile_px / tile_overlap — large-plate tiling (defaults 1024 / 0.15).

FssDinoDetector — few-shot (DINOv2 default, ungated)#

A clean-room reimplementation from the paper only of FSSDINO (arXiv:2602.07550, CC BY-NC-SA; the reference repo is all-rights-reserved and is not vendored). From a support set it builds n_clusters class-specific prototypes (k-means) plus a Gram matrix (channel co-occurrence), scores each query patch by cosine to the prototypes and a Gram-refined energy, combines the maps (mean ⊙ max) and assigns foreground vs background by argmax. It defaults to DINOv2 (ungated), so it runs gate-free.

from phenotypic.detect.nn import FssDinoDetector

# Default: bundled one-shot exemplar + ungated DINOv2.
det = FssDinoDetector(n_clusters=5)

# A true few-shot support set:
det = FssDinoDetector(
    support_images=["s1.tiff", "s2.tiff", "s3.tiff"],
    support_masks=["s1_mask.png", "s2_mask.png", "s3_mask.png"],
    n_clusters=5,
)

Key parameters:

  • support_images / support_masks — the few-shot support set (defaults to the bundled one-shot colony exemplar).

  • n_clusters — class-specific prototypes per class (paper default 5).

  • feature_layer — transformer hidden-state layer for the dense features. FSSDINO’s “Semantic Selection Gap” finding is that intermediate layers often beat the last, but cannot be reliably selected unsupervised — so the default is -1 (the last layer, the paper’s safe default).

  • dino_version / dino_size — backbone (2 = DINOv2 default/ungated, 3 = DINOv3 gated opt-in).

  • similarity_thresh — foreground-score floor on top of the fg-vs-bg argmax.

  • tile_px / tile_overlap — large-plate tiling (defaults 512 / 0.15).

Pipeline Integration#

GPU detectors work like any other PhenoTypic operation in a pipeline:

import phenotypic as pht
from phenotypic.detect.nn import Sam2Detector
from phenotypic.measure import SizeMeasurer

pipeline = pht.ImagePipeline(
    ops=[Sam2Detector(model_size="tiny", points_per_side=32)],
    measurer=SizeMeasurer(),
    name="sam2_colony_pipeline",
)

# Run the pipeline
results = pipeline.operate([image])
df = pipeline.measure([image])

JSON serialization#

Pipelines containing GPU detectors can be saved and loaded just like any other pipeline. The detector parameters are serialized; the model weights are not (they are re-downloaded or loaded from cache when needed):

# Save
pipeline.to_json("sam2_pipeline.json")

# Load -- works without torch installed (model loads lazily on apply)
restored = pht.ImagePipeline.from_json("sam2_pipeline.json")

Internal state (attributes prefixed with _, such as the loaded model) is excluded from serialization. The model is rebuilt transparently on the next call to apply.

Local Staged GPU Detection (CLI)#

When you run a pipeline through the CLI (python -m phenotypic) and it contains a GpuDetector, detection runs as three internal stages rather than invoking the GPU model once per image. The segmentation model is built once and every image is streamed through it — far more efficient than the notebook per-image path when processing a directory:

  1. Stage 1 — CPU preprocess. Every prior ImageOperation (enhancers, corrections) is applied per image and the result is saved to the normal per-image HDF (results/<dataset>/hdf/<stem>.h5).

  2. Stage 2 — resident-model GPU detect. The detector’s model is built once and kept resident while each staged HDF is streamed through preprocess → infer_batch. The labelled object map is written to a per-image .npy sidecar at results/<dataset>/objmap/<stem>.npy; the HDF is opened read-only here, so an interrupted run never corrupts it.

  3. Stage 3 — CPU merge + measure. The sidecar is merged back into the image through the object-map accessor, the post-detector refiners and the measurement queue run, the HDF is re-saved atomically, and the sidecar is deleted.

The output folder is identical to a single-pass run — staging is an internal optimization, not a different output contract.

Resume is content-defined. Re-running the same command skips any image whose work already exists: Stage 1 skips when the HDF exists, Stage 2 skips when the sidecar or the measurement parquet exists (Stage 3 deletes the sidecar, so the parquet is the durable “done” marker), and Stage 3 skips when the parquet exists. Progress is stage-tagged in the event log, so the run dashboard can show how far each image has moved through the three stages. If Stage 1 fails for an image (e.g. an unreadable file), Stages 2 and 3 skip it and record a structured failure instead of aborting the batch.

# Forward run: detection stages automatically because the pipeline has a GpuDetector
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/

# Export just the object maps (runs Stages 1-2, then writes one objmap PNG per image)
python -m phenotypic --mode process --layer objmap \
    --pipeline sam2_pipeline.json --input /plates/ -o /output/

SLURM Deployment#

When a pipeline contains a GpuDetector operation (either Sam2Detector or MicroSamDetector), the CLI automatically adapts:

Local execution: Forward GPU runs use the staged engine above (the model loads once and streams every image); see “Local Staged GPU Detection”. The legacy per-image path (measure-only and non-objmap layer exports) still forces sequential processing (n_jobs=1) to avoid multiple workers competing for the same GPU.

SLURM execution: Automatically adds --gpus-per-node=1 to the SLURM job if GPU resources were not explicitly requested.

# GPU resources are auto-requested when the pipeline contains a GpuDetector
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/

# Override with explicit SLURM GPU arguments
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/ \
    --slurm slurm_gpus_per_node=2 \
    --slurm slurm_partition=gpu

Pre-cache checkpoints on the login node before submitting (see “Downloading Model Checkpoints” above).

SLURM Staged GPU Detection#

A forward GPU run on SLURM (--slurm ... with a GpuDetector pipeline) runs as the staged engine above, submitted as a 3-link afterany dependency chain with per-stage resources:

  • Stage 1 — a CPU array over images (preprocess → staged HDF), on the --slurm profile.

  • Stage 2 — a GPU array over shards (--gpu-shards N): each task is one whole GPU running a resident model that streams its shard of HDFs to objmap sidecars, on the --gpu-slurm profile.

  • Stage 3 — a CPU array over images (merge sidecar → measure), on the --slurm profile.

afterany between stages means a handful of per-image failures never block the next stage. Because Stage 2 writes a .npy sidecar with the HDF opened read-only, there is no HDF5 write-locking on the GPU nodes; Stages 1 & 3 write the HDF atomically (temp + rename) on CPU nodes.

# CPU partition for Stages 1 & 3 (--slurm); GPU partition + 2 concurrent GPUs for Stage 2
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/ \
    --slurm slurm_partition=batch --slurm slurm_time=02:00:00 \
    --gpu-slurm slurm_partition=gpu --gpu-shards 2

The resource nesting is three levels: --gpu-shards (whole GPUs, across nodes) → --gpu-workers-per-gpu (replicas packed per GPU for small models) → --gpu-batch-size (images per forward pass; batchable models, auto in Spec 2). --gpu-slurm inherits/deltas over --slurm, so shared keys (account, qos, time) carry over and only the GPU partition/account need restating; one GPU is requested automatically.

Walltime survival. The Stage-2 script carries #SBATCH --signal=B:TERM@120, so SLURM sends SIGTERM shortly before the walltime. The shard-worker catches it and sbatch-resubmits its shard (afterany on itself); content-defined skip means the continuation processes only the remaining sidecar-less images, and it repeats until the shard is complete — so a TIMEOUT never loses work and never needs a manual restart.

Pre-staging gated weights. For offline compute nodes, download checkpoints on the login node first and export HF_HUB_OFFLINE=1 (and hf auth login for gated Hugging Face models). Gated foundation-model weights are never bundled; accept their license once via PHENOTYPIC_ACCEPT_MODEL_LICENSE=<model> (see the require_license_acceptance hook).

Custom detectors on SLURM. Each SLURM stage worker is a fresh process that deserializes the pipeline with ImagePipeline.from_json, which resolves operation classes from the phenotypic namespace. If your pipeline uses a detector defined outside the package, set PHENOTYPIC_PRELOAD_MODULES=your.module[,another.module] — the worker imports each before deserializing, so a self-registering module can make its class resolvable on the compute node. sbatch --export=ALL (the default) propagates the variable.

Device Selection#

Both detectors accept a device parameter that controls where inference runs.

Automatic detection (default)#

With device="auto" (the default), PhenoTypic probes accelerators in priority order:

  1. CUDA – NVIDIA GPUs

  2. MPS – Apple Silicon (macOS)

  3. XPU – Intel GPUs

  4. HPU – Habana Gaudi accelerators

If none is found, a RuntimeError is raised.

Explicit device#

# Force a specific device
Sam2Detector(device="cuda")   # NVIDIA GPU
Sam2Detector(device="mps")    # Apple Silicon
Sam2Detector(device="xpu")    # Intel GPU
Sam2Detector(device="cpu")    # CPU (very slow, but always available)

When an explicit accelerator is requested but unavailable, a RuntimeError is raised with a descriptive message.

resolve_device() utility#

The device resolution logic is available as a standalone function for custom workflows:

from phenotypic.detect.nn._checkpoint_manager import resolve_device

device = resolve_device("auto")           # raises if no accelerator
device = resolve_device("auto", allow_cpu=True)  # falls back to CPU with warning

Listing and Clearing Models#

List cached checkpoints#

python -m phenotypic.detect.nn list

This prints a table showing all cached SAM2 and micro-sam checkpoints with their file sizes and paths.

Clear cached checkpoints#

# Clear all cached checkpoints (prompts for confirmation)
python -m phenotypic.detect.nn clear

# Clear only SAM2 checkpoints
python -m phenotypic.detect.nn clear --model-type sam2

# Clear only micro-sam checkpoints
python -m phenotypic.detect.nn clear --model-type microsam

Troubleshooting#

ImportError: Sam2Detector requires the sam2 package#

PyTorch and the model packages are not installed. Install the torch extra:

uv add "phenotypic[torch]"

(Linux/macOS only — sam2 is not packaged for Windows.)

ImportError: MicroSamDetector requires the micro_sam package#

micro_sam is conda-only and must be installed separately. See Enabling micro_sam (optional, self-service) above.

RuntimeError: No accelerator available#

No GPU was detected. Options:

  • Ensure your GPU drivers and CUDA toolkit are installed correctly.

  • On macOS with Apple Silicon, ensure PyTorch >= 2.0 with MPS support.

  • Pass device="cpu" to force CPU inference (very slow):

Sam2Detector(device="cpu")

RuntimeError: device='cuda' requested but CUDA is not available#

CUDA was explicitly requested but is not available. Check:

  • nvidia-smi shows your GPU.

  • PyTorch was installed with CUDA support (torch.cuda.is_available() returns True).

  • On SLURM, the job was submitted to a GPU partition.

Out of memory (OOM) errors#

SAM models require significant GPU memory. To reduce VRAM usage:

  • Use a smaller model: Sam2Detector(model_size="tiny") instead of "large".

  • Use MicroSamDetector(model_type="vit_t_lm") for the smallest micro-sam model.

  • Reduce points_per_side (e.g., 16 instead of 32) to generate fewer candidate masks.

  • Process smaller images or downscale before detection.

Checkpoint not found on SLURM compute nodes#

Compute nodes often lack internet access. Pre-download checkpoints on the login node:

python -m phenotypic.detect.nn download --model-type sam2 --model-size tiny
python -m phenotypic.detect.nn download --model-type microsam --model-name vit_b_lm
python -m phenotypic.detect.nn list  # verify

Ensure TORCH_HOME and MICROSAM_CACHEDIR (if customised) point to a shared filesystem accessible from compute nodes.

Illegal instruction (core dumped) on a SLURM compute node#

A stage worker exits with code 132 and the SLURM .err log shows Illegal instruction (core dumped). The node’s CPU is too old for the installed numpy/scipy/torch wheels (a pre-AVX node on a heterogeneous partition). This affects any PhenoTypic SLURM run, not just the staged GPU engine. Pin jobs to modern nodes — use a homogeneous modern partition, or add a SLURM feature constraint that excludes the old CPUs, e.g.:

python -m phenotypic --pipeline p.json --input /plates/ -o /out/ \
    --slurm slurm_partition=<modern-partition> \
    --gpu-slurm slurm_partition=<gpu-partition>

(Stage 2’s GPU work runs on GPU nodes, which are typically consistent; the CPU Stages 1 & 3 are the ones exposed to a heterogeneous CPU partition.)