GPU-Accelerated Colony Detection#

Set up and use deep-learning-based colony detectors (SAM2, micro-sam, SAM3, DinoSam2) with GPU acceleration.

Installation#

The GPU detectors have different packaging constraints:

Detector	Package(s) needed	Available via	CUDA-capable?
`Sam2Detector`	`torch`, `torchvision`, `sam2`	PyPI (ships in `phenotypic[torch]`)	Yes — Linux + CUDA
`MicroSamDetector`	`micro_sam` (+ `torch`)	conda-forge only, not on PyPI	CPU by default; user-managed CUDA possible
`Sam3Detector`	`transformers`, `huggingface_hub` (+ `torch`)	PyPI (ships in `phenotypic[foundation]`); weights gated	Yes — Linux + CUDA
`DinoSam2Detector`	`transformers`, `sam2` (+ `torch`)	PyPI (ships in `phenotypic[foundation]`); DINOv2 weights ungated	Yes — Linux + CUDA

Per-model license posture#

Model	Code license	Weights license	Gated?
SAM2	Apache-2.0	Apache-2.0	No
SAM3	Apache-2.0 (`transformers`)	SAM License (commercial-OK)	Yes — accept on Hugging Face
DINOv2	Apache-2.0	Apache-2.0	No
DINOv3 (opt-in)	Apache-2.0	DINOv3 License (custom Meta)	Yes — accept on Hugging Face
INSID3 method (`Insid3Detector`)	Apache-2.0 (clean-room, no code vendored)	DINOv3-native backbone (gated)	Yes (via DINOv3)
FSSDINO method (`FssDinoDetector`)	paper CC BY-NC-SA (clean-room, no code vendored)	DINOv2 default (ungated) / DINOv3 opt-in	No (DINOv2) / Yes (DINOv3)

PhenoTypic does not redistribute model weights — each weight is downloaded by you from the upstream source under that model’s license, which you accept (see NOTICE and licenses/). The two semantic detectors (Insid3Detector, FssDinoDetector) carry no vendored upstream code: INSID3 is clean-room-reimplemented from its Apache-2.0 method (attributed), and FSSDINO is clean-room-reimplemented from the paper only (the reference repo is all-rights-reserved). When dino_version=3 is selected, PhenoTypic displays “Built with DINOv3” per the DINOv3 License.

PhenoTypic itself is distributed via PyPI and managed with uv. micro_sam is not published on PyPI, so it is not included in any phenotypic extra. Users who need MicroSamDetector must install micro_sam themselves; the tutorial recipe below uses pixi for that.

Installing the detector packages and downloading their weights is covered in the Deep Learning Detectors section of the Getting Started tutorial. In short: uv add "phenotypic[torch]" for SAM2, "phenotypic[foundation]" for the SAM3/DINO detectors, the [gpu] umbrella for all of them, and conda-forge micro_sam for MicroSamDetector. In a source checkout, use uv sync --extra foundation to install the same foundation-model dependencies. Gated SAM3/DINOv3 weights need a one-time Hugging Face license handshake (hf auth login after accepting the model license on Hugging Face).

MicroSamDetector is importable from phenotypic.detect.nn even when micro_sam is missing; the ImportError is deferred to the first apply() call and points back at the installation instructions.

Downloading Model Checkpoints#

Every detector downloads its checkpoints automatically on first use, and the phenotypic.detect.nn CLI (download / list / clear) pre-fetches and inspects them — essential before submitting SLURM jobs, since compute nodes often lack internet access. The full command set, cache-location environment variables (TORCH_HOME, MICROSAM_CACHEDIR, HF_HOME, PHENOTYPIC_ACCEPT_MODEL_LICENSE, …), and the login-node pre-staging workflow for gated foundation weights are documented in the Deep Learning Detectors tutorial section. The SLURM-specific staging is expanded under SLURM Deployment below.

Using Sam2Detector#

Sam2Detector wraps Meta’s SAM2 automatic mask generator. It lays a grid of prompt points over the RGB image, predicts masks at each point, filters by quality, and assembles a labelled object map.

from phenotypic.detect.nn import Sam2Detector

# Basic usage with default parameters
detector = Sam2Detector()

# Tuned for dense plates with small colonies
detector = Sam2Detector(
    model_size="small",
    points_per_side=48,
    pred_iou_thresh=0.6,
    min_mask_region_area=200,
)

# Apply to an image (downloads checkpoint on first use)
result = detector.apply(image)
print(result.num_objects)

Parameter tuning for colony detection#

points_per_side (default 32): Controls the density of the prompt grid. Use 16 for large, well-separated colonies. Increase to 48–64 for dense plates with many small colonies. Higher values increase inference time quadratically.
pred_iou_thresh (default 0.7): Minimum predicted IoU for keeping a mask. Raise to 0.85–0.95 for conservative detection (fewer false positives); lower to 0.5 to catch faint or ambiguous colonies.
stability_score_thresh (default 0.92): Filters masks by boundary stability. Higher values keep only masks with crisp edges.
min_mask_region_area (default 100): Minimum mask area in pixels. Increase to suppress agar texture, dust, and other small artefacts that SAM2 segments as objects. Typical range: 50–500 depending on image resolution.
model_size (default "tiny"): "tiny" is fastest and sufficient for most colony plates. Use "large" for maximum mask quality on publication figures.

Using MicroSamDetector#

MicroSamDetector uses SAM models finetuned on large-scale microscopy datasets. It is particularly effective for brightfield and darkfield microscopy images of agar plates.

from phenotypic.detect.nn import MicroSamDetector

# Default: ViT-Base light microscopy model
detector = MicroSamDetector()

# Use the larger model for higher accuracy
detector = MicroSamDetector(model_type="vit_l_lm")

result = detector.apply(image)

Model selection#

Light microscopy models (recommended for agar plate imaging):

"vit_t_lm" – ViT-Tiny, fastest, good for rapid screening
"vit_b_lm" – ViT-Base (default), best speed/accuracy trade-off
"vit_l_lm" – ViT-Large, highest accuracy, most VRAM

Electron microscopy models (for organelle segmentation):

"vit_b_em_organelles" – ViT-Base
"vit_l_em_organelles" – ViT-Large

Base SAM checkpoints (without microscopy finetuning):

"vit_t", "vit_b", "vit_l", "vit_h"

Using Sam3Detector#

Sam3Detector wraps Meta’s text-prompted SAM3 foundation model. Unlike SAM2’s dense point grid, SAM3 segments everything matching a short text prompt (default "colony") in one true batched forward pass, then assembles the predicted instance masks into a labelled objmap (output_kind="instance").

from phenotypic.detect.nn import Sam3Detector

# Override the prompt per run — SAM3 has no prompt-free "segment everything" mode.
det = Sam3Detector(prompt="yeast colony", score_thresh=0.5)

SAM3 weights are gated (SAM License). Accept the gate and authenticate once (see Deep Learning Detectors) before the first apply().

Dense plates. SAM3 caps at 200 instances per forward and runs at 1008 px internally, so Sam3Detector tiles large plates into fixed tile_px crops with tile_overlap, infers each tile, offsets the masks back to full coordinates, and merges cross-tile duplicates by IoU-NMS — all automatically. Images that fit one tile run un-tiled.

Key parameters:

prompt — free text describing the target ("colony", "bacterial colony").
score_thresh / mask_threshold — instance-confidence and mask-probability cutoffs.
min_mask_region_area — drop masks smaller than this (default 100).
tile_px / tile_overlap — dense-plate tiling controls (defaults 1008 / 0.15).

Using DinoSam2Detector#

DinoSam2Detector is a training-free instance detector that composes two ungated foundation models: SAM2’s automatic mask generator produces class-agnostic proposals, and a DINOv2 backbone (Apache-2.0, ungated by default) supplies dense patch features. Each proposal is scored by cosine similarity of its pooled DINO feature to a foreground prototype; background-like proposals are dropped, near-duplicates merged by IoU, and survivors painted into a labelled objmap.

from phenotypic.detect.nn import DinoSam2Detector

# Recommended config is fully ungated (SAM2 Apache + DINOv2 Apache).
det = DinoSam2Detector(dino_size="base", similarity_thresh=0.5)

dino_version selects the backbone generation: 2 = DINOv2 (default, ungated), 3 = DINOv3 (gated opt-in). Selecting dino_version=3 routes the snapshot pull through the DINOv3-License acceptance gate (pre-stage it with the download --model-type dinov3 --accept-license command from the Deep Learning Detectors tutorial section). The default DINOv3 checkpoint id is dinov3-vitb16-pretrain-lvd1689m.

Key parameters:

dino_version / dino_size — backbone generation (2/3) and size.
sam2_model_size — SAM2 variant for the proposal generator.
similarity_thresh — minimum cosine-to-prototype score to keep a proposal.
merge_iou_thresh — IoU above which two survivors are merged.

Semantic few-shot detectors (`Insid3Detector`, `FssDinoDetector`)#

Insid3Detector (one-shot, in-context) and FssDinoDetector (few-shot) are semantic detectors: they emit a binary objmask (output_kind="semantic"), not their own instance labels. The mask auto-labels into the shared objmap backend exactly like a threshold detector, so the repo’s downstream watershed (SeparateObjects) turns it into instances — pair them with a separation step in your pipeline, just as you would OtsuDetector. Both run on a frozen DINO backbone and write only objmask; objmap[:] > 0 then equals objmask[:].

Both ship a curated colony exemplar (a reference colony RGB + its mask, rendered once from load_synth_yeast_plate()) as the default reference / support set, so they work out of the box. Supply your own annotated exemplar to transfer to a new colony appearance.

`Insid3Detector` — one-shot in-context (DINOv3-native, gated)#

A faithful clean-room reimplementation of INSID3 (Apache-2.0). Given a single reference image + reference mask, it pools an in-context prototype and cosine-matches every query patch — but first removes DINOv3’s positional bias (estimated by SVD and projected out, INSID3’s defining step) so patches match on appearance, not position. It is DINOv3-native (gated, dino_version=3 default); a dino_version=2 opt-in runs gate-free (the debias is then a near-no-op).

from phenotypic.detect.nn import Insid3Detector

# Default: bundled exemplar + gated DINOv3 (accept the DINOv3 License first).
det = Insid3Detector(similarity_thresh=0.5)

# Override the in-context reference with your own annotated colony pair:
det = Insid3Detector(
    reference_image="ref_plate.tiff",
    reference_mask="ref_plate_mask.png",
    similarity_thresh=0.6,
)

# Gate-free DINOv2 variant (no Hugging Face token needed):
det = Insid3Detector(dino_version=2, dino_size="small")

Key parameters:

reference_image / reference_mask — the in-context exemplar (defaults to the bundled colony exemplar).
dino_version / dino_size — backbone (3 = DINOv3 default/gated, 2 = DINOv2).
similarity_thresh — cosine cutoff binarising the match map.
svd_components — INSID3’s positional-debias strength (leading SVD directions removed; default 4 ≈ DINOv3’s register-token count; 0 disables the debias).
tile_px / tile_overlap — large-plate tiling (defaults 1024 / 0.15).

`FssDinoDetector` — few-shot (DINOv2 default, ungated)#

A clean-room reimplementation from the paper only of FSSDINO (arXiv:2602.07550, CC BY-NC-SA; the reference repo is all-rights-reserved and is not vendored). From a support set it builds n_clusters class-specific prototypes (k-means) plus a Gram matrix (channel co-occurrence), scores each query patch by cosine to the prototypes and a Gram-refined energy, combines the maps (mean ⊙ max) and assigns foreground vs background by argmax. It defaults to DINOv2 (ungated), so it runs gate-free.

from phenotypic.detect.nn import FssDinoDetector

# Default: bundled one-shot exemplar + ungated DINOv2.
det = FssDinoDetector(n_clusters=5)

# A true few-shot support set:
det = FssDinoDetector(
    support_images=["s1.tiff", "s2.tiff", "s3.tiff"],
    support_masks=["s1_mask.png", "s2_mask.png", "s3_mask.png"],
    n_clusters=5,
)

Key parameters:

support_images / support_masks — the few-shot support set (defaults to the bundled one-shot colony exemplar).
n_clusters — class-specific prototypes per class (paper default 5).
feature_layer — transformer hidden-state layer for the dense features. FSSDINO’s “Semantic Selection Gap” finding is that intermediate layers often beat the last, but cannot be reliably selected unsupervised — so the default is -1 (the last layer, the paper’s safe default).
dino_version / dino_size — backbone (2 = DINOv2 default/ungated, 3 = DINOv3 gated opt-in).
similarity_thresh — foreground-score floor on top of the fg-vs-bg argmax.
tile_px / tile_overlap — large-plate tiling (defaults 512 / 0.15).

Pipeline Integration#

GPU detectors work like any other PhenoTypic operation in a pipeline:

import phenotypic as pht
from phenotypic.detect.nn import Sam2Detector
from phenotypic.measure import SizeMeasurer

pipeline = pht.ImagePipeline(
    ops=[Sam2Detector(model_size="tiny", points_per_side=32)],
    measurer=SizeMeasurer(),
    name="sam2_colony_pipeline",
)

# Run the pipeline
results = pipeline.operate([image])
df = pipeline.measure([image])

JSON serialization#

Pipelines containing GPU detectors can be saved and loaded just like any other pipeline. The detector parameters are serialized; the model weights are not (they are re-downloaded or loaded from cache when needed):

# Save
pipeline.to_json("sam2_pipeline.json")

# Load -- works without torch installed (model loads lazily on apply)
restored = pht.ImagePipeline.from_json("sam2_pipeline.json")

Internal state (attributes prefixed with _, such as the loaded model) is excluded from serialization. The model is rebuilt transparently on the next call to apply.

Local Staged GPU Detection (CLI)#

When you run a pipeline through the CLI (python -m phenotypic) and it contains a GpuDetector, detection runs as three internal stages rather than invoking the GPU model once per image. The segmentation model is built once and every image is streamed through it — far more efficient than the notebook per-image path when processing a directory:

Stage 1 — CPU preprocess. Every prior ImageOperation (enhancers, corrections) is applied per image and the result is saved to the normal per-image HDF (results/<dataset>/hdf/<stem>.h5).
Stage 2 — resident-model GPU detect. The detector’s model is built once and kept resident while each staged HDF is streamed through preprocess → infer_batch. The labelled object map is written to a per-image .npy sidecar at results/<dataset>/objmap/<stem>.npy; the HDF is opened read-only here, so an interrupted run never corrupts it.
Stage 3 — CPU merge + measure. The sidecar is merged back into the image through the object-map accessor, the post-detector refiners and the measurement queue run, the HDF is re-saved atomically, and the sidecar is deleted.

The output folder is identical to a single-pass run — staging is an internal optimization, not a different output contract.

Resume is content-defined. Re-running the same command skips any image whose work already exists: Stage 1 skips when the HDF exists, Stage 2 skips when the sidecar or the measurement parquet exists (Stage 3 deletes the sidecar, so the parquet is the durable “done” marker), and Stage 3 skips when the parquet exists. Progress is stage-tagged in the event log, so the run dashboard can show how far each image has moved through the three stages. If Stage 1 fails for an image (e.g. an unreadable file), Stages 2 and 3 skip it and record a structured failure instead of aborting the batch.

# Forward run: detection stages automatically because the pipeline has a GpuDetector
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/

# Export just the object maps (runs Stages 1-2, then writes one objmap PNG per image)
python -m phenotypic --mode process --layer objmap \
    --pipeline sam2_pipeline.json --input /plates/ -o /output/

SLURM Deployment#

When a pipeline contains a GpuDetector operation (either Sam2Detector or MicroSamDetector), the CLI automatically adapts:

Local execution: Forward GPU runs use the staged engine above (the model loads once and streams every image); see “Local Staged GPU Detection”. The legacy per-image path (measure-only and non-objmap layer exports) still forces sequential processing (n_jobs=1) to avoid multiple workers competing for the same GPU.

SLURM execution: Automatically adds --gpus-per-node=1 to the SLURM job if GPU resources were not explicitly requested.

# GPU resources are auto-requested when the pipeline contains a GpuDetector
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/

# Override with explicit SLURM GPU arguments
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/ \
    --slurm slurm_gpus_per_node=2 \
    --slurm slurm_partition=gpu

Pre-cache checkpoints on the login node before submitting (see “Downloading Model Checkpoints” above).

SLURM Staged GPU Detection#

A forward GPU run on SLURM (--slurm ... with a GpuDetector pipeline) runs as the staged engine above, submitted as a 3-link afterany dependency chain with per-stage resources:

Stage 1 — a CPU array over images (preprocess → staged HDF), on the --slurm profile.
Stage 2 — a GPU array over shards (--gpu-shards N): each task is one whole GPU running a resident model that streams its shard of HDFs to objmap sidecars, on the --gpu-slurm profile.
Stage 3 — a CPU array over images (merge sidecar → measure), on the --slurm profile.

afterany between stages means a handful of per-image failures never block the next stage. Because Stage 2 writes a .npy sidecar with the HDF opened read-only, there is no HDF5 write-locking on the GPU nodes; Stages 1 & 3 write the HDF atomically (temp + rename) on CPU nodes.

# CPU partition for Stages 1 & 3 (--slurm); GPU partition + 2 concurrent GPUs for Stage 2
python -m phenotypic --pipeline sam2_pipeline.json --input /plates/ -o /output/ \
    --slurm slurm_partition=batch --slurm slurm_time=02:00:00 \
    --gpu-slurm slurm_partition=gpu --gpu-shards 2

The resource nesting is three levels: --gpu-shards (whole GPUs, across nodes) → --gpu-workers-per-gpu (replicas packed per GPU for small models) → --gpu-batch-size (images per forward pass; batchable models, auto in Spec 2). --gpu-slurm inherits/deltas over --slurm, so shared keys (account, qos, time) carry over and only the GPU partition/account need restating; one GPU is requested automatically.

Walltime survival. The Stage-2 script carries #SBATCH --signal=B:TERM@120, so SLURM sends SIGTERM shortly before the walltime. The shard-worker catches it and sbatch-resubmits its shard (afterany on itself); content-defined skip means the continuation processes only the remaining sidecar-less images, and it repeats until the shard is complete — so a TIMEOUT never loses work and never needs a manual restart.

Pre-staging gated weights. For offline compute nodes, download checkpoints on the login node first and export HF_HUB_OFFLINE=1 (and hf auth login for gated Hugging Face models). Gated foundation-model weights are never bundled; accept their license once via PHENOTYPIC_ACCEPT_MODEL_LICENSE=<model> (see the require_license_acceptance hook).

Custom detectors on SLURM. Each SLURM stage worker is a fresh process that deserializes the pipeline with ImagePipeline.from_json, which resolves operation classes from the phenotypic namespace. If your pipeline uses a detector defined outside the package, set PHENOTYPIC_PRELOAD_MODULES=your.module[,another.module] — the worker imports each before deserializing, so a self-registering module can make its class resolvable on the compute node. sbatch --export=ALL (the default) propagates the variable.

Device Selection#

Both detectors accept a device parameter that controls where inference runs.

Automatic detection (default)#

With device="auto" (the default), PhenoTypic probes accelerators in priority order:

CUDA – NVIDIA GPUs
MPS – Apple Silicon (macOS)
XPU – Intel GPUs
HPU – Habana Gaudi accelerators

If none is found, a RuntimeError is raised.

Explicit device#

# Force a specific device
Sam2Detector(device="cuda")   # NVIDIA GPU
Sam2Detector(device="mps")    # Apple Silicon
Sam2Detector(device="xpu")    # Intel GPU
Sam2Detector(device="cpu")    # CPU (very slow, but always available)

When an explicit accelerator is requested but unavailable, a RuntimeError is raised with a descriptive message.

`resolve_device()` utility#

The device resolution logic is available as a standalone function for custom workflows:

from phenotypic.detect.nn._checkpoint_manager import resolve_device

device = resolve_device("auto")           # raises if no accelerator
device = resolve_device("auto", allow_cpu=True)  # falls back to CPU with warning

Listing and Clearing Models#

List cached checkpoints#

python -m phenotypic.detect.nn list

This prints a table showing all cached SAM2 and micro-sam checkpoints with their file sizes and paths.

Clear cached checkpoints#

# Clear all cached checkpoints (prompts for confirmation)
python -m phenotypic.detect.nn clear

# Clear only SAM2 checkpoints
python -m phenotypic.detect.nn clear --model-type sam2

# Clear only micro-sam checkpoints
python -m phenotypic.detect.nn clear --model-type microsam

Troubleshooting#

`ImportError: Sam2Detector requires the sam2 package`#

PyTorch and the model packages are not installed. Install the torch extra:

uv add "phenotypic[torch]"

(Linux/macOS only — sam2 is not packaged for Windows.)

`ImportError: MicroSamDetector requires the micro_sam package`#

micro_sam is conda-only and must be installed separately. See the Deep Learning Detectors tutorial section for the conda-forge / pixi recipe.

`RuntimeError: No accelerator available`#

No GPU was detected. Options:

Ensure your GPU drivers and CUDA toolkit are installed correctly.
On macOS with Apple Silicon, ensure PyTorch >= 2.0 with MPS support.
Pass device="cpu" to force CPU inference (very slow):

Sam2Detector(device="cpu")

`RuntimeError: device='cuda' requested but CUDA is not available`#

CUDA was explicitly requested but is not available. Check:

nvidia-smi shows your GPU.
PyTorch was installed with CUDA support (torch.cuda.is_available() returns True).
On SLURM, the job was submitted to a GPU partition.

Out of memory (OOM) errors#

SAM models require significant GPU memory. To reduce VRAM usage:

Use a smaller model: Sam2Detector(model_size="tiny") instead of "large".
Use MicroSamDetector(model_type="vit_t_lm") for the smallest micro-sam model.
Reduce points_per_side (e.g., 16 instead of 32) to generate fewer candidate masks.
Process smaller images or downscale before detection.

Checkpoint not found on SLURM compute nodes#

Compute nodes often lack internet access. Pre-download checkpoints on the login node:

python -m phenotypic.detect.nn download --model-type sam2 --model-size tiny
python -m phenotypic.detect.nn download --model-type microsam --model-name vit_b_lm
python -m phenotypic.detect.nn list  # verify

Ensure TORCH_HOME and MICROSAM_CACHEDIR (if customised) point to a shared filesystem accessible from compute nodes.

`Illegal instruction (core dumped)` on a SLURM compute node#

A stage worker exits with code 132 and the SLURM .err log shows Illegal instruction (core dumped). The node’s CPU is too old for the installed numpy/scipy/torch wheels (a pre-AVX node on a heterogeneous partition). This affects any PhenoTypic SLURM run, not just the staged GPU engine. Pin jobs to modern nodes — use a homogeneous modern partition, or add a SLURM feature constraint that excludes the old CPUs, e.g.:

python -m phenotypic --pipeline p.json --input /plates/ -o /out/ \
    --slurm slurm_partition=<modern-partition> \
    --gpu-slurm slurm_partition=<gpu-partition>

(Stage 2’s GPU work runs on GPU nodes, which are typically consistent; the CPU Stages 1 & 3 are the ones exposed to a heterogeneous CPU partition.)

GPU-Accelerated Colony Detection#

Installation#

Per-model license posture#

Downloading Model Checkpoints#

Using Sam2Detector#

Parameter tuning for colony detection#

Using MicroSamDetector#

Model selection#

Using Sam3Detector#

Using DinoSam2Detector#

Semantic few-shot detectors (Insid3Detector, FssDinoDetector)#

Insid3Detector — one-shot in-context (DINOv3-native, gated)#

FssDinoDetector — few-shot (DINOv2 default, ungated)#

Pipeline Integration#

JSON serialization#

Local Staged GPU Detection (CLI)#

SLURM Deployment#

SLURM Staged GPU Detection#

Device Selection#

Automatic detection (default)#

Explicit device#

resolve_device() utility#

Listing and Clearing Models#

List cached checkpoints#

Clear cached checkpoints#

Troubleshooting#

ImportError: Sam2Detector requires the sam2 package#

ImportError: MicroSamDetector requires the micro_sam package#

RuntimeError: No accelerator available#

RuntimeError: device='cuda' requested but CUDA is not available#

Out of memory (OOM) errors#

Checkpoint not found on SLURM compute nodes#

Illegal instruction (core dumped) on a SLURM compute node#

This Page

Semantic few-shot detectors (`Insid3Detector`, `FssDinoDetector`)#

`Insid3Detector` — one-shot in-context (DINOv3-native, gated)#

`FssDinoDetector` — few-shot (DINOv2 default, ungated)#

`resolve_device()` utility#

`ImportError: Sam2Detector requires the sam2 package`#

`ImportError: MicroSamDetector requires the micro_sam package`#

`RuntimeError: No accelerator available`#

`RuntimeError: device='cuda' requested but CUDA is not available`#

`Illegal instruction (core dumped)` on a SLURM compute node#