Writing a GPU detector#
This page is the reference for authoring a GpuDetector — the hooks you
implement, the capability fields you declare, and the rules that decide where in
a pipeline your detector may be placed.
Read it before subclassing GpuDetector, and again before adding an
OperationField to any operation that will contain one.
Why a GPU detector is not just an ObjectDetector
A GpuDetector in a CLI run does not execute per image. It triggers the
staged engine: CPU preprocess → resident-model GPU inference → CPU measure,
so a GPU node is never occupied by CPU work. Your detector is driven by the same
hooks in both the notebook path and the staged engine, but the staged engine
calls them from three different processes, on three different machines, at three
different times. Everything below follows from that.
The hook surface#
Subclass GpuDetector and implement two methods. Do not override
_operate — it is concrete, and the staged engine bypasses it entirely.
from pydantic import PrivateAttr
from phenotypic.abc_ import GpuDetector
from phenotypic.sdk_.typing_ import GpuInputLayer
class MyDetector(GpuDetector):
# Keep the alias, don't widen to `str` -- the Literal is what rejects a
# typo'd layer name at construction instead of at first inference.
input_layer: GpuInputLayer = "gray"
model_size: str = "small"
_model: object = PrivateAttr(default=None)
def _ensure_model_loaded(self) -> None:
"""Build the model on first use. Idempotent."""
if self._model is not None:
return
import torch # lazy: keep it out of import time
self._model = ...
def _infer_one(self, sample):
"""One preprocessed (H, W, 3) uint8 sample -> one result.
Return a uint16 labeled objmap for output_kind="instance", or a bool
mask for "semantic".
"""
return ...
The base class supplies the rest of the chain:
Hook |
Supplied behaviour |
Override when |
|---|---|---|
|
scales to |
rarely — your model wants a different tensor layout |
|
returns the list unchanged |
your model takes a true |
|
loops |
you overrode |
|
writes |
essentially never — the staged replay path depends on this exact behaviour |
Defer model construction to _ensure_model_loaded, never __init__. The
CLI constructs your detector to inspect it, serialise it, and split the pipeline
around it, in processes that have no GPU and may not have torch installed. A
model built in __init__ makes to_json() require a GPU.
And make it idempotent, because it is called more than once and from more than
one place. The base _infer_batch calls it before every batch, and the staged
engine also calls it directly, once per worker, before streaming a shard
(_cli_staged_strategy.py, _cli_staged_slurm_worker.py) — that up-front call
is what “resident model” means. Several shipped detectors call it from their own
helpers too. The if self._model is not None: return guard above is the whole
contract; without it the model is rebuilt per image on the GPU node.
If you override _infer_batch, the _ensure_model_loaded() call is yours to keep
It is not an implementation detail of the base method. In the notebook path
(op.apply(image) → _operate → _collate → _infer_batch) there is no
other caller, so an override that drops it runs against an unbuilt model.
The staged CLI engine will not tell you: it loads the model up front, before
streaming a shard, so the same detector works there. A bug that passes every
batch run and fails the first interactive apply() is close to the worst shape
available — the CLI is where the large runs happen, and the notebook is where
detectors get developed.
If you would rather not carry the obligation, override _infer_one and leave
_infer_batch alone; the base loops it for you.
Capability fields#
These are pydantic fields, so they serialise and round-trip. Declare them on the class; the engine reads them to decide how to drive you.
Field |
Meaning |
|---|---|
|
|
|
|
|
whether |
|
|
|
zero the border-plurality label before relabeling |
|
relabel by connected components after that |
|
|
drop_frame_background matters more than it looks. A class-agnostic segmenter
can emit the plate background as a positive-labelled mask framing the image;
left in place it survives into measurement and bridges every colony it touches,
collapsing all instances into one blob.
Determinism is a hard requirement#
The staged engine runs your detector’s inputs more than once. When your detector is nested, the operations ahead of it inside its own branch are applied twice: once by Stage 2, to a provenance-detached copy, purely to build the model input; and again by Stage 3, inside the enclosing operation, where they are the ones actually recorded in the journal. A retried Stage 3 replays the recorded result rather than re-inferring.
Seed anything stochastic — in your detector and in any operation you put ahead of it in the same branch. A non-deterministic prefix means Stage 2 inferred from an image Stage 3 never reconstructs, and nothing fails to say so.
Do not assume you see one image#
Your detector is invoked once per image today, but a GPU round is a
full-dataset sweep with the model resident, and the engine is free to
accumulate samples and issue one batched forward. Write _infer_one so it
depends only on its argument — no state carried between calls, no counters, no
“first image” special-casing.
Batching headroom
The engine currently calls _collate([sample]) with a one-element list, so
supports_batching buys nothing yet. That is a gap, not a decision. If your
model benefits from batching, implement _collate and _infer_batch properly
anyway — the cost is small and the sweep structure exists so this can be turned
on without touching detectors. An _infer_batch override must still call
_ensure_model_loaded() itself; see the warning under The hook surface.
Nesting: the container contract#
A GpuDetector may sit inside another operation, at any depth — most
usefully inside a CompositeDetector, where its mask is merged with a CPU
detector’s. The staged engine supports this by addressing the detector with a
tree path and substituting a replay stub at that path in Stage 3.
That path — ("CompositeDetector", "ops[0]"), say — is a single value used
three ways: it is the detector’s recorded pipeline_step_path, the address
Stage 3 substitutes at, and (sanitised, plus 8 hex of the exact path) the
<slot> directory its Stage-2 signal is written under, e.g.
.phenotypic/progress/stage2_raw/<dataset>/CompositeDetector__ops-0__dddf3676/<stem>.npy.
One addressing scheme, not three.
To split the pipeline it must answer one question about every container on the path:
What image does this container hand its children?
It cannot infer the answer — containers drive children in imperative _operate
bodies, and some pass a local intermediate that has no name in the operation
graph. So the engine keeps a small, closed table, and refuses anything not in
it.
The table is populated lazily on first use (_populate_child_contract), because
the module is imported by CLI argument validation that may never ask a GPU
question and phenotypic.detect/phenotypic.enhance are heavy. If you ever add
an entry, add it inside the existing single update() rather than assigning
key by key: the guard is if _CHILD_CONTRACT: return, so a half-built table
lets a second thread return early and refuse a placement the design permits —
and in the GUI, which runs threaded, that shows up as a refusal alert and a
disabled Run button on a pipeline that is perfectly valid.
Introduced with nested GPU staging
See docs/superpowers/specs/2026-09-15-nested-gpu-staging/design.md §4.3.
Only composition primitives may carry one#
Three classes, and nothing else:
ImagePipeline—"sequence"Each child receives the previous child’s output. Handled directly in
_child_contract, not by a table entry.CompositeDetector,CompositeEnhance—"parallel"Every child receives the container’s own input. Branches are parallel; order does not affect what any of them sees. These are the only two entries in
_CHILD_CONTRACT(phenotypic._cli._cli_validation).
A GpuDetector anywhere else is refused. That includes domain detectors
whose current code would classify cleanly — see below.
Entries match by isinstance, not by exact type. A subclass a user actually
holds — class Plotted(CompositeDetector, PlotImage) — is accepted; an
exact-type lookup refused every such subclass, which was the same defect the
table avoided for ImagePipeline. The safety argument is preserved by a second
rule: a subclass that overrides _operate is refused, because the
child-input contract was verified against the base’s _operate and a
replacement has been verified against nothing.
The contract answer is consumed by _branch_prefix in _cli_pipeline_split.py,
which dispatches on _child_contract and never on isinstance. A "sequence"
container contributes the ops preceding the step taken from it to
StagePlan.stage2_prefix; a "parallel" one contributes nothing, because its
children are branches and none runs “before” another; and the root pipeline
contributes nothing, because Stage 1 already applied and stored its ops.
This is a contract, not an observation#
A CompositeDetector is never sequential. Its branches each receive the
image the composite itself was given, and are combined afterwards by union,
intersection or overlap. That is what makes it a composite rather than a
pipeline — PhenoTypic already has ImagePipeline for the sequential case, and a
composite whose branches chained would be one. CompositeEnhance carries the
same contract for the same reason.
So the entry in the engine’s table is not a cached observation about today’s
_operate; it restates the type’s contract. If you are changing a composite
so its branches no longer see the same input, you are not editing a composite —
stop and use a pipeline.
The contract is tested, not just asserted#
Each of the two "parallel" primitives carries a behavioural probe in
tests/unit/detect/test_container_child_contracts.py, which puts two recording
operations in the container’s children and asserts what each received:
def test_composite_detector_branches_each_receive_the_composites_own_input():
image = load_synth_yeast_plate()
baseline = int(image.objmap[:].max())
assert baseline != _PROBE_OBJECT_COUNT, "fixture collision"
CompositeDetector(
ops=[_DetectProbe(tag="a"), _DetectProbe(tag="b")], mode="union"
).apply(image)
# Sequential branches would have "b" observing "a"'s objmap.
assert _SEEN == [("a", baseline), ("b", baseline)]
Assert the property, not a literal. load_synth_yeast_plate() arrives with
its 8×12 ground-truth objmap already populated, so a branch observes 96 on
entry, not 0. A test hard-coding 0 would be testing the fixture rather than
the contract, and a spike on this change was derailed by exactly that number
once already. The probe writes one deliberate square instead of thresholding,
because a threshold on the synthetic plate also yields ~96 objects and so would
not discriminate chaining from parallelism — hence the explicit
fixture-collision guard.
A declaration can lie and still pass every test that reads the table. A probe
cannot: if someone makes the branches chain, this fails immediately and points
at the contract. A companion test in
tests/unit/cli/test_gpu_detection_tree_wide.py asserts coverage of the table
itself, so a third "parallel" entry cannot arrive without its own probe.
Why domain detectors are excluded, even when they would classify#
FilamentousFungiDetector passes inoculum_detector the container’s own image
(_filamentous_fungi_detector.py:398), so it reads as "parallel". It is
still refused.
The reason is that a composition primitive’s child-input semantics is part of
what the class is, while a domain detector’s is incidental to what its
algorithm currently does. FilamentousFungiDetector._operate also runs an
inline ContrastStretching() (:418), a destructive _subtract_background,
and a del enhanced_work. Nothing about being a fungus detector constrains it
to keep handing its child the raw image, and if it stopped, the table would be
wrong with nothing to say so.
TwoKFilamentousDetector shows how tangled this gets inside one algorithm:
center_detector gets the original image
(_two_k_filamentous_detector.py:166), background_subtractor a derived
enhanced.copy() (:171), and branch_base mutates enhanced in place
(:204). Three fields, three inputs, none of it visible from outside.
So the rule is by kind of class, not by whether today’s code happens to be classifiable. A new detector needs no entry and no decision — it is refused by default, which is the right answer for it.
Both classes do route their children through apply_child, so their branches
record distinct pipeline_step_paths. That is a provenance-addressing fix and
not an admission to the staging table; do not read one from the other.
What you get instead of silence#
Every refusal below raises UnstageableGpuDetectorError, a ValueError
subclass, from find_gpu_detectors. The placement refusals are
unconditional, so they fire from pipeline_requires_gpu — before the run is
routed, rather than after it has already been sent somewhere. (The
more-than-one-detector refusal is the exception: it is gated on strict=True,
which only split_pipeline_at_gpu passes, so the GUI’s “is this a GPU
pipeline?” probe reports True for a two-detector pipeline instead of raising.)
Nested in something that is not a composition primitive:
a GpuDetector cannot be nested inside TwoKFilamentousDetector: only composition
primitives (ImagePipeline, CompositeDetector, CompositeEnhance) may carry one.
Lift the detector into a CompositeDetector branch, or into the top-level
pipeline.
Nested in a subclass of a listed composite that replaces _operate (here a
hypothetical ChainingComposite(CompositeDetector)):
a GpuDetector cannot be nested inside ChainingComposite: it subclasses
CompositeDetector but overrides _operate, so the 'parallel' child-input contract
verified for CompositeDetector does not carry over to it. Lift the detector into
a plain CompositeDetector branch, or into the top-level pipeline.
In a meas, post, filters or model slot of any pipeline — the root, or
(here) a nested pipeline keyed inner — which runs after the op chain:
GpuDetector at inner/meas:MeasureSymZones/center_detector cannot be staged: it
sits in the 'meas' slot of the pipeline at inner, which runs after the op chain
-- Stage 3 runs it on a CPU node, after GPU inference has finished. Move the
detector into that pipeline's ops.
And more than one detector anywhere in the tree — a deferred feature, not a limit of the design — naming every offending path so you know which branches to split:
staged execution does not support more than one GpuDetector per pipeline; found
2 at: CompositeDetector/ops[0], CompositeDetector/ops[1]
Each message names the way forward, because there almost always is one: run the
GPU detector as a CompositeDetector branch and feed its mask to the domain
detector, rather than nesting it inside.
A wrong answer here would mean Stage 2 reads a layer that was never prepared, and the run completes with different numbers. Refusing is the cheap outcome.
Pipeline slots, at any depth#
The walker descends every pipeline’s meas, post, filters and model
slots as well as its ops — the root pipeline’s and every nested one’s — and
names a slot entry with the slot as a colon namespace: meas:<key>,
post:<key>, filters:<key>, and model:<ClassName> for the single model.
A bare meas segment would collide with an ops key a user is free to
choose. The flip side is that an ops key may itself contain a colon, so
never classify a segment by parsing the string —
_operation_tree.pipeline_slot_of decides against the live pipeline.
Those slots run after the op chain. In the staged engine that is Stage 3, on a CPU node, after Stage 2’s GPU inference has finished; and a measurement may hand its nested detector a derived input, possibly once per object, that Stage 2 has no way to reproduce. So any path through a slot is refused.
For every shipped slot type the ancestor check above would refuse the shape
anyway — MeasureFeatures, PostMeasurement, SetAnalyzer and ModelFitter
are not composition primitives — but its message would blame the slot entry’s
class rather than the slot, which is why the slot check runs first. For one
shape the slot check is the only guard: a class that is both a slot type
and a composition primitive, such as
class MeasuringComposite(CompositeDetector, MeasureFeatures). It inherits
CompositeDetector._operate, so the ancestor check admits it, and without the
slot refusal its detector would be staged and inferred on the stored image —
the wrong input, with no error. If you write such a class, that refusal is what
keeps it honest; test_a_slot_entry_that_is_also_a_composite_is_refused pins
it.
How a refusal reaches the user#
The CLI checks placement right after it has parsed its options, before it
loads a manifest, clears the output directory for --overwrite, or exits for
--dry-run, and prints the message as a one-line usage error. A refused
pipeline therefore never touches --output, and --dry-run refuses exactly
what a real run would.
The GUI run console asks the same question when a pipeline is selected. A
refused pipeline shows the message in a red alert, in Local and SLURM mode
alike, and Run is disabled; Validate and Run are also refused server-side
before any run is registered. If you ever edit the probe
(_gui/run_console/_callbacks.py:_staged_gpu_capability), keep its
except UnstageableGpuDetectorError clause above the generic
except (OSError, ValueError, TypeError): the refusal is a ValueError, so
the other order silently treats it as “not a GPU pipeline”.
What Stage 3 actually runs: ReplayDetector#
Stage 3 does not call your detector. It builds a ReplayDetector
(_cli_replay_detector.py) holding your instance plus Stage 2’s recorded array,
substitutes it at gpu_path, and applies the resulting pipeline. Two
consequences for you:
The write still goes through your
_write_object_output. The stub delegates to it, sodrop_frame_backgroundandsplit_disconnected_labelsapply exactly as in a single pass. This is why overriding that hook is “essentially never” in the table above — a stub that assignedobjmapdirectly would skip both.The journal records your detector, not the stub.
ReplayDetectoroverrides four provenance hooks (operation_name,operation_class,parameters, and a duration offset carrying Stage 2’s inference time), so a staged run and a single-pass run of the same pipeline produce the same journal. Theparametersoverride is not only a parity concern: the stub holds the recorded array in anNdArrayField, and the defaultmodel_dumpwould serialise the whole objmap into the journal.
--mode process --layer objmap uses the same substitution, so a process export
runs the post-detector operation chain rather than dumping the raw array.
Testing your detector#
Use the shared CPU fake rather than writing another one:
tests/_fakes/fake_gpu_detector.py—FakeGpuDetector, a threshold detector with the full hook surface and notorch.For a test that round-trips a pipeline through JSON, register it first.
ImagePipeline.from_jsonresolves classes by bare name against thephenotypicnamespace, so a module-local fake raisesAttributeError. The in-process pattern is the autouse fixture attests/unit/cli/test_staged_routing.py:21;tests/_fakes/register_fake_gpu.pyexists for the live SLURM dispatch test, where worker processes cannot reach a fixture.
Gated weights and custom namespaces#
PHENOTYPIC_ACCEPT_MODEL_LICENSEplusrequire_license_acceptance(detect/nn/_checkpoint_manager.py) gate downloads of licence-restricted checkpoints.PHENOTYPIC_PRELOAD_MODULESnames self-registering modules: modules whose import attaches the class to thephenotypicnamespace (phenotypic.MyDetector = MyDetector). A module that only defines the class is not enough, because pipeline JSON records bare class names. Class resolution imports the listed modules in every process that deserializes a pipeline (the CLI, SLURM workers, local parallel workers, the finalizer), so a detector defined outside thephenotypicnamespace deserializes everywhere.A runtime path must never prompt: pass
interactive=Falsetorequire_license_acceptanceand to any download helper that can prompt.
Declaring what your detector needs#
The CLI’s run preflight reads preflight_requirements() from every operation
the run will execute, so a missing package, gated license, or uncached weight
is reported before any image is processed. For a fixed requirement, set the
class variables _requires_modules (import names, checked with find_spec and
never imported) and _requires_extra (the pyproject extra that provides
them). When a requirement depends on a field, override the method, call
super(), and extend the result with dataclasses.replace:
import dataclasses
from phenotypic.abc_ import OperationRequirements, WeightRequirement
def preflight_requirements(self) -> OperationRequirements:
requirements = super().preflight_requirements()
return dataclasses.replace(
requirements,
modules=("my_model_pkg", "torch"),
extra="torch",
weights=(WeightRequirement(model="my-model:base", license_key=None,
is_cached=lambda: my_cache_probe()),),
)
is_cached must answer from the file system alone: no torch import and no
network access, because the preflight runs on the submitting node. Return
None when the answer cannot be known. GpuDetector’s own override adds the
RGB requirement for input_layer="rgb"; keep it by calling super().