OME-Zarr store layout#

Everything about the on-disk shape of a per-image store lives in phenotypic.sdk_.ngff_: the directory layout, the pyramid geometry, the chunk/shard/codec policy, the attributes.phenotypic contract, the write-only OME projection, and the rename-promote commit primitive.

Nothing in this module reads or writes an Image. Keeping the geometry free of the image model is what lets the committed logic-validation script re-derive every numeric claim from numpy alone.

For the user-facing view of the same thing — what a store looks like, how to open one in napari or QuPath, and the CLI flags that govern it — see Store Results in OME-Zarr.

Layout constants#

phenotypic.sdk_.ngff_.STORE_SUFFIX: Final[str] = '.ome.zarr'#

Directory suffix for one per-image store.

phenotypic.sdk_.ngff_.STORE_ROOT_JSON: Final[str] = 'zarr.json'#

Zarr v3’s root metadata document, at the top of every store. Written last by the promote protocol, which is what lets a reader treat its presence as “this store is complete” and lets a completion marker fingerprint the store by this file alone.

phenotypic.sdk_.ngff_.STORE_SCHEMA_VERSION: Final[int] = 3#

Version of the PhenoTypic group and array layout. Distinct from metadata_schema_version, which versions the header namespace.

phenotypic.sdk_.ngff_.NGFF_VERSION: Final[str] = '0.5'#

NGFF specification version written into every ome block.

phenotypic.sdk_.ngff_.BIOFORMATS2RAW_LAYOUT: Final[int] = 3#

bioformats2raw.layout marker on the root group (named-series collection).

phenotypic.sdk_.ngff_.SERIES_ORDER: Final[tuple[str, str, str]] = ('rgb', 'gray', 'detect_mat')#

Canonical series order. rgb is omitted from a store when empty; the remaining names keep this relative order.

phenotypic.sdk_.ngff_.OBJMAP_LABEL: Final[str] = 'objmap'#

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

phenotypic.sdk_.ngff_.PYRAMID_STOP_PX: Final[int] = 512#

Halve pyramid levels until max(H, W) <= PYRAMID_STOP_PX.

Pyramid geometry#

Pyramid depth is a pure function of the level-0 shape — there is no user lever, which is what makes mixed-geometry drift within one output tree unreachable rather than merely unlikely.

phenotypic.sdk_.ngff_.pyramid_level_count(height: int, width: int, *, stop_px: int = 512) → int[source]#

Number of pyramid levels when halving until max(H, W) <= stop_px.

ceil, not floor: a floor-based formula terminates one level early and leaves a 4000x3000 plate’s smallest level at 1000x750.

Parameters:

  • height (int) – Level-0 height in pixels.

  • width (int) – Level-0 width in pixels.

  • stop_px (int) – Longest-edge threshold at which halving stops.

Returns:

A level count of at least 1.

Return type:

int

phenotypic.sdk_.ngff_.pyramid_level_shapes(shape: tuple[int, ...], levels: int) → tuple[tuple[int, ...], ...][source]#

Explicit shape per pyramid level, ceil-halving the two spatial axes.

A leading channel axis (3-D input) is carried through unchanged.

Parameters:

  • shape (tuple[int, ...]) – Level-0 shape, (y, x) or (c, y, x).

  • levels (int) – Number of levels to emit, including level 0.

Returns:

A tuple of levels shapes, starting with shape.

Return type:

tuple[tuple[int, …], …]

phenotypic.sdk_.ngff_.level_scale_vector(level0: tuple[int, ...], level_index: int) → list[float][source]#

Per-axis sampling factor after repeated 2x spatial reductions.

Ceil-halving changes the stored array extent but not the sampling operation: an odd 1025-pixel axis becomes 513 pixels after one 2x reduction, so its sampling factor is 2 rather than the shape ratio 1025 / 513. An axis saturates once it reaches one sample; leading channel axes are never sampled.

Parameters:

  • level0 (tuple[int, ...]) – Level-0 shape.

  • level_index (int) – Zero-based pyramid level.

Returns:

One float per axis, in axis order. Any leading channel axis is 1.0.

Return type:

list[float]

phenotypic.sdk_.ngff_.downsample_image(array: numpy.ndarray) → numpy.ndarray[source]#

2x block-mean downsample with edge replication, preserving dtype.

Edge replication (rather than zero padding) is what keeps an odd trailing row or column at its own brightness instead of darkening it toward zero. The spatial axes are the last two; any leading channel axis is preserved.

An integer result is rounded with np.rint – banker’s rounding, so an exact .5 mean goes to the nearest EVEN value, not always upward. Rounding rather than truncating is what keeps the pyramid unbiased: a plain astype drops the fraction at every level, so a uniform uint8 plate drifts 127.52 -> 126.02 over four levels and every thumbnail darkens.

Parameters:

array (numpy.ndarray) – 2-D (y, x) or 3-D (c, y, x) array.

Returns:

An array whose spatial extents are (n + 1) // 2.

Return type:

numpy.ndarray

phenotypic.sdk_.ngff_.downsample_label(array: numpy.ndarray) → numpy.ndarray[source]#

2x nearest-neighbour downsample (top-left of each 2x2 block).

A label map must never be mean-downsampled: averaging fabricates label values present at no level-0 pixel. Verified by claim C5 of the committed logic-validation script.

Parameters:

array (numpy.ndarray) – 2-D (y, x) integer label array.

Returns:

An array whose extents are (n + 1) // 2, with dtype preserved and no label value absent from array.

Return type:

numpy.ndarray

phenotypic.sdk_.ngff_.build_pyramid(array: numpy.ndarray, levels: int, *, kind: Literal['image', 'label']) → list[TypeAliasForwardRef('numpy.ndarray')][source]#

Materialise every pyramid level for one array.

Parameters:

  • array (numpy.ndarray) – Level-0 array.

  • levels (int) – Level count, including level 0.

  • kind (Literal['image', 'label']) – "image" downsamples by local mean; "label" by nearest-neighbour.

Returns:

A list of levels arrays, starting with array.

Return type:

list[TypeAliasForwardRef(‘numpy.ndarray’)]

Reading a store#

phenotypic.sdk_.ngff_.require_readable_store(store_path: Path) → dict[source]#

Read attributes.phenotypic, refusing a store this build cannot decode.

The gate is by value, not presence: a future store opened under today’s semantics is exactly what the 2026-08-19 ruling exists to prevent, and presence alone would let it through.

Every path that decodes store content goes through here, so the two halves of that guarantee – the check and its wording – cannot drift apart. read_phenotypic_attributes() stays ungated for the callers that must classify a store rather than read it: valid_staged_store answers False on a mismatch instead of raising, and that is what routes a stale store back to Stage 1 rather than aborting a run.

Parameters:

store_path (Path) – Path to a *.ome.zarr directory.

Returns:

The phenotypic block.

Raises:

  • FileNotFoundError – If the root zarr.json does not exist.

  • KeyError – If the root exists but carries no phenotypic block.

  • ValueError – If store_schema_version is not this build’s.

Return type:

dict

phenotypic.sdk_.ngff_.read_phenotypic_attributes(store_path: Path) → dict[source]#

Read the attributes.phenotypic block from a store root.

Parameters:

store_path (Path) – Path to a *.ome.zarr directory.

Returns:

The phenotypic block.

Raises:

  • FileNotFoundError – If the root zarr.json does not exist.

  • KeyError – If the root exists but carries no phenotypic block.

Return type:

dict

phenotypic.sdk_.ngff_.store_level0_shape(store_path: Path, member_path: str) → tuple[int, ...] | None[source]#

Return the level-0 shape of one member array, or None if absent.

Parameters:

  • store_path (Path) – Store root.

  • member_path (str) – Store-relative group path, e.g. "gray" or "rgb/labels/objmap".

Returns:

The level-0 array shape, or None when the level-0 array is missing.

Return type:

tuple[int, …] | None

phenotypic.sdk_.ngff_.primary_series(series_names: Sequence[str]) → str[source]#

Return the series a generic viewer should show, and labels attach to.

Parameters:

series_names (Sequence[str]) – Series present in the store.

Returns:

"rgb" when present, otherwise "gray".

Raises:

ValueError – If neither rgb nor gray is present.

Return type:

str

phenotypic.sdk_.ngff_.objmap_path(primary: str) → str[source]#

Return the store-relative path of the objmap label image.

Readers MUST take this from phenotypic.labels.objmap rather than hard-coding rgb/labels/objmap: when rgb is empty the primary series is gray and the label lives under it instead.

Parameters:

primary (str)

Return type:

str

phenotypic.sdk_.ngff_.valid_staged_store(path: Path) → bool[source]#

Return whether path holds the image layers Stage 2 requires.

Mirrors valid_staged_hdf case for case:

  • the root zarr.json parses and carries store_schema_version;

  • every entry in phenotypic.series and phenotypic.labels opens as a Zarr array group – objmap included, which Stage 1’s zeros write guarantees;

  • processed level-0 (y, x) extents agree and every extent is non-zero; the full decoded original may differ after geometry-changing pre-ops. A zero-size Zarr array is legal and must not pass.

The exception set is the HDF version’s (OSError, TypeError, ValueError) plus ``KeyError`` – which the attribute lookups need and the HDF version did not – plus ``AttributeError``. The root zarr.json is arbitrary JSON written by anyone, so phenotypic, phenotypic.series, and phenotypic.labels can each come back as a list rather than a mapping (another tool’s store, or a future schema); the .get/.values() calls below then raise AttributeError, which is a rejected store, not a crash in resume classification.

It does not need zarr.errors.BaseZarrError. The spec’s §3.6 argues the opposite (“none of zarr’s error types are ValueError subclasses”); that is inverted. BaseZarrError inherits directly from ``ValueError`` (https://zarr.readthedocs.io/en/stable/api/zarr/errors/), as do MetadataValidationError and every other zarr error except the four IndexError ones, none of which this function can raise. json.JSONDecodeError is likewise a ValueError and FileNotFoundError an OSError, so both are already covered. Keeping the shorter tuple also avoids importing zarr.errors in a function the resume planner calls once per image.

Parameters:

path (Path) – Candidate *.ome.zarr directory.

Returns:

True only for a store Stage 2 can consume.

Return type:

bool

The commit protocol#

A store is never written in place. Each publisher builds a .part sibling, writes arrays and chunks first and the root zarr.json last, then promotes the directory by rename. An interrupted write therefore leaves no valid root and reads as absent rather than partial.

Warning

Nothing may write into a promoted store. Both the per-image completion marker and the results viewer’s staleness scan identify a store by its root zarr.json alone, which is sound only because the promote writes that root last and replaces the directory wholesale. A code path that opens a promoted store for writing makes both report stale data as fresh, and neither can detect it. The guard is tests/unit/sdk_/test_ngff_promote.py::test_nothing_writes_into_a_promoted_store.

phenotypic.sdk_.ngff_.new_part_path(final: Path) → Path[source]#

Return a fresh, uuid-suffixed .part sibling of final.

The uuid – matching the attempt_id = uuid4().hex convention already used in _cli_staged_strategy.py (lines 148, 192, 225, 359) – is what keeps two concurrent writers from interleaving chunks into one directory. It is NOT what makes the promote itself benign; that is the retry loop in promote_store(). An un-suffixed .part would let two concurrent SLURM tasks interleave chunks into one directory and produce a store that validates. A PID is not enough: PIDs are reused.

Parameters:

final (Path)

Return type:

Path

phenotypic.sdk_.ngff_.promote_store(part: Path, final: Path, *, fsync: bool, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → Path[source]#

Atomically promote a fully written .part directory to final.

The caller is responsible for the write order inside part: all arrays and chunks first, then OME/zarr.json, then the root zarr.json last. An interrupted store therefore has no valid root and reads as absent. This function does not write the root zarr.json itself.

The move-aside is mandatory, not an optimization: os.replace onto a non-empty directory raises OSError (ENOTEMPTY) on POSIX, and on Windows MoveFileEx’s MOVEFILE_REPLACE_EXISTING cannot name a directory at all.

The whole exists -> move-aside -> replace sequence sits inside one retry loop and re-evaluates existence on every attempt. That is what makes duplicate execution benign: a uuid .part prevents two writers interleaving chunks, but it does nothing for the promote itself, where a check-then-act done once lets writer B skip the move-aside because A had not yet renamed, then hit ENOTEMPTY on a now-non-empty target.

On failure after a successful move-aside, that attempt’s trash is reconciled before retrying or raising. The previous store is rolled back when final is absent; if a concurrent writer has already published a new final, that winner remains authoritative and only the attempt’s superseded trash is removed. Every retry uses a fresh UUID trash path, so no attempt can collide with its predecessor’s move-aside directory.

Known weakening versus the single-file rename: the two renames are still not one atomic step, so a crash between them (as opposed to a raised error) leaves the image absent plus an orphaned .trash. Both are recoverable – absence reclassifies to the rebuilding stage, and sweep_orphan_parts() clears the leftovers.

Parameters:
Returns:

final.

Return type:

Path

phenotypic.sdk_.ngff_.sweep_orphan_parts(results_root: Path, *, min_age_seconds: float = 21600) → int[source]#

Remove stale orphaned .part / .trash directories.

A uuid identifies the attempt, not whether its process is alive. The staged SLURM engine explicitly assumes stale workers can still be running – that is what assert_active_epoch exists for – and under an array the tasks share one output root and start at different times. A sweep with no liveness signal would rmtree the .part directories its siblings are actively filling, which is the same defect a PID-based sweep has.

Two guards, both required:

  • age: only directories whose mtime is older than min_age_seconds are removed;

  • placement: the caller must run this from the controller before any worker is submitted, not from each worker’s start-up (see Phase 3).

The scan is bounded to results/<dataset>/zarr/ rather than recursive: rglob would descend into every store, which is the same ~400k-stat pathology the spec flags for the GUI’s discovery path.

Parameters:

  • results_root (Path) – The run’s results/ directory.

  • min_age_seconds (float) – Minimum age before a leftover is considered orphaned.

Returns:

Number of directories removed.

Return type:

int

Durability#

fsync before promote is a tri-state: unset auto-detects (on under SLURM, off locally), and --durable-writes / --no-durable-writes overrides it. The resolution happens in exactly one place so the flag and the sentence describing it cannot drift.

phenotypic.sdk_.ngff_.durable_writes_enabled(override: bool | None = None) → bool[source]#

Resolve whether the promote fsyncs before renaming.

write() returns once data is in the page cache. Without fsync the kernel may flush the root zarr.json before the chunk data it describes, so a node crash can leave a store that passes valid_staged_store() – metadata parses, shapes agree – while reading fill_value. That is silent wrong data, not a visible failure, and no amount of metadata validation catches it.

The dominant failure mode does not need it: a SLURM timeout kills the process, and the kernel survives and flushes normally. fsync buys protection only against node loss, power failure, and filesystem crash – which is exactly what a cluster job is exposed to and a laptop run is not.

Parameters:

override (bool | None) – --durable-writes / --no-durable-writes, or None to auto-detect.

Returns:

True when the promote should fsync.

Return type:

bool

Note

This checks SLURM_JOB_ID as well as SLURM_CPUS_PER_TASK. resolve_worker_count (_cli_utils.py:65-72) reads only the latter, so this is deliberately broader – not “exactly as” that helper does, which is what the spec’s §3.7 claims. A job that sets SLURM_JOB_ID without a per-task CPU count still gets durable writes.

phenotypic.sdk_.ngff_.describe_durability(override: bool | None = None) → str[source]#

One-line description of the resolved durability mode, for the start log.

The same command carries different guarantees in different places, which is a genuinely surprising thing to debug. Logging the resolved mode at run start is a required mitigation, not a nicety.

Shares _resolve_durability() with durable_writes_enabled(), so the flag and the sentence describing it cannot drift apart.

Parameters:

override (bool | None)

Return type:

str

phenotypic.sdk_.ngff_.fsync_tree(root: Path) → None[source]#

fsync every regular file under root, then every directory.

Both halves matter. On POSIX a durable file does not imply a durable directory entry, so flushing files plus the root alone would leave the nested gray/0/ and rgb/labels/objmap/0/ dirents unflushed – exactly the silent wrong-data mode §3.7 exists to close. Directories are flushed deepest-first so a parent’s entry is never made durable before the child it points at.

All directory flushes are POSIX-guarded: Windows cannot open a directory handle for flushing and relies on NTFS journaling instead.

Parameters:

root (Path)

Return type:

None