phenotypic.sdk_ #

overlays_dir(dataset: str) → Path[source]

Return path to overlays/<dataset>/.

Parameters:: dataset (str) – Dataset name.
Returns:: Directory path for overlay PNGs of the given dataset.
Return type:: Path

store_path(dataset: str, stem: str) → Path | None[source]

Full-res per-image OME-Zarr store for (dataset, stem), or None.

Parameters:

dataset (str) – Dataset name (subdirectory under results/).
stem (str) – Image stem (filename without extension).

Returns:

Resolved store path if the directory exists, otherwise None. Note the is_dir check: a store is a directory, so the is_file test the removed hdf_path used (it resolved a single per-image HDF file) would always return None here.

Return type:

Path | None

property curation_labels_parquet: Path: Return path to deliverables/qc/curation_labels.parquet (durable labels store).

property custom_categories_json: Path: Return path to deliverables/qc/custom_categories.json (custom-category registry).

deliverables_base: Path

property error_analysis_csv: Path: Return path to error_analysis.csv in the deliverables base.

property error_analysis_html: Path: Return path to error_analysis.html in the deliverables base.

property error_analysis_parquet: Path: Return path to error_analysis.parquet in the deliverables base.

property errors_dir: Path: Return path to the errors/ directory under the deliverables base.

property has_results: bool: Return True when a results/ directory exists under the output root.

property master_parquet: Path: Return path to master_measurements.parquet in the deliverables base.

property mirror_csv: Path: Return path to measurements.csv (post-applied mirror).

property mirror_parquet: Path: Return path to measurements.parquet (post-applied mirror).

output_root: Path | None

property pipeline_config_path: Path: Return path to pipeline.json in the deliverables base.

property plots_dir: Path: Return the resolved plots/ directory inside the bundle.

property qc_dir: Path

Return the QC directory, resolving legacy <output>/qc/ layouts.

Prefers deliverables/qc/ when it exists. Falls back to the legacy root <output>/qc/ when only that exists (pre-relocation runs). Returns the canonical deliverables/qc/ path for fresh writes when neither is present.

property qc_duckdb: Path: Return path to qc/qc.duckdb (the QC analysis database).

property qc_review_state_path: Path: Return path to deliverables/qc/review_state.json (GUI-owned review progress).

property resolved_pipeline_config_path: Path

Return the best existing pipeline config path inside the bundle.

Mirrors resolve_pipeline_config_path()’s precedence but anchored on deliverables_base (so a standalone bundle resolves inside itself without double-joining deliverables/): the canonical typed config when present, else the legacy plain pipeline.json when present, else the canonical path (so writers naturally create typed config files).

property results_dir: Path | None: Return the results/ directory, or None for a standalone bundle.

property verified_parquet: Path: Return path to verified.parquet (GUI-written verified-good archive).

class phenotypic.sdk_.ChunkManifestKey[source]

Bases: object

Keys inside <output>/.phenotypic/progress/chunk_manifest.json.

CHUNKS: Final[str] = 'chunks'

DATASETS: Final[str] = 'datasets'

NAME: Final[str] = 'name'

ROWS: Final[str] = 'rows'

TOTAL_ROWS: Final[str] = 'total_rows'

class phenotypic.sdk_.ChunkStateKey[source]

Bases: object

Keys inside <output>/.phenotypic/progress/chunk_state.json.

CHUNKED_FILES: Final[str] = 'chunked_files'

NEXT_CHUNK_ID: Final[str] = 'next_chunk_id'

class phenotypic.sdk_.DashboardManifestKey[source]

Bases: object

Keys inside <output>/.phenotypic/progress/manifest.json.

The manifest is built by _cli._dashboard._manifest_builder.build_manifest() and consumed by both the dashboard JS and the GUI run-console’s runs registry (_runs_registry.py). Writers and readers must reference these constants rather than spelling the bare string.

COMPLETED: Final[str] = 'completed'

DATASETS: Final[str] = 'datasets'

EVENT_DIAGNOSTICS: Final[str] = 'event_diagnostics'

EXECUTION_MODE: Final[str] = 'execution_mode'

FAILED: Final[str] = 'failed'

FAILURE_CATEGORIES: Final[str] = 'failure_categories'

GUI_RECORD_GENERATION: Final[str] = 'gui_record_generation': Exact GUI generation that published this canonical local manifest. Omitted from non-GUI and scheduler manifests for compatibility.

INPUT_PATH: Final[str] = 'input_path'

IS_COMPLETE: Final[str] = 'is_complete'

LAST_UPDATED: Final[str] = 'last_updated'

PENDING: Final[str] = 'pending'

PROCESSING_GENERATION: Final[str] = 'processing_generation'

SLURM_INFO: Final[str] = 'slurm_info'

STARTED: Final[str] = 'started'

START_TIME: Final[str] = 'start_time'

SUCCESS_RATE: Final[str] = 'success_rate'

TOTAL_IMAGES: Final[str] = 'total_images'

VERSION: Final[str] = 'version': Written, never read – and that is the correct state for it. A format version exists to be readable by something that does not exist yet, so “zero readers” is the expected condition of a healthy one, not evidence of death. P6 Task 7’s deletion ledger listed it as dead on a zero-reader count; it was removed and restored, because dropping it leaves the manifest with no schema discriminator and that cannot be added retroactively to trees written meanwhile. The criterion is right for a function and wrong for a format version.

class phenotypic.sdk_.DashboardManifestSlurmInfoKey[source]

Bases: object

Keys inside the slurm_info sub-dict of the dashboard manifest.

Distinct from JobMetadataKey, even when string values overlap — these describe the manifest contract, not the job-metadata sidecar.

ACTIVE_CHUNKS: Final[str] = 'active_chunks'

CHUNK_JOB_IDS: Final[str] = 'chunk_job_ids'

CHUNK_SCRIPTS: Final[str] = 'chunk_scripts'

COMPLETED_CHUNKS: Final[str] = 'completed_chunks'

PENDING_CHUNKS: Final[str] = 'pending_chunks'

TOTAL_CHUNKS: Final[str] = 'total_chunks'

class phenotypic.sdk_.EnvVar[source]

Bases: object

Environment variable names read or set by the CLI.

SLURM injects these into batch scripts; the CLI reads them to discover its execution context (job id, array task id, …) and to find node-local scratch storage.

SCRATCH: Final[str] = 'SCRATCH'

SLURM_ARRAY_JOB_ID: Final[str] = 'SLURM_ARRAY_JOB_ID'

SLURM_ARRAY_TASK_COUNT: Final[str] = 'SLURM_ARRAY_TASK_COUNT'

SLURM_ARRAY_TASK_ID: Final[str] = 'SLURM_ARRAY_TASK_ID'

SLURM_CPUS_PER_TASK: Final[str] = 'SLURM_CPUS_PER_TASK'

SLURM_JOB_ID: Final[str] = 'SLURM_JOB_ID'

SLURM_MEM_PER_NODE: Final[str] = 'SLURM_MEM_PER_NODE'

class phenotypic.sdk_.FootprintMixin[source]

Bases: object

Provides a mixin for creating morphological footprints for image processing.

The FootprintMixin class contains a static utility method to generate structuring elements (footprints) used in various image processing tasks. This functionality is particularly helpful in the context of analyzing microbial colonies on solid media agar plates. Morphological footprints are used to highlight specific features in images, such as colony edges, shapes, or connectivity, and can assist in segmentation, noise reduction, and feature extraction.

The mixin declares no attributes – it contributes a single static helper. (Do NOT write Attributes: followed by None to say so: napoleon renders that as an attribute named None, which registers a .None cross-reference target. Four such targets made every autodoc’d docstring returning None ambiguous, producing 23,934 of the docs build’s 24,585 warnings.)

class phenotypic.sdk_.GridInferenceMixin[source]

Bases: object

Mixin providing grid inference capabilities from binary masks.

Provides static methods for inferring grid structure from colony patterns using peak detection on row/column projections. Used by detectors and refiners that work with gridded plate images (96-well, 384-well formats, pinned cultures).

All methods are static to support parallelization in pipeline operations.

This is an internal utility for advanced users creating custom grid-based operations. Most users should use RoundPeaksDetector for detection or GridAlignmentRefiner for post-detection refinement directly.

class phenotypic.sdk_.HDF(filepath, name: str, mode: Literal['single', 'set'])[source]

Bases: object

Represents an interface to manage HDF5 files with support for single or set image modes, and ensures safe and compatible file access with retry and error-handling mechanisms.

The class facilitates operations on HDF5 files commonly used for storing phenotypic data in both single image and image set modes. This class includes utilities to handle locking errors and ensure compatibility by initializing proper HDF5 modes while providing safe access methods for writing.

Parameters:

name (str)
mode (Literal['single', 'set'])

filepath

Path to the HDF5 file on the filesystem.

Type:: Path

name

Name associated with the HDF5 resource, often used as an identifier.

Type:: str

mode

Specifies the mode for the HDF5 file, either single image or image set.

Type:: Literal[‘single’, ‘set’]

root_posix

The root path for the HDF5 resource, determined by the mode.

Type:: str

home_posix

The specific root directory of the HDF5 resource in the file, derived based on its mode.

Type:: str

set_data_posix

The subgroup path for the data entity in image set mode, if applicable.

Type:: str, optional

SINGLE_IMAGE_ROOT_POSIX

Base path for single image mode.

Type:: str

IMAGE_SET_ROOT_POSIX

Base path for image set mode.

Type:: str

IMAGE_SET_DATA_POSIX

Subgroup marker for image set data.

Type:: str

EXT

Set of valid file extensions used to recognize HDF5 files.

Type:: set

IMAGE_MEASUREMENT_SUBGROUP_KEY

Key for accessing measurements in an image’s group.

Type:: str

IMAGE_STATUS_SUBGROUP_KEY

Key for accessing statuses in an image’s group.

Type:: str

static get_group(handle: File, posix) → Group[source]

Retrieves or creates a group in an HDF5 file.

This method checks the validity of the provided HDF5 file handle and tries to retrieve the specified group based on the given posix path. If the group does not exist and the file is not opened in read-only mode, the group gets created. If the file is in read-only mode and the group does not exist, an error is raised.

Parameters:

handle (h5py.File) – The HDF5 file handle to operate on.
posix (str) – The posix path of the group to retrieve or create in the HDF5 file.

Returns:

The corresponding h5py group within the HDF5 file.

Return type:

h5py.Group

Raises:

ValueError – If the HDF5 file handle is invalid or no longer valid.
ValueError – If the file handle mode cannot be determined.
KeyError – If the specified group does not exist in read-only mode.

static save_array2hdf5(group, array, name, **kwargs)[source]

Saves a given numpy array to an HDF5 group. If a dataset with the specified name already exists in the group, it checks if the shapes match. If the shapes match, it updates the existing dataset; otherwise, it removes the existing dataset and creates a new one with the specified name. If a dataset with the given name doesn’t exist, it creates a new dataset.

Parameters:

group – h5py.Group The HDF5 group in which the dataset will be saved.
array – numpy.ndarray The data array to be stored in the dataset.
name – str The name of the dataset within the group.
**kwargs – dict Additional keyword arguments to pass when creating a new dataset.

__init__(filepath, name: str, mode: Literal['single', 'set'])[source]

Initializes a class instance to manage HDF5 file structures for single or set image data based on the given filepath, name of the resource, and operational mode.

filepath

Path to the HDF5 file.

Type:: Path

name

Identifier for the resource within the HDF5 file.

Type:: str

mode

Operational mode determining the structure and organization within the HDF5 file. Must be either ‘single’ or ‘set’.

Type:: Literal[‘single’, ‘set’]

root_posix

Posix path representing the root directory within the HDF5 file based on the mode.

Type:: str

home_posix

Posix path representing the home directory for the resource within the HDF5 file based on the mode.

Type:: str

set_data_posix

Posix path for the data subdirectory within the resource home directory. Only initialized in ‘set’ mode.

Type:: Optional[str]

Parameters:

filepath – Path to the target HDF5 file. Must have an HDF5-compatible extension, or a ValueError is raised.
name (str) – Name of the resource to be managed in the file. Used to construct the home directory for the resource within the HDF5 file.
mode (Literal['single', 'set']) – Operational mode. Specifies whether the resource represents a ‘single’ or ‘set’ image data. If the mode is invalid, a ValueError is raised.

Raises:

ValueError – If the filepath does not have an HDF5-compatible extension.
ValueError – If the mode is neither ‘single’ nor ‘set’.

get_data_group(handle)[source]

get_home(handle)[source]

Retrieves a specific group from an HDF file corresponding to single image data.

This method is used to fetch a predefined group from an HDF container, where the group is identified by a constant key related to single image data. The function provides a static interface allowing invocation without requiring an instance of the class.

Parameters:

handle – The HDF file handle from which the group should be retrieved.

Returns:

The group corresponding to single image data, retrieved based on the defined SINGLE_IMAGE_ROOT_POSIX.

Raises:

Appropriate exceptions may be raised by the underlying HDF.get_group() method, –
based on the implementation and provided handle or key. –

get_image_group(handle, image_name)[source]

get_image_measurement_subgroup(handle, image_name)[source]

get_protected_metadata_subgroup(handle: File, image_name: str) → Group[source]

Parameters:

handle (File)
image_name (str)

Return type:

Group

get_public_metadata_subgroup(handle: File, image_name: str) → Group[source]

Parameters:

handle (File)
image_name (str)

Return type:

Group

get_root_group(handle) → Group[source]

Return type:: Group

get_status_subgroup(handle, image_name)[source]

reader() → File[source]

Return type:: File

safe_writer() → File[source]

Returns a writer object that provides safe and controlled write access to an HDF5 file at the specified filepath or creates it if it doesn’t exist. Ensures that the file uses the ‘latest’ version of the HDF5 library for compatibility and performance.

Handles HDF5 file locking conflicts by attempting to clear consistency flags and retrying file opening with exponential backoff.

Returns:: A file writer object with append mode and ‘latest’ library version enabled.
Return type:: h5py.File
Raises:: OSError – If file cannot be opened after all retry attempts.

strict_writer() → File[source]

Provides access to an HDF5 file in read/write mode using the h5py library. This property is used to obtain an h5py.File object configured with the latest library version.

Note

If using SWMR mode, don’t forget to enable SWMR mode:

>>> hdf = HDF(filepath)
>>> with hdf.writer as writer:
...     writer.swmr_mode = True
...     # rest of your code

Returns:: An HDF5 file object opened in ‘r+’ mode, enabling reading and writing.
Return type:: h5py.File
Raises:: OSError – If the file cannot be opened or accessed.

swmr_reader() → File[source]

Return type:: File

swmr_writer() → File[source]

Returns a writer object that provides safe SWMR-compatible write access to an HDF5 file. Creates the file if it doesn’t exist and enables SWMR mode properly.

This method ensures proper SWMR mode initialization by creating the file with the correct settings from the start, avoiding cache conflicts that occur when trying to enable SWMR mode after opening.

Returns:: A file writer object with SWMR mode enabled.
Return type:: h5py.File
Raises:: OSError – If file cannot be opened after all retry attempts.

EXT = {'.h5', '.hdf', '.hdf5', '.he5'}

IMAGE_MEASUREMENT_SUBGROUP_KEY = 'measurements'

IMAGE_SET_DATA_POSIX = 'data'

IMAGE_SET_ROOT_POSIX = '/phenotypic/image_sets/'

IMAGE_STATUS_SUBGROUP_KEY = 'status'

PROTECTED_METADATA_SUBGROUP_KEY = 'protected_metadata'

PUBLIC_METADATA_SUBGROUP_KEY = 'public_metadata'

SINGLE_IMAGE_ROOT_POSIX = '/phenotypic/images/'

class phenotypic.sdk_.ImageState(work_id: str, dataset: str, image_stem: str, stages: Mapping[str, Mapping[str, object]], verdict: Literal['verified', 'unverified', 'failed'], reason: str | None = None)[source]

Bases: object

One image’s stages and whether its declared artifacts still match disk.

stages is the open map from spec §6.1 – stage1/stage2/ stage3/measured today, more later. Nothing here enumerates its keys; a caller asking “did stage 3 run?” reads "stage3" in state.stages, which is what makes a future stage additive rather than a schema break.

Under D-A there is no backfilled stage: per-store metadata is written in the store’s original promote, so there is nothing to record having happened afterwards.

Parameters:

work_id (str)
dataset (str)
image_stem (str)
stages (Mapping[str, Mapping[str, object]])
verdict (Literal['verified', 'unverified', 'failed'])
reason (str | None)

dataset: str

image_stem: str

reason: str | None = None

stages: Mapping[str, Mapping[str, object]]

verdict: Literal['verified', 'unverified', 'failed']: Spec §9 annotates images as “work_id -> stages + VERDICT”. A bool plus an unread reason was not that (SIMP-R1-09).

work_id: str

class phenotypic.sdk_.InputLayerMixin(*, input_layer: Literal['detect_mat', 'rgb'] = 'detect_mat')[source]

Bases: BaseModel

Adds an input_layer field selecting the operation’s source array.

Pointwise intensity curves are non-linear, so applying one to the three RGB channels and then collapsing to a detection matrix gives a different — often better — colony/background separation than collapsing first. This mixin exposes that choice without changing the output contract: the only layer an enhancer ever writes is still detect_mat.

When input_layer="rgb" the 3-D result is collapsed back to 2-D by projecting it through the image’s own detect_mode, so an upstream SetDetectMode(mode="MinRGB") is honoured.

The field is appended to the end of the subclass’s field order. When stacked with NormalizedOutputMixin, list this mixin first; the resulting order is […op params…, norm, input_layer].

Note

Reading rgb discards any enhancement a prior operation wrote to detect_mat — the same behaviour as SetDetectMode. This is documented, not enforced.

Parameters:: input_layer (Literal['detect_mat', 'rgb'])

classmethod __get_pydantic_json_schema__(core_schema: CoreSchema, handler: GetJsonSchemaHandler, /) → JsonSchemaValue

Hook into generating the model’s JSON schema.

Parameters:

core_schema (CoreSchema) – A pydantic-core CoreSchema. You can ignore this argument and call the handler with a new CoreSchema, wrap this CoreSchema ({‘type’: ‘nullable’, ‘schema’: current_schema}), or just call the handler with the original schema.
handler (GetJsonSchemaHandler) – Call into Pydantic’s internal JSON schema generation. This will raise a pydantic.errors.PydanticInvalidForJsonSchema if JSON schema generation fails. Since this gets called by BaseModel.model_json_schema you can override the schema_generator argument to that function to change JSON schema generation globally for a type.

Returns:

A JSON schema, as a Python object.

Return type:

JsonSchemaValue

classmethod __pydantic_init_subclass__(**kwargs: Any) → None[source]

Move input_layer to the end of the subclass’s field order.

Parameters:: kwargs (Any)
Return type:: None

classmethod __pydantic_on_complete__() → None

This is called once the class and its fields are fully initialized and ready to be used.

This typically happens when the class is created (just before [__pydantic_init_subclass__()][pydantic.main.BaseModel.__pydantic_init_subclass__] is called on the superclass), except when forward annotations are used that could not immediately be resolved. In that case, it will be called later, when the model is rebuilt automatically or explicitly using [model_rebuild()][pydantic.main.BaseModel.model_rebuild].

Return type:: None

classmethod construct(_fields_set: set[str] | None = None, **values: Any) → Self

Parameters:

_fields_set (set[str] | None)
values (Any)

Return type:

classmethod from_orm(obj: Any) → Self

Parameters:: obj (Any)
Return type:: Self

classmethod model_construct(_fields_set: set[str] | None = None, **values: Any) → Self

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note: model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.

Parameters:

_fields_set (set[str] | None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.
values (Any) – Trusted or pre-validated data dictionary.

Returns:

A new instance of the Model class with validated data.

Return type:

classmethod model_json_schema(by_alias: bool = True, ref_template: str = '#/$defs/{model}', schema_generator: type[~pydantic.json_schema.GenerateJsonSchema] = <class 'pydantic.json_schema.GenerateJsonSchema'>, mode: ~typing.Literal['validation', 'serialization'] = 'validation', *, union_format: ~typing.Literal['any_of', 'primitive_type_array'] = 'any_of') → dict[str, Any]

Generates a JSON schema for a model class.

Parameters:

by_alias (bool) – Whether to use attribute aliases or not.
ref_template (str) – The reference template.
union_format (Literal['any_of', 'primitive_type_array']) –
The format to use when combining schemas from unions together. Can be one of:
- ’any_of’: Use the [anyOf](https://json-schema.org/understanding-json-schema/reference/combining#anyOf)
keyword to combine schemas (the default). - ‘primitive_type_array’: Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.
schema_generator (type[GenerateJsonSchema]) – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications
mode (Literal['validation', 'serialization']) – The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

Return type:

classmethod model_parametrized_name(params: tuple[type[Any], ...]) → str

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:: params (tuple[type[Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.
Returns:: String representing the new class where params are passed to cls as type variables.
Raises:: TypeError – Raised when trying to generate concrete names for non-generic models.
Return type:: str

classmethod model_rebuild(*, force: bool = False, raise_errors: bool = True, _parent_namespace_depth: int = 2, _types_namespace: MappingNamespace | None = None) → bool | None

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

force (bool) – Whether to force the rebuilding of the model schema, defaults to False.
raise_errors (bool) – Whether to raise errors, defaults to True.
_parent_namespace_depth (int) – The depth level of the parent namespace, defaults to 2.
_types_namespace (MappingNamespace | None) – The types namespace, defaults to None.

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

Return type:

bool | None

Validate a pydantic model instance.

Parameters:

obj (Any) – The object to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
from_attributes (bool | None) – Whether to extract data from object attributes.
context (Any | None) – Additional context to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

Return type:

!!! abstract “Usage Documentation”: [JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

json_data (str | bytes | bytearray) – The JSON data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

Return type:

Validate the given object with string data against the Pydantic model.

Parameters:

obj (Any) – The object containing string data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Return type:

classmethod parse_file(path: str | Path, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) → Self

Parameters:

path (str | Path)
content_type (str | None)
encoding (str)
proto (DeprecatedParseProtocol | None)
allow_pickle (bool)

Return type:

Self

classmethod parse_obj(obj: Any) → Self

Parameters:: obj (Any)
Return type:: Self

classmethod parse_raw(b: str | bytes, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) → Self

Parameters:

b (str | bytes)
content_type (str | None)
encoding (str)
proto (DeprecatedParseProtocol | None)
allow_pickle (bool)

Return type:

Self

classmethod schema(by_alias: bool = True, ref_template: str = '#/$defs/{model}') → Dict[str, Any]

Parameters:

by_alias (bool)
ref_template (str)

Return type:

classmethod schema_json(*, by_alias: bool = True, ref_template: str = '#/$defs/{model}', **dumps_kwargs: Any) → str

Parameters:

by_alias (bool)
ref_template (str)
dumps_kwargs (Any)

Return type:

classmethod update_forward_refs(**localns: Any) → None

Parameters:: localns (Any)
Return type:: None

classmethod validate(value: Any) → Self

Parameters:: value (Any)
Return type:: Self

__copy__() → Self

Returns a shallow copy of the model.

Return type:: Self

__deepcopy__(memo: dict[int, Any] | None = None) → Self

Returns a deep copy of the model.

Parameters:: memo (dict[int, Any] | None)
Return type:: Self

__init__(**data: Any) → None

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:: data (Any)
Return type:: None

__iter__() → Generator[tuple[str, Any], None, None]

So dict(model) works.

Return type:: Generator[tuple[str, Any], None, None]

__pretty__(fmt: Callable[[Any], Any], **kwargs: Any) → Generator[Any]

Used by devtools (https://python-devtools.helpmanual.io/) to pretty print objects.

Parameters:

fmt (Callable[[Any], Any])
kwargs (Any)

Return type:

Generator[Any]

__repr_name__() → str

Name of the instance’s class, used in __repr__.

Return type:: str

__repr_recursion__(object: Any) → str

Returns the string representation of a recursive object.

Parameters:: object (Any)
Return type:: str

__rich_repr__() → RichReprResult

Used by Rich (https://rich.readthedocs.io/en/stable/pretty.html) to pretty print objects.

Return type:: RichReprResult

Returns a copy of the model.

!!! warning “Deprecated”: This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

include (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to include in the copied model.
exclude (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to exclude in the copied model.
update (Dict[str, Any] | None) – Optional dictionary of field-value pairs to override field values in the copied model.
deep (bool) – If True, the values of fields that are Pydantic models will be deep-copied.

Returns:

A copy of the model with included, excluded and updated fields as specified.

Return type:

Self

Parameters:

include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)

Return type:

Parameters:

include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)
encoder (Callable[[Any], Any] | None)
models_as_dict (bool)
dumps_kwargs (Any)

Return type:

model_copy(*, update: Mapping[str, Any] | None = None, deep: bool = False) → Self

!!! abstract “Usage Documentation”: [model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note: The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

update (Mapping[str, Any] | None) – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.
deep (bool) – Set to True to make a deep copy of the model.

Returns:

New model instance.

Return type:

!!! abstract “Usage Documentation”: [model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

mode (Literal['json', 'python'] | str) – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to include in the output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to exclude from the output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to use the field’s alias in the dictionary key if defined.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A dictionary representation of the model.

Return type:

!!! abstract “Usage Documentation”: [model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

indent (int | None) – Indentation to use in the JSON output. If None is passed, the output will be compact.
ensure_ascii (bool) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to include in the JSON output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to exclude from the JSON output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to serialize using field aliases.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A JSON string representation of the model.

Return type:

model_post_init(context: Any, /) → None

Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

Parameters:: context (Any)
Return type:: None

input_layer: InputLayer

model_computed_fields = {}

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property model_extra: dict[str, Any] | None

Get extra fields set during validation.

Returns:: A dictionary of extra fields, or None if config.extra is not set to “allow”.

model_fields = {'input_layer': FieldInfo(annotation=Literal['detect_mat', 'rgb'], required=False, default='detect_mat')}

property model_fields_set: set[str]

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,: i.e. that were not filled from defaults.

class phenotypic.sdk_.JobMetadataKey[source]

Bases: object

Keys inside <output>/.phenotypic/progress/job_metadata.json.

Writers (CLI execution strategies) and readers (recompile worker, sentinel, checkpoint handler, GUI runs registry) must reference these constants — never the bare string. Renaming a key here should fail fast at every site.

CHUNK_JOB_IDS: Final[str] = 'chunk_job_ids'

CHUNK_SCRIPTS: Final[str] = 'chunk_scripts'

DATASETS: Final[str] = 'datasets'

EXECUTION_MODE: Final[str] = 'execution_mode'

GUI_RECORD_GENERATION: Final[str] = 'gui_record_generation': GUI owner-record generation that initiated this scheduler launch. Together with slurm_generation this is the durable restart-safe binding between GUI identity and the CLI lifecycle epoch.

IMAGE_TASK_MAPPING: Final[str] = 'image_task_mapping'

IMAGE_TYPE: Final[str] = 'image_type'

INCLUDE_DATASET_COLUMN: Final[str] = 'include_dataset_column'

INPUT_PATH: Final[str] = 'input_path'

METADATA_CSV: Final[str] = 'metadata_csv'

NCOLS: Final[str] = 'ncols'

NO_QC: Final[str] = 'no_qc': Whether the recompile finalizer task should skip QC compute. Set on the SLURM recompile finalizer task dict alongside METADATA_CSV; read by _cli_recompile_worker._run_post_master_steps.

NROWS: Final[str] = 'nrows'

ORCHESTRATION_EPOCH: Final[str] = 'orchestration_epoch'

PIPELINE_PATH: Final[str] = 'pipeline_path'

PROCESSING_GENERATION: Final[str] = 'processing_generation'

SLURM_JOB_IDS: Final[str] = 'slurm_job_ids'

START_TIME: Final[str] = 'start_time'

class phenotypic.sdk_.LazyWidgetMixin[source]

Bases: object

Mixin providing a lazy ipywidget interface.

This mixin allows ImageOperation classes to automatically generate a Jupyter widget interface for parameter tuning and visualization.

The six lazily-populated UI handles this mixin uses (_ui, _param_widgets, _view_dropdown, _update_button, _output_widget, _image_ref) are declared as PrivateAttr on ImageOperation rather than here. A plain (non-BaseModel) mixin’s PrivateAttr declarations are not collected by pydantic, so this mixin stays a stateless methods-only class and the private attrs live on the pydantic model that mixes it in. This keeps the mixin safe to combine with any BaseModel in any MRO position.

__getstate__()[source]

Prepare the object for pickling by disposing of any widgets.

This ensures that UI components (which may contain unpickleable objects like input functions or thread locks) are cleaned up before serialization.

Note

This method modifies the object state by calling dispose_widgets(). Any active widgets will be detached from the object.

widget(image: Image | None = None, show: bool = False) → Widget[source]

Return (and optionally display) the root widget.

Parameters:

image (Image | None) – Optional image to visualize. If provided, visualization controls will be added to the widget.
show (bool) – Whether to display the widget immediately. Defaults to False.

Returns:

The root widget.

Return type:

ipywidgets.Widget

Raises:

ImportError – If ipywidgets or IPython are not installed.

class phenotypic.sdk_.MetadataMigrationAuthority(status_path: Path, terminal_receipt_path: Path, terminal_receipt_digest: str, plan_fingerprint: str, source_fingerprint: str, resulting_fingerprint: str, compatible_noop: bool)[source]

Bases: object

Stable terminal authority for the metadata migration stage.

Parameters:

status_path (Path)
terminal_receipt_path (Path)
terminal_receipt_digest (str)
plan_fingerprint (str)
source_fingerprint (str)
resulting_fingerprint (str)
compatible_noop (bool)

compatible_noop: bool

plan_fingerprint: str

resulting_fingerprint: str

source_fingerprint: str

status_path: Path

terminal_receipt_digest: str

terminal_receipt_path: Path

class phenotypic.sdk_.MetadataMigrationReport(source: str, status: Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed'], source_fingerprint: str, plan_fingerprint: str, targets: tuple[MetadataMigrationTarget, ...], conflicts: tuple[str, ...] = (), target_role: Literal['bundle_durable', 'bundle_all', 'exact_file'] | None = None)[source]

Bases: object

Immutable preflight result for a file, frame, or bundle.

Parameters:

source (str)
status (Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed'])
source_fingerprint (str)
plan_fingerprint (str)
targets (tuple[MetadataMigrationTarget, ...])
conflicts (tuple[str, ...])
target_role (Literal['bundle_durable', 'bundle_all', 'exact_file'] | None)

property blocked_count: int: Return the number of targets that cannot migrate losslessly.

property compatible_count: int: Return the number of already-canonical targets.

conflicts: tuple[str, ...] = ()

property migratable_count: int: Return the number of targets requiring migration.

plan_fingerprint: str

source: str

source_fingerprint: str

status: Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed']

target_role: Literal['bundle_durable', 'bundle_all', 'exact_file'] | None = None

targets: tuple[MetadataMigrationTarget, ...]

class phenotypic.sdk_.MetadataMigrationResult(status: Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed'], source: str, source_fingerprint: str, resulting_fingerprint: str | None, plan_fingerprint: str, receipt_path: Path | None, migrated_targets: tuple[str, ...] = (), skipped_targets: tuple[str, ...] = (), blocked_targets: tuple[str, ...] = (), conflicts: tuple[str, ...] = ())[source]

Bases: object

Immutable outcome of migration or rollback.

Parameters:

status (Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed'])
source (str)
source_fingerprint (str)
resulting_fingerprint (str | None)
plan_fingerprint (str)
receipt_path (Path | None)
migrated_targets (tuple[str, ...])
skipped_targets (tuple[str, ...])
blocked_targets (tuple[str, ...])
conflicts (tuple[str, ...])

blocked_targets: tuple[str, ...] = ()

conflicts: tuple[str, ...] = ()

migrated_targets: tuple[str, ...] = ()

plan_fingerprint: str

receipt_path: Path | None

resulting_fingerprint: str | None

skipped_targets: tuple[str, ...] = ()

source: str

source_fingerprint: str

status: Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed']

class phenotypic.sdk_.MetadataMigrationTarget(path: str, kind: Literal['csv', 'parquet', 'json', 'hdf', 'frame'], status: Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed'], source_fingerprint: str, proposed_header_map: tuple[tuple[str, str], ...] = (), needs_metadata_marker: bool = False, hdf_snapshot_fingerprint: str | None = None, conflicts: tuple[str, ...] = (), mixed_table: bool = False)[source]

Bases: object

Immutable preflight description of one migration target.

Parameters:

path (str)
kind (Literal['csv', 'parquet', 'json', 'hdf', 'frame'])
status (Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed'])
source_fingerprint (str)
proposed_header_map (tuple[tuple[str, str], ...])
needs_metadata_marker (bool)
hdf_snapshot_fingerprint (str | None)
conflicts (tuple[str, ...])
mixed_table (bool)

conflicts: tuple[str, ...] = ()

hdf_snapshot_fingerprint: str | None = None

kind: Literal['csv', 'parquet', 'json', 'hdf', 'frame']

mixed_table: bool = False

needs_metadata_marker: bool = False

path: str

proposed_header_map: tuple[tuple[str, str], ...] = ()

source_fingerprint: str

status: Literal['compatible', 'migratable', 'blocked', 'applied', 'rolled_back', 'failed']

class phenotypic.sdk_.ModulePath[source]

Bases: object

Importable module paths used in dynamic importlib.import_module dispatch.

Spelled out here so a renamed sub-package fails at type-check time (consumers reference ModulePath.POST — a typo there is caught by mypy) rather than silently at runtime.

ANALYSIS: Final[str] = 'phenotypic.analysis'

POST: Final[str] = 'phenotypic.post'

class phenotypic.sdk_.NormControlMixin[source]

Bases: object

Mixin for operations that need to disable normalization of inner operations.

Provides a method to create copies of ImageEnhancer or ImagePipeline instances with output normalization disabled. This is useful for composite operations where an inner enhancer operates on non-normalized data (e.g., variance-stabilized values from the Generalized Anscombe Transform, typically in the range ~1-32), where clipping or rescaling to [0, 1] would destroy the inverse transform.

The mixin uses duck typing to check for a norm attribute on operations. If an operation has one, the _disable_normalization method will create a shallow copy with norm=None. This preserves the original operation unchanged while allowing the copy to operate without output normalization. Operations that carry no norm field (e.g. BlurGauss) are returned unchanged.

Note

Renamed from ClipControlMixin in 0.18.0, when clip: bool became NormOut. The old name is gone.

Example

Creating a normalization-disabled copy of an enhancer:

>>> from phenotypic.abc_ import ImageEnhancer
>>> from phenotypic.sdk_ import NormalizedOutputMixin, NormControlMixin
>>>
>>> class Denoise(NormalizedOutputMixin, ImageEnhancer):
...     '''Denoise a colony plate.
...
...     Args:
...         sigma: Smoothing width in pixels.
...         norm: Output normalization policy.
...     '''
...
...     sigma: float = 1.0
...
...     def _operate(self, image):
...         return image
>>>
>>> enh = Denoise(sigma=5.0, norm="clip")
>>> copied = NormControlMixin._disable_normalization(enh)
>>> # Original unchanged, copy has norm=None
>>> enh.norm, copied.norm
('clip', None)

Creating a normalization-disabled copy of a pipeline:

>>> from phenotypic import ImagePipeline
>>> from phenotypic.enhance import BlurGauss
>>>
>>> pipeline = ImagePipeline(pipe_cfgs=[
...     BlurGauss(sigma=1.0),
...     Denoise(sigma=5.0, norm="clip")
... ])
>>> copied_pipe = NormControlMixin._disable_normalization(pipeline)
>>> # Only Denoise has a norm attribute, so only it is affected
>>> # _ops is a dict with operation names as keys
>>> [getattr(op, "norm", "absent") for op in copied_pipe._ops.values()]
['absent', None]

class phenotypic.sdk_.NormalizedOutputMixin(*, norm: Literal['clip', 'rescale'] | None = 'clip')[source]

Bases: BaseModel

Adds a norm field controlling how an operation’s output is range-guarded.

detect_mat is contractually [0, 1]. norm selects how an operation upholds that contract:

"clip" (default) saturates out-of-range values. It is the identity for in-range pixels, so absolute intensity is preserved and detect_mat stays comparable across a batch of plates.
"rescale" linearly remaps the full observed range onto [0, 1]. Ordering survives, absolute scale does not: a single specular highlight sets the max.
None passes values through untouched. Required inside a Generalized Anscombe Transform region (where the signal is deliberately not in [0, 1]) and by CompositeEnhance on non-normalized maps.

The field is appended to the end of the subclass’s field order rather than frontloaded, so an operation’s own parameters keep their natural position in model_json_schema() and to_json().

Note

Replaces the clip: bool field removed in 0.18.0. A bool cannot express "rescale", and the attribute name clip is claimed by NormControlMixin, which duck-types on it.

Parameters:: norm (Literal['clip', 'rescale'] | None)

classmethod __get_pydantic_json_schema__(core_schema: CoreSchema, handler: GetJsonSchemaHandler, /) → JsonSchemaValue

Hook into generating the model’s JSON schema.

Parameters:

core_schema (CoreSchema) – A pydantic-core CoreSchema. You can ignore this argument and call the handler with a new CoreSchema, wrap this CoreSchema ({‘type’: ‘nullable’, ‘schema’: current_schema}), or just call the handler with the original schema.
handler (GetJsonSchemaHandler) – Call into Pydantic’s internal JSON schema generation. This will raise a pydantic.errors.PydanticInvalidForJsonSchema if JSON schema generation fails. Since this gets called by BaseModel.model_json_schema you can override the schema_generator argument to that function to change JSON schema generation globally for a type.

Returns:

A JSON schema, as a Python object.

Return type:

JsonSchemaValue

classmethod __pydantic_init_subclass__(**kwargs: Any) → None[source]

Move norm to the end of the subclass’s field order.

Parameters:: kwargs (Any)
Return type:: None

classmethod __pydantic_on_complete__() → None

This is called once the class and its fields are fully initialized and ready to be used.

This typically happens when the class is created (just before [__pydantic_init_subclass__()][pydantic.main.BaseModel.__pydantic_init_subclass__] is called on the superclass), except when forward annotations are used that could not immediately be resolved. In that case, it will be called later, when the model is rebuilt automatically or explicitly using [model_rebuild()][pydantic.main.BaseModel.model_rebuild].

Return type:: None

classmethod construct(_fields_set: set[str] | None = None, **values: Any) → Self

Parameters:

_fields_set (set[str] | None)
values (Any)

Return type:

classmethod from_orm(obj: Any) → Self

Parameters:: obj (Any)
Return type:: Self

classmethod model_construct(_fields_set: set[str] | None = None, **values: Any) → Self

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note: model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.

Parameters:

_fields_set (set[str] | None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.
values (Any) – Trusted or pre-validated data dictionary.

Returns:

A new instance of the Model class with validated data.

Return type:

classmethod model_json_schema(by_alias: bool = True, ref_template: str = '#/$defs/{model}', schema_generator: type[~pydantic.json_schema.GenerateJsonSchema] = <class 'pydantic.json_schema.GenerateJsonSchema'>, mode: ~typing.Literal['validation', 'serialization'] = 'validation', *, union_format: ~typing.Literal['any_of', 'primitive_type_array'] = 'any_of') → dict[str, Any]

Generates a JSON schema for a model class.

Parameters:

by_alias (bool) – Whether to use attribute aliases or not.
ref_template (str) – The reference template.
union_format (Literal['any_of', 'primitive_type_array']) –
The format to use when combining schemas from unions together. Can be one of:
- ’any_of’: Use the [anyOf](https://json-schema.org/understanding-json-schema/reference/combining#anyOf)
keyword to combine schemas (the default). - ‘primitive_type_array’: Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.
schema_generator (type[GenerateJsonSchema]) – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications
mode (Literal['validation', 'serialization']) – The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

Return type:

classmethod model_parametrized_name(params: tuple[type[Any], ...]) → str

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:: params (tuple[type[Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.
Returns:: String representing the new class where params are passed to cls as type variables.
Raises:: TypeError – Raised when trying to generate concrete names for non-generic models.
Return type:: str

classmethod model_rebuild(*, force: bool = False, raise_errors: bool = True, _parent_namespace_depth: int = 2, _types_namespace: MappingNamespace | None = None) → bool | None

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

force (bool) – Whether to force the rebuilding of the model schema, defaults to False.
raise_errors (bool) – Whether to raise errors, defaults to True.
_parent_namespace_depth (int) – The depth level of the parent namespace, defaults to 2.
_types_namespace (MappingNamespace | None) – The types namespace, defaults to None.

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

Return type:

bool | None

Validate a pydantic model instance.

Parameters:

obj (Any) – The object to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
from_attributes (bool | None) – Whether to extract data from object attributes.
context (Any | None) – Additional context to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

Return type:

!!! abstract “Usage Documentation”: [JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

json_data (str | bytes | bytearray) – The JSON data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

Return type:

Validate the given object with string data against the Pydantic model.

Parameters:

obj (Any) – The object containing string data to validate.
strict (bool | None) – Whether to enforce types strictly.
extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context (Any | None) – Extra variables to pass to the validator.
by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.
by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Return type:

classmethod parse_file(path: str | Path, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) → Self

Parameters:

path (str | Path)
content_type (str | None)
encoding (str)
proto (DeprecatedParseProtocol | None)
allow_pickle (bool)

Return type:

Self

classmethod parse_obj(obj: Any) → Self

Parameters:: obj (Any)
Return type:: Self

classmethod parse_raw(b: str | bytes, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False) → Self

Parameters:

b (str | bytes)
content_type (str | None)
encoding (str)
proto (DeprecatedParseProtocol | None)
allow_pickle (bool)

Return type:

Self

classmethod schema(by_alias: bool = True, ref_template: str = '#/$defs/{model}') → Dict[str, Any]

Parameters:

by_alias (bool)
ref_template (str)

Return type:

classmethod schema_json(*, by_alias: bool = True, ref_template: str = '#/$defs/{model}', **dumps_kwargs: Any) → str

Parameters:

by_alias (bool)
ref_template (str)
dumps_kwargs (Any)

Return type:

classmethod update_forward_refs(**localns: Any) → None

Parameters:: localns (Any)
Return type:: None

classmethod validate(value: Any) → Self

Parameters:: value (Any)
Return type:: Self

__copy__() → Self

Returns a shallow copy of the model.

Return type:: Self

__deepcopy__(memo: dict[int, Any] | None = None) → Self

Returns a deep copy of the model.

Parameters:: memo (dict[int, Any] | None)
Return type:: Self

__init__(**data: Any) → None

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:: data (Any)
Return type:: None

__iter__() → Generator[tuple[str, Any], None, None]

So dict(model) works.

Return type:: Generator[tuple[str, Any], None, None]

__pretty__(fmt: Callable[[Any], Any], **kwargs: Any) → Generator[Any]

Used by devtools (https://python-devtools.helpmanual.io/) to pretty print objects.

Parameters:

fmt (Callable[[Any], Any])
kwargs (Any)

Return type:

Generator[Any]

__repr_name__() → str

Name of the instance’s class, used in __repr__.

Return type:: str

__repr_recursion__(object: Any) → str

Returns the string representation of a recursive object.

Parameters:: object (Any)
Return type:: str

__rich_repr__() → RichReprResult

Used by Rich (https://rich.readthedocs.io/en/stable/pretty.html) to pretty print objects.

Return type:: RichReprResult

Returns a copy of the model.

!!! warning “Deprecated”: This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

include (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to include in the copied model.
exclude (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to exclude in the copied model.
update (Dict[str, Any] | None) – Optional dictionary of field-value pairs to override field values in the copied model.
deep (bool) – If True, the values of fields that are Pydantic models will be deep-copied.

Returns:

A copy of the model with included, excluded and updated fields as specified.

Return type:

Self

Parameters:

include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)

Return type:

Parameters:

include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None)
by_alias (bool)
exclude_unset (bool)
exclude_defaults (bool)
exclude_none (bool)
encoder (Callable[[Any], Any] | None)
models_as_dict (bool)
dumps_kwargs (Any)

Return type:

model_copy(*, update: Mapping[str, Any] | None = None, deep: bool = False) → Self

!!! abstract “Usage Documentation”: [model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note: The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

update (Mapping[str, Any] | None) – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.
deep (bool) – Set to True to make a deep copy of the model.

Returns:

New model instance.

Return type:

!!! abstract “Usage Documentation”: [model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

mode (Literal['json', 'python'] | str) – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to include in the output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to exclude from the output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to use the field’s alias in the dictionary key if defined.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A dictionary representation of the model.

Return type:

!!! abstract “Usage Documentation”: [model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

indent (int | None) – Indentation to use in the JSON output. If None is passed, the output will be compact.
ensure_ascii (bool) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.
include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to include in the JSON output.
exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to exclude from the JSON output.
context (Any | None) – Additional context to pass to the serializer.
by_alias (bool | None) – Whether to serialize using field aliases.
exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.
exclude_defaults (bool) – Whether to exclude fields that are set to their default value.
exclude_none (bool) – Whether to exclude fields that have a value of None.
exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A JSON string representation of the model.

Return type:

model_post_init(context: Any, /) → None

Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

Parameters:: context (Any)
Return type:: None

model_computed_fields = {}

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property model_extra: dict[str, Any] | None

Get extra fields set during validation.

Returns:: A dictionary of extra fields, or None if config.extra is not set to “allow”.

model_fields = {'norm': FieldInfo(annotation=Union[Literal['clip', 'rescale'], NoneType], required=False, default='clip')}

property model_fields_set: set[str]

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,: i.e. that were not filled from defaults.

norm: NormOut

class phenotypic.sdk_.PhenotypicAttr[source]

Bases: object

Keys inside the namespaced attributes.phenotypic block.

Spelled out here so a renamed key fails at type-check time rather than silently at runtime, matching the JobMetadataKey pattern already used in phenotypic.sdk_._io_constants.

DETECT_MODE: Final[str] = 'detect_mode'

GAMMA: Final[str] = 'gamma'

GRID: Final[str] = 'grid'

ILLUMINANT: Final[str] = 'illuminant'

IMAGE_CLASS: Final[str] = 'image_class'

IMPORTED: Final[str] = 'imported'

LABELS: Final[str] = 'labels'

METADATA: Final[str] = 'metadata'

METADATA_TABLE: Final[str] = 'metadata_table': Which metadata snapshot the store’s embedded tables were built against (D-A), as {"snapshot_sha256": ..., "join_keys": [...], "join_kind": ...}. The key is metadata_table and not metadata, because METADATA directly above is already taken by the {protected, public, imported} image-metadata sections. This is the spelling’s single home: _run_state.py binds its private reader names to these two members rather than repeating the strings.

PHENOTYPIC_VERSION: Final[str] = 'phenotypic_version'

PROTECTED: Final[str] = 'protected'

PROVENANCE: Final[str] = 'provenance'

PUBLIC: Final[str] = 'public'

PUBLICATION_PROTOCOL: Final[str] = 'publication_protocol'

PYRAMID: Final[str] = 'pyramid'

ROOT: Final[str] = 'phenotypic'

SERIES: Final[str] = 'series'

SNAPSHOT_SHA256: Final[str] = 'snapshot_sha256'

STORE_SCHEMA_VERSION: Final[str] = 'store_schema_version'

TABLES: Final[str] = 'tables'

WORK_ID: Final[str] = 'work_id'

class phenotypic.sdk_.PreparedEmbeddedMeasurementTable(frame: pandas.DataFrame, measurement_columns: tuple[str, ...], join_status: Literal['not_requested', 'joined', 'no_common_keys'], join_keys: tuple[str, ...], metadata_snapshot_sha256: str)[source]

Bases: object

Joined payload plus stable provenance recorded with its Parquet file.

Parameters:

frame (pandas.DataFrame)
measurement_columns (tuple[str, ...])
join_status (Literal['not_requested', 'joined', 'no_common_keys'])
join_keys (tuple[str, ...])
metadata_snapshot_sha256 (str)

parquet_metadata() → dict[bytes, bytes][source]

Return replaceable join provenance as Arrow schema metadata.

Return type:: dict[bytes, bytes]

frame: DataFrame

join_keys: tuple[str, ...]

join_status: Literal['not_requested', 'joined', 'no_common_keys']

measurement_columns: tuple[str, ...]

metadata_snapshot_sha256: str

class phenotypic.sdk_.PreparedImageTables(measurements: pd.DataFrame, metadata: pd.DataFrame | None, measurement_columns: tuple[str, ...], join_status: JoinStatus, join_keys: tuple[str, ...], metadata_snapshot_sha256: str)[source]

Bases: object

One image’s measurement rows and, separately, its user metadata rows.

Spec §7.1-7.2’s inversion. measurements is the pre-join baseline – intrinsic identity (Metadata_ImageFile, Metadata_Dataset, the object label) plus the measured columns, and nothing that came from --metadata. metadata is the projection of the run’s metadata snapshot onto the join keys this image actually carries, or None when there is no such projection to make.

The join provenance below describes the metadata table, not the measurements table. After the inversion the measurements table carries no join at all, so its own recorded triple is not_requested / [] / "" – which is what measurements_payload() builds, and the one place that rule lives.

Parameters:

measurements (pd.DataFrame)
metadata (pd.DataFrame | None)
measurement_columns (tuple[str, ...])
join_status (JoinStatus)
join_keys (tuple[str, ...])
metadata_snapshot_sha256 (str)

measurements

The unjoined baseline, exactly the projection measurement_columns already named.

Type:: pandas.core.frame.DataFrame

metadata

User metadata rows for this image’s keys, or None when join_status is not_requested or no_common_keys.

Type:: pandas.core.frame.DataFrame | None

measurement_columns

Baseline column names, in writer order.

Type:: tuple[str, …]

join_status

Whether a metadata join was requested, possible, or done.

Type:: Literal[‘not_requested’, ‘joined’, ‘no_common_keys’]

join_keys

The common columns the metadata table is keyed on.

Type:: tuple[str, …]

metadata_snapshot_sha256

Digest of the metadata.csv snapshot this image was prepared against; "" when none was supplied.

Type:: str

measurements_payload() → PreparedEmbeddedMeasurementTable[source]

Return the measurements table’s own payload, carrying no join.

The triple is not_requested / () / "" unconditionally, because after the inversion the statement “this file is the result of a join” is false of tables/measurements/table.parquet on every store. That is also exactly the shape _valid_embedded_measurement_contract() already accepts, so the contract needs no change for this file.

Return type:: PreparedEmbeddedMeasurementTable

metadata_parquet_metadata() → dict[bytes, bytes][source]

Return the metadata table’s own join provenance as Arrow metadata.

The same key spellings the measurement table uses – one home for the names – minus measurement_columns, which says nothing about this file. This is what makes pht-metadata.parquet self-describing to a reader who has only the Parquet: it names the keys it is joined on, the direction of the join, and the snapshot it came from.

Return type:: dict[bytes, bytes]

join_keys: tuple[str, ...]

join_status: Literal['not_requested', 'joined', 'no_common_keys']

measurement_columns: tuple[str, ...]

measurements: DataFrame

metadata: DataFrame | None

metadata_snapshot_sha256: str

class phenotypic.sdk_.ProcessingStateKey[source]

Bases: object

Keys inside <output>/.phenotypic/processing_state.json.

Distinct from JobMetadataKey even where string values overlap (e.g. EXECUTION_MODE, INPUT_PATH) — these describe the processing_state.json contract, not the SLURM job metadata sidecar. Some values intentionally match across the two contracts so that a single field (like execution_mode) can be migrated atomically; the test_processing_state_keys_match_job_metadata_keys regression test asserts the overlap.

COMPLETED: Final[str] = 'completed'

CONFIG: Final[str] = 'config'

DATASETS: Final[str] = 'datasets'

ERRORS: Final[str] = 'errors'

EXECUTION_MODE: Final[str] = 'execution_mode'

FAILED: Final[str] = 'failed'

INITIAL_IMAGES: Final[str] = 'initial_images'

INPUT_PATH: Final[str] = 'input_path'

LAST_UPDATED: Final[str] = 'last_updated'

OUTPUT_DIR: Final[str] = 'output_dir'

PIPELINE_PATH: Final[str] = 'pipeline_path'

STARTED: Final[str] = 'started'

TIMESTAMP: Final[str] = 'timestamp'

VERSION: Final[str] = 'version'

class phenotypic.sdk_.RunDiagnostics(accepted: int, verified: int, failed: int)[source]

Bases: object

Counts derived from images. Nothing branches on these (§4.2, §9).

One-line projections over ImageState.verdict, not cached counts of a collection the caller already holds.

manifest.json’s counts and the event log’s presence were in an earlier draft of this dataclass and are dropped (U-5): verified zero consumers survive P6, and carrying demoted evidence into RunState is what keeps it alive as a quasi-evidence surface. The files remain on disk for a human debugging a run.

Parameters:

accepted (int)
verified (int)
failed (int)

accepted: int

failed: int

verified: int

class phenotypic.sdk_.RunIdentity(processing_generation: str, restart_epoch: int, scheduler_epoch: str | None, owner_generation: str | None, inventory_digest: str, scientific_config_digest: str, finalization_input_digest: str)[source]

Bases: object

The run-level identity of one run configuration (spec §5.1, §5.3).

Carries four of §5.1’s five identity tokens – the fifth, work_id, is per-image and lives on ImageState – plus §5.3’s three digests.

processing_generation is content-derived (from P2 onward), so resume and fencing are emergent rather than bookkeeping: two invocations with the same inputs mint the same identity without either having read the other’s state.

Note the count: spec §5.1 is headed “the six tokens” and its own amendment U-4 cuts publication_id, leaving five, of which work_id and processing_generation are the content-derived pair. Anything still saying “six tokens, three content-derived” predates U-4.

Parameters:

processing_generation (str)
restart_epoch (int)
scheduler_epoch (str | None)
owner_generation (str | None)
inventory_digest (str)
scientific_config_digest (str)
finalization_input_digest (str)

digest() → str[source]

Return a stable digest of the fencing-relevant tokens.

scheduler_epoch and owner_generation are excluded: they are liveness facts, not configuration, and folding them in would discard the verification cache every time a job is submitted against unchanged work.

Derived from IDENTITY_DIGEST_FIELDS, not from a second hand-written list (F6). The set was enumerated twice – here and in _run_state’s comparison – with nothing keeping them in step, so a sixth field added to one would have silently not been fenced by the other.

Uses canonical_digest() rather than a local ``json.dumps`` (F5). The hand-rolled copy matched on sort_keys and separators and differed on ensure_ascii, which is the one flag _digests calls load-bearing: every proof on disk was written with ensure_ascii=False. A dataset directory containing a non-ASCII character – plaque-café/ – was enough to make this digest disagree with the canonical spelling of the same value.

Return type:: str

finalization_input_digest: str

inventory_digest: str

owner_generation: str | None

processing_generation: str

restart_epoch: int

scheduler_epoch: str | None

scientific_config_digest: str

class phenotypic.sdk_.RunState(completion: Literal['complete', 'incomplete', 'failed', 'active'], identity: RunIdentity, images: Mapping[str, ImageState], advisories: tuple[str, ...], diagnostics: RunDiagnostics, depth: Literal['shallow', 'deep'], verified_at: datetime | None = None)[source]

Bases: object

The single answer to “is this run done?” (spec §4.3, §9).

Parameters:

completion (Literal['complete', 'incomplete', 'failed', 'active'])
identity (RunIdentity)
images (Mapping[str, ImageState])
advisories (tuple[str, ...])
diagnostics (RunDiagnostics)
depth (Literal['shallow', 'deep'])
verified_at (datetime | None)

advisories: tuple[str, ...]

completion: Literal['complete', 'incomplete', 'failed', 'active']

depth: Literal['shallow', 'deep']

diagnostics: RunDiagnostics

identity: RunIdentity

images: Mapping[str, ImageState]

verified_at: datetime | None = None

phenotypic.sdk_.__getattr__(name: str) → Any[source]

Resolve a heavy re-export on first access and cache it on the package.

Parameters:: name (str)
Return type:: Any

phenotypic.sdk_.aggregate_proof_is_current(output_dir: Path) → bool[source]

Return whether a valid aggregate proof still covers the deliverables.

The boolean half of _valid_aggregate_proof(), exported because the GUI’s core_readable asks exactly this question and spec §5.2 makes this module the public reader surface. Named to match run_proof_is_current(): the two ask the same shape of question about the run proof and the aggregate proof respectively.

Returns ``False`` on: an absent .phenotypic/aggregate_publication.json, one whose version is not AGGREGATE_PROOF_VERSION, one with an empty or non-mapping required_outputs, or one naming a required output whose bytes on disk no longer match the descriptor it was published with.

This is O(1) in images – three or four run-level deliverables – which is why callers re-ask it rather than caching the answer.

Parameters:: output_dir (Path)
Return type:: bool

phenotypic.sdk_.aggregate_publication_marker_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/aggregate_publication.json.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.aggregation_shard_dir(output_dir: Path, scheduler_epoch: str | None) → Path[source]

Return <progress>/aggregation_shards/<scheduler_epoch>/.

Spec §7.5: the fan-out’s measurement shards are per-invocation scratch, so a prior run’s shards can never be merged into this run’s master. Recompile already namespaces its shards this way under recompile/attempts/<attempt_id>/; this generalises the pattern to the forward path.

The namespace is not the correctness argument, and must not be read as one. _scheduler_epoch returns None for every local run, so consecutive local invocations share LOCAL_SCHEDULER_EPOCH and would collide. The fan-out therefore empties this directory when it starts, on both drivers, at the same logical point – which is strictly stronger than namespacing, since namespacing also leaves every prior run’s shards on disk accumulating forever. The epoch stays in the path because it costs nothing and keeps one path shape across the two drivers.

Pure path expression; callers mkdir when they intend to write.

Parameters:

output_dir (Path) – Run output root.
scheduler_epoch (str | None) – The active SLURM lifecycle generation, or None for a local run.

Returns:

The shard directory for this invocation.

Return type:

phenotypic.sdk_.analysis_full_parquet_path(progress_dir_: Path) → Path[source]

Return the legacy-named rolling measurement aggregation state.

Takes a progress_dir (not the run output root) since this file lives inside progress/. It is an internal resumable-run artifact, not a static-analysis sidecar or user-facing output.

Parameters:: progress_dir_ (Path)
Return type:: Path

phenotypic.sdk_.analysis_manifest_path(deliverables_base: Path) → Path[source]

Return the analysis manifest inside a deliverables directory.

Parameters:: deliverables_base (Path)
Return type:: Path

phenotypic.sdk_.assert_identity_current(output_dir: Path, identity: RunIdentity) → None[source]

Raise unless identity still describes output_dir’s state.

D6: a configuration change hard-errors, and it names the specific token that moved. A generic “identity changed” would make the content-derived generation a worse diagnostic than the uuid4 it replaces, which would be a strange thing to ship in a change whose argument is that content-derived identity is better.

Only the five tokens RunIdentity.digest() folds in are compared – see IDENTITY_DIGEST_FIELDS.

Parameters:

output_dir (Path) – Run output root.
identity (RunIdentity) – The identity the caller believes is current.

Raises:

RuntimeError – If the output has no readable processing state, or if any fenced token differs. The message names the first differing token and both values.

Return type:

None

phenotypic.sdk_.atomic_write_bytes(path: str | Path, data: bytes, *, pre_replace: Callable[[], None] | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → None[source]

Atomically write data to path (temp sibling + os.replace).

The bytes counterpart of atomic_write_text() for binary payloads (e.g. a serialized parquet buffer). Same crash-safety guarantees: an all-or-nothing replace and no partial/leftover temp file on failure.

Parameters:

path (str | Path) – The destination file path.
data (bytes) – The full binary payload to write.
pre_replace (Callable[[], None] | None) – Optional callback invoked immediately before publication.
commit_guard (Callable[[], AbstractContextManager[None]] | None)

Raises:

OSError – If the write or rename fails (the temp file is removed first).

Return type:

None

phenotypic.sdk_.atomic_write_json(path: str | Path, payload: Mapping[str, Any] | list[Any], *, indent: int = 2, sort_keys: bool = True, ensure_ascii: bool = False, pre_replace: Callable[[], None] | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → None[source]

Atomically write a JSON payload with deterministic formatting.

Parameters:

path (str | Path) – Destination JSON path.
payload (Mapping[str, Any] | list[Any]) – JSON-serializable mapping or list.
indent (int) – Indentation passed to json.dumps().
sort_keys (bool) – Whether mapping keys are sorted for deterministic output.
ensure_ascii (bool) – Whether non-ASCII characters are escaped.
pre_replace (Callable[[], None] | None) – Optional callback invoked immediately before publication.
commit_guard (Callable[[], AbstractContextManager[None]] | None)

Return type:

None

phenotypic.sdk_.atomic_write_parquet(path: str | Path, frame: Any, *, commit_guard: Callable[[], AbstractContextManager[None]] | None = None, **kwargs: Any) → None[source]

Atomically write a pandas-like frame with shared parquet defaults.

Parameters:

path (str | Path) – Destination parquet path.
frame (Any) – Object exposing to_parquet(path, **kwargs).
**kwargs (Any) – Per-call parquet writer overrides.
commit_guard (Callable[[], AbstractContextManager[None]] | None)
**kwargs

Return type:

None

phenotypic.sdk_.atomic_write_text(path: str | Path, text: str, *, encoding: str = 'utf-8', pre_replace: Callable[[], None] | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → None[source]

Atomically write text to path (temp sibling + os.replace).

A drop-in replacement for Path(path).write_text(text) that never leaves a half-written file: a concurrent reader sees either the old contents or the complete new ones, and an exception mid-write leaves any pre-existing file intact with no .tmp debris.

Parameters:

path (str | Path) – The destination file path.
text (str) – The full text payload to write.
encoding (str) – The text encoding (default "utf-8").
pre_replace (Callable[[], None] | None) – Optional callback invoked immediately before publication.
commit_guard (Callable[[], AbstractContextManager[None]] | None)

Raises:

OSError – If the write or rename fails (the temp file is removed first).

Return type:

None

phenotypic.sdk_.atomic_write_with_writer(path: str | Path, writer: Callable[[str], None], *, pre_replace: Callable[[], None] | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None, temp_suffix: str = '.tmp') → None[source]

Atomically write path using a callback that receives a temp path.

Parameters:

path (str | Path) – Final destination path.
writer (Callable[[str], None]) – Callable that writes complete output to a temporary path string.
pre_replace (Callable[[], None] | None) – Optional callback invoked after the temporary file is synced, immediately before it replaces path. An exception aborts publication and removes the temporary file.
commit_guard (Callable[[], AbstractContextManager[None]] | None)
temp_suffix (str)

Raises:

OSError – Propagated from the writer or rename after temp cleanup.

Return type:

None

phenotypic.sdk_.best_params_path(output_dir: Path) → Path[source]

Return <output>/deliverables/best_params.json (winner params sidecar).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.best_pipeline_path(output_dir: Path) → Path[source]

Return the canonical typed tuned-winner pipeline path.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.build_measurement_table_descriptor(table: PreparedEmbeddedMeasurementTable, *, objmap_target: str) → dict[str, object][source]

Build the stable root descriptor for one embedded measurement table.

Parameters:

table (PreparedEmbeddedMeasurementTable)
objmap_target (str)

Return type:

dict[str, object]

phenotypic.sdk_.build_metadata_table_descriptor(tables: PreparedImageTables) → dict[str, object][source]

Build the stable root descriptor for one embedded metadata table.

It records what the file is, not how it joins: the join keys are on the Parquet’s own key/value metadata and in the root’s metadata_table block, and repeating them a third time here would be a third home for one fact.

Parameters:: tables (PreparedImageTables)
Return type:: dict[str, object]

phenotypic.sdk_.bytes_fingerprint(data: bytes) → str[source]

Return a versioned SHA-256 fingerprint for exact bytes.

Parameters:: data (bytes) – Bytes to fingerprint.
Returns:: A "sha256:<hex>" content fingerprint.
Return type:: str

phenotypic.sdk_.canonical_metadata_order() → dict[str, int][source]

Global rank for every known metadata header (cluster then definition order).

Cluster-order major, enum definition-order minor. A header absent from this map is an unknown/uncategorized user tag; callers rank those last. The map is derived entirely from the import-time schema enums, so it is cached. The returned dict is read-only by contract; callers must not mutate it (mirrors header_to_module()).

Return type:: dict[str, int]

phenotypic.sdk_.checkpoint_lock_filename(checkpoint_type: Literal['manifest', 'finalize']) → str[source]

Filename of the SLURM-sentinel exclusive lock for a checkpoint task.

Parameters:: checkpoint_type (Literal['manifest', 'finalize']) – "manifest" or "finalize". Validated by the phenotypic.sdk_.typing_.CheckpointType Literal alias at type-check time.
Returns:: Filename relative to <progress>/ (hidden file, leading .).
Return type:: str

phenotypic.sdk_.checkpoint_lock_path(progress_dir_: Path, checkpoint_type: Literal['manifest', 'finalize']) → Path[source]

Return <progress>/.{checkpoint_type}_lock.

Parameters:

progress_dir_ (Path)
checkpoint_type (Literal['manifest', 'finalize'])

Return type:

phenotypic.sdk_.chunk_lock_path(progress_dir_: Path) → Path[source]

Return <progress>/.chunk_lock.

Parameters:: progress_dir_ (Path)
Return type:: Path

phenotypic.sdk_.chunk_manifest_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/progress/chunk_manifest.json.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.chunk_parquet_filename(chunk_id: int) → str[source]

Filename of a dashboard chunk Parquet (zero-padded chunk id).

Parameters:: chunk_id (int) – Zero-based chunk index, formatted {chunk_id:03d}.
Returns:: Filename relative to <progress>/chunks/.
Return type:: str

phenotypic.sdk_.chunk_parquet_path(progress_dir_: Path, chunk_id: int) → Path[source]

Return <progress>/chunks/chunk_<id:03d>.parquet.

Parameters:

progress_dir_ (Path)
chunk_id (int)

Return type:

phenotypic.sdk_.chunk_state_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/progress/chunk_state.json.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.chunks_dir(progress_dir_: Path) → Path[source]

Return <progress>/chunks/ for the dashboard chunk parquets.

Parameters:: progress_dir_ (Path)
Return type:: Path

phenotypic.sdk_.clear_machine_state(output_dir: Path) → bool[source]

Remove all of a run’s machine-state for a clean --restart.

Deletes current state inside .phenotypic/ (progress/, processing_state.json, processing_events.log, logs, and generated SLURM scripts) and any pre-migration root-level machine-state, while preserving _PRESERVED_ON_RESTART – the append-only terminal_failures.jsonl journal and restart_epoch.json, because a counter that resets on the operation it fences is not a fence – along with user-facing output artifacts (deliverables/, results/, qc/, …). Read that set’s membership rule before adding to it; it is now the only thing standing between --restart and every artifact under .phenotypic/. This is the difference between --restart (re-run the orchestration against clean state, keep outputs) and --overwrite (delete the whole output dir). Clearing the event log here is what stops a restart from appending to — and rebuilding its manifest/failure records from — the prior run’s events.

Returns:: True if any machine-state was removed, else False.
Parameters:: output_dir (Path)
Return type:: bool

phenotypic.sdk_.clear_verification_cache(output_dir: Path | None = None) → None[source]

Drop one output’s tier-1 entries, or every output’s.

Rule 4 of Task 3. Nothing in ``src/`` calls the scoped form. An earlier draft of this docstring said P2 wired it to clear_machine_state; it does not, and the paragraph below – which says the two are separate and why – is the accurate one. The two sentences contradicted each other three lines apart.

Left unwired deliberately. Adding the call is a behaviour change nobody has ruled on, and it is not needed: clear_machine_state deletes the on-disk tier by deleting the directory, and tier 1 is per-process memory that a restarting process does not have.

Tier 1 only, and deliberately. This function touches no file: it is a reader-side memory reset, and that is what lets _run_state re-export it without exporting a writer (INV-LAYER). clear_machine_state is what removes the on-disk tier, by removing every child of .phenotypic/. A caller that clears only this one and then resolves shallowly will still be served from disk – which is correct, since nothing about the run changed, but it is not what “clear the cache” sounds like.

Parameters:: output_dir (Path | None) – The output to forget. None forgets all of them, which is what a test fixture wants and what a process teardown wants; it is never what a run’s own state reset wants.
Return type:: None

phenotypic.sdk_.curation_labels_parquet_path(output_dir: Path) → Path[source]

Return <output>/deliverables/qc/curation_labels.parquet (durable labels store).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.custom_categories_json_path(output_dir: Path) → Path[source]

Return <output>/deliverables/qc/custom_categories.json (custom-category registry).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.dashboard_html_path(output_dir: Path) → Path[source]

Return <output>/deliverables/dashboard.html.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.dataset_measurements_dir(output_dir: Path, dataset: str) → Path[source]

Return <output>/results/<dataset>/measurements/.

Parameters:

output_dir (Path)
dataset (str)

Return type:

phenotypic.sdk_.dataset_overlays_dir(output_dir: Path, dataset: str) → Path[source]

Return <output>/deliverables/overlays/<dataset>/.

Parameters:

output_dir (Path)
dataset (str)

Return type:

phenotypic.sdk_.dataset_results_dir(output_dir: Path, dataset: str) → Path[source]

Return <output>/results/<dataset>/.

Parameters:

output_dir (Path)
dataset (str)

Return type:

phenotypic.sdk_.dataset_zarr_dir(output_dir: Path, dataset: str) → Path[source]

Return <output>/results/<dataset>/zarr/.

Parameters:

output_dir (Path)
dataset (str)

Return type:

phenotypic.sdk_.datasets_needing_migration(output_dir: Path) → list[str][source]

Datasets holding at least one .h5 result without a VALID store.

One predicate, so the CLI and the GUI cannot disagree about what “needs migrating” means.

Per-IMAGE, not per-dataset: the half-migrated tree this exists to catch has converted and unconverted images in the SAME dataset, so a dataset-level “has .h5 and has no zarr/ dir” test misses it entirely. That tree is the expected state after any interruption, because migration is resumable – and it is neither “only .h5” nor fully converted, so the older “only .h5” guard let it through and –mode full silently reprocessed every unconverted image from source.

Validity, not existence: valid_staged_store, not path.exists(). A store written at an older store_schema_version is present but the loader refuses it, so an existence test reads that tree as clean while every image fails to open.

Parameters:: output_dir (Path) – Run output root.
Returns:: Dataset names needing migration, sorted. Empty for a modern tree.
Return type:: list[str]

phenotypic.sdk_.default_output_dir_name(now: datetime | None = None) → str[source]

Default name for an auto-generated output directory.

Legacy helper for timestamped output-directory names.

Parameters:: now (datetime | None) – Override clock for tests; defaults to datetime.now().
Returns:: "phenotypic_results_YYYYMMDD_HHMMSS".
Return type:: str

phenotypic.sdk_.deliverables_dir(output_dir: Path) → Path[source]

Return <output>/deliverables/ — the user-facing-output folder.

Pure path expression; callers are responsible for mkdir when they intend to write into it. Writers that go through phenotypic.sdk_.atomic_write_with_writer() get the mkdir for free (it creates target.parent); direct write_text/write_bytes writers must mkdir explicitly.

Every artifact helper that previously rooted at <output>/ (master / measurements / per-feature split / analysis / dashboard / report / pipeline.json / README) now composes from here, so a future relocation is a one-line change.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.describe_durability(override: bool | None = None) → str[source]

One-line description of the resolved durability mode, for the start log.

The same command carries different guarantees in different places, which is a genuinely surprising thing to debug. Logging the resolved mode at run start is a required mitigation, not a nicety.

Shares _resolve_durability() with durable_writes_enabled(), so the flag and the sentence describing it cannot drift apart.

Parameters:: override (bool | None)
Return type:: str

phenotypic.sdk_.durable_writes_enabled(override: bool | None = None) → bool[source]

Resolve whether the promote fsyncs before renaming.

write() returns once data is in the page cache. Without fsync the kernel may flush the root zarr.json before the chunk data it describes, so a node crash can leave a store that passes valid_staged_store() – metadata parses, shapes agree – while reading fill_value. That is silent wrong data, not a visible failure, and no amount of metadata validation catches it.

The dominant failure mode does not need it: a SLURM timeout kills the process, and the kernel survives and flushes normally. fsync buys protection only against node loss, power failure, and filesystem crash – which is exactly what a cluster job is exposed to and a laptop run is not.

Parameters:: override (bool | None) – --durable-writes / --no-durable-writes, or None to auto-detect.
Returns:: True when the promote should fsync.
Return type:: bool

Note

This checks SLURM_JOB_ID as well as SLURM_CPUS_PER_TASK. resolve_worker_count (_cli_utils.py:65-72) reads only the latter, so this is deliberately broader – not “exactly as” that helper does, which is what the spec’s §3.7 claims. A job that sets SLURM_JOB_ID without a per-task CPU count still gets durable writes.

phenotypic.sdk_.embedded_measurement_columns(store_path: Path) → tuple[str, ...][source]

Return the column names one store’s embedded table carries.

The store enumerates its own columns in the descriptor, so this is the authoritative allow-list for anything that projects a single column – and it is what makes a column name a closed value set rather than a free-text parameter that reaches the filesystem.

Parameters:

store_path (Path) – Path to a *.ome.zarr directory.

Returns:

The declared column names, in the order the writer recorded them.

Raises:

OSError – If the store’s root zarr.json does not exist.
KeyError – If the store declares no measurement-table descriptor.

Return type:

tuple[str, …]

phenotypic.sdk_.ensure_metadata_prefix(name: str) → str[source]

Normalize a metadata name to the live schema’s emitted spelling.

Bare labels, canonical flat headers, and exact historical per-topic headers for known members all resolve centrally. Unknown bare labels receive the generic prefix. Unknown canonical metadata headers remain unchanged.

Parameters:: name (str)
Return type:: str

phenotypic.sdk_.ensure_typed_json_suffix(path: str | Path, suffix: str) → Path[source]

Return path with the canonical typed JSON suffix appended.

Bare stems receive the full typed suffix. Legacy .json paths receive only the typed tail, preserving the user-provided stem and case.

Parameters:

path (str | Path)
suffix (str)

Return type:

phenotypic.sdk_.error_analysis_csv_path(output_dir: Path) → Path[source]

Return <output>/deliverables/error_analysis.csv.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.error_analysis_html_path(output_dir: Path) → Path[source]

Return <output>/deliverables/error_analysis.html.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.error_analysis_parquet_path(output_dir: Path) → Path[source]

Return <output>/deliverables/error_analysis.parquet.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.error_category_parquet_path(output_dir: Path, category: str) → Path[source]

Return <output>/deliverables/errors/<category>.parquet.

Parameters:

output_dir (Path) – Run output directory.
category (str) – A bare, already-sanitized category token (e.g. "background_noise"). The caller is responsible for sanitization.

Return type:

phenotypic.sdk_.errors_dir(output_dir: Path) → Path[source]

Return <output>/deliverables/errors/ (per-category error parquets).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.event_log_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/processing_events.log.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.external_metadata_preserved_columns(measurement_columns: Iterable[object], metadata_columns: Iterable[object]) → frozenset[str][source]

Return the external metadata columns that keep their raw names in a join.

An external metadata table is joined onto a mixed-schema measurement frame. Its columns split in two: join keys the measurements already carry, which must keep their spelling or they match nothing, and attributes, which take the live Metadata_ spelling. A column keeps its raw name when it is not metadata-family (no known metadata member, not a Metadata_* header) and it either already exists in the measurement frame or is a known non-metadata schema header such as Grid_RowNum. Every other column is normalized with ensure_metadata_prefix().

Parameters:

measurement_columns (Iterable[object]) – Column names of the measurement frame.
metadata_columns (Iterable[object]) – Column names of the external metadata table.

Returns:

The subset of metadata_columns whose names must not be prefixed.

Return type:

frozenset[str]

phenotypic.sdk_.failures_jsonl_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/progress/failures.jsonl.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.file_fingerprint(path: Path) → str[source]

Return a versioned SHA-256 fingerprint for one file’s contents.

Parameters:: path (Path) – Existing regular file to fingerprint.
Returns:: A "sha256:<hex>" content fingerprint.
Return type:: str

phenotypic.sdk_.finalization_input_object(output_dir: Path) → dict[str, object][source]

Return the versioned finalization-input object (spec §5.5).

The object, not its digest: a caller that needs the digest passes this to phenotypic.sdk_._digests.canonical_digest(), and a caller that needs to explain a mismatch – “the metadata snapshot changed” rather than “the digest changed” – reads the fields.

schema_version is what makes adding a fourth input a reader-side bump instead of a second tree migration.

Parameters:: output_dir (Path) – Run output root.
Returns:: {"schema_version", "metadata_sha256", "include_dataset_column", "no_qc"}. Every value comes from config via .get and is None when the run never recorded it – a shape with no state file at all still returns the four keys.
Return type:: dict[str, object]

phenotypic.sdk_.generalization_path(output_dir: Path) → Path[source]

Return <output>/deliverables/generalization.json — the held-out report.

The winner’s generalization verdict (calibration vs held-out score, the gap, and the pass/fail margin), a user-facing deliverable. The held-out pass that writes it is Phase 4.5 part 2; this helper resolves the canonical location.

Parameters:: output_dir (Path) – The run output directory.
Returns:: <output_dir>/deliverables/generalization.json.
Return type:: Path

phenotypic.sdk_.generation_staging_path(target: Path, generation: str) → Path[source]

Return a sibling staging path for an explicit publication generation.

Raises:

ValueError – If generation is not a safe path component.

Parameters:

target (Path)
generation (str)

Return type:

phenotypic.sdk_.gui_launch_owner_path(output_dir: Path) → Path[source]

Return the canonical GUI launch-generation owner record path.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.has_config_suffix(path: str | Path, suffixes: Iterable[str]) → bool[source]

Return whether path ends with any configured suffix.

Matching is case-insensitive so callers can discover user-provided files from case-preserving filesystems without rewriting their names.

Parameters:

path (str | Path)
suffixes (Iterable[str])

Return type:

bool

phenotypic.sdk_.image_completion_marker_path(output_dir: Path, dataset: str, image_stem: str) → Path[source]

Return the general marker path for one dataset image stem.

Parameters:

output_dir (Path)
dataset (str)
image_stem (str)

Return type:

phenotypic.sdk_.image_record_path(output_dir: Path, dataset: str, image_stem: str) → Path[source]

Return <output>/.phenotypic/progress/images/<ds>/<stem>.json.

Parameters:

output_dir (Path)
dataset (str)
image_stem (str)

Return type:

MetadataMigrationResult

phenotypic.sdk_.is_binary_mask(arr: numpy.ndarray)[source]

Parameters:: arr (numpy.ndarray)

phenotypic.sdk_.is_metadata_header(col: str) → bool[source]

True if col is a metadata-family column.

Matches canonical Metadata_* columns and the exact finite set of historical per-topic headers. Arbitrary lookalikes such as MetadataFoo_Bar are rejected.

Parameters:: col (str)
Return type:: bool

phenotypic.sdk_.is_zarr_store_name(path: Path | str) → bool[source]

Return whether a path name uses a supported Zarr store suffix.

.ome.zarr remains the canonical PhenoTypic output suffix. Generic .zarr names are accepted as inputs so validity can be decided by the NGFF reader at the open/render boundary.

Parameters:: path (Path | str)
Return type:: bool

phenotypic.sdk_.job_metadata_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/progress/job_metadata.json.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.load_image_from_store(store_path: Path, *, fallback: ImageTypeName = 'Image') → _Image | _GridImage[source]

Read phenotypic.image_class from a store root and dispatch the loader.

Dispatches on image_class (Image / GridImage), which is the loader-dispatch field. It is not Metadata_ImageType, which is user-visible schema metadata and may be GridSection on a plain Image.

Bypasses the public :meth:`Image.load_zarr` guard, deliberately. That guard refuses a store carrying no image_class, because a user calling the public verb on such a store has almost certainly mistaken a --mode process export for a run bundle and wants imread instead. This function is the internal dispatcher: its caller supplies fallback and has therefore already made that determination itself (_cli_process_single passes the run’s own image type; the tune CLI passes "GridImage"). Routing through load_zarr would raise before the resolved class could ever be used, making fallback dead code. So the resolved class is asked to load the store directly.

A store with no bundle content still fails, one layer down and by its own error: _load_from_store subscripts the series mapping bare at series["gray"] and series["detect_mat"] (_image_io_handler.py), so a single-series process store raises KeyError: 'detect_mat'.

Parameters:

store_path (Path) – Path to a *.ome.zarr directory.
fallback (ImageTypeName) – Class name used when the block carries no image_class.

Returns:

An Image or GridImage loaded from the store.

Raises:

KeyError – If the store root carries no phenotypic block, or if it carries one but no bundle series.
ValueError – If store_schema_version is not this build’s.

Return type:

_Image | _GridImage

phenotypic.sdk_.logs_dir(output_dir: Path) → Path[source]

Return <output>/.phenotypic/logs/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.manifest_json_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/progress/manifest.json.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.master_carries_user_metadata(frame: pl.DataFrame) → bool[source]

Return whether this master predates the §7.3 inversion.

The one genuinely dangerous failure mode in §7 is a reader that filters or groups a master on a user-metadata column: against a v2 master that returns empty rather than raising. This predicate is what such a reader branches on.

A v1 run that was given no --metadata is indistinguishable from a v2 run by this test, and that is expected to be harmless – neither has anything to join. test_a_v1_metadata_free_master_is_indistinguishable_from_v2 is the designated falsifier for that expectation.

Parameters:: frame (pl.DataFrame) – A master measurements frame.
Returns:: True when the frame carries at least one user-metadata column.
Return type:: bool

phenotypic.sdk_.master_measurements_parquet_path(output_dir: Path) → Path[source]

Return <output>/deliverables/master_measurements.parquet.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.matches_any_suffix(path: str | Path, suffixes: Iterable[str]) → bool[source]

Return whether path ends with any suffix in suffixes.

Parameters:

path (str | Path)
suffixes (Iterable[str])

Return type:

bool

phenotypic.sdk_.measurements_by_feature_dir(output_dir: Path) → Path[source]

Return <output>/deliverables/measurements_by_feature/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.measurements_csv_path(output_dir: Path) → Path[source]

Return <output>/deliverables/measurements.csv (post-applied mirror).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.measurements_parquet_path(output_dir: Path) → Path[source]

Return <output>/deliverables/measurements.parquet (post-applied mirror).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.metadata_category_for_label(label: str) → str | None[source]

Return the shared category for a known label, or None.

Deprecated:: Use metadata_owner_for_label() or metadata_member_for_label() for semantic routing.

Parameters:: label (str)
Return type:: str | None

phenotypic.sdk_.metadata_category_prefixes() → tuple[str, ...][source]

Return the canonical namespace prefix.

Deprecated:: Use is_metadata_header() for namespace detection or owner lookup helpers for semantic routing.

Return type:: tuple[str, …]

phenotypic.sdk_.metadata_csv_deliverable_path(output_dir: Path) → Path[source]

Return <output>/deliverables/metadata.csv (co-located --metadata copy).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.metadata_member_for_header(header: str) → MetadataInfo | None[source]

Return the member for a bare, canonical, or exact historical header.

Parameters:: header (str)
Return type:: MetadataInfo | None

phenotypic.sdk_.metadata_member_for_label(label: str) → MetadataInfo | None[source]

Return the member for a bare, canonical, or exact historical label.

Parameters:: label (str)
Return type:: MetadataInfo | None

phenotypic.sdk_.metadata_migration_authority(source: str | Path | BundleLayout) → MetadataMigrationAuthority[source]

Load and validate the bundle’s published metadata-stage authority.

Parameters:: source (str | Path | BundleLayout)
Return type:: MetadataMigrationAuthority

phenotypic.sdk_.metadata_only_mask(df: pandas.DataFrame) → pandas.Series[source]

Mask of --metadata phantom rows; all-False when unknowable.

A phantom row is one the CLI’s --metadata left join carried through from the metadata CSV even though no measured object matched its key — every measurement/info column on it is null. Those rows are marked with the METADATA_ONLY (QC_MetadataOnly) boolean column.

The flag is CLI-only, so public analysis/post entry points that a user calls on a hand-built or measure() frame see no flag at all. This helper degrades to an all-False mask in that case, which reproduces exactly the pre-left-join behavior for every caller.

The dtype check is deliberately strict: only a real boolean column is trusted. An object/string column is rejected rather than coerced, because pd.Series(["False", "True"]).astype(bool) is [True, True] — the string "False" is truthy — which would silently mark every row a phantom. Rejecting costs nothing (it falls back to today’s behavior); a lenient coercion would corrupt every result. Both real CLI round-trips preserve the dtype: parquet stores a native bool, and polars’ write_csv emits true/false, which pd.read_csv parses to bool.

Parameters:: df (pandas.DataFrame) – Any measurement-shaped DataFrame.
Returns:: True where the row is a metadata-only phantom, False everywhere else (and everywhere when the flag column is absent or not a boolean column).
Return type:: Boolean Series aligned to df.index

Examples

>>> import pandas as pd
>>> from phenotypic.sdk_ import metadata_only_mask
>>> # A frame from a notebook ``image.measure()`` carries no flag.
>>> metadata_only_mask(pd.DataFrame({"Shape_Area": [10.0, 12.0]})).tolist()
[False, False]
>>> # A CLI mirror does: the undetected strain is flagged.
>>> mirror = pd.DataFrame({"QC_MetadataOnly": [False, True]})
>>> metadata_only_mask(mirror).tolist()
[False, True]

phenotypic.sdk_.metadata_owner_for_header(header: str) → type[MetadataInfo] | None[source]

Return the owner for a bare, canonical, or exact historical header.

Parameters:: header (str)
Return type:: type[MetadataInfo] | None

phenotypic.sdk_.metadata_owner_for_label(label: str) → type[MetadataInfo] | None[source]

Return the owner for a bare, canonical, or exact historical label.

Parameters:: label (str)
Return type:: type[MetadataInfo] | None

phenotypic.sdk_.migrate_legacy_machine_state(output_dir: Path) → bool[source]

Move a pre-migration run’s machine-state into .phenotypic/.

If legacy machine-state (progress/, processing_state.json, processing_events.log) is present at the output root, move each artifact into the .phenotypic/ cache so the run proceeds coherently against a single location. A no-op when no legacy state is present or everything is already migrated.

Robust to interruption and concurrency (the SLURM array case): each artifact is moved only when its source still exists and its destination does not, so a migration interrupted mid-move completes on the next call rather than leaving split state; and a lost move race (a concurrent worker moved the artifact first) is ignored rather than crashing. Keying per-artifact instead of on cache.exists() is what makes both safe.

Returns:: True if this call moved anything, else False.
Parameters:: output_dir (Path)
Return type:: bool

phenotypic.sdk_.migrate_legacy_qc(output_dir: Path) → bool[source]

Move a pre-relocation run’s <output>/qc/ into deliverables/qc/.

Hard cutover (MOVE, no duplication), mirroring migrate_legacy_machine_state(). A no-op when there is no legacy qc/ or when the canonical deliverables/qc/ already exists (the move is whole-directory; we never merge a half-written canonical with legacy).

Returns:: True if this call moved the directory, else False.
Parameters:: output_dir (Path)
Return type:: bool

phenotypic.sdk_.migrate_metadata_bundle(source: str | Path | BundleLayout, *, expected_plan_fingerprint: str, kinds: frozenset[str] | None = None, target_role: Literal['bundle_durable', 'bundle_all', 'exact_file'] | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → MetadataMigrationResult[source]

Migrate authoritative sources in a full or standalone bundle.

Re-running after an interruption is safe, by two mechanisms that are worth naming because “pass 1 is idempotent by content” is false – a parquet rewrite is not byte-idempotent and a re-applied rewrite changes every sha256. The real mechanisms are that an existing receipt short-circuits the re-run onto itself, and that an already-canonical bundle returns a compatible no-op that rewrites nothing. An executor who “optimizes” past the receipt check on the strength of the wrong reason breaks marker validity for the whole tree.

Parameters:

source (str | Path | BundleLayout) – Bundle path or resolved BundleLayout.
expected_plan_fingerprint (str) – Fingerprint from the matching preflight.
kinds (frozenset[str] | None) – Restrict the migration to these TargetKind values, and record that scope in the receipt. None means every kind.
target_role (Literal['bundle_durable', 'bundle_all', 'exact_file'] | None) – Explicit bundle ownership role. The migrate CLI passes bundle_durable; None preserves the generic full-bundle API.
commit_guard (Callable[[], AbstractContextManager[None]] | None)

Returns:

The migration result.

Return type:

phenotypic.sdk_.migrate_metadata_file(path: str | Path, *, expected_source_fingerprint: str) → MetadataMigrationResult[source]

Copy-on-write migrate one supported file after optimistic preflight.

Parameters:

path (str | Path)
expected_source_fingerprint (str)

Return type:

MetadataMigrationResult

phenotypic.sdk_.migrate_preflighted_metadata_bundle(source: str | Path | BundleLayout, *, report: MetadataMigrationReport, kinds: frozenset[str] | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → MetadataMigrationResult[source]

Migrate a bundle from one already-computed semantic preflight.

Parameters:

source (str | Path | BundleLayout)
report (MetadataMigrationReport)
kinds (frozenset[str] | None)
commit_guard (Callable[[], AbstractContextManager[None]] | None)

Return type:

MetadataMigrationResult

phenotypic.sdk_.migration_backup_dir(config_path: Path) → Path[source]

Return the dedicated sibling backup directory for a configuration.

Parameters:: config_path (Path)
Return type:: Path

phenotypic.sdk_.migration_backup_path(config_path: Path, *, timestamp: str, source_fingerprint: str) → Path[source]

Return a timestamped, fingerprinted backup path.

Parameters:

config_path (Path)
timestamp (str)
source_fingerprint (str)

Return type:

phenotypic.sdk_.migration_lock_path(config_path: Path) → Path[source]

Return the shared pipeline publication lock used by migrations.

This compatibility alias preserves the original SDK name while ensuring migrations coordinate with every ordinary canonical pipeline writer.

Parameters:: config_path (Path)
Return type:: Path

phenotypic.sdk_.migration_receipt_path(config_path: Path, *, resulting_fingerprint: str) → Path[source]

Return the durable receipt path for one migrated generation.

Parameters:

config_path (Path)
resulting_fingerprint (str)

Return type:

phenotypic.sdk_.named_analysis_csv_path(deliverables_base: Path, analysis_id: str) → Path[source]

Return the named CSV artifact path for analysis_id.

Parameters:

deliverables_base (Path)
analysis_id (str)

Return type:

phenotypic.sdk_.named_analysis_parquet_path(deliverables_base: Path, analysis_id: str) → Path[source]

Return the named Parquet artifact path for analysis_id.

Parameters:

deliverables_base (Path)
analysis_id (str)

Return type:

phenotypic.sdk_.named_analysis_paths(deliverables_base: Path, analysis_id: str) → AnalysisArtifactPaths[source]

Return all persisted paths associated with one analysis ID.

Parameters:

deliverables_base (Path)
analysis_id (str)

Return type:

AnalysisArtifactPaths

phenotypic.sdk_.new_part_path(final: Path) → Path[source]

Return a fresh, uuid-suffixed .part sibling of final.

The uuid – matching the attempt_id = uuid4().hex convention already used in _cli_staged_strategy.py (lines 148, 192, 225, 359) – is what keeps two concurrent writers from interleaving chunks into one directory. It is NOT what makes the promote itself benign; that is the retry loop in promote_store(). An un-suffixed .part would let two concurrent SLURM tasks interleave chunks into one directory and produce a store that validates. A PID is not enough: PIDs are reused.

Parameters:: final (Path)
Return type:: Path

phenotypic.sdk_.normalize_metadata_columns(frame: _FrameT) → _FrameT[source]

Normalize external metadata columns without mutating the input frame.

The input and output frame implementations match. Columns that normalize to one target are coalesced only when their dtypes are compatible and all overlapping non-null values agree. A conflict raises before any caller-owned state is changed.

Parameters:: frame (_FrameT)
Return type:: _FrameT

phenotypic.sdk_.order_measurement_columns(columns: Sequence[str]) → list[str][source]

Canonical measurement-frame column order.

[front metadata] -> [measurements] -> [IMAGE metadata] -> [info block].

Front (user/experimental) metadata is cluster/definition ordered via canonical_metadata_order(); unknown/uncategorized Metadata_* tags fall to the end of the front block alphabetically. The framework IMAGE-owned block is per-image provenance and trails the measurements. The per-object info block (Object_Label + Bbox_* / Grid_*) is detected by name and moves last. Measurements keep their incoming relative order.

Pure over column-name strings, so both the pandas (df[...]) and polars (df.select(...)) paths reuse it.

Parameters:: columns (Sequence[str])
Return type:: list[str]

phenotypic.sdk_.overlays_dir(output_dir: Path) → Path[source]

Return <output>/deliverables/overlays/ — the overlay package root.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.param_importance_path(output_dir: Path) → Path[source]

Return <output>/deliverables/param_importance.json (the report).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.pareto_best_pipeline_path(output_dir: Path, objective: str) → Path[source]

Return deliverables/pareto/best_<objective>.json (a per-axis winner).

The pipeline minimizing cost on the single objective axis of the Pareto front. objective is the objective name as it appears in objectives_json (a scorer-defined label, e.g. "Dice" or a composite child handle "s0").

Parameters:

output_dir (Path) – The run directory.
objective (str) – The objective-axis name (the best_<objective>.json stem).

Returns:

The per-objective best-pipeline path under pareto_dir().

Return type:

phenotypic.sdk_.pareto_dir(output_dir: Path) → Path[source]

Return <output>/deliverables/pareto/ — the multi-objective sub-folder.

Holds a multi-objective tune run’s Pareto front + per-objective best pipelines. A single-objective run never creates it (the back-compat lock).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.pareto_front_parquet_path(output_dir: Path) → Path[source]

Return <output>/deliverables/pareto/pareto_front.parquet (the front).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.pareto_importance_path(output_dir: Path, objective: str) → Path[source]

Return deliverables/pareto/param_importance_<objective>.json.

The per-objective RF-permutation importance report (the multi-objective sibling of param_importance_path()). objective is the objective name as it appears in objectives_json (a scorer-defined label, e.g. "Dice" or a composite child handle "s0").

Parameters:

output_dir (Path) – The run directory.
objective (str) – The objective-axis name (the filename’s <objective> slot).

Returns:

The per-objective importance-report path under pareto_dir().

Return type:

phenotypic.sdk_.paths_fingerprint(paths: Iterable[Path], *, root: Path | None = None) → str[source]

Fingerprint an ordered set of named filesystem entries without writes.

Files include their exact contents, directories include their normalized names, and missing entries are represented explicitly. Sorting by name makes the result independent of caller enumeration order.

Parameters:

paths (Iterable[Path]) – Files to include.
root (Path | None) – Optional anchor used to normalize names.

Returns:

A deterministic "sha256:<hex>" fingerprint.

Return type:

MetadataMigrationReport

phenotypic.sdk_.phenotypic_cache_dir(output_dir: Path) → Path[source]

Return <output>/.phenotypic/ — the hidden machine-state root.

Pure path expression; callers mkdir when they intend to write.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.phenotypic_cache_pipeline_json_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/pipeline.json.pht-pipe — the process-only run’s reproducibility copy. Distinct from pipeline_json_path(), which roots under deliverables/ (process-only writes no deliverables).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.pipeline_json_path(output_dir: Path) → Path[source]

Return the canonical typed pipeline config path under deliverables/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.pipeline_publication_lock(config_path: Path, *, timeout: float = 30.0) → Iterator[None][source]

Serialize a complete read/check/write pipeline transaction.

Callers must hold this lock across both their generation or staleness check and the final atomic replacement. Atomic rename prevents partial files; this lock prevents a valid concurrent generation from being silently overwritten after a check-then-replace race.

Parameters:

config_path (Path) – Canonical pipeline configuration being published.
timeout (float) – Maximum seconds to wait for a competing publisher.

Yields:

None while the shared publication lock is held.

Return type:

Iterator[None]

phenotypic.sdk_.pipeline_publication_lock_path(config_path: Path) → Path[source]

Return the shared interprocess lock path for pipeline publication.

Every production writer of a canonical output pipeline must acquire this lock before checking a source generation or replacing the file. The lock is intentionally about publication, not one particular migration, so CLI, QC, Analysis, and compatibility writers serialize against each other.

Legacy pipeline.json and canonical pipeline.json.pht-pipe paths intentionally map to the same output-level identity. This prevents a V1 reader/writer and a V2 writer from bypassing one another merely because they selected different compatibility filenames.

Parameters:: config_path (Path)
Return type:: Path

phenotypic.sdk_.plots_dir(output_dir: Path) → Path[source]

Return <output>/deliverables/plots/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.preflight_metadata_schema(source: Any, *, kinds: frozenset[str] | None = None, target_role: Literal['bundle_durable', 'bundle_all', 'exact_file'] | None = None) → MetadataMigrationReport[source]

Inspect a frame, supported file, or bundle without changing it.

Writes nothing. That is what makes --mode migrate’s pass-1 dry run free – not incidental, but the mechanism.

Parameters:

source (Any) – pandas/Polars frame, supported file path, run-output path, standalone deliverables path, or resolved BundleLayout.
kinds (frozenset[str] | None) – Restrict bundle discovery to these TargetKind values. None means every kind, so existing callers are unchanged. Ignored for a frame or a single file, which are already one explicit target.
target_role (Literal['bundle_durable', 'bundle_all', 'exact_file'] | None) – Explicit bundle ownership role. bundle_durable uses exact pipeline, aggregate, and standalone-master names without scanning per-image source directories. None preserves the generic bundle API’s complete historical target inventory.

Returns:

Immutable migration plan and compatibility status.

Return type:

phenotypic.sdk_.processing_report_html_path(output_dir: Path) → Path[source]

Return <output>/deliverables/processing_report.html.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.processing_state_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/processing_state.json.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.progress_dir(output_dir: Path) → Path[source]

Return <output>/.phenotypic/progress/.

Pure path expression; callers are responsible for mkdir when they intend to write into it.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.promote_store(part: Path, final: Path, *, fsync: bool, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → Path[source]

Atomically promote a fully written .part directory to final.

The caller is responsible for the write order inside part: all arrays and chunks first, then OME/zarr.json, then the root zarr.json last. An interrupted store therefore has no valid root and reads as absent. This function does not write the root zarr.json itself.

The move-aside is mandatory, not an optimization: os.replace onto a non-empty directory raises OSError (ENOTEMPTY) on POSIX, and on Windows MoveFileEx’s MOVEFILE_REPLACE_EXISTING cannot name a directory at all.

The whole exists -> move-aside -> replace sequence sits inside one retry loop and re-evaluates existence on every attempt. That is what makes duplicate execution benign: a uuid .part prevents two writers interleaving chunks, but it does nothing for the promote itself, where a check-then-act done once lets writer B skip the move-aside because A had not yet renamed, then hit ENOTEMPTY on a now-non-empty target.

On failure after a successful move-aside, that attempt’s trash is reconciled before retrying or raising. The previous store is rolled back when final is absent; if a concurrent writer has already published a new final, that winner remains authoritative and only the attempt’s superseded trash is removed. Every retry uses a fresh UUID trash path, so no attempt can collide with its predecessor’s move-aside directory.

Known weakening versus the single-file rename: the two renames are still not one atomic step, so a crash between them (as opposed to a raised error) leaves the image absent plus an orphaned .trash. Both are recoverable – absence reclassifies to the rebuilding stage, and sweep_orphan_parts() clears the leftovers.

Parameters:

part (Path) – Fully written .part directory.
final (Path) – Target store path.
fsync (bool) – Whether to flush part before renaming (see durable_writes_enabled()).
commit_guard (Callable[[], AbstractContextManager[None]] | None)

Returns:

final.

Return type:

phenotypic.sdk_.publication_commit(commit_guard: Callable[[], AbstractContextManager[None]] | None) → AbstractContextManager[None][source]

Return the narrow context guarding one canonical filesystem mutation.

Parameters:: commit_guard (Callable[[], AbstractContextManager[None]] | None)
Return type:: AbstractContextManager[None]

phenotypic.sdk_.qc_dir(output_dir: Path) → Path[source]

Return <output>/deliverables/qc/ — durable QC + curation state.

Relocated under deliverables/ so a deliverables bundle is self-contained and portable. For reads that must honour the legacy root <output>/qc/ of pre-relocation runs, use BundleLayout.qc_dir, which resolves the same three branches and is the one with callers. A second module-level resolver existed here and was deleted in P6 Task 7 – two implementations of one fallback, only one of them reachable.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.qc_duckdb_path(output_dir: Path) → Path[source]

Return <output>/deliverables/qc/qc.duckdb.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.qc_review_state_path(output_dir: Path) → Path[source]

Return <output>/deliverables/qc/review_state.json (GUI-owned review progress).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.read_embedded_measurement_column(store_path: Path, column: str) → dict[int, float | None][source]

Project one measurement column out of a store’s embedded table.

Returns the column keyed by the descriptor’s own target.column – Object_Label – rather than by a positional index or an assumed key name, so the value a caller paints onto a colony is the value measured for that object. The join key is read from the store; it is never assumed.

column is checked against embedded_measurement_columns() before the Parquet is opened. A name the store does not declare therefore never reaches the filesystem, which is what lets a request-facing caller pass a user-supplied name through without it becoming a probe.

Only two of the table’s ~130 columns are read. Parquet is columnar, so the other 128 are never decoded – that is what makes a per-request projection affordable.

Parameters:

store_path (Path) – Path to a *.ome.zarr directory.
column (str) – Name of the column to project. Must appear in the store’s declared measurement_columns.

Returns:

A mapping {object_label: value}. A null cell maps to None.

Raises:

OSError – If the store’s root zarr.json does not exist.
KeyError – If the store declares no measurement-table descriptor.
ValueError – If column is not one the store declares.
TypeError – If column holds values that are not numbers – a colour hex string, say. Measurement display scales them, and a silent None would hide the mismatch.

Return type:

dict[int, float | None]

phenotypic.sdk_.read_embedded_measurement_descriptor(store_path: Path) → dict[str, object][source]

Return one store’s tables.measurements descriptor.

The descriptor is the store’s own account of its embedded table: the payload path, the measurement_columns list, and the target naming the join column and the label image it indexes. Reading it costs one small JSON parse and never opens the Parquet payload, so a caller that only needs the column list – a column picker, say – pays nothing for the ~130 columns it does not want.

Parameters:

store_path (Path) – Path to a *.ome.zarr directory.

Returns:

The descriptor mapping, exactly as build_measurement_table_descriptor() wrote it.

Raises:

OSError – If the store’s root zarr.json does not exist.
KeyError – If the root carries no phenotypic block, or the block declares no tables.measurements descriptor. An absent descriptor is a normal state, not a fault: a --mode process run never measures, and a store written before embedded tables has none.
ValueError – If the store’s store_schema_version is not this build’s – the same refusal every other content reader makes.

Return type:

dict[str, object]

Examples

>>> import tempfile
>>> from pathlib import Path
>>> from phenotypic import GridImage
>>> from phenotypic.data import load_synth_yeast_plate
>>> img = GridImage(load_synth_yeast_plate())
>>> with tempfile.TemporaryDirectory() as tmp:
...     store = img.save2zarr(Path(tmp) / 'plate.ome.zarr')
...     try:
...         read_embedded_measurement_descriptor(store)
...     except KeyError:
...         print('no embedded table')
no embedded table

phenotypic.sdk_.read_image_record(output_dir: Path, dataset: str, image_stem: str) → dict[str, object] | None[source]

Return one image’s record, or None when it cannot be read.

Every failure returns None rather than raising – INV-VERDICT’s degrade half: an unreadable record must make an image look less finished, never make a caller explode. A truncated file, a JSON array where an object belongs, and an absent file are all the same answer, because a caller that must distinguish them is asking the wrong question of this function.

Parameters:

output_dir (Path) – Run output root.
dataset (str) – Dataset name.
image_stem (str) – Source image stem.

Returns:

The record as a mapping, or None.

Return type:

dict[str, object] | None

phenotypic.sdk_.read_phenotypic_attributes(store_path: Path) → dict[source]

Read the attributes.phenotypic block from a store root.

Parameters:

store_path (Path) – Path to a *.ome.zarr directory.

Returns:

The phenotypic block.

Raises:

FileNotFoundError – If the root zarr.json does not exist.
KeyError – If the root exists but carries no phenotypic block.

Return type:

dict

phenotypic.sdk_.read_run_manifest(output_dir: Path) → dict | None[source]

Read the run manifest if present, resolving legacy layouts.

Reads <output>/.phenotypic/progress/manifest.json, falling back to the pre-migration <output>/progress/manifest.json for legacy runs (via resolve_manifest_json_path()). Replaces 4 inline json.loads(manifest_path.read_text()) blocks.

Parameters:: output_dir (Path) – Run output directory containing progress/.
Returns:: Parsed manifest dict, or None when the file is missing or unparseable (callers can decide whether absence is fatal).
Return type:: dict | None

phenotypic.sdk_.readme_md_path(output_dir: Path) → Path[source]

Return <output>/deliverables/README.md.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.recompile_dir(progress_dir_: Path) → Path[source]

Return <progress>/recompile/.

Parameters:: progress_dir_ (Path)
Return type:: Path

phenotypic.sdk_.recompile_status_dir(progress_dir_: Path) → Path[source]

Return <progress>/recompile/status/.

Parameters:: progress_dir_ (Path)
Return type:: Path

phenotypic.sdk_.record_provenance(record: object) → str[source]

Return a record’s provenance, defaulting to "forward" (U-10).

Absent means forward, and that is the strict reading. A record written before this field existed – or by any writer that forgets it – must be fenced on work_id like any other, so the default is the value that keeps the fence. Defaulting to "migrated", or reading a bare record["provenance"] and treating the KeyError as “unmarked”, would strip the fence from every tree written before P3.

A function rather than the record.get("provenance", "forward") the plan prescribes: the rule is a default, and a default restated at each call site is one edit away from being two defaults. This is the same reason stage3_markers_required is a live gate finding.

Parameters:: record (object) – A record mapping, or anything at all – a non-mapping is treated as unmarked rather than raising, so a caller that already has read_image_record()’s None need not branch twice.
Returns:: PROVENANCE_MIGRATED only when the record says so explicitly; PROVENANCE_FORWARD in every other case.
Return type:: str

phenotypic.sdk_.record_rejection(record: Mapping[str, object], *, work_id: str, dataset: str, image_stem: str) → str | None[source]

Return why record cannot certify this image, or None.

The single implementation of per-image record validity, and the one reason this function exists rather than the check being written twice. _cli_completion.valid_image_success and _run_state’s deep path both ask it, exactly as they both asked marker_rejection() of the marker this record replaces. Splitting them again – after gate finding IMPL-F3 spent a whole increment merging them – would look like progress in a diff and be the same defect returning.

A sentence rather than a bool, because the sentence lands in ImageState.reason and is what makes “which images are missing, and why?” answerable without re-running anything.

Two clauses are worth reading twice:

The ``work_id`` comparison is skipped for a migrated record (U-10). A pre-markers tree never had a work_id to match, so comparing it unconditionally would reject every migrated image. The relaxation is per-record and read through record_provenance(), so an absent or unrecognized value keeps the fence.
A record with no artifacts certifies nothing (CAN-23), and after the collapse that is one missing check away from being wrong. A Stage-2 worker writes stages.stage2 and no artifacts into this same file; before the collapse the two facts lived in two trees and mistaking one for the other was impossible.

Parameters:

record (Mapping[str, object]) – The record mapping, as returned by read_image_record().
work_id (str) – The identity this image is expected to carry.
dataset (str) – The dataset the caller is asking about.
image_stem (str) – The image stem the caller is asking about.

Returns:

A sentence naming the first failed clause, or None when the record may certify this image. Artifact contents are not checked here – that is fenced_artifact_path’s half, kept separate so a caller asking “is this record even about my image?” pays no I/O.

Return type:

str | None

phenotypic.sdk_.rembi_manifest_path(output_dir: Path) → Path[source]

Return <output>/deliverables/rembi.yaml — the REMBI run manifest.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.replace_embedded_measurement_table(store_path: Path, table: PreparedEmbeddedMeasurementTable, *, objmap_target: str | None = None, durable: bool | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → Path[source]

Replace one store’s authoritative table without recomputing pixel arrays.

A complete root-last store transaction whose unchanged files are hard-linked where the platform permits.

Superseded by replace_image_tables() on every forward path. It survives for the one consumer that still reads and rewrites pre-inversion stores byte-exactly – --mode migrate, whose reclaim authority compares a store’s bytes against a joined payload. (--mode recompile was the other, until its per-store rewrite was removed outright on 2026-09-11; it now writes no store byte.) Retire it with the last of those call sites. It deliberately leaves metadata_table and tables.metadata exactly as it found them: it has no metadata payload to describe, and clearing a block it knows nothing about would be a guess.

Parameters:

store_path (Path)
table (PreparedEmbeddedMeasurementTable)
objmap_target (str | None)
durable (bool | None)
commit_guard (Callable[[], AbstractContextManager[None]] | None)

Return type:

phenotypic.sdk_.replace_image_tables(store_path: Path, tables: PreparedImageTables, *, objmap_target: str | None = None, durable: bool | None = None, commit_guard: Callable[[], AbstractContextManager[None]] | None = None) → Path[source]

Refresh one store’s measurement AND metadata tables together.

The --mode measure analogue of the promote-time writer: both tables and the root’s metadata_table block move as one root-last transaction, so the store never certifies a table it does not have or a snapshot it was not built against.

Parameters:

store_path (Path) – A promoted *.ome.zarr store.
tables (PreparedImageTables) – The split payload to write.
objmap_target (str | None) – Store-relative path of the label image the measurement table indexes. None reads it from the store.
durable (bool | None) – fsync before promoting. None auto-detects SLURM.
commit_guard (Callable[[], AbstractContextManager[None]] | None) – Publication guard, checked at the commit point.

Returns:

The store path.

Raises:

ValueError – If the store declares no objmap target.

Return type:

phenotypic.sdk_.resolve_event_log_path(output_dir: Path) → Path[source]

Return the event log sibling of the resolved progress dir.

The event log lives beside progress/ (D14): in .phenotypic/ for a migrated/new run, at the output root for a not-yet-migrated legacy read. Read-only helper for resume/discovery; never mutates the run dir.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.resolve_execution_mode(job_meta: dict | None) → Literal['local', 'slurm'][source]

Extract ExecutionMode from job metadata, defaulting to "local".

Replaces a 5-site copy-paste of the job_meta.get("execution_mode", "local") if job_meta else "local" pattern.

Silent coercion: any value that isn’t exactly "slurm" collapses to "local" — including None (no metadata file), {} (no key), garbage strings (e.g. "validate", ""), and the literal None value. The function never raises. Callers who need to detect an unknown mode and warn / refuse should inspect job_meta directly before calling this helper.

Parameters:: job_meta (dict | None) – Parsed .phenotypic/progress/job_metadata.json content, or None when the file is absent.
Returns:: "local" or "slurm".
Return type:: Literal[‘local’, ‘slurm’]

phenotypic.sdk_.resolve_manifest_json_path(output_dir: Path) → Path[source]

Return <progress>/manifest.json resolving the progress dir for legacy runs.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.resolve_pipeline_config_path(output_dir: Path) → Path[source]

Return the best existing pipeline config path for output_dir.

Resolution prefers the canonical typed path, falls back to legacy pipeline.json when present, and returns the canonical path when neither exists so writers naturally create typed config files.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.resolve_processing_state_path(output_dir: Path) → Path[source]

Return the processing-state file that exists, preferring .phenotypic/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.resolve_progress_dir(output_dir: Path) → Path[source]

Return the progress dir that exists, preferring .phenotypic/.

Read-only helper for resume/discovery so a pre-migration run (progress at the output root) is still found. Falls back to the new location when neither exists (the default for fresh writes).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.resolve_run_state(output_dir: Path, *, depth: Literal['shallow', 'deep'] = 'deep') → RunState[source]

Resolve one run’s completion state (spec §4.3, §9).

Verdict precedence is total and ordered (OPEN-QUESTIONS Q2): complete > active > failed > incomplete. First match wins. contradictory does not exist.

complete outranks active because a run proof covers the current inventory: a live worker at that point is either fenced by restart_epoch or belongs to a new invocation that has already changed the inventory, in which case rule 1 does not fire and this is not the case being decided. active outranks failed so that a failure from a previous attempt cannot mask an attempt currently retrying it.

depth="shallow" re-stats the verification cache’s recorded tuples – tier 1 in process, tier 2 from .phenotypic/verification_cache.json when tier 1 is cold (U-11) – and falls through to a deep pass for any image that is absent from the cache, moved, minted under a different identity, or unreadable. It never yields a positive verdict from a cache entry alone (INV-VERDICT): a cached entry can only ever license skipping a re-verification the caller already performed, and the run-level proofs are re-verified on every call regardless. A pass that deep-verified anything rewrites tier 2; a fully warm one writes nothing.

Parameters:

output_dir (Path) – Run output root. May be any directory, including one this package has never written to.
depth (Literal['shallow', 'deep']) – "deep" re-verifies every declared artifact’s content and repopulates the cache. "shallow" re-stats instead. See spec §9’s caller/depth table.

Returns:

A RunState. Never raises for an unreadable or absent tree – every parse failure degrades toward incomplete (INV-VERDICT’s degrade half). RunState.depth reports the depth actually performed, which for a cold "shallow" call is "deep".

Return type:

RunState

phenotypic.sdk_.resolve_split_assignment_path(output_dir: Path) → Path[source]

Return the split assignment that exists, preferring .pht-tune-cache/.

Read-only resolver mirroring resolve_progress_dir(). The held-out split is checked in the hidden tune cache FIRST, THEN at the legacy output root — a missing split silently RE-DERIVES a fresh held-out partition on resume (a reproducibility / held-out-leak bug), so resume MUST find a legacy-root split.json. Falls back to the new location when neither exists (the default for a fresh derive-and-write).

Parameters:: output_dir (Path) – The run output directory.
Returns:: The split-assignment path that exists, else the new cache location.
Return type:: Path

phenotypic.sdk_.resolve_study_db_path(output_dir: Path) → Path[source]

Return the study DB that exists, preferring .pht-tune-cache/.

Read-only resolver: a relocated run keeps study.db under the hidden tune cache; a legacy run kept it at the output root. Falls back to the new location when neither exists (so a cold sampler restart from a missing study.db is harmless — no migration is performed).

Parameters:: output_dir (Path) – The run output directory.
Returns:: The study DB path that exists, else the new cache location.
Return type:: Path

phenotypic.sdk_.resolve_tuning_spec_path(output_dir: Path) → Path[source]

Return the best existing tuning spec path for output_dir.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.restart_epoch_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/restart_epoch.json.

Pure path expression. The readers and the writer are phenotypic._cli._cli_identity.read_restart_epoch() and bump_restart_epoch() – the writer lives in _cli because spec §5.2 keeps every publisher out of sdk_, and only the path belongs here.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.results_dir(output_dir: Path) → Path[source]

Return <output>/results/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.rollback_metadata_migration(receipt_path: str | Path) → MetadataMigrationResult[source]

Restore every applied target recorded by a migration receipt.

Parameters:: receipt_path (str | Path)
Return type:: MetadataMigrationResult

phenotypic.sdk_.run_completion_marker_path(output_dir: Path) → Path[source]

Return the canonical generation-bearing completion marker path.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.run_identity(output_dir: Path) → RunIdentity | None[source]

Return this output’s run identity, or None when it has no state.

In P1 the tokens are read from the fields today’s writers already produce: processing_generation (still a uuid4().hex until P2), pipeline_sha256, the three finalization inputs, and work_ids. restart_epoch defaults to 0 because P2 introduces its writer. That is what makes this phase independently landable – the reader works on today’s trees, before any writer moves.

scheduler_epoch and owner_generation come from the two liveness records and are deliberately outside RunIdentity.digest(): they are facts about processes, not about configuration, and folding them in would discard the verification cache every time a job is submitted against unchanged work.

Parameters:: output_dir (Path) – Run output root. May be any directory, including one this package has never written to.
Returns:: A RunIdentity, or None when there is no readable processing state. Never raises.
Return type:: RunIdentity | None

phenotypic.sdk_.sentinel_resubmitted_path(progress_dir_: Path) → Path[source]

Return <progress>/sentinel_resubmitted marker file path.

Parameters:: progress_dir_ (Path)
Return type:: Path

phenotypic.sdk_.shard_parquet_filename(shard_id: int) → str[source]

Filename of a per-shard Parquet inside the recompile worker.

Parameters:: shard_id (int) – Zero-based shard index.
Returns:: Filename relative to the recompile shard directory.
Return type:: str

phenotypic.sdk_.slurm_lifecycle_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/progress/slurm_lifecycle.json.

The mutable active-generation fence. _cli_slurm_lifecycle owns every write; this helper exists because the run-state reader must be able to ask who is in flight without importing a writer (INV-LAYER).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.slurm_scripts_dir(output_dir: Path) → Path[source]

Return <output>/.phenotypic/slurm_scripts/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.source_cache_key(source: Path, fingerprint: str) → str[source]

Return an opaque cache key bound to canonical source and content.

Parameters:

source (Path)
fingerprint (str)

Return type:

phenotypic.sdk_.source_image_stem(path: Path) → str[source]

Return the canonical artifact stem for a source image path.

OME-Zarr source images use a double suffix, so their identity strips the complete .ome.zarr suffix. Every other source keeps the standard pathlib.Path.stem contract, including ordinary multi-dot files.

Parameters:: path (Path) – Source image file or OME-Zarr store path.
Returns:: The canonical source-image stem.
Return type:: str

phenotypic.sdk_.source_image_suffix(path: Path) → str[source]

Return the canonical suffix for a source image path.

Parameters:: path (Path) – Source image file or OME-Zarr store path.
Returns:: .ome.zarr for a store source, otherwise the standard final suffix.
Return type:: str

phenotypic.sdk_.store_publication_token(store: Path, *, root_dir_fd: int | None = None) → str | None[source]

Return the root-last token for a PhenoTypic-published store.

PhenoTypic promotes an immutable store by replacing zarr.json last. Its root bytes and file identity therefore identify the complete generation without touching every chunk on GPFS. Inode and ctime close the gap where a byte-identical replacement preserves the old mtime. A generic third-party store has no such publication contract and returns None so the caller uses the conservative recursive snapshot fallback.

Parameters:

store (Path) – Published store path. Used for ordinary path-based inspection.
root_dir_fd (int | None) – Optional held descriptor for the store root. When given, zarr.json is opened relative to that identity with O_NOFOLLOW so a route can keep validation and serving bound to one directory generation.

Returns:

The publication token, or None when the protocol is not declared.

Return type:

str | None

phenotypic.sdk_.store_revision_identity(path: Path) → str[source]

Return a stable revision identity for one OME-Zarr store.

PhenoTypic-published immutable generations use the explicit root-last publication token and touch only zarr.json. Generic third-party stores have no publication invariant, so the conservative fallback hashes framed relative paths, member types, sizes, and nanosecond mtimes twice to reject an unstable snapshot. It intentionally does not read chunk contents: CLI work and completion use a separate content-digest contract.

Parameters:

path (Path) – Existing *.ome.zarr directory.

Returns:

A versioned SHA-256 metadata identity.

Raises:

OSError – If the store is unstable or contains a symlink or another non-regular member.
ValueError – If path is not named as an OME-Zarr store.

Return type:

phenotypic.sdk_.store_stem(store_path: Path) → str[source]

Return the image stem of an *.ome.zarr or *.zarr directory.

Path.stem is WRONG here — it strips one suffix and leaves img.ome, which is a plausible-looking wrong name rather than an error: it propagates into parquet filenames and completion markers, and zarr_store_path(out, ds, "img.ome") then resolves to a store that does not exist, so every image reprocesses forever.

Parameters:: store_path (Path) – A <stem>.ome.zarr or <stem>.zarr directory.
Returns:: The bare stem, e.g. "img" for img.ome.zarr.
Raises:: ValueError – If store_path does not end in .zarr. It raises rather than falling back to .stem, because a silent fallback is exactly the failure being prevented.
Return type:: str

phenotypic.sdk_.sweep_orphan_parts(results_root: Path, *, min_age_seconds: float = 21600) → int[source]

Remove stale orphaned .part / .trash directories.

A uuid identifies the attempt, not whether its process is alive. The staged SLURM engine explicitly assumes stale workers can still be running – that is what assert_active_epoch exists for – and under an array the tasks share one output root and start at different times. A sweep with no liveness signal would rmtree the .part directories its siblings are actively filling, which is the same defect a PID-based sweep has.

Two guards, both required:

age: only directories whose mtime is older than min_age_seconds are removed;
placement: the caller must run this from the controller before any worker is submitted, not from each worker’s start-up (see Phase 3).

The scan is bounded to results/<dataset>/zarr/ rather than recursive: rglob would descend into every store, which is the same ~400k-stat pathology the spec flags for the GUI’s discovery path.

Parameters:

results_root (Path) – The run’s results/ directory.
min_age_seconds (float) – Minimum age before a leftover is considered orphaned.

Returns:

Number of directories removed.

Return type:

int

phenotypic.sdk_.task_status_filename(task_index: int) → str[source]

Filename of a per-task SLURM-recompile status JSON.

Parameters:: task_index (int) – Zero-based recompile task index.
Returns:: Filename relative to <progress>/recompile/status/.
Return type:: str

phenotypic.sdk_.task_status_path(output_dir: Path, task_index: int) → Path[source]

Return <progress>/recompile/status/task_<idx>.json.

Parameters:

output_dir (Path)
task_index (int)

Return type:

phenotypic.sdk_.terminal_failures_jsonl_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/terminal_failures.jsonl.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.timed_execution(func)[source]: Decorator to measure and print the execution time of a function.

phenotypic.sdk_.trials_parquet_path(output_dir: Path) → Path[source]

Return <output>/trials.parquet (the trial journal; output-dir root).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.tune_cache_dir(output_dir: Path) → Path[source]

Return <output>/.pht-tune-cache/ — the tune run’s machine-state root.

The tune-side sibling of phenotypic_cache_dir(). Pure path expression; callers mkdir when they intend to write.

Parameters:: output_dir (Path) – The run output directory.
Returns:: <output_dir>/.pht-tune-cache/.
Return type:: Path

phenotypic.sdk_.tune_cache_journal_path(output_dir: Path) → Path[source]

Return <output>/.pht-tune-cache/journal.log for a tune fleet.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.tune_cache_run_marker_path(output_dir: Path) → Path[source]

Return <output>/.pht-tune-cache/run.json — the tune-run marker.

Written at run START (before any deliverable lands) so a live or finished tune output is GUI-discoverable. See RUN_MARKER_JSON.

Parameters:: output_dir (Path) – The run output directory.
Returns:: <output_dir>/.pht-tune-cache/run.json.
Return type:: Path

phenotypic.sdk_.tune_cache_split_assignment_path(output_dir: Path) → Path[source]

Return <output>/.pht-tune-cache/splits/split.json — the held-out split.

The persisted calibration / held-out partition (plate names + split kind + dataset identity + seed entropy). Read-if-exists-else-derive on resume, so a re-run reuses the original partition regardless of the new master seed. A legacy run wrote it under <output>/splits/; use resolve_split_assignment_path() to read either location.

Parameters:: output_dir (Path) – The run output directory.
Returns:: <output_dir>/.pht-tune-cache/splits/split.json.
Return type:: Path

phenotypic.sdk_.tune_cache_splits_dir(output_dir: Path) → Path[source]

Return <output>/.pht-tune-cache/splits/ — the held-out split folder.

Machine state that must survive a fresh-master rewrite and gate resume, so it lives in the hidden tune cache, not under deliverables_dir(). Pure path expression; callers mkdir when they intend to write.

Parameters:: output_dir (Path) – The run output directory.
Returns:: <output_dir>/.pht-tune-cache/splits/.
Return type:: Path

phenotypic.sdk_.tune_cache_study_db_path(output_dir: Path) → Path[source]

Return <output>/.pht-tune-cache/study.db (the Optuna study DB).

The canonical SQLite-WAL storage for the Optuna-backed OptunaStudyStore when the tune extra is installed, relocated into the hidden tune cache. A legacy run wrote it at the output root; use resolve_study_db_path() to read either location.

Parameters:: output_dir (Path) – The run output directory.
Returns:: <output_dir>/.pht-tune-cache/study.db.
Return type:: Path

phenotypic.sdk_.tuning_spec_path(output_dir: Path) → Path[source]

Return the canonical typed tuning spec path under deliverables/.

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.user_metadata_headers(columns: Iterable[str]) → tuple[str, ...][source]

Return the headers that could only have come from --metadata.

A master’s intrinsic metadata is the identity the image carries about itself: the IMAGE-owned per-image provenance block, plus EXPERIMENT.DATASET, which the CLI inserts from the dataset directory name rather than from any CSV. Everything else in the metadata namespace was joined in from the run’s metadata.csv.

Ownership, not the prefix. Metadata_Strain is a real schema member (GENETIC.STRAIN), so “carries a Metadata_* column” does not separate the two shapes – a v2 master carries Metadata_Dataset and Metadata_ImageName and would be misread as v1 by that test. Namespace detection goes through is_metadata_header() and routing through metadata_owner_for_header(), never through prefix parsing.

Parameters:: columns (Iterable[str]) – Column names of a master frame.
Returns:: The user-metadata headers present, in the order given.
Return type:: tuple[str, …]

phenotypic.sdk_.valid_staged_store(path: Path) → bool[source]

Return whether path holds the image layers Stage 2 requires.

Mirrors valid_staged_hdf case for case:

the root zarr.json parses and carries store_schema_version;
every entry in phenotypic.series and phenotypic.labels opens as a Zarr array group – objmap included, which Stage 1’s zeros write guarantees;
processed level-0 (y, x) extents agree and every extent is non-zero; the full decoded original may differ after geometry-changing pre-ops. A zero-size Zarr array is legal and must not pass.

The exception set is the HDF version’s (OSError, TypeError, ValueError) plus ``KeyError`` – which the attribute lookups need and the HDF version did not – plus ``AttributeError``. The root zarr.json is arbitrary JSON written by anyone, so phenotypic, phenotypic.series, and phenotypic.labels can each come back as a list rather than a mapping (another tool’s store, or a future schema); the .get/.values() calls below then raise AttributeError, which is a rejected store, not a crash in resume classification.

It does not need zarr.errors.BaseZarrError. The spec’s §3.6 argues the opposite (“none of zarr’s error types are ValueError subclasses”); that is inverted. BaseZarrError inherits directly from ``ValueError`` (https://zarr.readthedocs.io/en/stable/api/zarr/errors/), as do MetadataValidationError and every other zarr error except the four IndexError ones, none of which this function can raise. json.JSONDecodeError is likewise a ValueError and FileNotFoundError an OSError, so both are already covered. Keeping the shorter tuple also avoids importing zarr.errors in a function the resume planner calls once per image.

Parameters:: path (Path) – Candidate *.ome.zarr directory.
Returns:: True only for a store Stage 2 can consume.
Return type:: bool

phenotypic.sdk_.validate_analysis_id(analysis_id: str) → str[source]

Validate an analysis ID used as a deliverables artifact stem.

Parameters:

analysis_id (str) – Candidate stable analysis identity.

Returns:

The unchanged validated ID.

Raises:

TypeError – If analysis_id is not a string.
ValueError – If the ID is unsafe or collides with a canonical table.

Return type:

phenotypic.sdk_.validated_published_metadata_migration_targets(receipt_path: str | Path) → tuple[tuple[Path, str, str], ...][source]

Return receipt-certified artifact fingerprint transitions.

A target is returned only after the complete receipt and its current disk state pass the migration engine’s normal validation. prepared targets whose atomic replacement reached disk before the receipt state update are included when their current bytes match the prepared post fingerprint.

Parameters:

receipt_path (str | Path) – Durable metadata-migration receipt to validate.

Returns:

Tuples of (path, source_fingerprint, post_fingerprint) for published target transitions.

Raises:

OSError – The receipt or one of its targets cannot be read.
ValueError – The receipt, its authority scope, or current target bytes fail validation.

Return type:

tuple[tuple[Path, str, str], …]

phenotypic.sdk_.verification_cache_path(output_dir: Path) → Path[source]

Return <output>/.phenotypic/verification_cache.json.

Pure path expression; the caller decides whether to write. Note that the cache’s writer deliberately does not mkdir this path’s parent – see phenotypic.sdk_._verification_cache.persist_states().

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.verified_parquet_path(output_dir: Path) → Path[source]

Return <output>/deliverables/verified.parquet (GUI-written, §9).

Parameters:: output_dir (Path)
Return type:: Path

phenotypic.sdk_.write_embedded_measurement_table(store_part: Path, table: PreparedEmbeddedMeasurementTable) → Path[source]

Write the prepared Parquet payload and its two Zarr v3 groups.

Parameters:

store_part (Path)
table (PreparedEmbeddedMeasurementTable)

Return type:

phenotypic.sdk_.write_image_tables(store_part: Path, tables: PreparedImageTables, *, objmap_target: str) → dict[str, object][source]

Write both of one image’s tables into a part and return the fragment.

Parameters:

store_part (Path) – An unpromoted *.ome.zarr.part directory. Both tables land here, before the root zarr.json (D-A).
tables (PreparedImageTables) – The split payload.
objmap_target (str) – Store-relative path of the label image the measurement table’s Object_Label column indexes.

Returns:

The attributes.phenotypic fragment to apply to the root document with apply_image_tables_attributes().

Return type:

dict[str, object]

phenotypic.sdk_.write_metadata_table(store_part: Path, tables: PreparedImageTables) → Path[source]

Write tables/metadata/pht-metadata.parquet and its Zarr v3 group.

The metadata analogue of write_embedded_measurement_table(), and called from the same place: the store’s own .part, before the root zarr.json. That is what makes D-A’s backfill step unnecessary – no path writes into a store that already carries a content proof.

The payload is self-describing from the file alone (spec §7.2): its Parquet key/value metadata names the join keys, the join kind, and the metadata snapshot it was projected from, so a third party who opens only this Parquet can still say what it is and how it attaches.

Parameters:

store_part (Path) – An unpromoted *.ome.zarr.part directory.
tables (PreparedImageTables) – The split payload. tables.metadata must not be None.

Returns:

The written Parquet path.

Raises:

ValueError – If tables carries no metadata frame.

Return type:

phenotypic.sdk_.zarr_store_path(output_dir: Path, dataset: str, stem: str) → Path[source]

Return <output>/results/<dataset>/zarr/<stem>.ome.zarr/.

The single place .ome.zarr is joined to an image stem. Callers must never hand-join the suffix, and must take the stem back off a store with store_stem() rather than Path.stem. Both rules are enforced by tests/unit/test_ome_zarr_invariants.py (test_store_suffix_is_joined_in_exactly_one_place and test_path_stem_is_never_taken_of_a_store_directory).

Parameters:

output_dir (Path) – Run output root.
dataset (str) – Dataset name.
stem (str) – Image filename without extension.

Returns:

The per-image store path. Existence is not checked.

Return type: