phenotypic.sdk_ package#

Developer tools shared across fungal colony plate workflows.

Lightweight helpers for timing, mask validation, constants, color conversions, error handling, and HDF storage used by the processing pipeline. Includes a timed execution decorator, mask validators, colourspace utilities, custom exceptions, and HDF helpers for persisting plate datasets and measurements.

Advanced users can access GridInferenceMixin and FootprintMixin for creating custom grid-based operations and morphological footprints.

The register submodule provides registry utilities for extensible components like plotters and analysis plugins.

The _io_constants submodule is the single source of truth for CLI ↔ GUI artifact filenames, directory names, JSON contract keys, and path helpers (re-exported here at package level for convenience).

class phenotypic.sdk_.ChunkManifestKey[source]#

Bases: object

Keys inside <output>/progress/chunk_manifest.json.

CHUNKS: Final[str] = 'chunks'#
DATASETS: Final[str] = 'datasets'#
NAME: Final[str] = 'name'#
ROWS: Final[str] = 'rows'#
TOTAL_ROWS: Final[str] = 'total_rows'#
class phenotypic.sdk_.ChunkStateKey[source]#

Bases: object

Keys inside <output>/progress/chunk_state.json.

CHUNKED_FILES: Final[str] = 'chunked_files'#
NEXT_CHUNK_ID: Final[str] = 'next_chunk_id'#
class phenotypic.sdk_.ClipControlMixin[source]#

Bases: object

Mixin for operations that need to control clipping behavior of inner operations.

Provides a method to create copies of ImageEnhancer or ImagePipeline instances with clipping disabled. This is useful for composite operations where an inner enhancer operates on non-normalized data (e.g., variance-stabilized values from the Generalized Anscombe Transform, typically in the range ~1-32).

The mixin uses duck typing to check for a clip attribute on operations. If an operation has clip=True, the _disable_clipping method will create a shallow copy with clip=False. This preserves the original operation unchanged while allowing the copy to operate without output clipping.

Example

Creating a clip-disabled copy of an enhancer:

>>> from phenotypic.sdk_ import ClipControlMixin
>>> from phenotypic.enhance import LocalEdgeDenoise
>>>
>>> enh = LocalEdgeDenoise(sigma_spatial=5, clip=True)
>>> copied = ClipControlMixin._disable_clipping(enh)
>>> # Original unchanged, copy has clip=False
>>> enh.clip, copied.clip
(True, False)

Creating a clip-disabled copy of a pipeline:

>>> from phenotypic import ImagePipeline
>>> from phenotypic.enhance import GaussianBlur, LocalEdgeDenoise
>>>
>>> pipeline = ImagePipeline(pipe_cfgs=[
...     GaussianBlur(sigma=1.0),
...     LocalEdgeDenoise(sigma_spatial=5, clip=True)
... ])
>>> copied_pipe = ClipControlMixin._disable_clipping(pipeline)
>>> # Only LocalEdgeDenoise has clip attribute, so only it is affected
>>> # _ops is a dict with operation names as keys
>>> list(copied_pipe._ops.values())[1].clip
False
class phenotypic.sdk_.DashboardManifestKey[source]#

Bases: object

Keys inside <output>/progress/manifest.json (dashboard manifest).

The manifest is built by _cli._dashboard._manifest_builder.build_manifest() and consumed by both the dashboard JS and the GUI run-console’s runs registry (_runs_registry.py). Writers and readers must reference these constants rather than spelling the bare string.

ANALYSIS_DATA_VERSION: Final[str] = 'analysis_data_version'#
COMPLETED: Final[str] = 'completed'#
DATASETS: Final[str] = 'datasets'#
EXECUTION_MODE: Final[str] = 'execution_mode'#
FAILED: Final[str] = 'failed'#
FAILURE_CATEGORIES: Final[str] = 'failure_categories'#
INPUT_PATH: Final[str] = 'input_path'#
IS_COMPLETE: Final[str] = 'is_complete'#
LAST_UPDATED: Final[str] = 'last_updated'#
PENDING: Final[str] = 'pending'#
SLURM_INFO: Final[str] = 'slurm_info'#
STARTED: Final[str] = 'started'#
START_TIME: Final[str] = 'start_time'#
SUCCESS_RATE: Final[str] = 'success_rate'#
TOTAL_IMAGES: Final[str] = 'total_images'#
VERSION: Final[str] = 'version'#
class phenotypic.sdk_.DashboardManifestSlurmInfoKey[source]#

Bases: object

Keys inside the slurm_info sub-dict of the dashboard manifest.

Distinct from JobMetadataKey, even when string values overlap — these describe the manifest contract, not the job-metadata sidecar.

ACTIVE_CHUNKS: Final[str] = 'active_chunks'#
CHUNK_JOB_IDS: Final[str] = 'chunk_job_ids'#
CHUNK_SCRIPTS: Final[str] = 'chunk_scripts'#
COMPLETED_CHUNKS: Final[str] = 'completed_chunks'#
PENDING_CHUNKS: Final[str] = 'pending_chunks'#
TOTAL_CHUNKS: Final[str] = 'total_chunks'#
class phenotypic.sdk_.EnvVar[source]#

Bases: object

Environment variable names read or set by the CLI.

SLURM injects these into batch scripts; the CLI reads them to discover its execution context (job id, array task id, …) and to find node-local scratch storage.

SCRATCH: Final[str] = 'SCRATCH'#
SLURM_ARRAY_JOB_ID: Final[str] = 'SLURM_ARRAY_JOB_ID'#
SLURM_ARRAY_TASK_COUNT: Final[str] = 'SLURM_ARRAY_TASK_COUNT'#
SLURM_ARRAY_TASK_ID: Final[str] = 'SLURM_ARRAY_TASK_ID'#
SLURM_CPUS_PER_TASK: Final[str] = 'SLURM_CPUS_PER_TASK'#
SLURM_JOB_ID: Final[str] = 'SLURM_JOB_ID'#
SLURM_MEM_PER_NODE: Final[str] = 'SLURM_MEM_PER_NODE'#
class phenotypic.sdk_.FootprintMixin[source]#

Bases: object

Provides a mixin for creating morphological footprints for image processing.

The FootprintMixin class contains a static utility method to generate structuring elements (footprints) used in various image processing tasks. This functionality is particularly helpful in the context of analyzing microbial colonies on solid media agar plates. Morphological footprints are used to highlight specific features in images, such as colony edges, shapes, or connectivity, and can assist in segmentation, noise reduction, and feature extraction.

None#
class phenotypic.sdk_.GridInferenceMixin[source]#

Bases: object

Mixin providing grid inference capabilities from binary masks.

Provides static methods for inferring grid structure from colony patterns using peak detection on row/column projections. Used by detectors and refiners that work with gridded plate images (96-well, 384-well formats, pinned cultures).

All methods are static to support parallelization in pipeline operations.

This is an internal utility for advanced users creating custom grid-based operations. Most users should use RoundPeaksDetector for detection or GridAlignmentRefiner for post-detection refinement directly.

class phenotypic.sdk_.HDF(filepath, name: str, mode: Literal['single', 'set'])[source]#

Bases: object

Represents an interface to manage HDF5 files with support for single or set image modes, and ensures safe and compatible file access with retry and error-handling mechanisms.

The class facilitates operations on HDF5 files commonly used for storing phenotypic data in both single image and image set modes. This class includes utilities to handle locking errors and ensure compatibility by initializing proper HDF5 modes while providing safe access methods for writing.

Parameters:
filepath#

Path to the HDF5 file on the filesystem.

Type:

Path

name#

Name associated with the HDF5 resource, often used as an identifier.

Type:

str

mode#

Specifies the mode for the HDF5 file, either single image or image set.

Type:

Literal[‘single’, ‘set’]

root_posix#

The root path for the HDF5 resource, determined by the mode.

Type:

str

home_posix#

The specific root directory of the HDF5 resource in the file, derived based on its mode.

Type:

str

set_data_posix#

The subgroup path for the data entity in image set mode, if applicable.

Type:

str, optional

SINGLE_IMAGE_ROOT_POSIX#

Base path for single image mode.

Type:

str

IMAGE_SET_ROOT_POSIX#

Base path for image set mode.

Type:

str

IMAGE_SET_DATA_POSIX#

Subgroup marker for image set data.

Type:

str

EXT#

Set of valid file extensions used to recognize HDF5 files.

Type:

set

IMAGE_MEASUREMENT_SUBGROUP_KEY#

Key for accessing measurements in an image’s group.

Type:

str

IMAGE_STATUS_SUBGROUP_KEY#

Key for accessing statuses in an image’s group.

Type:

str

static assert_swmr_on(g: Group) None[source]#

Assert that SWMR mode is enabled on the group’s file.

Parameters:

g (Group) – HDF5 group to check.

Raises:

RuntimeError – If SWMR mode is not enabled.

Return type:

None

static close_handle(handle: File | Group) None[source]#
Parameters:

handle (File | Group)

Return type:

None

static get_group(handle: File, posix) Group[source]#

Retrieves or creates a group in an HDF5 file.

This method checks the validity of the provided HDF5 file handle and tries to retrieve the specified group based on the given posix path. If the group does not exist and the file is not opened in read-only mode, the group gets created. If the file is in read-only mode and the group does not exist, an error is raised.

Parameters:
  • handle (h5py.File) – The HDF5 file handle to operate on.

  • posix (str) – The posix path of the group to retrieve or create in the HDF5 file.

Returns:

The corresponding h5py group within the HDF5 file.

Return type:

h5py.Group

Raises:
  • ValueError – If the HDF5 file handle is invalid or no longer valid.

  • ValueError – If the file handle mode cannot be determined.

  • KeyError – If the specified group does not exist in read-only mode.

static get_uncompressed_sizes_for_group(group: Group) tuple[dict[str, int], int][source]#

Recursively collect the uncompressed (logical) sizes of SWMR-compatible datasets.

This function walks the provided HDF5 group and inspects every dataset without reading any data. For each dataset that is compatible with SWMR writing rules (i.e., chunked layout and no variable-length data types), it computes the uncompressed size in bytes as: dtype.itemsize * number_of_elements.

Notes

  • This works regardless of whether datasets are stored compressed on disk; the reported size is the logical size when uncompressed in memory.

  • Variable-length strings (and datasets containing variable-length fields) are excluded because they are not SWMR-write friendly and their uncompressed size cannot be determined from metadata alone.

  • The operation is safe under SWMR: it only reads object metadata, creates no new refine, and does not modify the file.

Parameters:

group (Group) – The root h5py.Group to traverse.

Returns:

  • sizes: dict mapping absolute dataset paths (e.g., ‘/grp/ds’) to uncompressed size in bytes.

  • total_bytes: sum of all values in sizes.

Return type:

A tuple (sizes, total_bytes) where

static load_frame(group: Group, *, require_swmr: bool = False) DataFrame[source]#

Load a pandas DataFrame from HDF5 storage.

Parameters:
  • group (Group) – HDF5 group containing the DataFrame data.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Returns:

Reconstructed pandas DataFrame.

Raises:
Return type:

DataFrame

static load_series(group: Group, *, dataset: str = 'values', index_dataset: str = 'index', require_swmr: bool = False) Series[source]#

Load a pandas Series from HDF5 storage.

Reconstructs the Series with original name, index names, order, and missingness. Respects logical length from attributes.

Parameters:
  • group (Group) – HDF5 group containing the Series data.

  • dataset (str) – Name of the values dataset.

  • index_dataset (str) – Name of the index dataset.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Returns:

Reconstructed pandas Series.

Raises:
Return type:

Series

static preallocate_frame_layout(group: Group, dataframe: DataFrame, *, chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100, require_swmr: bool = False) None[source]#

Preallocate HDF5 layout for a pandas DataFrame without writing data.

Creates layout for shared index and column series using Series preallocation.

Parameters:
  • group (Group) – HDF5 group to write to.

  • dataframe (DataFrame) – pandas DataFrame to create layout for.

  • chunks (tuple[int, ...]) – Chunk shape for datasets.

  • compression (str) – Compression algorithm.

  • preallocate (int) – Initial allocation size.

  • string_fixed_length (int) – Character length for fixed-length strings.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Raises:
  • RuntimeError – If require_swmr=True and SWMR mode not enabled. If group.file.swmr_mode is True and datasets don’t exist.

  • ValueError – If DataFrame validation fails.

Return type:

None

static preallocate_series_layout(group: Group, series: Series, *, dataset: str = 'values', index_dataset: str = 'index', chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100) None[source]#

Preallocate HDF5 layout for a pandas Series without writing data.

Creates resizable, chunked, compressed datasets with initial shape (preallocate,) and maxshape (None,). Initializes masks to zeros and sets len=0.

Parameters:
  • group (Group) – HDF5 group to write to.

  • series (Series) – pandas Series to create layout for (used for schema).

  • dataset (str) – Name for the values dataset.

  • index_dataset (str) – Name for the index dataset.

  • chunks (tuple[int, ...]) – Chunk shape for datasets.

  • compression (str) – Compression algorithm.

  • preallocate (int) – Initial allocation size.

  • string_fixed_length (int) – Character length for fixed-length strings.

Raises:
  • RuntimeError – If require_swmr=True and SWMR mode not enabled. If group.file.swmr_mode is True and datasets don’t exist.

  • ValueError – If series validation fails.

Return type:

None

static save_array2hdf5(group, array, name, **kwargs)[source]#

Saves a given numpy array to an HDF5 group. If a dataset with the specified name already exists in the group, it checks if the shapes match. If the shapes match, it updates the existing dataset; otherwise, it removes the existing dataset and creates a new one with the specified name. If a dataset with the given name doesn’t exist, it creates a new dataset.

Parameters:
  • group – h5py.Group The HDF5 group in which the dataset will be saved.

  • array – numpy.ndarray The data array to be stored in the dataset.

  • name – str The name of the dataset within the group.

  • **kwargs – dict Additional keyword arguments to pass when creating a new dataset.

static save_frame_append(group: Group, dataframe: DataFrame, *, require_swmr: bool = True) None[source]#

Append a pandas DataFrame to existing HDF5 datasets.

Parameters:
  • group (Group) – HDF5 group containing existing datasets.

  • dataframe (DataFrame) – pandas DataFrame to append.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Raises:
  • RuntimeError – If require_swmr=True and SWMR mode not enabled.

  • ValueError – If validation fails or schema mismatch.

Return type:

None

static save_frame_new(group: Group, dataframe: DataFrame, *, chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100, require_swmr: bool = False) None[source]#

Create datasets and write a pandas DataFrame to HDF5.

Parameters:
  • group (Group) – HDF5 group to write to.

  • dataframe (DataFrame) – pandas DataFrame to persist.

  • chunks (tuple[int, ...]) – Chunk shape for new datasets.

  • compression (str) – Compression algorithm for new datasets.

  • preallocate (int) – Initial allocation size for new datasets.

  • string_fixed_length (int) – Character length for fixed-length strings.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Raises:
  • RuntimeError – If require_swmr=True and SWMR mode not enabled.

  • ValueError – If DataFrame validation fails.

Return type:

None

static save_frame_update(group: Group, dataframe: DataFrame, *, start: int = 0, require_swmr: bool = True) None[source]#

Update a pandas DataFrame in HDF5 at specified position.

Parameters:
  • group (Group) – HDF5 group containing existing datasets.

  • dataframe (DataFrame) – pandas DataFrame to write.

  • start (int) – Starting position for the update.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Raises:
  • RuntimeError – If require_swmr=True and SWMR mode not enabled.

  • ValueError – If validation fails or schema mismatch.

Return type:

None

static save_series_append(group: Group, series: Series, *, dataset: str = 'values', index_dataset: str = 'index', require_swmr: bool = True) None[source]#

Append a pandas Series to existing HDF5 datasets.

Appends at the end using current logical length. Resizes datasets if needed and updates logical length.

Parameters:
  • group (Group) – HDF5 group containing existing datasets.

  • series (Series) – pandas Series to append.

  • dataset (str) – Name of the values dataset.

  • index_dataset (str) – Name of the index dataset.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Raises:
  • RuntimeError – If require_swmr=True and SWMR mode not enabled.

  • ValueError – If validation fails or schema mismatch.

Return type:

None

static save_series_new(group: Group, series: Series, *, dataset: str = 'values', index_dataset: str = 'index', chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100, require_swmr: bool = False) None[source]#

Create datasets and write a pandas Series to HDF5.

Creates new datasets or reuses existing preallocated layout. Writes the first len(series) elements and sets logical length.

Parameters:
  • group (Group) – HDF5 group to write to.

  • series (Series) – pandas Series to persist.

  • dataset (str) – Name for the values dataset.

  • index_dataset (str) – Name for the index dataset.

  • chunks (tuple[int, ...]) – Chunk shape for new datasets.

  • compression (str) – Compression algorithm for new datasets.

  • preallocate (int) – Initial allocation size for new datasets.

  • string_fixed_length (int) – Character length for fixed-length strings.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Raises:
Return type:

None

static save_series_update(group: Group, series: Series, *, start: int = 0, dataset: str = 'values', index_dataset: str = 'index', require_swmr: bool = True) None[source]#

Update a pandas Series in HDF5 at specified position.

Overwrites [start:start+len(series)] and updates logical length to the largest contiguous written extent.

Parameters:
  • group (Group) – HDF5 group containing existing datasets.

  • series (Series) – pandas Series to write.

  • start (int) – Starting position for the update.

  • dataset (str) – Name of the values dataset.

  • index_dataset (str) – Name of the index dataset.

  • require_swmr (bool) – If True, assert SWMR mode is enabled.

Raises:
  • RuntimeError – If require_swmr=True and SWMR mode not enabled.

  • ValueError – If validation fails or schema mismatch.

Return type:

None

__init__(filepath, name: str, mode: Literal['single', 'set'])[source]#

Initializes a class instance to manage HDF5 file structures for single or set image data based on the given filepath, name of the resource, and operational mode.

filepath#

Path to the HDF5 file.

Type:

Path

name#

Identifier for the resource within the HDF5 file.

Type:

str

mode#

Operational mode determining the structure and organization within the HDF5 file. Must be either ‘single’ or ‘set’.

Type:

Literal[‘single’, ‘set’]

root_posix#

Posix path representing the root directory within the HDF5 file based on the mode.

Type:

str

home_posix#

Posix path representing the home directory for the resource within the HDF5 file based on the mode.

Type:

str

set_data_posix#

Posix path for the data subdirectory within the resource home directory. Only initialized in ‘set’ mode.

Type:

Optional[str]

Parameters:
  • filepath – Path to the target HDF5 file. Must have an HDF5-compatible extension, or a ValueError is raised.

  • name (str) – Name of the resource to be managed in the file. Used to construct the home directory for the resource within the HDF5 file.

  • mode (Literal['single', 'set']) – Operational mode. Specifies whether the resource represents a ‘single’ or ‘set’ image data. If the mode is invalid, a ValueError is raised.

Raises:
  • ValueError – If the filepath does not have an HDF5-compatible extension.

  • ValueError – If the mode is neither ‘single’ nor ‘set’.

get_data_group(handle)[source]#
get_home(handle)[source]#

Retrieves a specific group from an HDF file corresponding to single image data.

This method is used to fetch a predefined group from an HDF container, where the group is identified by a constant key related to single image data. The function provides a static interface allowing invocation without requiring an instance of the class.

Parameters:

handle – The HDF file handle from which the group should be retrieved.

Returns:

The group corresponding to single image data, retrieved based on the defined SINGLE_IMAGE_ROOT_POSIX.

Raises:
  • Appropriate exceptions may be raised by the underlying HDF.get_group() method,

  • based on the implementation and provided handle or key.

get_image_group(handle, image_name)[source]#
get_image_measurement_subgroup(handle, image_name)[source]#
get_protected_metadata_subgroup(handle: File, image_name: str) Group[source]#
Parameters:
  • handle (File)

  • image_name (str)

Return type:

Group

get_public_metadata_subgroup(handle: File, image_name: str) Group[source]#
Parameters:
  • handle (File)

  • image_name (str)

Return type:

Group

get_root_group(handle) Group[source]#
Return type:

Group

get_status_subgroup(handle, image_name)[source]#
reader() File[source]#
Return type:

File

safe_writer() File[source]#

Returns a writer object that provides safe and controlled write access to an HDF5 file at the specified filepath or creates it if it doesn’t exist. Ensures that the file uses the ‘latest’ version of the HDF5 library for compatibility and performance.

Handles HDF5 file locking conflicts by attempting to clear consistency flags and retrying file opening with exponential backoff.

Returns:

A file writer object with append mode and ‘latest’ library version enabled.

Return type:

h5py.File

Raises:

OSError – If file cannot be opened after all retry attempts.

strict_writer() File[source]#

Provides access to an HDF5 file in read/write mode using the h5py library. This property is used to obtain an h5py.File object configured with the latest library version.

Note

If using SWMR mode, don’t forget to enable SWMR mode:

>>> hdf = HDF(filepath)
>>> with hdf.writer as writer:
...     writer.swmr_mode = True
...     # rest of your code
Returns:

An HDF5 file object opened in ‘r+’ mode, enabling reading and writing.

Return type:

h5py.File

Raises:

OSError – If the file cannot be opened or accessed.

swmr_reader() File[source]#
Return type:

File

swmr_writer() File[source]#

Returns a writer object that provides safe SWMR-compatible write access to an HDF5 file. Creates the file if it doesn’t exist and enables SWMR mode properly.

This method ensures proper SWMR mode initialization by creating the file with the correct settings from the start, avoiding cache conflicts that occur when trying to enable SWMR mode after opening.

Returns:

A file writer object with SWMR mode enabled.

Return type:

h5py.File

Raises:

OSError – If file cannot be opened after all retry attempts.

EXT = {'.h5', '.hdf', '.hdf5', '.he5'}#
IMAGE_MEASUREMENT_SUBGROUP_KEY = 'measurements'#
IMAGE_SET_DATA_POSIX = 'data'#
IMAGE_SET_ROOT_POSIX = '/phenotypic/image_sets/'#
IMAGE_STATUS_SUBGROUP_KEY = 'status'#
PROTECTED_METADATA_SUBGROUP_KEY = 'protected_metadata'#
PUBLIC_METADATA_SUBGROUP_KEY = 'public_metadata'#
SINGLE_IMAGE_ROOT_POSIX = '/phenotypic/images/'#
class phenotypic.sdk_.HdfAttr[source]#

Bases: object

Top-level attribute keys on per-image HDF5 files.

PHENOTYPIC_CLASS: Final[str] = 'phenotypic_class'#
class phenotypic.sdk_.JobMetadataKey[source]#

Bases: object

Keys inside <output>/progress/job_metadata.json.

Writers (CLI execution strategies) and readers (recompile worker, sentinel, checkpoint handler, GUI runs registry) must reference these constants — never the bare string. Renaming a key here should fail fast at every site.

CHUNK_JOB_IDS: Final[str] = 'chunk_job_ids'#
CHUNK_SCRIPTS: Final[str] = 'chunk_scripts'#
DATASETS: Final[str] = 'datasets'#
EXECUTION_MODE: Final[str] = 'execution_mode'#
IMAGE_TASK_MAPPING: Final[str] = 'image_task_mapping'#
INCLUDE_DATASET_COLUMN: Final[str] = 'include_dataset_column'#
INPUT_PATH: Final[str] = 'input_path'#
METADATA_CSV: Final[str] = 'metadata_csv'#
NO_QC: Final[str] = 'no_qc'#

Whether the recompile finalizer task should skip QC compute. Set on the SLURM recompile finalizer task dict alongside METADATA_CSV; read by _cli_recompile_worker._run_post_master_steps.

SLURM_JOB_IDS: Final[str] = 'slurm_job_ids'#
START_TIME: Final[str] = 'start_time'#
class phenotypic.sdk_.LazyWidgetMixin[source]#

Bases: object

Mixin providing a lazy ipywidget interface.

This mixin allows ImageOperation classes to automatically generate a Jupyter widget interface for parameter tuning and visualization.

The six lazily-populated UI handles this mixin uses (_ui, _param_widgets, _view_dropdown, _update_button, _output_widget, _image_ref) are declared as PrivateAttr on ImageOperation rather than here. A plain (non-BaseModel) mixin’s PrivateAttr declarations are not collected by pydantic, so this mixin stays a stateless methods-only class and the private attrs live on the pydantic model that mixes it in. This keeps the mixin safe to combine with any BaseModel in any MRO position.

__getstate__()[source]#

Prepare the object for pickling by disposing of any widgets.

This ensures that UI components (which may contain unpickleable objects like input functions or thread locks) are cleaned up before serialization.

Note

This method modifies the object state by calling dispose_widgets(). Any active widgets will be detached from the object.

widget(image: Image | None = None, show: bool = False) Widget[source]#

Return (and optionally display) the root widget.

Parameters:
  • image (Image | None) – Optional image to visualize. If provided, visualization controls will be added to the widget.

  • show (bool) – Whether to display the widget immediately. Defaults to False.

Returns:

The root widget.

Return type:

ipywidgets.Widget

Raises:

ImportError – If ipywidgets or IPython are not installed.

class phenotypic.sdk_.ModulePath[source]#

Bases: object

Importable module paths used in dynamic importlib.import_module dispatch.

Spelled out here so a renamed sub-package fails at type-check time (consumers reference ModulePath.POST — a typo there is caught by mypy) rather than silently at runtime.

ANALYSIS: Final[str] = 'phenotypic.analysis'#
POST: Final[str] = 'phenotypic.post'#
class phenotypic.sdk_.ProcessingStateKey[source]#

Bases: object

Keys inside <output>/processing_state.json (resume-state file).

Distinct from JobMetadataKey even where string values overlap (e.g. EXECUTION_MODE, INPUT_PATH) — these describe the processing_state.json contract, not the SLURM job metadata sidecar. Some values intentionally match across the two contracts so that a single field (like execution_mode) can be migrated atomically; the test_processing_state_keys_match_job_metadata_keys regression test asserts the overlap.

COMPLETED: Final[str] = 'completed'#
CONFIG: Final[str] = 'config'#
DATASETS: Final[str] = 'datasets'#
ERRORS: Final[str] = 'errors'#
EXECUTION_MODE: Final[str] = 'execution_mode'#
FAILED: Final[str] = 'failed'#
INITIAL_IMAGES: Final[str] = 'initial_images'#
INPUT_PATH: Final[str] = 'input_path'#
LAST_UPDATED: Final[str] = 'last_updated'#
OUTPUT_DIR: Final[str] = 'output_dir'#
PIPELINE_PATH: Final[str] = 'pipeline_path'#
STARTED: Final[str] = 'started'#
TIMESTAMP: Final[str] = 'timestamp'#
VERSION: Final[str] = 'version'#
phenotypic.sdk_.analysis_csv_path(output_dir: Path) Path[source]#

Return <output>/deliverables/analysis.csv.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.analysis_full_parquet_path(progress_dir_: Path) Path[source]#

Return <progress>/analysis_full.parquet (incremental combined frame).

Takes a progress_dir (not the run output root) since this file lives inside progress/ — it’s an internal mid-run artifact, not a user- facing output.

Parameters:

progress_dir_ (Path)

Return type:

Path

phenotypic.sdk_.analysis_html_path(output_dir: Path) Path[source]#

Return <output>/deliverables/analysis.html.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.analysis_parquet_path(output_dir: Path) Path[source]#

Return <output>/deliverables/analysis.parquet.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.analysis_scatter_json_path(progress_dir_: Path) Path[source]#

Return <progress>/analysis_scatter.json.

Parameters:

progress_dir_ (Path)

Return type:

Path

phenotypic.sdk_.atomic_write_bytes(path: str | Path, data: bytes) None[source]#

Atomically write data to path (temp sibling + os.replace).

The bytes counterpart of atomic_write_text() for binary payloads (e.g. a serialized parquet buffer). Same crash-safety guarantees: an all-or-nothing replace and no partial/leftover temp file on failure.

Parameters:
  • path (str | Path) – The destination file path.

  • data (bytes) – The full binary payload to write.

Raises:

OSError – If the write or rename fails (the temp file is removed first).

Return type:

None

phenotypic.sdk_.atomic_write_text(path: str | Path, text: str, *, encoding: str = 'utf-8') None[source]#

Atomically write text to path (temp sibling + os.replace).

A drop-in replacement for Path(path).write_text(text) that never leaves a half-written file: a concurrent reader sees either the old contents or the complete new ones, and an exception mid-write leaves any pre-existing file intact with no .tmp debris.

Parameters:
  • path (str | Path) – The destination file path.

  • text (str) – The full text payload to write.

  • encoding (str) – The text encoding (default "utf-8").

Raises:

OSError – If the write or rename fails (the temp file is removed first).

Return type:

None

phenotypic.sdk_.best_params_path(output_dir: Path) Path[source]#

Return <output>/deliverables/best_params.json (winner params sidecar).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.best_pipeline_path(output_dir: Path) Path[source]#

Return the canonical typed tuned-winner pipeline path.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.checkpoint_lock_filename(checkpoint_type: Literal['manifest', 'finalize']) str[source]#

Filename of the SLURM-sentinel exclusive lock for a checkpoint task.

Parameters:

checkpoint_type (Literal['manifest', 'finalize']) – "manifest" or "finalize". Validated by the phenotypic.sdk_.typing_.CheckpointType Literal alias at type-check time.

Returns:

Filename relative to <progress>/ (hidden file, leading .).

Return type:

str

phenotypic.sdk_.checkpoint_lock_path(progress_dir_: Path, checkpoint_type: Literal['manifest', 'finalize']) Path[source]#

Return <progress>/.{checkpoint_type}_lock.

Parameters:
  • progress_dir_ (Path)

  • checkpoint_type (Literal['manifest', 'finalize'])

Return type:

Path

phenotypic.sdk_.chunk_lock_path(progress_dir_: Path) Path[source]#

Return <progress>/.chunk_lock.

Parameters:

progress_dir_ (Path)

Return type:

Path

phenotypic.sdk_.chunk_manifest_path(output_dir: Path) Path[source]#

Return <output>/progress/chunk_manifest.json.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.chunk_parquet_filename(chunk_id: int) str[source]#

Filename of a dashboard chunk Parquet (zero-padded chunk id).

Parameters:

chunk_id (int) – Zero-based chunk index, formatted {chunk_id:03d}.

Returns:

Filename relative to <progress>/chunks/.

Return type:

str

phenotypic.sdk_.chunk_parquet_path(progress_dir_: Path, chunk_id: int) Path[source]#

Return <progress>/chunks/chunk_<id:03d>.parquet.

Parameters:
  • progress_dir_ (Path)

  • chunk_id (int)

Return type:

Path

phenotypic.sdk_.chunk_state_path(output_dir: Path) Path[source]#

Return <output>/progress/chunk_state.json.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.chunks_dir(progress_dir_: Path) Path[source]#

Return <progress>/chunks/ for the dashboard chunk parquets.

Parameters:

progress_dir_ (Path)

Return type:

Path

phenotypic.sdk_.clear_machine_state(output_dir: Path) bool[source]#

Remove all of a run’s machine-state for a clean --restart.

Deletes the .phenotypic/ cache (progress/, processing_state.json, processing_events.log) and any pre-migration root-level machine-state, while leaving user-facing output artifacts (deliverables/, results/, qc/, logs/, …) untouched. This is the difference between --restart (re-run the orchestration against clean state, keep outputs) and --overwrite (delete the whole output dir). Clearing the event log here is what stops a restart from appending to — and rebuilding its manifest/failure records from — the prior run’s events.

Returns:

True if any machine-state was removed, else False.

Parameters:

output_dir (Path)

Return type:

bool

phenotypic.sdk_.curation_labels_parquet_path(output_dir: Path) Path[source]#

Return <output>/qc/curation_labels.parquet (durable labels store).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.custom_categories_json_path(output_dir: Path) Path[source]#

Return <output>/qc/custom_categories.json (custom-category registry).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.dashboard_html_path(output_dir: Path) Path[source]#

Return <output>/deliverables/dashboard.html.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.dataset_hdf_dir(output_dir: Path, dataset: str) Path[source]#

Return <output>/results/<dataset>/hdf/.

Parameters:
Return type:

Path

phenotypic.sdk_.dataset_measurements_dir(output_dir: Path, dataset: str) Path[source]#

Return <output>/results/<dataset>/measurements/.

Parameters:
Return type:

Path

phenotypic.sdk_.dataset_overlays_dir(output_dir: Path, dataset: str) Path[source]#

Return <output>/results/<dataset>/overlays/.

Parameters:
Return type:

Path

phenotypic.sdk_.dataset_results_dir(output_dir: Path, dataset: str) Path[source]#

Return <output>/results/<dataset>/.

Parameters:
Return type:

Path

phenotypic.sdk_.default_output_dir_name(now: datetime | None = None) str[source]#

Default name for an auto-generated output directory.

Legacy helper for timestamped output-directory names.

Parameters:

now (datetime | None) – Override clock for tests; defaults to datetime.now().

Returns:

"phenotypic_results_YYYYMMDD_HHMMSS".

Return type:

str

phenotypic.sdk_.deliverables_dir(output_dir: Path) Path[source]#

Return <output>/deliverables/ — the user-facing-output folder.

Pure path expression; callers are responsible for mkdir when they intend to write into it. Writers that go through phenotypic._cli._cli_output_manager._atomic_write() get the mkdir for free (it creates target.parent); direct write_text/write_bytes writers must mkdir explicitly.

Every artifact helper that previously rooted at <output>/ (master / measurements / per-feature split / analysis / dashboard / report / pipeline.json / README) now composes from here, so a future relocation is a one-line change.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.ensure_typed_json_suffix(path: str | Path, suffix: str) Path[source]#

Return path with the canonical typed JSON suffix appended.

Bare stems receive the full typed suffix. Legacy .json paths receive only the typed tail, preserving the user-provided stem and case.

Parameters:
Return type:

Path

phenotypic.sdk_.error_analysis_csv_path(output_dir: Path) Path[source]#

Return <output>/deliverables/error_analysis.csv.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.error_analysis_html_path(output_dir: Path) Path[source]#

Return <output>/deliverables/error_analysis.html.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.error_analysis_parquet_path(output_dir: Path) Path[source]#

Return <output>/deliverables/error_analysis.parquet.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.error_category_parquet_path(output_dir: Path, category: str) Path[source]#

Return <output>/deliverables/errors/<category>.parquet.

Parameters:
  • output_dir (Path) – Run output directory.

  • category (str) – A bare, already-sanitized category token (e.g. "background_noise"). The caller is responsible for sanitization.

Return type:

Path

phenotypic.sdk_.errors_dir(output_dir: Path) Path[source]#

Return <output>/deliverables/errors/ (per-category error parquets).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.event_log_path(output_dir: Path) Path[source]#

Return <output>/.phenotypic/processing_events.log.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.failures_jsonl_path(output_dir: Path) Path[source]#

Return <output>/progress/failures.jsonl.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.generalization_path(output_dir: Path) Path[source]#

Return <output>/deliverables/generalization.json — the held-out report.

The winner’s generalization verdict (calibration vs held-out score, the gap, and the pass/fail margin), a user-facing deliverable. The held-out pass that writes it is Phase 4.5 part 2; this helper resolves the canonical location.

Parameters:

output_dir (Path) – The run output directory.

Returns:

<output_dir>/deliverables/generalization.json.

Return type:

Path

phenotypic.sdk_.has_config_suffix(path: str | Path, suffixes: Iterable[str]) bool[source]#

Return whether path ends with any configured suffix.

Matching is case-insensitive so callers can discover user-provided files from case-preserving filesystems without rewriting their names.

Parameters:
Return type:

bool

phenotypic.sdk_.is_binary_mask(arr: numpy.ndarray)[source]#
Parameters:

arr (numpy.ndarray)

phenotypic.sdk_.job_metadata_path(output_dir: Path) Path[source]#

Return <output>/progress/job_metadata.json.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.load_image_from_hdf(hdf_path: Path, *, fallback: ImageTypeName = 'Image') _Image | _GridImage[source]#

Open an HDF5, read its phenotypic_class attr, dispatch to the right Image class.

Replaces the 3 ad-hoc h5py.File(...) fh.attrs.get('phenotypic_class', 'Image') GridImage if cls_attr == 'GridImage' else Image patterns in _cli_recompile_worker, _cli_execution_strategies, and phenotypicCLI.

Parameters:
  • hdf_path (Path) – Path to a per-image HDF5 file.

  • fallback (ImageTypeName) – Image class name to use when the HDF lacks the phenotypic_class attribute (legacy files). Type-checked (statically) against ImageTypeName — there is no runtime validation; the only effect of an unrecognized string is that the dispatch falls through to Image.

Returns:

An Image or GridImage instance loaded from the HDF.

Return type:

_Image | _GridImage

phenotypic.sdk_.load_master_measurements(output_dir: Path) 'pl.DataFrame' | None[source]#

Read <output>/master_measurements.csv into a polars DataFrame.

Parameters:

output_dir (Path) – Run output directory.

Returns:

DataFrame, or None when the file is missing.

Return type:

Optional[‘pl.DataFrame’]

phenotypic.sdk_.logs_dir(output_dir: Path) Path[source]#

Return <output>/logs/ (SLURM stdout/stderr + sbatch scripts).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.manifest_json_path(output_dir: Path) Path[source]#

Return <output>/progress/manifest.json (dashboard manifest).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.master_measurements_csv_path(output_dir: Path) Path[source]#

Return <output>/deliverables/master_measurements.csv.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.master_measurements_parquet_path(output_dir: Path) Path[source]#

Return <output>/deliverables/master_measurements.parquet.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.matches_any_suffix(path: str | Path, suffixes: Iterable[str]) bool[source]#

Return whether path ends with any suffix in suffixes.

Parameters:
Return type:

bool

phenotypic.sdk_.measurements_by_feature_dir(output_dir: Path) Path[source]#

Return <output>/deliverables/measurements_by_feature/.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.measurements_csv_path(output_dir: Path) Path[source]#

Return <output>/deliverables/measurements.csv (post-applied mirror).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.measurements_parquet_path(output_dir: Path) Path[source]#

Return <output>/deliverables/measurements.parquet (post-applied mirror).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.migrate_legacy_machine_state(output_dir: Path) bool[source]#

Move a pre-migration run’s machine-state into .phenotypic/.

If legacy machine-state (progress/, processing_state.json, processing_events.log) is present at the output root, move each artifact into the .phenotypic/ cache so the run proceeds coherently against a single location. A no-op when no legacy state is present or everything is already migrated.

Robust to interruption and concurrency (the SLURM array case): each artifact is moved only when its source still exists and its destination does not, so a migration interrupted mid-move completes on the next call rather than leaving split state; and a lost move race (a concurrent worker moved the artifact first) is ignored rather than crashing. Keying per-artifact instead of on cache.exists() is what makes both safe.

Returns:

True if this call moved anything, else False.

Parameters:

output_dir (Path)

Return type:

bool

phenotypic.sdk_.overlay_manifest_path(output_dir: Path) Path[source]#

Return <output>/progress/overlay_manifest.json.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.param_importance_path(output_dir: Path) Path[source]#

Return <output>/deliverables/param_importance.json (the report).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.pareto_best_pipeline_path(output_dir: Path, objective: str) Path[source]#

Return deliverables/pareto/best_<objective>.json (a per-axis winner).

The pipeline maximizing the single objective axis on the Pareto front. objective is the objective name as it appears in objectives_json (a scorer-defined label, e.g. "Dice" or a composite child handle "s0").

Parameters:
  • output_dir (Path) – The run directory.

  • objective (str) – The objective-axis name (the best_<objective>.json stem).

Returns:

The per-objective best-pipeline path under pareto_dir().

Return type:

Path

phenotypic.sdk_.pareto_dir(output_dir: Path) Path[source]#

Return <output>/deliverables/pareto/ — the multi-objective sub-folder.

Holds a multi-objective tune run’s Pareto front + per-objective best pipelines. A single-objective run never creates it (the back-compat lock).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.pareto_front_parquet_path(output_dir: Path) Path[source]#

Return <output>/deliverables/pareto/pareto_front.parquet (the front).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.pareto_importance_path(output_dir: Path, objective: str) Path[source]#

Return deliverables/pareto/param_importance_<objective>.json.

The per-objective RF-permutation importance report (the multi-objective sibling of param_importance_path()). objective is the objective name as it appears in objectives_json (a scorer-defined label, e.g. "Dice" or a composite child handle "s0").

Parameters:
  • output_dir (Path) – The run directory.

  • objective (str) – The objective-axis name (the filename’s <objective> slot).

Returns:

The per-objective importance-report path under pareto_dir().

Return type:

Path

phenotypic.sdk_.phenotypic_cache_dir(output_dir: Path) Path[source]#

Return <output>/.phenotypic/ — the hidden machine-state root.

Pure path expression; callers mkdir when they intend to write.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.phenotypic_cache_pipeline_json_path(output_dir: Path) Path[source]#

Return <output>/.phenotypic/pipeline.json — the process-only run’s reproducibility copy. Distinct from pipeline_json_path(), which roots under deliverables/ (process-only writes no deliverables).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.pipeline_json_path(output_dir: Path) Path[source]#

Return the canonical typed pipeline config path under deliverables/.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.processing_report_html_path(output_dir: Path) Path[source]#

Return <output>/deliverables/processing_report.html.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.processing_state_path(output_dir: Path) Path[source]#

Return <output>/.phenotypic/processing_state.json.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.progress_dir(output_dir: Path) Path[source]#

Return <output>/.phenotypic/progress/.

Pure path expression; callers are responsible for mkdir when they intend to write into it.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.qc_config_json_path(output_dir: Path) Path[source]#

Return <output>/qc/qc_config.json.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.qc_dir(output_dir: Path) Path[source]#

Return <output>/qc/.

Pure path expression; run_qc is responsible for mkdir when it writes into it (via _atomic_write()).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.qc_members_parquet_path(output_dir: Path) Path[source]#

Return <output>/qc/qc_members.parquet.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.qc_review_state_path(output_dir: Path) Path[source]#

Return <output>/qc/review_state.json (GUI-owned review progress).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.qc_summary_parquet_path(output_dir: Path) Path[source]#

Return <output>/qc/qc_summary.parquet.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.read_run_manifest(output_dir: Path) dict | None[source]#

Read the run manifest if present, resolving legacy layouts.

Reads <output>/.phenotypic/progress/manifest.json, falling back to the pre-migration <output>/progress/manifest.json for legacy runs (via resolve_manifest_json_path()). Replaces 4 inline json.loads(manifest_path.read_text()) blocks.

Parameters:

output_dir (Path) – Run output directory containing progress/.

Returns:

Parsed manifest dict, or None when the file is missing or unparseable (callers can decide whether absence is fatal).

Return type:

dict | None

phenotypic.sdk_.readme_md_path(output_dir: Path) Path[source]#

Return <output>/deliverables/README.md.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.recompile_dir(progress_dir_: Path) Path[source]#

Return <progress>/recompile/.

Parameters:

progress_dir_ (Path)

Return type:

Path

phenotypic.sdk_.recompile_status_dir(progress_dir_: Path) Path[source]#

Return <progress>/recompile/status/.

Parameters:

progress_dir_ (Path)

Return type:

Path

phenotypic.sdk_.resolve_best_pipeline_path(output_dir: Path) Path[source]#

Return the best existing tuned-winner pipeline path for output_dir.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.resolve_event_log_path(output_dir: Path) Path[source]#

Return the event log sibling of the resolved progress dir.

The event log lives beside progress/ (D14): in .phenotypic/ for a migrated/new run, at the output root for a not-yet-migrated legacy read. Read-only helper for resume/discovery; never mutates the run dir.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.resolve_execution_mode(job_meta: dict | None) Literal['local', 'slurm'][source]#

Extract ExecutionMode from job metadata, defaulting to "local".

Replaces a 5-site copy-paste of the job_meta.get("execution_mode", "local") if job_meta else "local" pattern.

Silent coercion: any value that isn’t exactly "slurm" collapses to "local" — including None (no metadata file), {} (no key), garbage strings (e.g. "validate", ""), and the literal None value. The function never raises. Callers who need to detect an unknown mode and warn / refuse should inspect job_meta directly before calling this helper.

Parameters:

job_meta (dict | None) – Parsed progress/job_metadata.json content, or None when the file is absent.

Returns:

"local" or "slurm".

Return type:

Literal[‘local’, ‘slurm’]

phenotypic.sdk_.resolve_manifest_json_path(output_dir: Path) Path[source]#

Return <progress>/manifest.json resolving the progress dir for legacy runs.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.resolve_pipeline_config_path(output_dir: Path) Path[source]#

Return the best existing pipeline config path for output_dir.

Resolution prefers the canonical typed path, falls back to legacy pipeline.json when present, and returns the canonical path when neither exists so writers naturally create typed config files.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.resolve_processing_state_path(output_dir: Path) Path[source]#

Return the processing-state file that exists, preferring .phenotypic/.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.resolve_progress_dir(output_dir: Path) Path[source]#

Return the progress dir that exists, preferring .phenotypic/.

Read-only helper for resume/discovery so a pre-migration run (progress at the output root) is still found. Falls back to the new location when neither exists (the default for fresh writes).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.resolve_split_assignment_path(output_dir: Path) Path[source]#

Return the split assignment that exists, preferring .pht-tune-cache/.

Read-only resolver mirroring resolve_progress_dir(). The held-out split is checked in the hidden tune cache FIRST, THEN at the legacy output root — a missing split silently RE-DERIVES a fresh held-out partition on resume (a reproducibility / held-out-leak bug), so resume MUST find a legacy-root split.json. Falls back to the new location when neither exists (the default for a fresh derive-and-write).

Parameters:

output_dir (Path) – The run output directory.

Returns:

The split-assignment path that exists, else the new cache location.

Return type:

Path

phenotypic.sdk_.resolve_study_db_path(output_dir: Path) Path[source]#

Return the study DB that exists, preferring .pht-tune-cache/.

Read-only resolver: a relocated run keeps study.db under the hidden tune cache; a legacy run kept it at the output root. Falls back to the new location when neither exists (so a cold sampler restart from a missing study.db is harmless — no migration is performed).

Parameters:

output_dir (Path) – The run output directory.

Returns:

The study DB path that exists, else the new cache location.

Return type:

Path

phenotypic.sdk_.resolve_tuning_spec_path(output_dir: Path) Path[source]#

Return the best existing tuning spec path for output_dir.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.results_dir(output_dir: Path) Path[source]#

Return <output>/results/.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.sentinel_resubmitted_path(progress_dir_: Path) Path[source]#

Return <progress>/sentinel_resubmitted marker file path.

Parameters:

progress_dir_ (Path)

Return type:

Path

phenotypic.sdk_.shard_parquet_filename(shard_id: int) str[source]#

Filename of a per-shard Parquet inside the recompile worker.

Parameters:

shard_id (int) – Zero-based shard index.

Returns:

Filename relative to the recompile shard directory.

Return type:

str

phenotypic.sdk_.slurm_scripts_dir(output_dir: Path) Path[source]#

Return <output>/slurm_scripts/.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.task_status_filename(task_index: int) str[source]#

Filename of a per-task SLURM-recompile status JSON.

Parameters:

task_index (int) – Zero-based recompile task index.

Returns:

Filename relative to <progress>/recompile/status/.

Return type:

str

phenotypic.sdk_.task_status_path(output_dir: Path, task_index: int) Path[source]#

Return <progress>/recompile/status/task_<idx>.json.

Parameters:
  • output_dir (Path)

  • task_index (int)

Return type:

Path

phenotypic.sdk_.timed_execution(func)[source]#

Decorator to measure and print the execution time of a function.

phenotypic.sdk_.trials_parquet_path(output_dir: Path) Path[source]#

Return <output>/trials.parquet (the trial journal; output-dir root).

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.tune_cache_dir(output_dir: Path) Path[source]#

Return <output>/.pht-tune-cache/ — the tune run’s machine-state root.

The tune-side sibling of phenotypic_cache_dir(). Pure path expression; callers mkdir when they intend to write.

Parameters:

output_dir (Path) – The run output directory.

Returns:

<output_dir>/.pht-tune-cache/.

Return type:

Path

phenotypic.sdk_.tune_cache_run_marker_path(output_dir: Path) Path[source]#

Return <output>/.pht-tune-cache/run.json — the tune-run marker.

Written at run START (before any deliverable lands) so a live or finished tune output is GUI-discoverable. See RUN_MARKER_JSON.

Parameters:

output_dir (Path) – The run output directory.

Returns:

<output_dir>/.pht-tune-cache/run.json.

Return type:

Path

phenotypic.sdk_.tune_cache_split_assignment_path(output_dir: Path) Path[source]#

Return <output>/.pht-tune-cache/splits/split.json — the held-out split.

The persisted calibration / held-out partition (plate names + split kind + dataset identity + seed entropy). Read-if-exists-else-derive on resume, so a re-run reuses the original partition regardless of the new master seed. A legacy run wrote it under <output>/splits/; use resolve_split_assignment_path() to read either location.

Parameters:

output_dir (Path) – The run output directory.

Returns:

<output_dir>/.pht-tune-cache/splits/split.json.

Return type:

Path

phenotypic.sdk_.tune_cache_splits_dir(output_dir: Path) Path[source]#

Return <output>/.pht-tune-cache/splits/ — the held-out split folder.

Machine state that must survive a fresh-master rewrite and gate resume, so it lives in the hidden tune cache, not under deliverables_dir(). Pure path expression; callers mkdir when they intend to write.

Parameters:

output_dir (Path) – The run output directory.

Returns:

<output_dir>/.pht-tune-cache/splits/.

Return type:

Path

phenotypic.sdk_.tune_cache_study_db_path(output_dir: Path) Path[source]#

Return <output>/.pht-tune-cache/study.db (the Optuna study DB).

The canonical SQLite-WAL storage for the Optuna-backed OptunaStudyStore when the tune extra is installed, relocated into the hidden tune cache. A legacy run wrote it at the output root; use resolve_study_db_path() to read either location.

Parameters:

output_dir (Path) – The run output directory.

Returns:

<output_dir>/.pht-tune-cache/study.db.

Return type:

Path

phenotypic.sdk_.tuning_spec_path(output_dir: Path) Path[source]#

Return the canonical typed tuning spec path under deliverables/.

Parameters:

output_dir (Path)

Return type:

Path

phenotypic.sdk_.verified_parquet_path(output_dir: Path) Path[source]#

Return <output>/deliverables/verified.parquet (GUI-written, §9).

Parameters:

output_dir (Path)

Return type:

Path

Subpackages#

Submodules#