phenotypic.sdk_#
Developer tools shared across fungal colony plate workflows.
Lightweight helpers for timing, mask validation, constants, color conversions, error handling, and HDF storage used by the processing pipeline. Includes a timed execution decorator, mask validators, colourspace utilities, custom exceptions, and HDF helpers for persisting plate datasets and measurements.
Advanced users can access GridInferenceMixin and FootprintMixin for creating custom grid-based operations and morphological footprints.
The register submodule provides registry utilities for extensible components
like plotters and analysis plugins.
The _io_constants submodule is the single source of truth for CLI ↔ GUI
artifact filenames, directory names, JSON contract keys, and path helpers
(re-exported here at package level for convenience).
Functions
Atomically write |
|
Atomically write |
|
Global rank for every known metadata header (cluster then definition order). |
|
Prefix a bare metadata label with its schema category, else generic. |
|
True if |
|
Category that owns a bare label ( |
|
All metadata category prefixes (e.g. |
|
Canonical measurement-frame column order. |
|
Decorator to measure and print the execution time of a function. |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return the canonical typed tuned-winner pipeline path. |
|
Return |
|
Return |
|
Return |
|
Filename of a dashboard chunk Parquet (zero-padded chunk id). |
|
Return |
|
Return |
|
Return |
|
Filename of the SLURM-sentinel exclusive lock for a checkpoint task. |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Default name for an auto-generated output directory. |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return whether |
|
Return |
|
Open an HDF5, read its |
|
Read |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return whether |
|
Move a pre-migration run's machine-state into |
|
Move a pre-relocation run's |
|
Remove all of a run's machine-state for a clean |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return the canonical typed pipeline config path under |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Read the run manifest if present, resolving legacy layouts. |
|
Return |
|
Return |
|
Return |
|
Return the best existing tuned-winner pipeline path for |
|
Return the event log sibling of the resolved progress dir. |
|
Extract |
|
Return |
|
Return the best existing pipeline config path for |
|
Return the processing-state file that exists, preferring |
|
Return the progress dir that exists, preferring |
|
Return the qc dir that exists, preferring |
|
Return the split assignment that exists, preferring |
|
Return the study DB that exists, preferring |
|
Return the best existing tuning spec path for |
|
Return |
|
Return |
|
Filename of a per-shard Parquet inside the recompile worker. |
|
Return |
|
Filename of a per-task SLURM-recompile status JSON. |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return |
|
Return the canonical typed tuning spec path under |
|
Return |
Classes
Mixin for operations that need to control clipping behavior of inner operations. |
|
Provides a mixin for creating morphological footprints for image processing. |
|
Mixin providing grid inference capabilities from binary masks. |
|
Represents an interface to manage HDF5 files with support for single or set image modes, and ensures safe and compatible file access with retry and error-handling mechanisms. |
|
Mixin providing a lazy ipywidget interface. |
|
Keys inside |
|
Keys inside |
|
Keys inside |
|
Keys inside the |
|
Environment variable names read or set by the CLI. |
|
Top-level attribute keys on per-image HDF5 files. |
|
Keys inside |
|
Importable module paths used in dynamic |
|
Keys inside |
|
Resolved on-disk topology of a run output or a standalone deliverables bundle. |
- class phenotypic.sdk_.BundleLayout(deliverables_base: Path, output_root: Path | None)[source]
Bases:
objectResolved on-disk topology of a run output or a standalone deliverables bundle.
Separates the deliverables base (the folder directly holding
master_measurements.parquet) from the optional output root (the parent that also holdsresults/and.phenotypic/). A standalone bundle hasoutput_root is None; deliverables-internal artefacts always resolve fromdeliverables_baseso the bundle is portable.- deliverables_base
Folder containing
master_measurements.parquet.- Type:
- output_root
Parent run directory holding
results/+ machine state, orNonefor a standalone (deliverables-only) bundle.- Type:
pathlib.Path | None
- classmethod detect(path: Path) BundleLayout[source]
Classify
pathas a run output dir or a standalone deliverables bundle.Case 1 —
pathdirectly holdsmaster_measurements.parquet: treat it as the deliverables base. Promotepath.parenttooutput_rootONLY whenpathis literally nameddeliverablesAND a siblingresults/exists (the “pointed at the deliverables subdir of a full run” case); this guard stops a renamed standalone bundle from adopting an unrelated siblingresults/.Case 2 —
pathcontainsdeliverables/master_measurements.parquet:deliverables_base = path/deliverablesandoutput_root = path.- Parameters:
path (Path) – Either a run output directory (containing a
deliverables/subdirectory) or a standalone deliverables folder (directly containingmaster_measurements.parquet).- Returns:
A
BundleLayoutwith resolveddeliverables_baseandoutput_root.- Raises:
FileNotFoundError –
pathis neither a run output directory nor a deliverables bundle.- Return type:
- hdf_path(dataset: str, stem: str) Path | None[source]
Full-res per-image HDF for
(dataset, stem), orNoneif unavailable.
- property analysis_csv: Path
Return path to
analysis.csvin the deliverables base.
- property analysis_parquet: Path
Return path to
analysis.parquetin the deliverables base.
- property curation_labels_parquet: Path
Return path to
deliverables/qc/curation_labels.parquet(durable labels store).
- property custom_categories_json: Path
Return path to
deliverables/qc/custom_categories.json(custom-category registry).
- deliverables_base: Path
- property error_analysis_csv: Path
Return path to
error_analysis.csvin the deliverables base.
- property error_analysis_html: Path
Return path to
error_analysis.htmlin the deliverables base.
- property error_analysis_parquet: Path
Return path to
error_analysis.parquetin the deliverables base.
- property errors_dir: Path
Return path to the
errors/directory under the deliverables base.
- property has_results: bool
Return
Truewhen aresults/directory exists under the output root.
- property master_csv: Path
Return path to
master_measurements.csvin the deliverables base.
- property master_parquet: Path
Return path to
master_measurements.parquetin the deliverables base.
- property mirror_csv: Path
Return path to
measurements.csv(post-applied mirror).
- property mirror_parquet: Path
Return path to
measurements.parquet(post-applied mirror).
- property pipeline_config_path: Path
Return path to
pipeline.jsonin the deliverables base.
- property qc_dir: Path
Return the QC directory, resolving legacy
<output>/qc/layouts.Prefers
deliverables/qc/when it exists. Falls back to the legacy root<output>/qc/when only that exists (pre-relocation runs). Returns the canonicaldeliverables/qc/path for fresh writes when neither is present.
- property qc_duckdb: Path
Return path to
qc/qc.duckdb(the QC analysis database).
- property qc_review_state_path: Path
Return path to
deliverables/qc/review_state.json(GUI-owned review progress).
- property resolved_pipeline_config_path: Path
Return the best existing pipeline config path inside the bundle.
Mirrors
resolve_pipeline_config_path()’s precedence but anchored ondeliverables_base(so a standalone bundle resolves inside itself without double-joiningdeliverables/): the canonical typed config when present, else the legacy plainpipeline.jsonwhen present, else the canonical path (so writers naturally create typed config files).
- property verified_parquet: Path
Return path to
verified.parquet(GUI-written verified-good archive).
- class phenotypic.sdk_.ChunkManifestKey[source]
Bases:
objectKeys inside
<output>/progress/chunk_manifest.json.
- class phenotypic.sdk_.ChunkStateKey[source]
Bases:
objectKeys inside
<output>/progress/chunk_state.json.
- class phenotypic.sdk_.ClipControlMixin[source]
Bases:
objectMixin for operations that need to control clipping behavior of inner operations.
Provides a method to create copies of ImageEnhancer or ImagePipeline instances with clipping disabled. This is useful for composite operations where an inner enhancer operates on non-normalized data (e.g., variance-stabilized values from the Generalized Anscombe Transform, typically in the range ~1-32).
The mixin uses duck typing to check for a clip attribute on operations. If an operation has clip=True, the _disable_clipping method will create a shallow copy with clip=False. This preserves the original operation unchanged while allowing the copy to operate without output clipping.
Example
Creating a clip-disabled copy of an enhancer:
>>> from phenotypic.sdk_ import ClipControlMixin >>> from phenotypic.enhance import LocalEdgeDenoise >>> >>> enh = LocalEdgeDenoise(sigma_spatial=5, clip=True) >>> copied = ClipControlMixin._disable_clipping(enh) >>> # Original unchanged, copy has clip=False >>> enh.clip, copied.clip (True, False)
Creating a clip-disabled copy of a pipeline:
>>> from phenotypic import ImagePipeline >>> from phenotypic.enhance import GaussianBlur, LocalEdgeDenoise >>> >>> pipeline = ImagePipeline(pipe_cfgs=[ ... GaussianBlur(sigma=1.0), ... LocalEdgeDenoise(sigma_spatial=5, clip=True) ... ]) >>> copied_pipe = ClipControlMixin._disable_clipping(pipeline) >>> # Only LocalEdgeDenoise has clip attribute, so only it is affected >>> # _ops is a dict with operation names as keys >>> list(copied_pipe._ops.values())[1].clip False
- class phenotypic.sdk_.DashboardManifestKey[source]
Bases:
objectKeys inside
<output>/progress/manifest.json(dashboard manifest).The manifest is built by
_cli._dashboard._manifest_builder.build_manifest()and consumed by both the dashboard JS and the GUI run-console’s runs registry (_runs_registry.py). Writers and readers must reference these constants rather than spelling the bare string.
- class phenotypic.sdk_.DashboardManifestSlurmInfoKey[source]
Bases:
objectKeys inside the
slurm_infosub-dict of the dashboard manifest.Distinct from
JobMetadataKey, even when string values overlap — these describe the manifest contract, not the job-metadata sidecar.
- class phenotypic.sdk_.EnvVar[source]
Bases:
objectEnvironment variable names read or set by the CLI.
SLURM injects these into batch scripts; the CLI reads them to discover its execution context (job id, array task id, …) and to find node-local scratch storage.
- class phenotypic.sdk_.FootprintMixin[source]
Bases:
objectProvides a mixin for creating morphological footprints for image processing.
The FootprintMixin class contains a static utility method to generate structuring elements (footprints) used in various image processing tasks. This functionality is particularly helpful in the context of analyzing microbial colonies on solid media agar plates. Morphological footprints are used to highlight specific features in images, such as colony edges, shapes, or connectivity, and can assist in segmentation, noise reduction, and feature extraction.
- None
- class phenotypic.sdk_.GridInferenceMixin[source]
Bases:
objectMixin providing grid inference capabilities from binary masks.
Provides static methods for inferring grid structure from colony patterns using peak detection on row/column projections. Used by detectors and refiners that work with gridded plate images (96-well, 384-well formats, pinned cultures).
All methods are static to support parallelization in pipeline operations.
This is an internal utility for advanced users creating custom grid-based operations. Most users should use RoundPeaksDetector for detection or GridAlignmentRefiner for post-detection refinement directly.
- class phenotypic.sdk_.HDF(filepath, name: str, mode: Literal['single', 'set'])[source]
Bases:
objectRepresents an interface to manage HDF5 files with support for single or set image modes, and ensures safe and compatible file access with retry and error-handling mechanisms.
The class facilitates operations on HDF5 files commonly used for storing phenotypic data in both single image and image set modes. This class includes utilities to handle locking errors and ensure compatibility by initializing proper HDF5 modes while providing safe access methods for writing.
- filepath
Path to the HDF5 file on the filesystem.
- Type:
Path
- name
Name associated with the HDF5 resource, often used as an identifier.
- Type:
- mode
Specifies the mode for the HDF5 file, either single image or image set.
- Type:
Literal[‘single’, ‘set’]
- root_posix
The root path for the HDF5 resource, determined by the mode.
- Type:
- home_posix
The specific root directory of the HDF5 resource in the file, derived based on its mode.
- Type:
- set_data_posix
The subgroup path for the data entity in image set mode, if applicable.
- Type:
str, optional
- SINGLE_IMAGE_ROOT_POSIX
Base path for single image mode.
- Type:
- IMAGE_SET_ROOT_POSIX
Base path for image set mode.
- Type:
- IMAGE_SET_DATA_POSIX
Subgroup marker for image set data.
- Type:
- EXT
Set of valid file extensions used to recognize HDF5 files.
- Type:
- IMAGE_MEASUREMENT_SUBGROUP_KEY
Key for accessing measurements in an image’s group.
- Type:
- IMAGE_STATUS_SUBGROUP_KEY
Key for accessing statuses in an image’s group.
- Type:
- static assert_swmr_on(g: Group) None[source]
Assert that SWMR mode is enabled on the group’s file.
- Parameters:
g (Group) – HDF5 group to check.
- Raises:
RuntimeError – If SWMR mode is not enabled.
- Return type:
None
- static close_handle(handle: File | Group) None[source]
- Parameters:
handle (File | Group)
- Return type:
None
- static get_group(handle: File, posix) Group[source]
Retrieves or creates a group in an HDF5 file.
This method checks the validity of the provided HDF5 file handle and tries to retrieve the specified group based on the given posix path. If the group does not exist and the file is not opened in read-only mode, the group gets created. If the file is in read-only mode and the group does not exist, an error is raised.
- Parameters:
- Returns:
The corresponding h5py group within the HDF5 file.
- Return type:
- Raises:
ValueError – If the HDF5 file handle is invalid or no longer valid.
ValueError – If the file handle mode cannot be determined.
KeyError – If the specified group does not exist in read-only mode.
- static get_uncompressed_sizes_for_group(group: Group) tuple[dict[str, int], int][source]
Recursively collect the uncompressed (logical) sizes of SWMR-compatible datasets.
This function walks the provided HDF5 group and inspects every dataset without reading any data. For each dataset that is compatible with SWMR writing rules (i.e., chunked layout and no variable-length data types), it computes the uncompressed size in bytes as: dtype.itemsize * number_of_elements.
Notes
This works regardless of whether datasets are stored compressed on disk; the reported size is the logical size when uncompressed in memory.
Variable-length strings (and datasets containing variable-length fields) are excluded because they are not SWMR-write friendly and their uncompressed size cannot be determined from metadata alone.
The operation is safe under SWMR: it only reads object metadata, creates no new refine, and does not modify the file.
- Parameters:
group (Group) – The root h5py.Group to traverse.
- Returns:
sizes: dict mapping absolute dataset paths (e.g., ‘/grp/ds’) to uncompressed size in bytes.
total_bytes: sum of all values in sizes.
- Return type:
A tuple (sizes, total_bytes) where
- static load_frame(group: Group, *, require_swmr: bool = False) DataFrame[source]
Load a pandas DataFrame from HDF5 storage.
- Parameters:
group (Group) – HDF5 group containing the DataFrame data.
require_swmr (bool) – If True, assert SWMR mode is enabled.
- Returns:
Reconstructed pandas DataFrame.
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If data validation fails.
- Return type:
DataFrame
- static load_series(group: Group, *, dataset: str = 'values', index_dataset: str = 'index', require_swmr: bool = False) Series[source]
Load a pandas Series from HDF5 storage.
Reconstructs the Series with original name, index names, order, and missingness. Respects logical length from attributes.
- Parameters:
- Returns:
Reconstructed pandas Series.
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If data validation fails.
- Return type:
Series
- static preallocate_frame_layout(group: Group, dataframe: DataFrame, *, chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100, require_swmr: bool = False) None[source]
Preallocate HDF5 layout for a pandas DataFrame without writing data.
Creates layout for shared index and column series using Series preallocation.
- Parameters:
group (Group) – HDF5 group to write to.
dataframe (DataFrame) – pandas DataFrame to create layout for.
compression (str) – Compression algorithm.
preallocate (int) – Initial allocation size.
string_fixed_length (int) – Character length for fixed-length strings.
require_swmr (bool) – If True, assert SWMR mode is enabled.
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled. If group.file.swmr_mode is True and datasets don’t exist.
ValueError – If DataFrame validation fails.
- Return type:
None
- static preallocate_series_layout(group: Group, series: Series, *, dataset: str = 'values', index_dataset: str = 'index', chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100) None[source]
Preallocate HDF5 layout for a pandas Series without writing data.
Creates resizable, chunked, compressed datasets with initial shape (preallocate,) and maxshape (None,). Initializes masks to zeros and sets len=0.
- Parameters:
group (Group) – HDF5 group to write to.
series (Series) – pandas Series to create layout for (used for schema).
dataset (str) – Name for the values dataset.
index_dataset (str) – Name for the index dataset.
compression (str) – Compression algorithm.
preallocate (int) – Initial allocation size.
string_fixed_length (int) – Character length for fixed-length strings.
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled. If group.file.swmr_mode is True and datasets don’t exist.
ValueError – If series validation fails.
- Return type:
None
- static save_array2hdf5(group, array, name, **kwargs)[source]
Saves a given numpy array to an HDF5 group. If a dataset with the specified name already exists in the group, it checks if the shapes match. If the shapes match, it updates the existing dataset; otherwise, it removes the existing dataset and creates a new one with the specified name. If a dataset with the given name doesn’t exist, it creates a new dataset.
- Parameters:
group – h5py.Group The HDF5 group in which the dataset will be saved.
array – numpy.ndarray The data array to be stored in the dataset.
name – str The name of the dataset within the group.
**kwargs – dict Additional keyword arguments to pass when creating a new dataset.
- static save_frame_append(group: Group, dataframe: DataFrame, *, require_swmr: bool = True) None[source]
Append a pandas DataFrame to existing HDF5 datasets.
- Parameters:
group (Group) – HDF5 group containing existing datasets.
dataframe (DataFrame) – pandas DataFrame to append.
require_swmr (bool) – If True, assert SWMR mode is enabled.
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If validation fails or schema mismatch.
- Return type:
None
- static save_frame_new(group: Group, dataframe: DataFrame, *, chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100, require_swmr: bool = False) None[source]
Create datasets and write a pandas DataFrame to HDF5.
- Parameters:
group (Group) – HDF5 group to write to.
dataframe (DataFrame) – pandas DataFrame to persist.
compression (str) – Compression algorithm for new datasets.
preallocate (int) – Initial allocation size for new datasets.
string_fixed_length (int) – Character length for fixed-length strings.
require_swmr (bool) – If True, assert SWMR mode is enabled.
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If DataFrame validation fails.
- Return type:
None
- static save_frame_update(group: Group, dataframe: DataFrame, *, start: int = 0, require_swmr: bool = True) None[source]
Update a pandas DataFrame in HDF5 at specified position.
- Parameters:
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If validation fails or schema mismatch.
- Return type:
None
- static save_series_append(group: Group, series: Series, *, dataset: str = 'values', index_dataset: str = 'index', require_swmr: bool = True) None[source]
Append a pandas Series to existing HDF5 datasets.
Appends at the end using current logical length. Resizes datasets if needed and updates logical length.
- Parameters:
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If validation fails or schema mismatch.
- Return type:
None
- static save_series_new(group: Group, series: Series, *, dataset: str = 'values', index_dataset: str = 'index', chunks: tuple[int, ...] = (25,), compression: str = 'gzip', preallocate: int = 100, string_fixed_length: int = 100, require_swmr: bool = False) None[source]
Create datasets and write a pandas Series to HDF5.
Creates new datasets or reuses existing preallocated layout. Writes the first len(series) elements and sets logical length.
- Parameters:
group (Group) – HDF5 group to write to.
series (Series) – pandas Series to persist.
dataset (str) – Name for the values dataset.
index_dataset (str) – Name for the index dataset.
compression (str) – Compression algorithm for new datasets.
preallocate (int) – Initial allocation size for new datasets.
string_fixed_length (int) – Character length for fixed-length strings.
require_swmr (bool) – If True, assert SWMR mode is enabled.
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If series validation fails.
- Return type:
None
- static save_series_update(group: Group, series: Series, *, start: int = 0, dataset: str = 'values', index_dataset: str = 'index', require_swmr: bool = True) None[source]
Update a pandas Series in HDF5 at specified position.
Overwrites [start:start+len(series)] and updates logical length to the largest contiguous written extent.
- Parameters:
- Raises:
RuntimeError – If require_swmr=True and SWMR mode not enabled.
ValueError – If validation fails or schema mismatch.
- Return type:
None
- __init__(filepath, name: str, mode: Literal['single', 'set'])[source]
Initializes a class instance to manage HDF5 file structures for single or set image data based on the given filepath, name of the resource, and operational mode.
- filepath
Path to the HDF5 file.
- Type:
Path
- name
Identifier for the resource within the HDF5 file.
- Type:
- mode
Operational mode determining the structure and organization within the HDF5 file. Must be either ‘single’ or ‘set’.
- Type:
Literal[‘single’, ‘set’]
- root_posix
Posix path representing the root directory within the HDF5 file based on the mode.
- Type:
- home_posix
Posix path representing the home directory for the resource within the HDF5 file based on the mode.
- Type:
- set_data_posix
Posix path for the data subdirectory within the resource home directory. Only initialized in ‘set’ mode.
- Type:
Optional[str]
- Parameters:
filepath – Path to the target HDF5 file. Must have an HDF5-compatible extension, or a ValueError is raised.
name (str) – Name of the resource to be managed in the file. Used to construct the home directory for the resource within the HDF5 file.
mode (Literal['single', 'set']) – Operational mode. Specifies whether the resource represents a ‘single’ or ‘set’ image data. If the mode is invalid, a ValueError is raised.
- Raises:
ValueError – If the filepath does not have an HDF5-compatible extension.
ValueError – If the mode is neither ‘single’ nor ‘set’.
- get_data_group(handle)[source]
- get_home(handle)[source]
Retrieves a specific group from an HDF file corresponding to single image data.
This method is used to fetch a predefined group from an HDF container, where the group is identified by a constant key related to single image data. The function provides a static interface allowing invocation without requiring an instance of the class.
- Parameters:
handle – The HDF file handle from which the group should be retrieved.
- Returns:
The group corresponding to single image data, retrieved based on the defined SINGLE_IMAGE_ROOT_POSIX.
- Raises:
Appropriate exceptions may be raised by the underlying HDF.get_group() method, –
based on the implementation and provided handle or key. –
- get_image_group(handle, image_name)[source]
- get_image_measurement_subgroup(handle, image_name)[source]
- get_protected_metadata_subgroup(handle: File, image_name: str) Group[source]
- Parameters:
handle (File)
image_name (str)
- Return type:
Group
- get_public_metadata_subgroup(handle: File, image_name: str) Group[source]
- Parameters:
handle (File)
image_name (str)
- Return type:
Group
- get_root_group(handle) Group[source]
- Return type:
Group
- get_status_subgroup(handle, image_name)[source]
- reader() File[source]
- Return type:
File
- safe_writer() File[source]
Returns a writer object that provides safe and controlled write access to an HDF5 file at the specified filepath or creates it if it doesn’t exist. Ensures that the file uses the ‘latest’ version of the HDF5 library for compatibility and performance.
Handles HDF5 file locking conflicts by attempting to clear consistency flags and retrying file opening with exponential backoff.
- strict_writer() File[source]
Provides access to an HDF5 file in read/write mode using the h5py library. This property is used to obtain an h5py.File object configured with the latest library version.
Note
If using SWMR mode, don’t forget to enable SWMR mode:
>>> hdf = HDF(filepath) >>> with hdf.writer as writer: ... writer.swmr_mode = True ... # rest of your code
- swmr_reader() File[source]
- Return type:
File
- swmr_writer() File[source]
Returns a writer object that provides safe SWMR-compatible write access to an HDF5 file. Creates the file if it doesn’t exist and enables SWMR mode properly.
This method ensures proper SWMR mode initialization by creating the file with the correct settings from the start, avoiding cache conflicts that occur when trying to enable SWMR mode after opening.
- EXT = {'.h5', '.hdf', '.hdf5', '.he5'}
- IMAGE_MEASUREMENT_SUBGROUP_KEY = 'measurements'
- IMAGE_SET_DATA_POSIX = 'data'
- IMAGE_SET_ROOT_POSIX = '/phenotypic/image_sets/'
- IMAGE_STATUS_SUBGROUP_KEY = 'status'
- PROTECTED_METADATA_SUBGROUP_KEY = 'protected_metadata'
- PUBLIC_METADATA_SUBGROUP_KEY = 'public_metadata'
- SINGLE_IMAGE_ROOT_POSIX = '/phenotypic/images/'
- class phenotypic.sdk_.HdfAttr[source]
Bases:
objectTop-level attribute keys on per-image HDF5 files.
- class phenotypic.sdk_.JobMetadataKey[source]
Bases:
objectKeys inside
<output>/progress/job_metadata.json.Writers (CLI execution strategies) and readers (recompile worker, sentinel, checkpoint handler, GUI runs registry) must reference these constants — never the bare string. Renaming a key here should fail fast at every site.
- class phenotypic.sdk_.LazyWidgetMixin[source]
Bases:
objectMixin providing a lazy ipywidget interface.
This mixin allows ImageOperation classes to automatically generate a Jupyter widget interface for parameter tuning and visualization.
The six lazily-populated UI handles this mixin uses (
_ui,_param_widgets,_view_dropdown,_update_button,_output_widget,_image_ref) are declared asPrivateAttronImageOperationrather than here. A plain (non-BaseModel) mixin’sPrivateAttrdeclarations are not collected by pydantic, so this mixin stays a stateless methods-only class and the private attrs live on the pydantic model that mixes it in. This keeps the mixin safe to combine with anyBaseModelin any MRO position.- __getstate__()[source]
Prepare the object for pickling by disposing of any widgets.
This ensures that UI components (which may contain unpickleable objects like input functions or thread locks) are cleaned up before serialization.
Note
This method modifies the object state by calling dispose_widgets(). Any active widgets will be detached from the object.
- widget(image: Image | None = None, show: bool = False) Widget[source]
Return (and optionally display) the root widget.
- Parameters:
image (Image | None) – Optional image to visualize. If provided, visualization controls will be added to the widget.
show (bool) – Whether to display the widget immediately. Defaults to False.
- Returns:
The root widget.
- Return type:
ipywidgets.Widget
- Raises:
ImportError – If ipywidgets or IPython are not installed.
- class phenotypic.sdk_.ModulePath[source]
Bases:
objectImportable module paths used in dynamic
importlib.import_moduledispatch.Spelled out here so a renamed sub-package fails at type-check time (consumers reference
ModulePath.POST— a typo there is caught by mypy) rather than silently at runtime.
- class phenotypic.sdk_.ProcessingStateKey[source]
Bases:
objectKeys inside
<output>/processing_state.json(resume-state file).Distinct from
JobMetadataKeyeven where string values overlap (e.g.EXECUTION_MODE,INPUT_PATH) — these describe the processing_state.json contract, not the SLURM job metadata sidecar. Some values intentionally match across the two contracts so that a single field (likeexecution_mode) can be migrated atomically; thetest_processing_state_keys_match_job_metadata_keysregression test asserts the overlap.
- phenotypic.sdk_.analysis_csv_path(output_dir: Path) Path[source]
Return
<output>/deliverables/analysis.csv.
- phenotypic.sdk_.analysis_full_parquet_path(progress_dir_: Path) Path[source]
Return
<progress>/analysis_full.parquet(incremental combined frame).Takes a progress_dir (not the run output root) since this file lives inside
progress/— it’s an internal mid-run artifact, not a user- facing output.
- phenotypic.sdk_.analysis_html_path(output_dir: Path) Path[source]
Return
<output>/deliverables/analysis.html.
- phenotypic.sdk_.analysis_parquet_path(output_dir: Path) Path[source]
Return
<output>/deliverables/analysis.parquet.
- phenotypic.sdk_.analysis_scatter_json_path(progress_dir_: Path) Path[source]
Return
<progress>/analysis_scatter.json.
- phenotypic.sdk_.atomic_write_bytes(path: str | Path, data: bytes) None[source]
Atomically write
datatopath(temp sibling +os.replace).The bytes counterpart of
atomic_write_text()for binary payloads (e.g. a serialized parquet buffer). Same crash-safety guarantees: an all-or-nothing replace and no partial/leftover temp file on failure.
- phenotypic.sdk_.atomic_write_text(path: str | Path, text: str, *, encoding: str = 'utf-8') None[source]
Atomically write
texttopath(temp sibling +os.replace).A drop-in replacement for
Path(path).write_text(text)that never leaves a half-written file: a concurrent reader sees either the old contents or the complete new ones, and an exception mid-write leaves any pre-existing file intact with no.tmpdebris.
- phenotypic.sdk_.best_params_path(output_dir: Path) Path[source]
Return
<output>/deliverables/best_params.json(winner params sidecar).
- phenotypic.sdk_.best_pipeline_path(output_dir: Path) Path[source]
Return the canonical typed tuned-winner pipeline path.
- phenotypic.sdk_.canonical_metadata_order() dict[str, int][source]
Global rank for every known metadata header (cluster then definition order).
Cluster-order major, enum definition-order minor. A header absent from this map is an unknown/uncategorized user tag; callers rank those last. The map is derived entirely from the import-time schema enums, so it is cached. The returned dict is read-only by contract; callers must not mutate it (mirrors
header_to_module()).
- phenotypic.sdk_.checkpoint_lock_filename(checkpoint_type: Literal['manifest', 'finalize']) str[source]
Filename of the SLURM-sentinel exclusive lock for a checkpoint task.
- phenotypic.sdk_.checkpoint_lock_path(progress_dir_: Path, checkpoint_type: Literal['manifest', 'finalize']) Path[source]
Return
<progress>/.{checkpoint_type}_lock.
- phenotypic.sdk_.chunk_manifest_path(output_dir: Path) Path[source]
Return
<output>/progress/chunk_manifest.json.
- phenotypic.sdk_.chunk_parquet_filename(chunk_id: int) str[source]
Filename of a dashboard chunk Parquet (zero-padded chunk id).
- phenotypic.sdk_.chunk_parquet_path(progress_dir_: Path, chunk_id: int) Path[source]
Return
<progress>/chunks/chunk_<id:03d>.parquet.
- phenotypic.sdk_.chunk_state_path(output_dir: Path) Path[source]
Return
<output>/progress/chunk_state.json.
- phenotypic.sdk_.chunks_dir(progress_dir_: Path) Path[source]
Return
<progress>/chunks/for the dashboard chunk parquets.
- phenotypic.sdk_.clear_machine_state(output_dir: Path) bool[source]
Remove all of a run’s machine-state for a clean
--restart.Deletes the
.phenotypic/cache (progress/,processing_state.json,processing_events.log, logs, and generated SLURM scripts) and any pre-migration root-level machine-state, while leaving user-facing output artifacts (deliverables/,results/,qc/, …) untouched. This is the difference between--restart(re-run the orchestration against clean state, keep outputs) and--overwrite(delete the whole output dir). Clearing the event log here is what stops a restart from appending to — and rebuilding its manifest/failure records from — the prior run’s events.
- phenotypic.sdk_.curation_labels_parquet_path(output_dir: Path) Path[source]
Return
<output>/deliverables/qc/curation_labels.parquet(durable labels store).
- phenotypic.sdk_.custom_categories_json_path(output_dir: Path) Path[source]
Return
<output>/deliverables/qc/custom_categories.json(custom-category registry).
- phenotypic.sdk_.dashboard_html_path(output_dir: Path) Path[source]
Return
<output>/deliverables/dashboard.html.
- phenotypic.sdk_.dataset_hdf_dir(output_dir: Path, dataset: str) Path[source]
Return
<output>/results/<dataset>/hdf/.
- phenotypic.sdk_.dataset_measurements_dir(output_dir: Path, dataset: str) Path[source]
Return
<output>/results/<dataset>/measurements/.
- phenotypic.sdk_.dataset_overlays_dir(output_dir: Path, dataset: str) Path[source]
Return
<output>/deliverables/overlays/<dataset>/.
- phenotypic.sdk_.dataset_results_dir(output_dir: Path, dataset: str) Path[source]
Return
<output>/results/<dataset>/.
- phenotypic.sdk_.default_output_dir_name(now: datetime | None = None) str[source]
Default name for an auto-generated output directory.
Legacy helper for timestamped output-directory names.
- phenotypic.sdk_.deliverables_dir(output_dir: Path) Path[source]
Return
<output>/deliverables/— the user-facing-output folder.Pure path expression; callers are responsible for
mkdirwhen they intend to write into it. Writers that go throughphenotypic._cli._cli_output_manager._atomic_write()get themkdirfor free (it createstarget.parent); directwrite_text/write_byteswriters mustmkdirexplicitly.Every artifact helper that previously rooted at
<output>/(master / measurements / per-feature split / analysis / dashboard / report / pipeline.json / README) now composes from here, so a future relocation is a one-line change.
- phenotypic.sdk_.ensure_metadata_prefix(name: str) str[source]
Prefix a bare metadata label with its schema category, else generic.
Strain -> MetadataGenetic_Strain(the owning enum’s category); an unknownFoo -> Metadata_Foo(kept, uncategorized). Names that already carry a metadata prefix – anyMetadata<Topic>_category prefix or the genericMetadata_– pass through unchanged.
- phenotypic.sdk_.ensure_typed_json_suffix(path: str | Path, suffix: str) Path[source]
Return
pathwith the canonical typed JSON suffix appended.Bare stems receive the full typed suffix. Legacy
.jsonpaths receive only the typed tail, preserving the user-provided stem and case.
- phenotypic.sdk_.error_analysis_csv_path(output_dir: Path) Path[source]
Return
<output>/deliverables/error_analysis.csv.
- phenotypic.sdk_.error_analysis_html_path(output_dir: Path) Path[source]
Return
<output>/deliverables/error_analysis.html.
- phenotypic.sdk_.error_analysis_parquet_path(output_dir: Path) Path[source]
Return
<output>/deliverables/error_analysis.parquet.
- phenotypic.sdk_.error_category_parquet_path(output_dir: Path, category: str) Path[source]
Return
<output>/deliverables/errors/<category>.parquet.
- phenotypic.sdk_.errors_dir(output_dir: Path) Path[source]
Return
<output>/deliverables/errors/(per-category error parquets).
- phenotypic.sdk_.event_log_path(output_dir: Path) Path[source]
Return
<output>/.phenotypic/processing_events.log.
- phenotypic.sdk_.failures_jsonl_path(output_dir: Path) Path[source]
Return
<output>/progress/failures.jsonl.
- phenotypic.sdk_.generalization_path(output_dir: Path) Path[source]
Return
<output>/deliverables/generalization.json— the held-out report.The winner’s generalization verdict (calibration vs held-out score, the gap, and the pass/fail margin), a user-facing deliverable. The held-out pass that writes it is Phase 4.5 part 2; this helper resolves the canonical location.
- phenotypic.sdk_.has_config_suffix(path: str | Path, suffixes: Iterable[str]) bool[source]
Return whether
pathends with any configured suffix.Matching is case-insensitive so callers can discover user-provided files from case-preserving filesystems without rewriting their names.
- phenotypic.sdk_.is_binary_mask(arr: numpy.ndarray)[source]
- Parameters:
arr (numpy.ndarray)
- phenotypic.sdk_.is_metadata_header(col: str) bool[source]
True if
colis a metadata-family column.Matches any per-topic
Metadata<Topic>_prefix (e.g.MetadataGenetic_) and the genericMetadata_fallback used for uncategorized user metadata, per the namespace design (spec §10.3): the wholeMetadatafamily is recognized, measurement/identity columns (Shape_,Object_, …) are not.
- phenotypic.sdk_.job_metadata_path(output_dir: Path) Path[source]
Return
<output>/progress/job_metadata.json.
- phenotypic.sdk_.load_image_from_hdf(hdf_path: Path, *, fallback: ImageTypeName = 'Image') _Image | _GridImage[source]
Open an HDF5, read its
phenotypic_classattr, dispatch to the right Image class.Replaces the 3 ad-hoc
h5py.File(...) → fh.attrs.get('phenotypic_class', 'Image') → GridImage if cls_attr == 'GridImage' else Imagepatterns in_cli_recompile_worker,_cli_execution_strategies, andphenotypicCLI.- Parameters:
hdf_path (Path) – Path to a per-image HDF5 file.
fallback (ImageTypeName) – Image class name to use when the HDF lacks the
phenotypic_classattribute (legacy files). Type-checked (statically) againstImageTypeName— there is no runtime validation; the only effect of an unrecognized string is that the dispatch falls through toImage.
- Returns:
An
ImageorGridImageinstance loaded from the HDF.- Return type:
_Image | _GridImage
- phenotypic.sdk_.load_master_measurements(output_dir: Path) 'pl.DataFrame' | None[source]
Read
<output>/master_measurements.csvinto a polars DataFrame.- Parameters:
output_dir (Path) – Run output directory.
- Returns:
DataFrame, or
Nonewhen the file is missing.- Return type:
Optional[‘pl.DataFrame’]
- phenotypic.sdk_.manifest_json_path(output_dir: Path) Path[source]
Return
<output>/progress/manifest.json(dashboard manifest).
- phenotypic.sdk_.master_measurements_csv_path(output_dir: Path) Path[source]
Return
<output>/deliverables/master_measurements.csv.
- phenotypic.sdk_.master_measurements_parquet_path(output_dir: Path) Path[source]
Return
<output>/deliverables/master_measurements.parquet.
- phenotypic.sdk_.matches_any_suffix(path: str | Path, suffixes: Iterable[str]) bool[source]
Return whether
pathends with any suffix insuffixes.
- phenotypic.sdk_.measurements_by_feature_dir(output_dir: Path) Path[source]
Return
<output>/deliverables/measurements_by_feature/.
- phenotypic.sdk_.measurements_csv_path(output_dir: Path) Path[source]
Return
<output>/deliverables/measurements.csv(post-applied mirror).
- phenotypic.sdk_.measurements_parquet_path(output_dir: Path) Path[source]
Return
<output>/deliverables/measurements.parquet(post-applied mirror).
- phenotypic.sdk_.metadata_category_for_label(label: str) str | None[source]
Category that owns a bare label (
'Strain' -> 'MetadataGenetic'), or None.
- phenotypic.sdk_.metadata_category_prefixes() tuple[str, ...][source]
All metadata category prefixes (e.g.
'MetadataGenetic_') in cluster order.Ordered by the bio-semantic cluster order (
_METADATA_CLUSTER_ORDER), then deduplicated, so callers building bucket-priority lists get a stable, canonical ordering. REMBI is a separate axis (seeby_module/header_to_module).
- phenotypic.sdk_.metadata_csv_deliverable_path(output_dir: Path) Path[source]
Return
<output>/deliverables/metadata.csv(co-located--metadatacopy).
- phenotypic.sdk_.migrate_legacy_machine_state(output_dir: Path) bool[source]
Move a pre-migration run’s machine-state into
.phenotypic/.If legacy machine-state (
progress/,processing_state.json,processing_events.log) is present at the output root, move each artifact into the.phenotypic/cache so the run proceeds coherently against a single location. A no-op when no legacy state is present or everything is already migrated.Robust to interruption and concurrency (the SLURM array case): each artifact is moved only when its source still exists and its destination does not, so a migration interrupted mid-move completes on the next call rather than leaving split state; and a lost move race (a concurrent worker moved the artifact first) is ignored rather than crashing. Keying per-artifact instead of on
cache.exists()is what makes both safe.
- phenotypic.sdk_.migrate_legacy_qc(output_dir: Path) bool[source]
Move a pre-relocation run’s
<output>/qc/intodeliverables/qc/.Hard cutover (MOVE, no duplication), mirroring
migrate_legacy_machine_state(). A no-op when there is no legacyqc/or when the canonicaldeliverables/qc/already exists (the move is whole-directory; we never merge a half-written canonical with legacy).
- phenotypic.sdk_.order_measurement_columns(columns: Sequence[str]) list[str][source]
Canonical measurement-frame column order.
[front metadata] -> [measurements] -> [MetadataImage_*] -> [info block].Front (user/experimental) metadata is cluster/definition ordered via
canonical_metadata_order(); unknown/uncategorizedMetadata_*tags fall to the end of the front block alphabetically. The frameworkMetadataImage_*block is per-image provenance and trails the measurements. The per-object info block (Object_Label+Bbox_*/Grid_*) is detected by name and moves last. Measurements keep their incoming relative order.Pure over column-name strings, so both the pandas (
df[...]) and polars (df.select(...)) paths reuse it.
- phenotypic.sdk_.overlay_manifest_path(output_dir: Path) Path[source]
Return
<output>/deliverables/overlays/overlay_manifest.json.
- phenotypic.sdk_.overlays_dir(output_dir: Path) Path[source]
Return
<output>/deliverables/overlays/— the overlay package root.
- phenotypic.sdk_.param_importance_path(output_dir: Path) Path[source]
Return
<output>/deliverables/param_importance.json(the report).
- phenotypic.sdk_.pareto_best_pipeline_path(output_dir: Path, objective: str) Path[source]
Return
deliverables/pareto/best_<objective>.json(a per-axis winner).The pipeline maximizing the single
objectiveaxis on the Pareto front.objectiveis the objective name as it appears inobjectives_json(a scorer-defined label, e.g."Dice"or a composite child handle"s0").- Parameters:
- Returns:
The per-objective best-pipeline path under
pareto_dir().- Return type:
- phenotypic.sdk_.pareto_dir(output_dir: Path) Path[source]
Return
<output>/deliverables/pareto/— the multi-objective sub-folder.Holds a multi-objective tune run’s Pareto front + per-objective best pipelines. A single-objective run never creates it (the back-compat lock).
- phenotypic.sdk_.pareto_front_parquet_path(output_dir: Path) Path[source]
Return
<output>/deliverables/pareto/pareto_front.parquet(the front).
- phenotypic.sdk_.pareto_importance_path(output_dir: Path, objective: str) Path[source]
Return
deliverables/pareto/param_importance_<objective>.json.The per-objective RF-permutation importance report (the multi-objective sibling of
param_importance_path()).objectiveis the objective name as it appears inobjectives_json(a scorer-defined label, e.g."Dice"or a composite child handle"s0").- Parameters:
- Returns:
The per-objective importance-report path under
pareto_dir().- Return type:
- phenotypic.sdk_.phenotypic_cache_dir(output_dir: Path) Path[source]
Return
<output>/.phenotypic/— the hidden machine-state root.Pure path expression; callers
mkdirwhen they intend to write.
- phenotypic.sdk_.phenotypic_cache_pipeline_json_path(output_dir: Path) Path[source]
Return
<output>/.phenotypic/pipeline.json.pht-pipe— the process-only run’s reproducibility copy. Distinct frompipeline_json_path(), which roots underdeliverables/(process-only writes no deliverables).
- phenotypic.sdk_.pipeline_json_path(output_dir: Path) Path[source]
Return the canonical typed pipeline config path under
deliverables/.
- phenotypic.sdk_.processing_report_html_path(output_dir: Path) Path[source]
Return
<output>/deliverables/processing_report.html.
- phenotypic.sdk_.processing_state_path(output_dir: Path) Path[source]
Return
<output>/.phenotypic/processing_state.json.
- phenotypic.sdk_.progress_dir(output_dir: Path) Path[source]
Return
<output>/.phenotypic/progress/.Pure path expression; callers are responsible for
mkdirwhen they intend to write into it.
- phenotypic.sdk_.qc_dir(output_dir: Path) Path[source]
Return
<output>/deliverables/qc/— durable QC + curation state.Relocated under
deliverables/so a deliverables bundle is self-contained and portable. Useresolve_qc_dir()for reads that must honour the legacy root<output>/qc/layout of pre-relocation runs.
- phenotypic.sdk_.qc_duckdb_path(output_dir: Path) Path[source]
Return
<output>/deliverables/qc/qc.duckdb.
- phenotypic.sdk_.qc_review_state_path(output_dir: Path) Path[source]
Return
<output>/deliverables/qc/review_state.json(GUI-owned review progress).
- phenotypic.sdk_.read_run_manifest(output_dir: Path) dict | None[source]
Read the run manifest if present, resolving legacy layouts.
Reads
<output>/.phenotypic/progress/manifest.json, falling back to the pre-migration<output>/progress/manifest.jsonfor legacy runs (viaresolve_manifest_json_path()). Replaces 4 inlinejson.loads(manifest_path.read_text())blocks.
- phenotypic.sdk_.readme_md_path(output_dir: Path) Path[source]
Return
<output>/deliverables/README.md.
- phenotypic.sdk_.recompile_status_dir(progress_dir_: Path) Path[source]
Return
<progress>/recompile/status/.
- phenotypic.sdk_.rembi_manifest_path(output_dir: Path) Path[source]
Return
<output>/deliverables/rembi.yaml— the REMBI run manifest.
- phenotypic.sdk_.resolve_best_pipeline_path(output_dir: Path) Path[source]
Return the best existing tuned-winner pipeline path for
output_dir.
- phenotypic.sdk_.resolve_event_log_path(output_dir: Path) Path[source]
Return the event log sibling of the resolved progress dir.
The event log lives beside
progress/(D14): in.phenotypic/for a migrated/new run, at the output root for a not-yet-migrated legacy read. Read-only helper for resume/discovery; never mutates the run dir.
- phenotypic.sdk_.resolve_execution_mode(job_meta: dict | None) Literal['local', 'slurm'][source]
Extract
ExecutionModefrom job metadata, defaulting to"local".Replaces a 5-site copy-paste of the
job_meta.get("execution_mode", "local") if job_meta else "local"pattern.Silent coercion: any value that isn’t exactly
"slurm"collapses to"local"— includingNone(no metadata file),{}(no key), garbage strings (e.g."validate",""), and the literalNonevalue. The function never raises. Callers who need to detect an unknown mode and warn / refuse should inspectjob_metadirectly before calling this helper.
- phenotypic.sdk_.resolve_manifest_json_path(output_dir: Path) Path[source]
Return
<progress>/manifest.jsonresolving the progress dir for legacy runs.
- phenotypic.sdk_.resolve_pipeline_config_path(output_dir: Path) Path[source]
Return the best existing pipeline config path for
output_dir.Resolution prefers the canonical typed path, falls back to legacy
pipeline.jsonwhen present, and returns the canonical path when neither exists so writers naturally create typed config files.
- phenotypic.sdk_.resolve_processing_state_path(output_dir: Path) Path[source]
Return the processing-state file that exists, preferring
.phenotypic/.
- phenotypic.sdk_.resolve_progress_dir(output_dir: Path) Path[source]
Return the progress dir that exists, preferring
.phenotypic/.Read-only helper for resume/discovery so a pre-migration run (progress at the output root) is still found. Falls back to the new location when neither exists (the default for fresh writes).
- phenotypic.sdk_.resolve_qc_dir(output_dir: Path) Path[source]
Return the qc dir that exists, preferring
deliverables/qc/.Read-only resolver: deliverables/qc if present, else legacy root qc if present, else the canonical deliverables/qc (for fresh writes).
- phenotypic.sdk_.resolve_split_assignment_path(output_dir: Path) Path[source]
Return the split assignment that exists, preferring
.pht-tune-cache/.Read-only resolver mirroring
resolve_progress_dir(). The held-out split is checked in the hidden tune cache FIRST, THEN at the legacy output root — a missing split silently RE-DERIVES a fresh held-out partition on resume (a reproducibility / held-out-leak bug), so resume MUST find a legacy-rootsplit.json. Falls back to the new location when neither exists (the default for a fresh derive-and-write).
- phenotypic.sdk_.resolve_study_db_path(output_dir: Path) Path[source]
Return the study DB that exists, preferring
.pht-tune-cache/.Read-only resolver: a relocated run keeps
study.dbunder the hidden tune cache; a legacy run kept it at the output root. Falls back to the new location when neither exists (so a cold sampler restart from a missingstudy.dbis harmless — no migration is performed).
- phenotypic.sdk_.resolve_tuning_spec_path(output_dir: Path) Path[source]
Return the best existing tuning spec path for
output_dir.
- phenotypic.sdk_.sentinel_resubmitted_path(progress_dir_: Path) Path[source]
Return
<progress>/sentinel_resubmittedmarker file path.
- phenotypic.sdk_.shard_parquet_filename(shard_id: int) str[source]
Filename of a per-shard Parquet inside the recompile worker.
- phenotypic.sdk_.slurm_scripts_dir(output_dir: Path) Path[source]
Return
<output>/.phenotypic/slurm_scripts/.
- phenotypic.sdk_.task_status_filename(task_index: int) str[source]
Filename of a per-task SLURM-recompile status JSON.
- phenotypic.sdk_.task_status_path(output_dir: Path, task_index: int) Path[source]
Return
<progress>/recompile/status/task_<idx>.json.
- phenotypic.sdk_.timed_execution(func)[source]
Decorator to measure and print the execution time of a function.
- phenotypic.sdk_.trials_parquet_path(output_dir: Path) Path[source]
Return
<output>/trials.parquet(the trial journal; output-dir root).
- phenotypic.sdk_.tune_cache_dir(output_dir: Path) Path[source]
Return
<output>/.pht-tune-cache/— the tune run’s machine-state root.The tune-side sibling of
phenotypic_cache_dir(). Pure path expression; callersmkdirwhen they intend to write.
- phenotypic.sdk_.tune_cache_run_marker_path(output_dir: Path) Path[source]
Return
<output>/.pht-tune-cache/run.json— the tune-run marker.Written at run START (before any deliverable lands) so a live or finished tune output is GUI-discoverable. See
RUN_MARKER_JSON.
- phenotypic.sdk_.tune_cache_split_assignment_path(output_dir: Path) Path[source]
Return
<output>/.pht-tune-cache/splits/split.json— the held-out split.The persisted calibration / held-out partition (plate names + split kind + dataset identity + seed entropy). Read-if-exists-else-derive on resume, so a re-run reuses the original partition regardless of the new master seed. A legacy run wrote it under
<output>/splits/; useresolve_split_assignment_path()to read either location.
- phenotypic.sdk_.tune_cache_splits_dir(output_dir: Path) Path[source]
Return
<output>/.pht-tune-cache/splits/— the held-out split folder.Machine state that must survive a fresh-master rewrite and gate resume, so it lives in the hidden tune cache, not under
deliverables_dir(). Pure path expression; callersmkdirwhen they intend to write.
- phenotypic.sdk_.tune_cache_study_db_path(output_dir: Path) Path[source]
Return
<output>/.pht-tune-cache/study.db(the Optuna study DB).The canonical SQLite-WAL storage for the Optuna-backed
OptunaStudyStorewhen thetuneextra is installed, relocated into the hidden tune cache. A legacy run wrote it at the output root; useresolve_study_db_path()to read either location.
- phenotypic.sdk_.tuning_spec_path(output_dir: Path) Path[source]
Return the canonical typed tuning spec path under
deliverables/.
- phenotypic.sdk_.verified_parquet_path(output_dir: Path) Path[source]
Return
<output>/deliverables/verified.parquet(GUI-written, §9).
Modules
|
|
|
PhenoTypic Constants |
|
PhenoTypic Exceptions |
|
Napari-based interactive tools for PhenoTypic. |
|
Registry utilities for PhenoTypic components. |
|
Shared SLURM utilities for array job chunking, sbatch submission, and dispatching. |
|
SLURM headroom calculation and submission validation utilities. |