Distributed Tuning on HPCC Clusters#

Run one Optuna tuning study across many SLURM compute nodes without deploying a database service. A distributed PhenoTypic tune run uses one GPFS-visible append-only journal and a terminal publisher owned by the run lifecycle.

Start a distributed study#

Use an Optuna sampler and add --slurm. When no storage URL is supplied, PhenoTypic records an absolute shared journal URL at <output>/.pht-tune-cache/journal.log; every worker opens that same journal with Optuna’s symlink-lock backend.

uv run phenotypic-tune run spec.json -i ./plates -o ./out \
    --strategy tpe --n-trials 200 --n-workers 16 \
    --slurm slurm_partition=short --slurm slurm_mem=8G --slurm slurm_time=04:00:00

The output directory and its journal must be on storage visible to every compute node (for example your /bigdata allocation), not a node-local /scratch path. The command returns after submitting the worker array and its one dependent terminal finalizer; it does not run a separate scheduler sidecar.

Storage policy#

For an Optuna strategy, storage resolves in this order:

  1. --storage-url on the command line.

  2. strategy.storage_url in the tuning spec.

  3. $PHENOTYPIC_TUNE_STORAGE_URL.

  4. The mode default: run-local SQLite for a local run, or the absolute run-local journal:// URL for --slurm.

Do not use SQLite for a multi-node fleet. --slurm rejects an explicit SQLite URL before it creates run artifacts. Password-bearing URLs are also rejected: they would otherwise be copied into the run marker and generated worker script. An explicitly configured password-less RDB URL remains available when your environment already operates one, but it is not required for the normal HPCC workflow.

What SLURM submits#

The submitter writes the resolved tuning spec, the recorded storage URL, a held-out split, and one lifecycle generation before it submits work. It then submits exactly these compute roles:

  • One worker array. Each task reloads the resolved spec and calibration images, then claims trials from the shared journal until the common terminal budget is reached.

  • One non-array terminal finalizer with an afterany dependency on that array. It reopens the recorded backend without create semantics, requires the expected number of completed-or-pruned terminal trials and a valid winner, then publishes the tuning artifacts.

The finalizer writes trials.parquet, best_pipeline.json, best_params.json, importance/Pareto outputs where applicable, and the held-out generalization report. best_params.json is the final completion signal. If the journal is missing, corrupt, incomplete, or has no valid terminal winner, publication fails closed and the lifecycle records the failed generation.

Monitor and recover#

The run marker at .pht-tune-cache/run.json records the selected backend and expected terminal budget. The lifecycle state and append-only ledger live under .phenotypic/progress/; use them together with your scheduler’s normal job tools to follow the worker and finalizer roles.

After a completed or interrupted distributed run is quiescent, republish from the exact recorded backend:

uv run phenotypic-tune finalize ./out

Normal manual finalization refuses an active lifecycle generation. If a failed generation remains active, --force first requests cancellation and proceeds only after scheduler quiescence has been proven and no unresolved scheduler tokens remain:

uv run phenotypic-tune finalize ./out --force

Publication is lifecycle-locked from ownership validation through all writes. Running finalize again after a successful publication is safe and produces byte-identical published artifacts. Re-running run against an output with an active lifecycle is not a replacement for finalization; wait for the finalizer or use the explicit recovery command after quiescence.