Celebi logo Celebi

Celebi / Releases

CelebiChrono v1.0.0b3

Raw data

  • register-data <runner> <remote_path>: data already on a compute farm is registered without touching your network — the MD5 is computed on the runner, the data is copied into Yuki’s managed impressions area on that runner, and the local task becomes a pointer to it. Runs as a background job with live progress; re-registering the same path is idempotent, and the task’s default runner is set to the hosting runner.
  • Data commands renamed: use-data → attach-data (adopt a DITE impression, zero transfer) and send → upload-data (local directory to DITE with a progress bar). The old names are removed.
  • verify-data: recomputes the MD5 of the current data task and compares it with the registered UUID (re-hashed on the host runner for remote data).
  • cache_on_runner replaces use_eos: REANA copies results to EOS, SSH copies them to the runner’s managed impressions area, native/dry is unaffected.
  • Raw-data tasks: ls no longer shows Memory limit / Validated; status renders the stageout file table (with in-Yuki marks); archived displays as [coda][archived].

Runners

  • register-runner/update-runner accept ssh options (--ssh-host, --ssh-user, --ssh-key-path, --ssh-port, --remote-workdir) and native options (--workdir, --cores, --mem-mb, --conda-path, --snakemake-path).
  • SSH keys are uploaded automatically at register/update; without a key path, ~/.ssh/id_rsa / ~/.ssh/id_ed25519 are tried.
  • test-runner <name> probes connectivity, snakemake, conda and the remote working directory; results persist and show as a HEALTH column in runners.
  • runner-envs <name> lists the conda environments on an ssh/native runner.
  • The Chern shell gains test_runner, runner_envs, register_data, cache_on_runner, and runner-name tab completion; register_runner in the shell is ssh-aware.

Engine logs

  • engine-logs now renders the ssh/native log schema; the workflow log section is titled YUKI LOG.
  • engine-logs --fetch generates the engine logs on the server when missing; without --fetch the Yuki workflow log is still served when available. Error bodies show as errors instead of an empty N/A table.

Fixes & quality

  • DITE port fallback corrected to 127.0.0.1:3315; add_host wrapper arity fix; the runner tab-completion cache is now populated.
  • The tqdm dependency is declared in pyproject.toml.
  • Full-project pylint score 10.00/10 with the project rcfile.

Yuki (DITE server)

  • Remote data registration: register-data runs on Yuki as a background job (hashing → copying → registering); the impression goes running → archived and remote-hosted data is listed through file_status.
  • verify-data recomputes MD5s (on the host runner for remote data) and compares with the registered UUID.
  • SSH workflows: remote layout workflows/<project>/<uuid>/ with a managed impressions/ cache; raw-data inputs cache automatically on the runner and are staged by the Snakefile setup rule — the same mechanism REANA uses for EOS.
  • cache_on_runner (replaces use_eos): EOS on REANA, runner impressions on SSH, no-op on native/dry.
  • Runner probing endpoints: /test-runner, /runner-health, /runner-envs; runner settings/health storage; ssh keys stored under $YUKIDIR/keys/.
  • Engine logs: /engine-log?fetch=true generates logs on demand and serves the Yuki workflow.log otherwise.
  • Docker consolidation (non-root multi-stage image, unified build.sh), selective collect/file visibility, and full pylint cleanup.

Runner cache management

The runner’s managed impressions cache (<remote-workdir>/impressions/...) is now manageable from the shell, symmetric in both directions:

  • cache-results <runner> saves the current impression’s runner-resident results — the workflow’s stageout — into the runner’s managed cache. It is the manual version of the workflow’s own cache rule: the copy runs on the runner with the reflink → hardlink → rsync → copy chain and is made read-only. Runs as a background job on Yuki with live polling; impressions whose job never finished, or whose stageout is missing on the runner, are skipped with a reason instead of failing the batch.
  • purge-ssh-runner-cache <runner> evicts the current impression’s cache entry from an ssh runner (read-only entries are made writable first) and clears the matching local bookkeeping so nothing dangles. It never purges the whole runner cache, skips registrations still in flight, and warns to restore registered data with register-data.
  • whereabouts reports where the current impression’s data lives — yuki storage, each runner’s workflow/cache states, and the registration host — in aligned columns, read from the persisted distribution registry.

All three commands exist in the Chern shell (do_*), celebi-cli, and the shell-function layer. Run inside a folder, they fan out over every impressed subobject (one confirmation for the whole batch). The Yuki server CLI gains yuki purge-ssh-runner-cache for server-side purges.

Failure handling

  • A backend failure no longer stamps every execution job failed: jobs already in a terminal state keep their status, so a failed remote workflow start leaves jobs dissonance and never clobbers a previously completed status. This fixes the spurious “Blocked: upstream input … is failed” cascades observed after a workflow died at startup.
  • SSH workflow start failures now include the tail of the runner’s snakemake.log (“Remote Snakemake failed: … (exit N)”) instead of an empty detail, making remote failures diagnosable.

New server endpoints

  • POST /purge-runner-cache, POST /cache-results + GET /cache-results/<job_id>, and GET /whereabouts/<project>/<impression> back the new commands.
  • Cache job state lives in $YUKIDIR/cache-jobs/, generalizing the register-jobs helpers to per-purpose directories.