SPECTER Labs
Technical Docs ops

Run Lake (Cross-Run DuckDB)

The run lake is the cross-run analysis harness for wonton-soup. It reconciles run artifacts into DuckDB and exports stable datasets for downstream dashboards and reports.

Why It Exists

Single-run logs answer local debugging questions. The lake answers cross-run questions:

  • how effects vary across providers/backends/interventions
  • whether patterns persist across repeated seeds
  • how metrics behave against explicitly selected references

Storage Layout

Lake root resolves via analysis.lake.db.resolve_lake_paths() and runtime_paths.resolve_artifacts_root().

  • If SPECTER_ARTIFACT_ROOT is unset:
    • active path: dossiers/wonton-soup/outputs/lake/
  • If SPECTER_ARTIFACT_ROOT is set:
    • active local staging path: $SPECTER_RUNTIME_ROOT/wonton-soup/artifacts/lake/ when SPECTER_RUNTIME_ROOT is set, otherwise tmp/runtime-artifacts/wonton-soup/artifacts/lake/
    • remote target root: $SPECTER_ARTIFACT_ROOT/wonton-soup/artifacts/lake/

Inside active root:

  • lake.duckdb
  • exports/
  • jobs/

Stable IDs

Lake keys are deterministic:

  • root_id: hash of resolved log-root path
  • run_key: hash of (root_id, rel_run_dir)

Canonical Workflows

Daily Incremental Update

uv run python wonton.py postprocess --logs-dir logs
uv run python wonton.py lake reconcile --logs-dir logs
uv run python wonton.py lake export --format specter-viz

Use when adding fresh runs locally.

Backfill / Repair Missing Metrics

uv run python wonton.py postprocess --logs-dir logs
uv run python wonton.py lake reconcile --logs-dir logs

Use when old runs are missing postprocess-heavy fields.

Export-Only Refresh

uv run python wonton.py lake export --format specter-viz

Use when source lake tables are already current.

Site Dashboard Release (Hybrid Publish Path)

uv run python wonton.py lake job release-site-dashboard \
  --config dossiers/wonton-soup/analysis/lake/presets/72_site_dashboard_superset_v1.json \
  --logs-dir logs

Use when publishing a reproducible compiled dashboard cohort to site/dashboards/wonton-soup/data. This path is intentionally hybrid: release materialization is human-triggered, while CI validates and publishes static site assets.

Durable Sync Path

uv run python wonton.py sync now
uv run python wonton.py sync durability-status
uv run python wonton.py sync durable-push

Use when local and remote artifact roots must converge canonically.

What Reconcile Extracts

High-level extracted surfaces:

  • run registry and file registry
  • theorem/intervention summaries from summary.json.gz
  • theorem artifact index
  • graph node/edge fact tables
  • MCTS trace summaries
  • basin tables from basin-only or summary-backed runs
  • optional postprocess outputs
  • goal-outcome aggregates
  • optional reference-scored K outputs

Primary implementation modules:

  • analysis/lake/db.py
  • analysis/lake/reconcile.py
  • analysis/lake/extract_basin.py

Determinism and Repro Guarantees

  • DB connections enforce single-thread mode for deterministic exports.
  • Exports use explicit ordering and deterministic JSON encoding.
  • Reference populations must be explicit for leakage-safe K scoring.

Reference-selection guardrail ADR:

Common Failure Modes

Reconcile misses expected runs

Checks:

  • confirm --logs-dir points at intended run root
  • confirm runs have run_status.json
  • check for partial/failed runs excluded by downstream filters

Export output looks stale

Checks:

  • rerun postprocess then reconcile
  • confirm lake.duckdb under active artifact root is the one being queried

Dashboard export aborts on selected runs

Checks:

  • confirm the selected runs have summary.json(.gz) at the run root
  • if the error says basin-only run, keep that run in the lake but exclude it from dashboard jobs
  • do not rely on implicit skipping; mixed selections now fail loudly so site cohorts cannot shrink silently

Durability drift persists

Checks:

  • run sync durability-status
  • inspect log/artifact roots from runtime_paths
  • rerun sync durable-push after resolving root mismatch

Job-Based Materialization

For explicit dataset jobs, reference selection, and manifest provenance: