Scheduling & runs

Once a pipeline is deployed, the control plane executes it — on demand or on a schedule — over a durable, Postgres-backed job queue with a multi-worker pool.

Serving the control plane

valved serve runs the supervisor: the FastAPI app, the cron scheduler, the crash-recovery reaper, the archiver, and a worker pool.

valved serve                     # everything in one process
valved worker                    # or run standalone workers that drain the queue

Workers claim jobs with optimistic-claim semantics, emit heartbeats (so the reaper can reclaim a crashed worker’s jobs after a missed-heartbeat threshold), retry transient failures with backoff, and write structured per-run logs. Supervisor-loop intervals are tunable in valved/runtime.toml.

On-demand runs

valved run <pipeline> --watch    # enqueue a run and stream logs until it finishes
valved runs                      # recent runs (--limit, --pipeline)
valved runs --recovery <run_id>  # render a run's recovery-attempt chain
valved logs <run_id>             # logs for a specific run

You can also run just the extract-load (dlt) half of a pipeline:

valved el run <name> --watch
valved el verify <name>          # schema + smoke test, no data load

How a run ends

Each step is judged on evidence of its own: a dlt step on the load package it wrote, a dbt step on run_results.json, and a python step — or a dbt run-operation — on its exit code (see steps). The run’s outcome follows from its steps:

Run status	When
success	Every step succeeded.
warning	Every step finished and none failed, but at least one finished with warnings — a dbt test configured severity: warn fired.
failed	A step failed — including one whose failure_mode let the run carry on past it.

A run can also end cancelled, when someone stops it, or superseded, when the scheduler declines a duplicate trigger.

A warning run did its work, and has something to read. Its downstream steps ran, nothing was retried, and it is not handed to recovery, so no investigation opens and its schedule is not paused. The warning stays in view instead: yellow in the console, in valved runs and in the static UI, printed when valved run --watch ends, and kept in the run’s error_message. The run emits run.succeeded with "status": "warning" in its payload, so a consumer counting successes counts it, and one that cares can tell the two apart. valved run --watch and valved logs --follow exit 0 for it.

Waiting on runs from a script?

Treat warning as finished. A loop that stops only at success or failed waits on a warning run forever.

Scheduling is data, not code

The live schedule lives in the schedules table, and valved schedule changes it instantly — no code change, no PR. Every change is audited in schedule_changes and takes effect on the next scheduler tick.

valved schedule list
valved schedule show <pipeline>
valved schedule set-cron daily_revenue "0 3 * * *" --timezone America/New_York
valved schedule pause  daily_revenue --reason "investigating upstream outage"
valved schedule resume daily_revenue

set-cron upserts: if the pipeline has no schedule row yet it creates one (at the prod target unless you pass --target-pipeline), so a schedule is stood up end to end from the CLI. Bad cron expressions and unknown IANA timezones are rejected up front, before anything is written.

[seed_schedule] is not yet applied to anything

A pipeline’s optional [seed_schedule] block is parsed and validated — a bad cron there fails valved pipelines validate — but nothing currently writes it into the schedules table. The reconciler that would seed it is not shipped, and valved schedule reseed <pipeline> is a placeholder that prints a notice and exits non-zero.

Today, the only thing that creates or changes a live schedule is valved schedule set-cron (or its REST/MCP equivalent). Treat [seed_schedule] as a declaration of intent for a future release, not as a way to schedule a pipeline.

Load plausibility

A load that succeeds but is implausibly small never silently finalizes as a success. On every run, the runtime records per-resource actuals — rows loaded and each resource’s write disposition (replace / merge / append) — onto the run. After each successful load, it judges those actuals against an expectation:

• First successful run — actuals are compared against the plan’s own row estimates (the ones valved plan printed).
• Subsequent runs — actuals are compared against the previous successful run, disposition-aware: replace/merge resources are comparable run-over-run, but an append/incremental resource is never compared against prior totals — an incremental run legitimately loads a tiny fraction of the first load. Once it has history, an incremental resource is judged only by an explicitly declared min_rows floor.
• Graceful degradation — no estimates and no history means the actuals are recorded but no judgment is made. Missing inputs never block a run.

The consequence is graduated:

• Suspect — a moderate deviation (by default, actuals below half the expectation). The run still finalizes success, but it’s marked with a ⚠ suspect badge in valved runs and a durable run.suspect event is emitted per tripped resource.
• Implausible — a high-confidence trip (by default, a ≥10× collapse in a replace/merge resource, or first-run actuals far below the estimate). The run finalizes failed with an error carrying the stable Implausible load prefix, and flows through the normal failure path: the schedule auto-pauses, an investigation records the diagnosis, and recovery classifies it as data-shaped — no automated code fix is attempted; check the source or extraction.

Thresholds, per-pipeline overrides, and the explicit opt-out live under [runner.plausibility] — see the configuration reference.

Observing

Every run, log line, status transition, and cost number is queryable from any surface — CLI, REST, MCP, or the static UI. External agents can subscribe to run-completion webhooks. Roll up cost and success/failure with:

valved metrics ...               # token→USD cost, run success/failure, per-agent usage

When a scheduled run fails after exhausting retries, Valved’s recovery engine steps in — see Recovery.