Scheduling & runs

Once a pipeline is deployed, the control plane executes it — on demand or on a schedule — over a durable, Postgres-backed job queue with a multi-worker pool.

Serving the control plane

valved serve runs the supervisor: the FastAPI app, the cron scheduler, the crash-recovery reaper, the archiver, and a worker pool.

valved serve                     # everything in one process
valved worker                    # or run standalone workers that drain the queue

Workers claim jobs with optimistic-claim semantics, emit heartbeats (so the reaper can reclaim a crashed worker’s jobs after a missed-heartbeat threshold), retry transient failures with backoff, and write structured per-run logs. Supervisor-loop intervals are tunable in valved/runtime.toml.

On-demand runs

valved run <pipeline> --watch    # enqueue a run and stream logs until it finishes
valved runs list                 # recent runs
valved runs list --recovery      # render a recovery chain
valved logs <run_id>             # logs for a specific run

You can also run just the extract-load (dlt) half of a pipeline:

valved el run <name> --watch
valved el verify <name>          # schema + smoke test, no data load

Scheduling is data, not code

A pipeline’s optional [seed_schedule] block seeds the schedules table only at first registration. Thereafter the live schedule is changed instantly via valved schedule — no code change, no PR — and every change is audited in schedule_changes, effective on the next scheduler tick.

valved schedule list
valved schedule show <pipeline>
valved schedule set-cron daily_revenue "0 3 * * *" --timezone America/New_York
valved schedule pause  daily_revenue --reason "investigating upstream outage"
valved schedule resume daily_revenue
valved schedule reseed <pipeline>    # re-apply an edited [seed_schedule] from code

Note

Editing the [seed_schedule] block in a pipeline’s TOML is a no-op on an already-registered pipeline unless you run valved schedule reseed <pipeline>. That’s the one path by which a code edit to the seed takes effect — the live schedule is otherwise the source of truth.

Observing

Every run, log line, status transition, and cost number is queryable from any surface — CLI, REST, MCP, or the static UI. External agents can subscribe to run-completion webhooks. Roll up cost and success/failure with:

valved metrics ...               # token→USD cost, run success/failure, per-agent usage

When a scheduled run fails after exhausting retries, Valved’s recovery engine steps in — see Recovery.