Recovery

When a scheduled run fails, Valved doesn’t just alert and stop. The recovery engineer — a subagent in the harness — diagnoses the failure, grounded in the real exceptions dlt and dbt raise, and proposes a reviewable fix. It never writes to prod autonomously.

The auto-fix loop

On a run failure, the recovery engineer runs a bounded diagnose-then-delegate loop:

1. Diagnose the failure from the actual run exception and logs.
2. Delegate a fix to the relevant domain subagent (DLT or dbt engineer).
3. Verify by executing — the same execute-until-green discipline as a normal build.

The loop is budget-bounded. In valved/runner.toml:

[auto_fix]
enabled = true
max_attempts = 3

Control-plane recovery — the trigger that fires on a scheduled run’s failure — is governed by a separate [recovery] block with two settings: an install-wide enabled switch (default on; when off, valved serve wires no recovery trigger at all) and a daily spend cap, daily_token_budget_usd (default $5.00). When the cap is reached, Valved logs the failure and skips diagnosis entirely — no model invocation — until the budget resets at UTC midnight.

The cap counts all agent spend, and [recovery] isn’t file-configurable yet

The cap does not measure what recovery has spent. It reads the day’s total agent-attributed cost — every plan, build, engineer, reviewer and ask recorded since UTC midnight — and compares that single number to the ceiling. Reviewer invocations began landing in that table in 0.3.0, so review-heavy days now count against it where they previously didn’t, and the cap trips sooner than it used to. A busy day of building can silence scheduled recovery for the rest of that day.

Neither setting can be changed from a file today. The config loader reads no [recovery] section from valved.toml or any file in valved/, so such a block is silently ignored and the two defaults above are always what apply. (This is distinct from [auto_fix] above, which is read from valved/runner.toml.)

You can also opt a single run out of the loop:

valved el run <name> --no-auto-fix
valved el run <name> --max-fix-attempts 1

How failures are classified

Diagnosis starts from the real exceptions dlt raises, not from guesswork. Each failure is classified into a category that decides whether a fix is proposed — and to which engineer:

Category	Propose a fix?
Schema-contract drift (column add/remove, type change)	yes → DLT engineer (add/remove column, relax contract, type hint)
Terminal load failure (NOT NULL, PK conflict, bad type)	no — data-quality; surface the logs
Missing relation / destination scaffolding	partial → SQL specialist (provision the missing container)
Transient (network / rate-limit / timeout) after retry exhaustion	no fix — recommend retry tuning
Destination outage (multiple pipelines failing at once)	no — infrastructure; no pause
Credentials expired / revoked	no — instructs a credential refresh; never touches secrets
Model refusal (the LLM declined as policy)	no — terminal and non-retryable. A refusal is deterministic: the identical prompt on the identical model refuses again, so it is never retried on the auto-fix budget.
Implausible load (the plausibility gate tripped — the load succeeded but was implausibly small against its own history or estimate)	no — data-shaped; the analyst owns it. The load ran fine, it just wasn’t credible — check the source or extraction. There is no automated code fix; recovery records an honest summary instead of a misdiagnosis.
Novel / unclassified	no — record the logs; the analyst owns it

An implausible-load failure arrives here through the normal path: the run finalizes failed with the stable Implausible load error prefix, the schedule auto-pauses, and an investigation records the diagnosis — classified data-shaped, so no auto-fix is attempted.

Investigations

When retries are exhausted, the recovery engineer records an investigation — a durable failure record carrying a diagnosis plus a proposed Plan. That plan flows through the normal build → deploy path: you review it, build it, and deploy it as a PR. Nothing lands in prod without your merge.

valved investigations list --status proposed
valved investigations show <id> --all-runs
valved investigations dismiss <id> --reason "upstream fixed itself"

Resolve an investigation as part of a deploy — which also auto-resumes the pipeline’s schedule:

valved deploy <pipeline> --investigation <id>

The principle

Recovery embodies Valved’s core safety stance: the harness diagnoses and proposes; a human approves. The recovery engineer has the same terminal-grade tools and verify-by-execution discipline as the build agents, but its output is always a reviewable artifact — never an autonomous production write.