Recovery

When a scheduled run fails, Valved doesn’t just alert and stop. The recovery engineer — a subagent in the harness — diagnoses the failure, grounded in the real exceptions dlt and dbt raise, and proposes a reviewable fix. It never writes to prod autonomously.

The auto-fix loop

On a run failure, the recovery engineer runs a bounded diagnose-then-delegate loop:

1. Diagnose the failure from the actual run exception and logs.
2. Delegate a fix to the relevant domain subagent (DLT or dbt engineer).
3. Verify by executing — the same execute-until-green discipline as a normal build.

The loop is budget-bounded. In valved/runner.toml:

[auto_fix]
enabled = true
max_attempts = 3

And a daily spend cap in valved/runtime.toml ([recovery]) — when exceeded, Valved logs the failure but does not spend more on diagnosis until the budget resets. Recovery can be disabled per-pipeline.

You can also opt a single run out of the loop:

valved el run <name> --no-auto-fix
valved el run <name> --max-fix-attempts 1

Investigations

When retries are exhausted, the recovery engineer records an investigation — a durable failure record carrying a diagnosis plus a proposed Plan. That plan flows through the normal build → deploy path: you review it, build it, and deploy it as a PR. Nothing lands in prod without your merge.

valved investigations list --status proposed
valved investigations show <id> --all-runs
valved investigations dismiss <id> --reason "upstream fixed itself"

Resolve an investigation as part of a deploy — which also auto-resumes the pipeline’s schedule:

valved deploy <pipeline> --investigation <id>

The principle

Recovery embodies Valved’s core safety stance: the harness diagnoses and proposes; a human approves. The recovery engineer has the same terminal-grade tools and verify-by-execution discipline as the build agents, but its output is always a reviewable artifact — never an autonomous production write.