Self-Healing Supervisor
Erlang-inspired restart strategies, budget tracking, exponential backoff, and process supervision for fault-tolerant agent management.
The self-healing supervisor provides Erlang-inspired fault tolerance for WeftOS agents. When an agent crashes, the supervisor applies a configurable restart strategy, respects a budget to prevent restart storms, and uses exponential backoff to avoid thundering herds.
Feature: os-patterns (off by default: crates/clawft-kernel/Cargo.toml). Without it the strategy, budget and tracker types below are not compiled.
Source: crates/clawft-kernel/src/supervisor.rs
Phase: K1
Restart Strategies
Five strategies determine what happens when an agent fails:
pub enum RestartStrategy {
OneForOne, // Restart only the failed child (default)
OneForAll, // Restart all children if one fails
RestForOne, // Restart failed child + all started after it
Permanent, // Do not restart — let the agent stay dead
Transient, // Restart only if exit code is non-zero
}Strategy Selection Guide
| Strategy | Use Case |
|---|---|
OneForOne | Independent agents that don't share state |
OneForAll | Tightly coupled agents where partial state is invalid |
RestForOne | Pipeline stages where downstream depends on upstream |
Permanent | One-shot tasks (batch jobs, migrations) |
Transient | Agents that may exit cleanly (0) but should restart on crash |
Restart Budget
Prevents infinite restart loops by capping restarts within a time window:
pub struct RestartBudget {
pub max_restarts: u32, // Default: 5
pub within_secs: u64, // Default: 60
}record_restart returns false once the restart count in the window exceeds max_restarts. The window resets after within_secs elapses (check_window). On exhaustion AgentSupervisor::handle_exit logs a warning ("restart budget exhausted for pid, escalating") and returns no restarts; there is no parent supervisor in the code to notify, so "escalation" here means the agent stays down.
Exponential Backoff
Each restart increases the delay exponentially:
100ms → 200ms → 400ms → 800ms → ... → 30s (cap)Formula: 100ms * 2^(restart_count - 1), capped at 30 seconds (next_backoff_ms). With a trained RestartStrategyModel (EML), next_backoff_ms_with_model uses the learned delay instead; untrained or absent, the result is the formula above (supervisor.rs).
Restart Tracker
The RestartTracker maintains per-agent restart state:
pub struct RestartTracker {
pub restart_count: u32,
pub window_start: Instant,
pub last_restart: Option<Instant>,
pub backoff_ms: u64,
}Key methods:
| Method | Description |
|---|---|
record_restart(budget) | Record a restart, returns false if budget exceeded |
is_exhausted(budget) | Check if budget is used up |
remaining(budget) | Restarts remaining in current window |
next_backoff_ms() | Calculate next backoff delay |
should_restart(strategy, exit_code) | Policy check for restart decision |
Supervisor integration
Strategy and budget are set once on the supervisor, not per spawn:
let supervisor = AgentSupervisor::new(/* ... */)
.with_restart_config(RestartStrategy::OneForOne, RestartBudget { max_restarts: 3, within_secs: 30 });There is no spawn_supervised method. When an agent exits, the daemon calls AgentSupervisor::handle_exit(pid, exit_code):
- Link and monitor signals are delivered (
monitor.rs), withNormalfor exit code 0 andCrashotherwise. RestartTracker::should_restart(strategy, exit_code)decides:Permanentnever,Transientonly on a non-zero code, the others always.record_restartcharges the per-PID budget; over budget, nothing is restarted.- Within budget, an
agent.self_heal_restartevent (strategy, exit code, backoff) is appended to the chain whenexochainis on, then the strategy is applied:OneForOneandTransientrestart the failed process,OneForAllrestarts it and its siblings,RestForOnerestarts it and those started after it.
handle_exit records the computed backoff_ms and logs it; reading the function, the delay is not slept inside it, so the caller must honour the returned backoff. The page that preceded this one said the agent "sleeps for backoff_ms then restarts"; that was not found in handle_exit.
A separate watchdog_sweep reaps finished tasks: processes still shown as running are moved to Exited(-2) (watchdog) or Exited(-3) (panic) (supervisor.rs, native feature).
Configuration
No [supervisor] table in weave.toml exists in the code at 0.8.4 (a git grep for [supervisor], default_strategy and restart_window_secs over crates/ finds only unrelated project-certificate settings in clawft-types). Per-project child-kernel supervision has its own settings, restart_max, restart_window_secs and idle_stop_secs, described in the project topology guide (Project topology); those restart whole project kernels, not agents.