Observability
Dead letter queue, metrics registry, log service, and timer service for kernel-wide monitoring and diagnostics.
WeftOS provides four observability components created during kernel boot when the os-patterns feature is on. That feature is off by default (crates/clawft-kernel/Cargo.toml: os-patterns = ["exochain"], not in default), so a default build has none of them and the accessors return None. Boot creates each as a shared object and exposes it through an accessor; the boot block (boot.rs, "Initialize os-patterns observability modules") does not register them in the ServiceRegistry.
Feature: os-patterns
Phase: K1/K5
Dead Letter Queue
Captures messages that could not be delivered (target not found, inbox full, routing failure). Essential for diagnosing message loss in multi-agent systems.
Source: crates/clawft-kernel/src/dead_letter.rs
pub struct DeadLetterQueue { /* bounded queue of DeadLetter { message, reason, timestamp } */ }
// DeadLetterQueue::new(max_size) or with_default_capacity(); with_chain(cm) logs intake to the chainFailure reasons
pub enum DeadLetterReason {
TargetNotFound { pid: Pid },
InboxFull { pid: Pid },
Timeout { duration_ms: u64 },
GovernanceDenied { reason: String },
AgentExited { pid: Pid },
}Query API
| Method | Description |
|---|---|
intake(message, reason) | Capture a failed message |
query_by_target(pid) | Find dead letters for a specific agent |
query_by_reason(name) | Filter by failure reason |
query_by_time_range(start, end) | Time-windowed queries |
take_for_retry(msg_id) | Remove and return for retry |
re_add(letter) | Return a letter after failed retry |
snapshot() | Get all entries (read-only) |
len(), capacity(), clear() | Size and housekeeping |
Metrics Registry
Three metric types: counters, gauges, and histograms. Thread-safe via atomics and locks.
Source: crates/clawft-kernel/src/metrics.rs
Built-in metrics
MetricsRegistry::with_builtins() registers these names (constants in metrics.rs):
| Metric | Type |
|---|---|
kernel.messages_sent, kernel.messages_delivered, kernel.messages_dropped | Counter |
kernel.agent_spawns, kernel.agent_crashes, kernel.tool_executions | Counter |
kernel.active_agents, kernel.active_services, kernel.chain_length | Gauge |
kernel.ipc_latency_ms, kernel.tool_execution_ms, kernel.governance_evaluation_ms | Histogram |
Boot also sets a few gauges with dotted names (kernel.process.count, kernel.agent.count, kernel.chain.height, kernel.uptime_secs) that differ from the constants above (boot.rs). Which names an exporter should rely on is not settled in the code, so read registry.list_names() on a running kernel.
API
// Counters
registry.counter_inc("name");
registry.counter_add("name", 5);
registry.counter_get("name") -> u64;
// Gauges
registry.gauge_set("name", 42);
registry.gauge_inc("name", 1);
registry.gauge_dec("name", 1);
registry.gauge_get("name") -> i64;
// Histograms
registry.histogram_register("name", &[1.0, 5.0, 10.0, 50.0]);
registry.histogram_record("name", 7.3);
registry.histogram_percentile("name", 0.99) -> Option<f64>;
// Export
registry.snapshot_all() -> Vec<MetricSnapshot>;
registry.list_names() -> Vec<String>;Log Service
Structured kernel-level logging with per-agent attribution, trace correlation, and queryable history.
Source: crates/clawft-kernel/src/log_service.rs
pub struct LogEntry {
pub timestamp: DateTime<Utc>,
pub level: LogLevel,
pub source_pid: Option<Pid>,
pub source_service: Option<String>,
pub message: String,
pub fields: HashMap<String, serde_json::Value>,
pub trace_id: Option<String>,
}Builder pattern:
LogEntry::new(LogLevel::Info, "agent spawned")
.with_pid(pid)
.with_service("supervisor")
.with_trace_id("req-123")
.with_field("strategy", json!("OneForOne"))Queryable via LogQuery with filters for PID, service, minimum level, since and until, trace id, and a limit.
Timer Service
Provides one-shot and repeating timers with sub-second precision. It complements CronService (cron expressions, minute granularity). Each timer delivers a KernelMessage to its owner's PID when it fires (timer.rs header).
Source: crates/clawft-kernel/src/timer.rs
The earlier claim that the cognitive tick, persistence auto-save and mesh heartbeat use this service was not supported by the code (the persistence helper has no timer; see Persistence).
Self-Healing Supervisor
Erlang-inspired restart strategies, budget tracking, exponential backoff, and process supervision for fault-tolerant agent management.
Persistence
What is persisted to disk at 0.8.4: the chain and tree checkpoints, cluster and app catalogs, and the JSON save/load helpers for the causal graph and HNSW index.