clawft

Observability

Dead letter queue, metrics registry, log service, and timer service for kernel-wide monitoring and diagnostics.

WeftOS provides four observability components created during kernel boot when the os-patterns feature is on. That feature is off by default (crates/clawft-kernel/Cargo.toml: os-patterns = ["exochain"], not in default), so a default build has none of them and the accessors return None. Boot creates each as a shared object and exposes it through an accessor; the boot block (boot.rs, "Initialize os-patterns observability modules") does not register them in the ServiceRegistry.

Feature: os-patterns Phase: K1/K5

Dead Letter Queue

Captures messages that could not be delivered (target not found, inbox full, routing failure). Essential for diagnosing message loss in multi-agent systems.

Source: crates/clawft-kernel/src/dead_letter.rs

pub struct DeadLetterQueue { /* bounded queue of DeadLetter { message, reason, timestamp } */ }
// DeadLetterQueue::new(max_size) or with_default_capacity(); with_chain(cm) logs intake to the chain

Failure reasons

pub enum DeadLetterReason {
    TargetNotFound { pid: Pid },
    InboxFull { pid: Pid },
    Timeout { duration_ms: u64 },
    GovernanceDenied { reason: String },
    AgentExited { pid: Pid },
}

Query API

MethodDescription
intake(message, reason)Capture a failed message
query_by_target(pid)Find dead letters for a specific agent
query_by_reason(name)Filter by failure reason
query_by_time_range(start, end)Time-windowed queries
take_for_retry(msg_id)Remove and return for retry
re_add(letter)Return a letter after failed retry
snapshot()Get all entries (read-only)
len(), capacity(), clear()Size and housekeeping

Metrics Registry

Three metric types: counters, gauges, and histograms. Thread-safe via atomics and locks.

Source: crates/clawft-kernel/src/metrics.rs

Built-in metrics

MetricsRegistry::with_builtins() registers these names (constants in metrics.rs):

MetricType
kernel.messages_sent, kernel.messages_delivered, kernel.messages_droppedCounter
kernel.agent_spawns, kernel.agent_crashes, kernel.tool_executionsCounter
kernel.active_agents, kernel.active_services, kernel.chain_lengthGauge
kernel.ipc_latency_ms, kernel.tool_execution_ms, kernel.governance_evaluation_msHistogram

Boot also sets a few gauges with dotted names (kernel.process.count, kernel.agent.count, kernel.chain.height, kernel.uptime_secs) that differ from the constants above (boot.rs). Which names an exporter should rely on is not settled in the code, so read registry.list_names() on a running kernel.

API

// Counters
registry.counter_inc("name");
registry.counter_add("name", 5);
registry.counter_get("name") -> u64;

// Gauges
registry.gauge_set("name", 42);
registry.gauge_inc("name", 1);
registry.gauge_dec("name", 1);
registry.gauge_get("name") -> i64;

// Histograms
registry.histogram_register("name", &[1.0, 5.0, 10.0, 50.0]);
registry.histogram_record("name", 7.3);
registry.histogram_percentile("name", 0.99) -> Option<f64>;

// Export
registry.snapshot_all() -> Vec<MetricSnapshot>;
registry.list_names() -> Vec<String>;

Log Service

Structured kernel-level logging with per-agent attribution, trace correlation, and queryable history.

Source: crates/clawft-kernel/src/log_service.rs

pub struct LogEntry {
    pub timestamp: DateTime<Utc>,
    pub level: LogLevel,
    pub source_pid: Option<Pid>,
    pub source_service: Option<String>,
    pub message: String,
    pub fields: HashMap<String, serde_json::Value>,
    pub trace_id: Option<String>,
}

Builder pattern:

LogEntry::new(LogLevel::Info, "agent spawned")
    .with_pid(pid)
    .with_service("supervisor")
    .with_trace_id("req-123")
    .with_field("strategy", json!("OneForOne"))

Queryable via LogQuery with filters for PID, service, minimum level, since and until, trace id, and a limit.

Timer Service

Provides one-shot and repeating timers with sub-second precision. It complements CronService (cron expressions, minute granularity). Each timer delivers a KernelMessage to its owner's PID when it fires (timer.rs header).

Source: crates/clawft-kernel/src/timer.rs

The earlier claim that the cognitive tick, persistence auto-save and mesh heartbeat use this service was not supported by the code (the persistence helper has no timer; see Persistence).

On this page