clawft

Clustering

How WeftOS nodes track each other as a cluster at v0.8.4: node identity and state, the liveness ping and the heartbeat tracker, signed node facts, the service registry that placement reads, and which of the replicated-state designs (process table, chain and tree sync) are library code only.

Clustering is what the kernel does once it has peers: it tracks who is alive, shares what each node offers, and keeps state in step. The transport and admission are on Mesh networking; how peers are found is on Discovery.

Source: crates/clawft-kernel/src/cluster.rs, mesh_liveness.rs, mesh_heartbeat.rs, mesh_service_adv.rs, mesh_process.rs, mesh_chain.rs, mesh_tree.rs. Features: mesh and cluster, both in the kernel's default features (crates/clawft-kernel/Cargo.toml).

What runs and what is library code

Several of the K6 clustering designs are implemented and unit-tested but are not called by the daemon. This page separates them so that you do not plan around a feature that does not run. Method: for each symbol, a search of crates/ for files other than its defining module, test files and lib.rs re-exports that mention it.

PieceStatus at v0.8.4
Node identity, node idIn use everywhere
NodeState and cluster membership eventsIn use
Liveness ping and pong (mesh_liveness.rs)Started by the daemon (node_facts_rpc.rs)
HeartbeatTracker (SWIM-style states)In use, fed by the liveness pongs, when discovery = true
ClusterServiceRegistryUsed by the workload placement control plane (workload_ctl/plane.rs)
Signed node factsIn use; read by placement and weaver cluster nodes --facts
ChainSyncRequest, ChainSyncResponseA request builder and response type are in mesh_runtime.rs; the fork detector detect_chain_fork has no caller outside its module
DistributedProcessTable, CrdtGossipState, ConsistentHashRing, ProcessAdvertisementNo caller outside mesh_process.rs
compare_tree_roots, MerkleProof, TreeSyncRequest (tree sync)No caller outside mesh_tree.rs
ConsensusRole (Raft-style role)No caller outside mesh_process.rs

An earlier version of this page described the process table, the consistent-hash ring, chain fork detection and Merkle tree sync as the clustering behaviour. They are the intended design (K6); the daemon does not run them yet.

Node identity

A node's identity is an Ed25519 key pair. The node id is hex(SHA-256(pubkey)[..16]), 32 lowercase hex characters, derived in one place (crates/clawft-kernel/src/node_id.rs) and used by the mesh, the cluster, the registry and the substrate. ADR-103 decision D11 chose SHA-256 because it changes the fewest live ids, and amends ADR-025 to match. (The cluster id, a separate value derived from the genesis payload, is still SHAKE-256 in ADR-025.)

Keys certify keys: the machine key certifies a user key, and a user key certifies a project key, with no derivation, so projects can move between machines (Project topology).

Node state

NodeState (cluster.rs) has six values:

Joining, Active, Suspect, Unreachable, Leaving, Left

Liveness and heartbeats

Each node pings every verified peer on the mesh.ping control topic, by default every 10 seconds, and forgets an unanswered ping after 30 seconds (LivenessConfig::default, mesh_liveness.rs). A pong counts only when it arrives on the verified connection of the peer that was pinged and echoes a nonce this node sent and has not yet seen answered. A counted pong updates a smoothed round-trip time and feeds the heartbeat tracker. Unverified peers are never pinged. A pong may carry the responder's load sample. Everything is in memory.

The tracker (mesh_heartbeat.rs) moves a peer through Alive, Suspect and Dead. It exists in the mesh runtime when [kernel.mesh] discovery = true, and its settings default to a 1 second probe interval, 500 ms ping timeout, 1 second indirect timeout, 3 indirect witnesses and a 5 second suspect timeout (HeartbeatConfig::default). Health transitions are emitted as cluster membership events, and peers the tracker considers dead can be disconnected.

Service registry and exposures

ClusterServiceRegistry holds service advertisements per service name, merged last-writer-wins per (service name, node id). The placement control plane builds its view of which nodes serve workload-host and which expose inference from it (workload_ctl/plane.rs). A service that is exposed is advertised, and an exposure that is not announced is refused.

Signed node facts

Placement needs to know what a node can do, not only what it is doing. Each node publishes signed node facts: CPU architecture, memory, runtimes, accelerators and libraries, each with a provenance of claimed, probed or measured. They are signed by the node key, which proves who said it and not that it is true, and they carry a TTL. See Governed placement.

Admission and the cluster owner

Whether a peer is a cluster member is a verdict, not a label. Under admission = "enforce" the mesh listener asks a verdict source, which is a designated cluster-owner daemon (cluster_owner_uid) that answers fail-closed with a 10-minute grace for already-admitted peers (ADR-103 amendment A6). The genesis hash is a cluster label and not a credential. See Mesh networking.

Commands

weaver cluster status          # node count, shard count, consensus
weaver cluster nodes           # every node in the cluster
weaver cluster nodes --facts   # with signed capability facts
weaver cluster health          # per-node health
weaver cluster shards          # shard distribution
weaver cluster join <ADDRESS>  # add a node (see --help for platform and name)
weaver cluster leave <NODE_ID> # remove a node
weaver fleet status            # every known node: trust, last announce, placed cogs, location

weaver fleet location set records an operator-set site and room for a node; the change is chained and needs Admin. The fleet view is described in docs/guides/fleet-manager.md.

On this page