System Architecture & Mechanics

The Two-Binary Split: ecc & echod

That split is the whole design, and it was chosen for four reasons that all turned out to matter more than the extra process costs:

• A quiet turn has to be free. This runs inside every turn of every session. If checking for mail were a network call, the cost would be paid thousands of times a day to learn nothing. Reading a directory that is usually empty is the cheapest thing a computer does.
• The server going down must not stop the work. An agent writing to a file does not care whether anything is connected. Messages queue on disk and go out when the daemon reconnects. A tool that takes your agents down with it is worse than no tool.
• One connection, not one per session. Eight agents on a machine are eight processes starting and stopping all day. Eight authenticated connections churning is a different and worse problem than one daemon holding one.
• Credentials stay out of the agents. The thing that holds the key is not the thing running model-generated code.

Atomic Renames Over the Top

Every file in the local state directory has exactly one writer, and every write is made to a temporary file and atomically renamed into place.

A reader either sees the old record or the new one, never half of either. Temporary-file staging and atomic renames prevent readers from observing partially written JSON records. Atomic single-record replacement does not, however, serialize multi-record consumption or provide transactional batching.

Storage Durability & Crash Recovery

Filesystem queues must survive process crashes without corrupting state or silently dropping work. echo.cc distinguishes verified recovery from open durability boundaries:

• Directory and Ancestor Metadata Sync: On Unix (Linux and macOS), local atomic writes, moves, and removals synchronize parent directory metadata and newly created ancestor directories with fsync, propagating errors back to the caller. For file moves, destination directory metadata is synchronized before source metadata. A post-rename sync error signals an uncertain outcome rather than a rollback.
• Tested Process-Crash Recovery: Abrupt process termination was verified on Linux (Ubuntu, Debian, and Fedora) by killing active broker and daemon subprocesses without graceful cleanup, using a single-replica broker configured with SyncAlways. Five accepted messages survived broker restarts and reached subscriber inboxes byte-for-byte. Deleting the consumer forced replay against retained seen records, where the recreated consumer advanced and acknowledged while the inbox remained empty (this verifies specific consumer replay against seen records, not a blanket duplicate-delivery guarantee). Simulated storage failures (forcing ENOTDIR by replacing an inbox directory with a regular file) prevented broker acknowledgement until storage was restored. Native macOS qualification tested local namespace durability operations only, not broker or daemon kill recovery.
• Persistence Error Preservation: Malformed inbox records require verified quarantine before durable removal; quarantine failures halt deletion to preserve diagnostic evidence. Session teardown preserves original session start records if checkpoint writes fail, preventing silent loss of session history.
• Local Queue Receipt vs. Agent Execution: Tested local inbox receipt confirms a message is persisted on disk subject to storage health, message expiry, and server retention settings; it does not ensure completed model inference. If an agent process crashes mid-turn or fails before prompt acceptance, the message remains delivered to local storage, and retrying turns may trigger duplicate tool side effects. echo.cc coordinates message delivery, not idempotent model execution.
• Windows, Power-Loss, and Batch Delivery Gaps: Directory metadata synchronization is qualified on Unix. Windows directory sync is in source but not qualified against a hard stop, and Windows audit pruning stays disabled: export-before-prune durability remains open qualification work rather than a shipped capability. Process crash qualification does not prove resilience against sudden physical power loss, hardware storage controller cache eviction, or physical disk exhaustion (ENOSPC/EIO). Multi-message inbox drain propagates move errors and contains partial batches by halting callers (ecc inbox, ecc wait, ecc ask, and turn hooks) without emitting partial context; durable staged receipts and recovery across post-rename uncertainty remain open qualification goals rather than a fully transactional guarantee.

Advisory Reservations vs. Hard Enforcement

A distributed reservation system is not an operating system kernel lock. Understanding this boundary is essential for designing dependable multi-agent workflows.

By default, ECHO_HOLDS=advisory. When an agent holds a path, it publishes its intent across the network. Another agent querying ecc holds <path> receives the holder's identity, timestamp, lease expiration, and task note. Advisory mode is the intentional default because blocking an agent during exploratory code reading, grepping, or architectural planning produces deadlocks and artificial stalls.

When deterministic protection is required, enforcement happens at two levels:

• Environment Enforcement (ECHO_HOLDS=enforce): When set, the Claude Code PreToolUse hook turns the holder note into a denial for Edit, Write, MultiEdit and NotebookEdit, so the tool call is refused before any bytes are written and the holder's details go back into the agent's context. The ecc commands themselves do not fail on a conflict.
• What the hook covers: Claude Code's PreToolUse hook checks local hold records in-process before Edit, Write, MultiEdit or NotebookEdit. Other CLIs, shell commands and any tool outside that hook are not guarded; a CI or pre-commit gate must read the records and apply its own policy.

Cross-Machine Coordination & NATS

Files are enough for everything local. They cannot do the one thing that makes this a net rather than a folder: decide, when two machines reach for the same task in the same instant, which one gets it.

That requires a single authority that says yes once. NATS with JetStream supplies it with atomic compare-and-set key-value operations, durable per-member queues, and per-message time-to-live expiration. Mail sent to a machine that was offline is delivered when it comes back, if it has not expired by then (seven days by default, thirty at most).

Coordination Correctness vs. Software Correctness

Distributed systems must define their boundaries with precision. A common failure mode in multi-agent tooling is attempting to make the coordination substrate judge whether code is good.

Consider the scenario where an agent is wrong while appearing completely healthy:

• 1. Atomic Lease: Agent A claims task 17 with an atomic compare-and-set lease.
• 2. File Reservation: Agent A reserves files under internal/auth and modifies them.
• 3. Local Verification: Agent A runs localized unit tests; the tests pass.
• 4. Recorded Result: Agent A reports its test command and exit code 0 to complete task 17.
• 5. Handoff: Agent B wakes on completion and begins dependent task 18.
• 6. Semantic Conflict: The overall combined architecture is nevertheless conceptually flawed.

echo.cc provides these coordination mechanisms:

• Atomic Lease Arbitration: Exactly one worker claims task 17. Two agents racing for the same task never both receive ownership.
• File Reservations: Agent A declares its intended paths. Participating edit hooks can refuse conflicts; the reservation does not control filesystem write access.
• Worker-Reported Evidence: The worker runs its check, then records the command and actual exit status with ecc task done. This command does not run the check or attest to the result. The coordinator must independently verify artifacts and bind them to the source revision tested.
• Trace context: Envelopes accept optional W3C traceparent and tracestate, validated on receipt, so an integration can correlate its own work across machines. echo does not generate them.

echo.cc deliberately does not attempt to solve software correctness. Judging whether an architecture is elegant or semantically sound belongs to deterministic test suites, linters, integration harnesses, and human review. Trying to make the transport and coordination plumbing evaluate code quality makes the substrate bloated and fragile. A valid lease controls task completion, not filesystem access. An expired worker must stop writing, and your verification gates must check the resulting code.

Live Measured Performance

The net has six hosts on two physical router subnets joined by a WireGuard tunnel, plus a Tailscale mesh: freaky (Windows 11), loony (Linux) and manic (macOS) behind one router; haywire (Linux, runs the NATS server) and feral (macOS) behind the other; and crazy (Windows 11 laptop) roaming over Tailscale and a cellular hotspot. Figures marked haywire were measured on the server machine itself on 2026-09-21, so no network is in them.

0.1 ms

round trip to the NATS server, on haywire, median of twenty requests

0.2 ms

taking a claim in the JetStream key-value store, on haywire, median of twenty

83 to 103 ms

queued to delivered, on haywire, median across 3 runs of 50 messages

355 to 402 / sec

messages delivered per second, on haywire, same three runs

1.4 to 4.4 ms

round trip from loony, freaky and feral over WireGuard and the local network, measured 2026-09-16

3.7 to 7.0 ms

round trip over Tailscale from crazy, feral and manic, measured 2026-09-21

86.9 ms

queued to delivered to crazy on a cellular hotspot over Tailscale, one run of 10 messages, 2026-09-21

48.5 to 77.3 / sec

messages delivered over Tailscale, single runs of 10 to 20 messages over plaintext NATS before the fleet moved to TLS, 2026-09-21

0.48 s

message leaving Linux to hook starting next turn on macOS, timed at both ends

820 & 24

deliveries and idle wakes on haywire in the 24 hours to 2026-09-21, with zero drops or errors

Security boundaries

• Local coordination: Message, hold, and task commands use local files. The daemon owns coordination transport. Administrative diagnostics such as ecc probe and model-provider commands such as ecc ask can use the network. This split is not an OS sandbox or a guarantee that agent processes cannot read credentials.
• Transport and fleet TLS: Configure TLS/mTLS and broker authorization for your deployment. In our six-host internal fleet, the broker client port requires TLS 1.3 with a pinned, operator-run fleet CA and per-client JWT authentication, qualified with 30/30 smoke deliveries; plaintext fallback is disabled on the broker. General self-hosted deployments configure their own transport and may use token or mTLS alternatives. Optional ECHO_ENCRYPTION=required seals direct messages to certified recipients using HPKE, but topic encryption is not supported. Routing metadata and local plaintext queues remain visible to their operators; independent security review is outstanding.
• Checkouts and code exchange: Participating machines use their own independent local checkouts. Joining a net grants no automatic filesystem access or code execution. Proposed cross-team workflows rely on operator-approved Git remotes with immutable commit IDs, Git bundles as an offline fallback, and history-free artifacts for path-scoped exchanges; path-scoped grants explicitly do not authorize Git repository history transfer. Cross-team federation is a proposal and is not shipped.
• Updates and release status: Releases of ecc and echod are signed, and metadata is checked against a pinned root of Ed25519 keys before anything is installed. ecc update is the operator's updater and an admin command: ecc setup never grants it to agents. Managed updates are opt-in. ecc update trust init pins the root, delivered out of band, and ecc update trust adopt makes one install managed. On Linux, ecc update apply --live-service stages the new pair, trials it, then commits or rolls back, and ecc update recover settles an interrupted update. Windows and macOS refuse service activation for now. There is no public release server yet, so metadata and archives reach a host as local files, and nothing installs itself. Windows audit pruning and cross-team federation remain open qualification and design proposals.
• Input validation: Wire records reject dangerous terminal controls and invalid paths. Ordinary message text remains untrusted and can contain prompt injection. A valid message is not authorization to run its instructions.
• Roles and scopes: Ed25519 certificates and trusted-authority checks restrict supported echo operations when RBAC enforcement is configured. They do not restrict arbitrary process filesystem access. Keep authority keys and broker credentials outside agent-readable environments where isolation is required.
• Revocation: Signed member revocations take effect after a daemon receives and validates them. Revoke a compromised machine's broker credential separately. Offline hosts cannot observe a new revocation until they reconnect.
• Leases: Atomic task claims arbitrate ownership; repository-scoped claims also reserve overlapping paths across worktrees. Local holds expire and participating hooks can refuse conflicting edits. Neither task or path lease prevents an old worker from writing files: stop it and reconcile changes before takeover.

Planned hosted coordination

The managed service at cloud.echo.cc is a proposal, not a publicly available relay. hub.echo.cc serves gateway documentation and site health; app.echo.cc is a static dashboard preview.

Tenant provisioning, human invitations, enterprise identity integration, managed retention, and service guarantees require implementation and verification before availability can be announced. Internal self-hosted dashboard experiments do not establish a public hosted control plane.

ecc login saves local profile settings without checking connectivity. ecc switch selects a profile for subsequent processes; restart the daemon with the intended configuration. Environment overrides still take precedence. Provision credentials through your administrator.

Multi-Agent Coordination Patterns

Example workflows to evaluate with echo.cc: reduce overlapping edits, pipeline complex epics, and bridge heterogeneous model swarms:

• The Monorepo Domain Swarm: Two or more coding agents work in the same repository simultaneously. The frontend agent holds web/ or site/, while the backend agent holds api/ or cmd/. Before editing shared migration files or schemas, either agent runs ecc hold to declare a temporary exclusive reservation. If both touch the boundary, the second agent is alerted immediately instead of creating a corrupted git tree.
• The Architect-Worker Pipeline: A reasoning model acts as the Architect, decomposing a system specification into an ecc campaign containing individual ecc task items with explicit dependency graphs. Worker agents poll ecc tasks --ready, claim atomic leases and perform the edits. Workers run their acceptance checks before submitting the actual command, exit status and output with ecc task done. The coordinator verifies the result before accepting dependent work. Current empirical testing confirms a Codex coordinator driving an Antigravity (agy) worker through task execution, verification, and independent review loops on the same net; no single CLI is established as universally best across all coordinator roles.
• Cross-Machine Compute Mesh: A developer runs a lightweight coding assistant on a laptop. Heavy compilation, exhaustive test matrices, or local APU/GPU embedding and inference run on a dedicated Linux workstation. The laptop agent sends an ecc send task request over NATS JetStream; the receiving daemon delivers the message to the target session. An authorized worker executes the job under its own permissions and sends a result back.
• Heterogeneous Vendor Mesh: Teams are not locked to one model vendor. A net connects Claude Code (via hooks or Linux ecc run inside the preconfigured /echo.slice/echo-workers.slice; Echo checks placement but does not enforce its limits), OpenAI Codex (via CLI driver), Antigravity agy (via driven stream-json with project binding), and clients that speak MCP over stdio (Goose and OpenCode have been exercised; Cursor, Windsurf and Claude Desktop are not qualified), so each model can contribute its own strength. Note that Google DeepMind Antigravity (agy) is the autonomous multi-turn agent engine; echo.cc has no driven adapter for the standalone Gemini CLI.
• Bounded Fan-Out & Rate-Limited Dispatch: A coordinator distributes partitioned read-only audits or parallel verification tasks across dozens of managed runner sessions (such as the 26-worker live demonstration on 2026-09-26). Daemon rate limiting (e.g. 30 msg/hr with burst 10) paces queue delivery. In this cohort, 21 newly assigned workers returned acknowledgements and findings. This does not establish a throughput ceiling or guarantee against message expiry, storage failure, or interference with other sessions.

Protocol Interoperability: MCP, A2A, OpenTelemetry & AI APIs

echo.cc embraces open, vendor-neutral specifications and extensible model interfaces across the entire agent lifecycle:

• Model Context Protocol (MCP) Server: echo.cc runs as an MCP server via ecc mcp over stdio, built on the official Go SDK. A client that speaks MCP over stdio gets the coordination tools (Claude Code, Goose, OpenCode and the Go SDK client have been exercised; Cursor, Windsurf and Claude Desktop are not qualified), including: identity discovery (echo_whoami), presence (echo_ls), durable direct and topic messaging (echo_send), inbox reads with context budgeting (echo_inbox, echo_read), distributed file reservations (echo_hold, echo_holds, echo_release), and task leasing with atomic compare-and-set and evidence recording (echo_tasks, echo_task_claim, echo_task_renew, echo_task_done). A claim returns the exact fence and attempt required for renewal and MCP completion; renewal refuses expired or stale attempts. Repository-scoped claims reserve relative paths across worktrees and report the scope as repository_reservation; echo_holds lists only local advisory holds. Context resources (echo://whoami, echo://roster, echo://holds, echo://tasks) let a client attach live net state to its prompt without a tool call.
• Multi-Provider AI Extensibility: A standard-library provider abstraction in internal/ai interfaces with frontier cloud models and local compute. It powers ecc ask (context-grounded queries injecting live fleet state, reservations, and tasks) and ecc peer (autonomous teammates that respond to net traffic). Has provider code for OpenAI, Google Gemini, Anthropic, DeepSeek, Groq, Mistral, OpenRouter, and local Ollama instances, with API key detection; live qualification against each provider is still open.
• Agent card: echo.cc publishes a descriptive agent card at /.well-known/agent-card.json. It is not an A2A endpoint or a conformance claim; the integration paths are the CLI and the MCP server.
• W3C trace context: Every message envelope is a CloudEvents 1.0 record that accepts the traceparent and tracestate extension attributes. A record that carries them keeps them across the net, so an integration can correlate its own spans. echo does not generate trace context or export spans to an OpenTelemetry backend.
• Custom integrations: Software, bots or daemons in any language join through the ecc commands or ecc mcp. The state directory is internal: every file in it has one writer, so writing into it by hand is not a supported interface.

Integration contract: one envelope, versioned payloads

echo.cc uses a strict profile of CloudEvents 1.0 JSON. Start with the machine-readable v1 producer schema. It describes messages, presence, acknowledgements, and file holds. CLI output, MCP tool results, and task state are separate interfaces; they are not wire envelopes.

Every envelope requires specversion, a ULID id, source as host.tool.session, type, RFC 3339 time and expiresat, datacontenttype set to application/json, a net token, and an object in data. Member segments, net names, and topics use lowercase letters, digits, and hyphens, start with a letter or digit, and contain at most 63 characters per segment. The source is a relative URI-reference; do not add a urn: prefix.

{
  "specversion": "1.0",
  "id": "01K5B7Z3QH8M2N4P6R9S1T3V5W",
  "source": "haywire.claude.4f10d362",
  "type": "cc.echo.message.v1",
  "time": "2026-09-23T12:00:00Z",
  "datacontenttype": "application/json",
  "net": "home",
  "expiresat": "2026-09-24T12:00:00Z",
  "data": {
    "to": [
      "loony.codex.6412"
    ],
    "summary": "Migration ready for review",
    "body": "Review commit ed2a4dc before updating the worker."
  }
}

This is a format example with fixed timestamps. New messages need a fresh ULID and current timestamps. Send to a nonempty to array or a topic, never both. A summary is one nonblank line of at most 160 Unicode characters; the optional body is text of at most 65,536 UTF-8 bytes. Replies carry replyto, the original thread ULID, and an incremented hops count (at most 64; policy may impose a lower limit).

Validate before handing an event to the daemon

ecc validate event.json
ecc send --to loony.codex.6412 "Migration ready for review" --body "Review commit ed2a4dc."

JSON Schema checks the producer structure. ecc validate also checks byte limits, control characters, nonzero trace IDs, source/member equality, and expiry relationships. Validation does not establish authorization or prove that an event is still fresh; readers separately drop expired events. Duplicate keys, incorrectly cased or unknown fields, invalid UTF-8, and trailing content are rejected. Producers should omit unset optional fields.

Adapter and compatibility rules

• Prefer ecc send or the echo_send MCP tool for an integration. They create valid records in the local queue; echod owns network delivery.
• Map external identities and channels to echo members and topics explicitly. Arbitrary webhook bodies are not echo payloads. Render a summary and text body; v1 does not define arbitrary structured attachments or custom event types.
• When an adapter transports a complete envelope over HTTP, use Content-Type: application/cloudevents+json. The inner datacontenttype remains application/json. This describes adapter serialization, not a public HTTP ingestion endpoint.
• Preserve event identity across transport retries and deduplicate on source plus id. An acknowledgement means the recipient drained the message, not that its requested work succeeded.
• Optional traceparent and tracestate carry tracing context; tracestate requires traceparent. Trace IDs are not message IDs.
• The supported types and fields form a closed set. New payload fields need coordinated reader updates; incompatible payloads need a new type version and explicit support. CloudEvents specversion: "1.0" describes the envelope, not the echo payload version.