# Echo recovery qualification and coding comparison protocol

Protocol draft 0.2, 6 October 2026 UTC. The isolated recovery qualification is runnable. The upstream comparative study below remains planned and unrun. A separate original synthetic development diagnostic completed 48 local-model attempts with zero accepted pairs; all failures are retained. A separately qualified tool-contract revision on the same exposed synthetic tasks completed another 48 attempts: single 8/16, ordinary two-worker messaging 0/16, Echo 0/16. Neither run establishes fresh benchmark performance or coordination benefit. This is not a preregistered coding-result claim.

## Independently runnable recovery qualification

Prerequisites: reviewed Echo source checkout/source bundle, Python 3, matching Go toolchain from go.mod, and already populated module cache. Public source/binary release remains gated, so obtaining a permitted source bundle is an access prerequisite; these instructions do not imply an existing public binary release. The fixture builds a separate ecc binary from the supplied source and does not install or replace the user's ecc/echod.

```bash
python3 scripts/research-recovery-fixture.py --repo . --out /tmp/echo-recovery-result.json
```

On hosts where `/tmp` is mounted noexec, add `--work-root /path/to/owned/executable/cache`. The runner creates and removes its own private child directory there; never point at another worker's temporary directory. Offline dependency/toolchain preparation is a separate setup step, not a hidden network operation during the fixture.

The Python runner launches actual ecc subprocesses in a private temporary workspace and state directory. It checks owned identity registration, advisory hold persistence/expiry, task persistence across fresh CLI processes, failure-closed claim without an owning daemon, and refusal of unowned completion. It additionally runs selected existing source tests for broker claim exclusivity, stale fences, lease expiry, actual command-handler lifecycle, recovery decisions, unsafe recovery refusal, replay after audit persistence and stale attempts, and logical-cross-host claim exclusion.

Go dependencies are offline (GOPROXY/GOSUMDB off, GOTOOLCHAIN local); selected broker tests use owned loopback JetStream, not the user's broker. The subprocess environment contains explicit runtime settings and no provider credentials. State and diagnostic report are application-private. No model work occurs. The test cases exercise exact source behavior and scripted crash boundaries, not actual arbitrary third-party agent reasoning or released-binary distributed installation. Logical hosts inside a local test are not separate physical hosts. Worker-supplied verification commands are recorded evidence; Echo does not independently execute/attest those commands.

Use a permitted clean source checkout for an independent result; a locally modified tree is explicitly flagged. Record source commit, curated-file hashes, runner hash, Go version, OS and result JSON. A reviewer should independently run it from their own permitted clean checkout. Passing on Cisco's machine is qualification, not outside reproduction or proof coordination improves coding.

## Coding question and frozen design choices

Question: at equal declared resources, does Echo coordination improve the rate of two-feature tasks accepted by hidden execution checks compared with simpler coordination? Primary contrast is Echo versus two ordinary isolated worktrees with plain messaging. A single agent handling both features is a separate baseline.

Conditions: (A) one model agent performs both features; (B) two same-model agents, separate worktrees and a simple durable message mechanism, no Echo task/hold policies; (C) two same-model agents, same worktree isolation, Echo messaging/tasks/advisory holds. Hold enforcement must be identical advisory behavior, not extra filesystem protection present only in C. Prompts differ only where the coordination interface requires it. Every condition gets the same aggregate model-token cap, wall-time cap, permitted commands and task information. Team budgets include both workers, coordination context and retries; no hidden planner or free integration agent.

Pilot: eight previously unseen feature pairs, two stochastic trials per condition. Primary endpoint: fraction of task pairs whose final merged artifact passes both feature hidden tests within caps. Secondary: feature-level pass rate, merge conflict/failure, elapsed time, aggregate model input/output tokens, tool commands, operator interventions, recovery/duplicate work. Aggregate token cap: 25,000 per task pair, wall cap 30 minutes, integration grace inside that cap. Exact provider model version, decoding settings, cost prices and adapters must be frozen before execution. No paid runs are authorized by this protocol.

Selection: freeze a permitted upstream revision, then compute SHA-256(seed `echo-pilot-v01` + canonical repo/task/feature-pair ID), sort and choose the first eight eligible pairs across at least two repositories. Eligibility is determined by artifact-license clearance, environment feasibility and untouched-base/combined-oracle grading checks, without looking at agent successes. Development pairs must be disjoint. Every exclusion and failed environment preparation enters the manifest; do not replace hard tasks after seeing outcomes. Pilot is for feasibility/variance; no leaderboard or powered-generalization conclusion.

Hidden tests and gold implementations are inaccessible to agents. The evaluator may use gold only to sanity-check grading, never to merge agent code. Integration uses one declared ordinary merge procedure across B/C. An unresolved conflict is failure in the main endpoint; a separately labeled salvage arm may measure operator/model merge work under an equal budget. Changes to CooperBench's original merging procedure must be explicit, not represented as an exact replication.

A separate recovery arm injects one owned-worker termination after the twentieth aggregate successful tool command, if reached, and measures safe ownership/restart/final acceptance. No-fault tasks form the main capability comparison. Do not equate a recovery fixture PASS with acceptance of code; attempts that never reach the injection point remain in their denominators with injection marked not reached.

Statistics: paired task-seed outcome differences for primary B/C contrast, cluster uncertainty by originating repository/task set to account for overlapping features. Pilot intervals and all attempts reported; a larger study's sample size follows variance and a prespecified meaningful effect. Zero accepted tasks means cost per accepted task undefined, not zero. Missing usage is unknown, not free. Infrastructure failures, refusals and timeout are retained as labeled failed/inconclusive attempts with costs.

## CooperBench source and artifact licensing

Candidate upstream revision observed 6 October 2026: `63b9d44d9f39a02fccf5bf0052db48a917a011fd`; this is an observation, not yet a frozen task/image manifest. Primary [dataset README](https://github.com/cooperbench/CooperBench/blob/63b9d44d9f39a02fccf5bf0052db48a917a011fd/dataset/README.md) describes per-feature prompts, gold patches and hidden tests, with version-matched dataset/harness/images. Its metadata declares MIT, and [package metadata](https://github.com/cooperbench/CooperBench/blob/63b9d44d9f39a02fccf5bf0052db48a917a011fd/pyproject.toml) declares MIT. The attempted main LICENSE URL was absent; metadata is not a completed artifact-rights audit.

The [paper's repository table](https://cooperbench.com/static/pdfs/main.pdf) distinguishes MIT, MIT-CMU, BSD-3 and Apache-2.0 upstream repositories. Preserve each task's original repository license/notices; verify image/distribution terms and exact source commits. Underlying task code is not automatically covered by benchmark-package metadata. Pin container digests and grading scripts before execution. No task data or model outputs were downloaded for this protocol preparation.

## Independent review packet

Supply: source/runner hashes and permitted installation bundle, fixture instructions and results, protocol version, exact task/model/image/config manifests, license/notice matrix, acceptance rubric, sanitized per-attempt results and complete accounting. Provide reviewer access to permitted artifacts, not credentials or raw personal transcripts. Ask them to rerun recovery qualification, challenge the closest-baseline design, verify hidden-test isolation, and reproduce a small permitted subset. Record the reviewer identity/method, exceptions and failures with consent. Automated model critiques are useful advisory reviews; they do not count as an independent researcher reproducing results. Do not send this packet or outreach without the user's explicit send authorization.


## Retained original synthetic development diagnostic

The first version used eight original function pairs, two seeds and three conditions, with a pinned local qwen3-coder:30b model and 25,000 aggregate-token / 30-minute artifact caps. All 48 attempts are retained: single, ordinary two-worker messaging and Echo each accepted zero of 16 pairs. Accounting reconciled 138 model requests, 126,116 prompt tokens and 32,975 returned tokens; usage was complete for every attempt. External paid spend was zero; hardware and electricity are unmeasured. With zero completions, cost per accepted pair is undefined.

This negative diagnostic exposed tool-contract failures and supplies no comparative coding gain or equivalence evidence. It is not CooperBench, a representative coding benchmark or the upstream pilot proposed above. A V2 contract uses explicit operation arguments, mutually exclusive structured payloads and whole-batch validation. Its separate scripted qualification passed 48 attempts, including 24 golden checks, 24 expected incomplete-base failures, 24 actual recovery faults and seven VM controls. Scripted qualification is not model performance. V2 reuses exposed tasks; its development outcomes must remain separate from V1 and cannot support an isolated causal or fresh-holdout claim. [Dated report and evidence limits](https://tuneharness.com/research/2026-10-06/#mathematical-reference-and-echo-qualification).


V2 model execution and metadata analysis are complete. The 48 attempts used 385 requests, 409,184 prompt tokens and 23,387 returned tokens; usage was complete for every attempt. Single-context acceptance was 8/16; ordinary two-worker and Echo conditions each accepted 0/16. Thirteen merge conflicts and all failed attempts remain in their denominators. The all-failure two-worker contrast establishes neither equivalence nor benefit; the single condition is a separate descriptive baseline. [Public aggregate results and commitments](/research/original-coding-development-results.json). No model fault arm was run.