# Pre-registration — Study 2 Phase B: the independence factor — 2026-07-03

> **Amendment (pre-data, 2026-07-03).** After filing and before any item was
> analyzed, `run_debate_swarm` gained an optional `max_workers` argument that
> issues a round's independent per-agent calls concurrently. It is
> measurement-neutral: responses are collected back into per-agent index order,
> so the answer list, majority vote, first-appearance tiebreak, and logs are
> identical to the sequential path (default `max_workers=1`, so every prior study
> is unchanged). This run uses `max_workers=7` purely to cut wall-clock on a
> single stalled provider call. No registered quantity is affected.

Filed before the run. Two prior contact experiments have now shown that *contact*
does not discipline the debate swarm's overconfidence on recall items: Phase A
(§5.6) found a code sandbox trimmed the confidence a debate round *adds* but did
not rescue calibration where it concentrates, and Phase B′ (§5.7) found that even
the *right* channel — a retrieval tool, used on 98% of turns and returning
relevant evidence — left overconfidence unmoved (+0.206 vs no-tool +0.227), and
on the items the swarm got wrong it stayed confidently wrong (+0.82 → +0.65).
Contact was present and used; the coherence did not track it.

Condorcet's jury theorem names *two* conditions for a vote to beat its members:
better-than-chance competence, and **independent** errors. Every swarm run so far
has been homogeneous — one base-model family — and therefore violates the second
by construction: agents that share weights share blind spots, so debate can
launder correlated error into confident consensus rather than correcting it.
Phase B tests the factor the whole program was blocked on until a second family
was wired in: does genuine **independence** discipline the overconfidence that
contact could not? This is the independence main effect (tools off); the full
independence × contact 2×2 is deferred.

## 1. Design (pinned)

Runner: `scripts/run_study2_independence.py` at the commit that files this
document. Same debate machinery as every prior study (`run_debate_swarm`,
unchanged), **7 agents, 2 rounds, temperature 0.7**, majority aggregator emitting
the modal answer's share as the convergence-confidence. No tools in either cell.
Two independent cells per item (no state crosses cells):

- **HOMO** — 7× `claude-haiku-4-5-20251001`. The homogeneous arm; identical model
  set and code path to Study 1, the replicate, and Phase A's NT cell. Routed
  entirely to the Anthropic client.
- **HETERO** — 4× `claude-haiku-4-5-20251001` + 3× `gpt-4o-mini-2024-07-18`,
  routed per-agent by family. Genuinely mixed families, so the agents' errors are
  (partly) independent — the manipulation.

Both families are pinned `supports_logprobs=False`, so the confidence construct
(verbalized number + majority self-consistency) and the confidence *signal type*
are matched across families: **independence is the only thing that varies** between
the cells. gpt-4o-mini is chosen as a capability-comparable, non-reasoning peer to
claude-haiku (both small/fast tier, both take temperature) precisely so any
calibration difference is attributable to independence rather than to a capability
gap or a reasoning-vs-not regime difference.

## 2. Battery (frozen)

n = **90** fresh TriviaQA `rc.nocontext` recall items, ids frozen in
`preregistration/2026-07-03_study2-independence-frozen-battery.json` at this
commit (sha256 recorded there; the runner rebuilds and verifies the fingerprint
before spending). Selection rule, deterministic: the first 90 items in loader
order (`load_triviaqa(limit=400)`) whose `task_id` is **not** in the union of
every prior run's per-item CSV — Study 1, its correction, the replicate, Phase A,
Phase B′, and the difficulty-calibration grading pass. Recall is the locus every
prior study located the overconfidence in, so it is where independence gets its
fairest test. Scoring is the official TriviaQA alias metric, external; ground
truth never enters agent context.

## 3. Registered predictions (stated so they can fail)

**P-I1 — PRIMARY.** Independence disciplines calibration: the HETERO cell's
post-debate overconfidence is **lower than HOMO's** — paired bootstrap 95% CI of
(overconf(HOMO_post) − overconf(HETERO_post)) lies above 0. *Falsified if* the CI
includes or lies below 0. Reported either way, same prominence. (If it fails —
if a heterogeneous swarm is just as overconfident — that is the strongest form of
the project's thesis: coherence without contact survives even independence.)

**P-I2 — debate-inflation.** Independence blunts the confidence a debate round
*manufactures*: the round-0→post overconfidence rise is smaller in HETERO than in
HOMO — paired bootstrap 95% CI of ([overconf(HOMO_post)−overconf(HOMO_r0)] −
[overconf(HETERO_post)−overconf(HETERO_r0)]) lies above 0. *Falsified if* it
includes or lies below 0.

**P-I3 — manipulation check.** The mixed families actually disagree more than
same-family agents: mean round-0 answer diversity is higher in HETERO than HOMO —
paired bootstrap 95% CI of (diversity(HETERO) − diversity(HOMO)) lies above 0.
*If this fails, the manipulation did not produce independence and P-I1 is
uninterpretable* — reported as a precondition, not a result.

**P-I4 — accuracy (estimation only).** accuracy(HETERO_post) − accuracy(HOMO_post),
reported as a magnitude with its 95% CI, **not** a significance test. Explicitly
confounded: any capability gap between `claude-haiku` and `gpt-4o-mini` moves
accuracy independently of independence, so this is descriptive context for P-I1,
not a claim about independence and accuracy.

**Registered descriptives:** per-cell accuracy / confidence / overconfidence /
ECE / MCE / Brier / AUC for the r0 and post arms; round-0 unanimity and diversity
by round per cell; per-cell tokens.

## 4. Analysis conventions (pinned)

10-bin equal-width ECE; tie-corrected Mann-Whitney AUC; paired item-level
percentile bootstrap, B = 10,000, seed 20260703. `instrument/metrics.py`
unchanged. Analysis script (`scripts/study2_independence_analysis.py`) committed
with this filing, before the run completes; published with the run. No slicing
beyond the registered descriptives.

## 5. Cost, gates, honesty

Worst-case ≈ 2,520 calls (7 agents × 2 rounds × 2 cells × 90 items), split
~1,980 Anthropic + ~540 OpenAI; well under the key-level cap and cheap
(gpt-4o-mini is inexpensive). Checkpointed, resumable, mock-validated before any
real token. Anthropic and OpenAI both accept no usable determinism handle for
this call shape; model ids, temperature, prompts, frozen item ids, agent seeds,
and the analysis seed are pinned instead. Known limit, disclosed now: this is the
independence *main effect* on TriviaQA recall with one heterogeneous mix (4:3,
two families); the full independence × contact 2×2 and additional family mixes
are future work. All Section-6 stop-rules of the 2026-06-25 registration apply.