# Pre-registration — Study 2 Phase B′: the retrieval contact channel — 2026-07-03

Filed before the run. Phase A (§5.6) found that a code sandbox disciplined the
confidence debate manufactures *in aggregate* but did **not** rescue calibration
on the recall/knowledge items where the overconfidence concentrates, because
code cannot retrieve. It sharpened the "contact is dominant" claim to "the
*right* contact is dominant." Phase B′ tests that directly: give the same swarm a
**retrieval** tool over each item's own provided evidence and ask whether *that*
channel disciplines calibration where code did not. Inherits Amendment 1 and all
Section-6 stop-rules of the 2026-06-25 registration.

## 1. Design (pinned)

Runner: `scripts/run_study2_phaseB.py` at the commit that files this document.
Model `claude-haiku-4-5-20251001`, temperature 0.7, **7 agents, 2 rounds**, equal
votes. Three independent cells per item (no state crosses cells):

- **NT** — no tool (control, as in Phase A / the replicate's confirmed arm).
- **CODE** — one sandboxed Python computation per turn (Phase A's channel, re-run
  on these items as a within-study control for "the wrong tool").
- **RETRIEVAL** — one evidence query per turn: an agent emits a `search` block;
  the harness returns query-ranked evidence sentences from the item's own
  provided documents, with **every scored answer alias redacted** by the same
  normalization the scorer uses (`instrument/contact/retrieval.py`).

## 2. Battery (frozen)

n = **90** fresh TriviaQA `rc` recall items **with provided evidence**, item ids
frozen in `preregistration/2026-07-03_study2-phaseB-frozen-battery.json` at this commit — the recall
population where Phase A located the overconfidence. Every item id used in
Study 1, the replicate, or Phase A is excluded (deterministic; verified stable
across rebuild). Scoring is the official TriviaQA alias metric, external; ground
truth never enters agent context.

## 3. Leak-safety (a hard gate, not a hope)

The retrieval channel is only valid if it can never hand an agent the scored
answer. Enforced three ways: (a) redaction by normalized-alias match, with a
final assembled-output redaction pass and a safe fallback (a sentence that would
still leak is withheld); (b) a regression test (`tests/test_retrieval_leak.py`)
incl. the cross-snippet-join failure mode; (c) a **pre-flight leak audit in the
runner that aborts before any paid call** if any item's retriever leaks. Filing
condition: the audit reports 0 leaks on the frozen battery (verified: 0). Reason
from redacted evidence to the answer is contact *working*, not a leak.

## 4. Registered predictions (stated so they can fail)

**P-R1 — PRIMARY.** Retrieval disciplines calibration where code did not: the
RETRIEVAL cell's post-debate overconfidence is **lower than NT's** — paired
bootstrap 95% CI of (overconf(NT_post) − overconf(RETRIEVAL_post)) lies above 0.
*Falsified if* the CI includes or lies below 0. Reported either way, same
prominence.

**P-R2 — accuracy.** Retrieval improves accuracy over NT on these recall items:
paired bootstrap 95% CI of (accuracy(RETRIEVAL_post) − accuracy(NT_post)) lies
above 0. *Falsified if* it includes or lies below 0.

**P-R3 — the "wrong tool" control.** CODE does *not* meaningfully help here:
its overconfidence and accuracy are within noise of NT (CIs of the CODE−NT
contrasts include 0), reproducing Phase A's finding on this fresh recall slice.
*Complicated if* CODE helps as much as RETRIEVAL (then the effect is generic tool
access, not retrieval-as-contact).

**P-R4 — mechanism (may be underpowered).** Within RETRIEVAL, grounded final
answers (a final-round modal-answer agent issued a query) calibrate better than
ungrounded ones. Reported UNDERPOWERED if retrieval uptake is near-universal
(as in Phase A) or either group has < 8 items.

**Registered descriptives:** per-cell accuracy / confidence / overconfidence /
ECE / MCE / Brier / AUC; round-0 unanimity and diversity by round per cell;
retrieval- and code-uptake rate; per-cell tokens.

## 5. Analysis conventions (pinned)

10-bin equal-width ECE; tie-corrected Mann-Whitney AUC; paired item-level
percentile bootstrap, B = 10,000, seed 20260703. `instrument/metrics.py`
unchanged. Analysis script committed before the run completes; published with
the run. No slicing beyond the registered descriptives.

## 6. Cost, gates, honesty

Worst-case ≈ 7,560 calls (~$20–30 at Haiku pricing; realistic lower with partial
tool uptake), under the key-level cap; checkpointed, resumable, mock-validated,
and real-1-item-smoked (100% retrieval uptake, 0 leaks, correct answer) before
the full run. Anthropic accepts no seed parameter; model id, temperature,
prompts, frozen item ids, redaction rule, and analysis seed are pinned instead.
Known limit, disclosed now: this tests retrieval on TriviaQA recall only (MMLU
knowledge items ship no evidence corpus); a general retrieval channel is future
work. All Section-6 stop-rules of the 2026-06-25 registration apply.