Final Year Design Project — Research Proposal
Implementation status (Nov 2026). P0 is engineering readiness: 615 tests, 13/13 gate checks passed. No efficacy result has been measured. Planned work is labelled as planned throughout.
Multi-Agent Debate (MAD) improves the factual accuracy of large language models (LLMs) by having several agents argue toward a shared answer, yet it suffers a documented failure mode: inter-agent sycophancy, where a confident-but-wrong majority pressures a correct minority agent into abandoning its position, and a plain majority vote then locks in the wrong answer. This proposal introduces a trust-calibrated deliberation framework in which a panel of heterogeneous open-weight LLMs debate a scientific question while each agent's influence is dynamically re-weighted by a bounded trust score driven by the verifiability of its claims against retrieved scientific literature. The final answer is produced by trust-weighted aggregation rather than a headcount, allowing an evidence-backed minority to outweigh an unsupported majority. Alongside the mechanism, the project targets a reproducible sycophancy-stress evaluation harness — a controlled injection protocol, three metrics (CCR, MPR, ECR), a controlled benchmark, and a pre-registered causal contrast (B8 vs B8-no-trust) — with four executable arms now, a six-arm target, and three Core datasets (GPQA-Main, MMLU-Pro STEM, HLE as an abstention control). The planned benchmark follows a scale ladder of 180 items as the floor, 300 as the target, 600 as a stretch, and 800 as a no-go. The work runs entirely at inference time (no fine-tuning) on a single-GPU budget. Expected outcomes are stated as pre-registered success criteria, not as results: a statistically significant reduction in collapse rate and an open-source evaluation suite reusable by the research community.
Getting several large language models (LLMs) to debate a question before agreeing on an answer — called Multi-Agent Debate (MAD) — is a widely used way to improve factual accuracy and reasoning quality (Du et al., 2023; Chan et al., 2024). By exposing each agent to the critiques of others, MAD encourages diverse reasoning paths and lets the group recover from individual errors. However, these debates carry a real and well-documented weakness: an agent frequently drops a correct answer simply to align with a confident-sounding majority, so the group converges on a confidently wrong consensus (Yao et al., 2025). Because contemporary models are trained on heavily overlapping corpora, they tend to make correlated mistakes; a plain majority vote can therefore quietly bury a correct minority. Empirically, the minority turns out to be right in roughly one in four disagreements (He et al., 2026). In scientific, medical, or clinical question answering, a confidently wrong group answer is more harmful than an honest “I am not sure,” because it is delivered with the apparent authority of a consensus.
As LLM systems are increasingly deployed as research assistants, tutors, and decision-support tools, this false consensus among cooperating agents becomes a trust and safety problem, not merely an accuracy statistic — it fails silently and confidently. Sycophancy is cited as an open problem in over 50 peer-reviewed papers since 2023. In multi-agent systems specifically, it directly undermines the reason MAD is used: if debate collapses to whichever position sounds most confident rather than which is most correct, it adds computational cost without adding reliability. Existing responses fall into three camps: diagnosis (Yao et al., 2025; He et al., 2026), which measures the problem without proposing a mitigation; efficiency optimisation (iMAD, Fan et al., 2026, AAAI Oral), which learns when to trigger a debate rather than whom to trust during it; and static aggregation (Mixture-of-Agents, Wang et al., 2025, ICLR Poster), which combines model outputs without dynamic evidence grounding. Crucially, none dynamically re-weights agent influence during a debate based on whether that agent's claims can be verified against external evidence. This proposal targets exactly that gap, on the argument that the only reliable way to break a social-pressure collapse is to tie influence to something external and independent of the debate's internal dynamics.
Two recent published mitigations deserve particular attention. ConsensAgent (Pitre et al., Findings of ACL 2025) reduces sycophancy via static and dynamic prompt refinement — a textual intervention that does not produce a numeric trust score and does not use external evidence retrieval. DebUnc (Yoffe et al., Findings of EMNLP 2025) modulates in-loop influence using self-reported uncertainty signals, which are themselves susceptible to the same social pressure that drives sycophancy. Our approach is differentiated from both by its reliance on external, independent evidence verification as the basis for trust updates, rather than textual or self-reported signals. The theoretical foundation for majority-convergence in multi-LLM debate is established by Estornell and Liu (NeurIPS 2024), who prove a “tyranny of the majority” result and propose pruning-style interventions, but stop short of a deployable, evidence-grounded system.
In current multi-agent debate systems, an agent's influence on the final answer depends on how many other agents agree with it — not on whether what it says is factually supported. There is no mechanism that adjusts each agent's trust score during the debate based on the verifiability of its claims against retrieved scientific literature. As a consequence, a hallucinating majority can systematically override a correct, well-supported minority. This inter-agent sycophancy undermines the very reason MAD is used: if a debate collapses to whichever position sounds most confident rather than which is most correct, it adds computational cost without adding reliability. The problem is most acute in high-stakes scientific and clinical applications, where the cost of a confidently wrong consensus is highest.
The problem of inter-agent sycophancy has been formally diagnosed but not solved with an in-session, evidence-grounded mechanism. We position our work against the closest lines of prior research:
| Prior work | What it does | Why the gap remains |
|---|---|---|
| Yao et al. (2025) · He et al. (2026) | Define and quantify inter-agent sycophancy (minority correct in ~25% of disagreements). | Diagnostic only — measures the failure, proposes no mitigation. |
| iMAD — Fan et al. (2026), AAAI Oral | Learns when to trigger a debate for token efficiency. | Decides whether to debate, not whom to trust once debating; falls back to majority vote. |
| MoA — Wang et al. (2025), ICLR Poster | Layered multi-agent aggregation of diverse LLMs. | Static, equal-weight — no evidence grounding, no dynamic trust. |
| ConsensAgent — Pitre et al. (2025), ACL Findings | First published sycophancy mitigation, via prompt refinement. | Textual intervention — no numeric trust, no external evidence, no adversarial-injection testing. |
| DebUnc — Yoffe et al. (2025), EMNLP Findings | Modulates in-loop influence using uncertainty signals. | Uses self-reported uncertainty — the very signal social pressure can manipulate. |
| Estornell & Liu (2024), NeurIPS | Proves majority-convergence (“tyranny of the majority”); pruning-style interventions. | Theoretical; not a deployable, evidence-grounded system. |
| Minority Sentinel (2026), SIGIR Workshop | Post-hoc correction based on log-level features after debate. | Intervenes only after debate ends; no in-session trust adjustment. |
Research gap (one sentence): no existing framework dynamically re-weights agent influence within an active debate session based on the real-time verifiability of claims against external evidence with a formally bounded numeric trust score. Prompt refinement (ConsensAgent), self-reported uncertainty (DebUnc), theoretical pruning (Estornell & Liu), and post-hoc correction (Minority Sentinel) each address adjacent problems, but none ties influence to external evidence verification during the debate. Our closest published competitor is ConsensAgent; our sharpest differentiation is that our trust signal is external, evidence-grounded, numeric, and formally bounded, rather than a textual prompt tweak.
Research question. When a multi-agent debate faces a planted, confident-but-wrong majority, does evidence-grounded trust weighting preserve correct minority answers better than a plain majority vote and other standard aggregation methods?
Central hypothesis. If an agent's influence in aggregation is dynamically re-weighted during the debate based on external evidence verification, then sycophantic collapse — agents abandoning correct answers under social pressure — will measurably decrease relative to standard majority-vote debate, without any model fine-tuning. This decomposes into three pre-registered targets, to be tested rather than assumed:
To answer the research question, the objectives are to:
We propose a trust-calibrated deliberation system in which a panel of N = 3 heterogeneous open-weight LLMs from different model families debate a scientific question over K = 3 rounds. As the agents argue, each agent's claims are decomposed into atomic factual statements and checked against partitioned scientific sources. Every agent i carries a trust score Si that rises when its claims are supported by retrieved evidence and falls when they are contradicted, and the final answer is produced by trust-weighted aggregation rather than a simple headcount. The system operates entirely at inference time with no fine-tuning required.
The architecture follows a sequential pipeline with a feedback loop. The input (a scientific question) first enters a lightweight confidence gate. If the gate determines the question is straightforward, a direct answer is returned immediately. Otherwise, three heterogeneous agents each produce an independent initial position. Each agent's answer is decomposed into atomic claims, which are routed to partitioned scientific corpora (PubMed, arXiv, Semantic Scholar) for evidence retrieval. A cross-encoder reranker scores each claim as supported, contradicted, or unverifiable. The trust scores are updated based on these verdicts. Agents then revise their positions in light of peer arguments and their own trust standing. The retrieval→trust-update→ revision loop repeats for K rounds, after which trust-weighted aggregation produces the final answer with citations and a full trust trajectory log.
The end-to-end flow consists of nine stages: (i) a lightweight confidence gate answers easy questions directly and routes only genuinely uncertain ones into debate; (ii) the three agents state independent initial positions (Round 0); (iii) claim decomposition converts each answer into tagged atomic propositions; (iv) source-partitioned retrieval routes each agent's claims to a distinct corpus (Agent A → PubMed, B → arXiv, C → Semantic Scholar); (v) a cross-encoder scores each claim as supported / contradicted / unverifiable; (vi) the trust update converts those verdicts into new trust scores; (vii) agents revise their positions, aware of peers' arguments and their own trust standing; steps (iii)–(vii) repeat for K rounds; and (viii) trust-weighted aggregation emits the final answer plus citations, the full trust trajectory, and per-agent reasoning traces. Every intermediate output is logged to enable post-hoc analysis of
A cheap gate avoids wasting the full ~9-forward-pass pipeline on easy questions where all agents would already agree correctly. It uses a three-sample agreement check (one fast pass per agent); unanimous, high-confidence agreement returns a direct answer, while any divergence or low confidence triggers the full debate. Mis-gating an easy question wastes only compute, never correctness, since the evaluation sets are pre-filtered to genuinely divergent cases. The gate can be implemented via self-reported confidence or a simple agreement-rate heuristic; a trained classifier was considered and rejected as unnecessary complexity for the FYDP scope.
We use three agents from different model families to ensure genuine heterogeneity in reasoning styles and failure modes. This is critical because same-family model copies share correlated blind spots and hallucination patterns, reducing the diversity benefit of multi-agent debate. We use a two-phase model strategy: during development (Ph 0–1) the pipeline runs Qwen3.5-9B, Gemma 4 12B, and Ministral-3-14B-Instruct on an RTX A6000 48GB, keeping iteration cost low. For the main causal matrix (Core and Full) we serve the same three families at 24–27B in 4-bit: Qwen3.8-27B-AWQ-INT4 (~21.0 GB), Gemma-4-26B-A4B AWQ (~17.2 GB), and Mistral-24B-AWQ (15.06 GB) — ~53.3 GB total on one 80 GB A100 (~$0.136/GPU-h). The pinned 73.68 GB FP8 stack (Qwen3.8-27B-FP8 29.95 GB / Gemma FP8 28.67 GB / Mistral AWQ 15.06 GB) on a 96 GB card is reserved for the N1 hardware smoke and a small GPQA precision replication, not the main Full matrix. All three families (Alibaba, Google, Mistral) are genuinely distinct for true heterogeneity. The pipeline code is model-agnostic — only the model config changes between phases. The N = 3 configuration is chosen as the minimum needed to create a genuine “1 vs 2” minority-suppression scenario, which is the core phenomenon under study. Scaling ablations at N ∈ {2, 3, 5} are planned to probe sensitivity to panel size.
Each agent's response is decomposed into atomic factual statements using structured chain-of-thought prompting with explicit XML-style claim tags. This enables the downstream evidence scorer to address individual propositions rather than entire responses. Each atomic claim is a verifiable statement such as “Drug X reduces mortality by 30% in patients over 65.” Compound claims are split into separate checks for the effect, magnitude, and qualifier.
The retrieval subsystem routes each agent's claims to a distinct corpus: Agent A’s claims are checked against PubMed (biomedical literature), Agent B’s against arXiv (quantitative sciences), and Agent C’s against Semantic Scholar (broad scientific coverage). This source-partitioning prevents all three agents from retrieving from the same narrow slice of evidence, which would reduce the system to “whoever searched better” rather than “whoever is actually supported.” Retrieved passages are reranked using a cross-encoder (ms-marco-MiniLM) and classified as supported, contradicted, or unverifiable. Claims marked as unverifiable are abstained from the trust update and do not affect the agent's score. If the primary retrieval API is unavailable, an OpenAlex fallback is used.
For each agent i at round t, let Vi(t) be its count of supported claims, Hi(t) its count of contradicted claims, and ni(t) the number of claims checked. The update accumulates evidence in logit space, claim-count normalised so that verbosity cannot buy influence:
The accumulated logits pass through one softmax, followed by a floor/ceiling simplex projection. The operator order is fixed: logit accumulation → softmax → simplex projection — not softmax → clamp → renormalise. The old clamp order is wrong because the final renormalisation can push a weight back below the floor.
With floor = 0.1 and ceiling = 0.9, the projection guarantees mini Ti ≥ floor, Σi Ti = 1, and maxi Ti ≤ max(ceiling, 1/N). The floor gives no-silencing (a contradicted agent keeps a non-zero weight and can recover), and the effective ceiling gives no-domination. Neutral evidence is an exact identity: when support and contradiction counts are both zero, trust does not drift toward uniform. Dividing by the claim count makes the update invariant to verbosity — duplicating every claim leaves the trust vector unchanged.
These are bounded design properties, not global convergence theorems. The boundedness test covers 10,000 random draws for N ∈ {1, 3, 5} and is part of the 514-test P0 suite; the empirical trajectory is logged for every debate. The defaults in the current pipeline are α = 0.5 and β = 0.3, with floor/ceiling sensitivity checked before the matrix is frozen.
Instead of a plain majority vote, the final answer a* is the option that maximises the total trust mass of the agents supporting it at the final round K, where pi is agent i's position and 1[·] is the indicator function:
This is the mechanism that should prevent sycophantic collapse: a correct, evidence-backed minority agent can outweigh two unsupported majority agents when its trust score is sufficiently higher — the exact scenario a headcount gets wrong.
The complete deliberation follows a structured multi-round protocol. After the initial positions are established and initial trust scores are set uniformly at 1/3, each round proceeds as follows: (1) claims are extracted and decomposed; (2) source-partitioned retrieval is executed; (3) evidence verdicts are assigned; (4) trust scores are updated via Equations 1–3; (5) for adversarial evaluation only, a fabricated “expert consensus” message is injected between Rounds 1 and 2 to simulate social pressure; (6) agents revise positions with awareness of peer arguments and their own trust standing. AfterK rounds, the trust-weighted aggregation (Equation 4) produces the final answer. Full trust trajectories, per-round evidence verdicts, and per-agent reasoning traces are logged for every debate, enabling both the ECR calibration analysis and post-hoc failure diagnosis.
| Component | Selection | Rationale |
|---|---|---|
| Agent 1 | Qwen3.5-9B (Dev) → Qwen3.8-27B (Final) | Two-phase: fast dev on RTX A6000 48GB; 27B served AWQ 4-bit (~21.0 GB) for the Core/Full main matrix on one 80 GB A100. FP8 checkpoint (~29.95 GB) only for the 96 GB precision check. |
| Agent 2 | Gemma 4 12B (Dev) → Gemma-4-26B-A4B (Final) | Dev: encoder-free proxy. Final: MoE, served AWQ 4-bit (~17.2 GB) on the 80 GB A100; FP8 checkpoint (28.67 GB) only for the precision check. |
| Agent 3 | Ministral-3-14B-Instruct (Dev) → Mistral-24B-AWQ (Final) | Dev: Mistral-family 14B, served as AWQ 4-bit on the A6000 (the official FP8 checkpoint cannot run on Ampere). Final: third distinct family, AWQ 4-bit (15.06 GB) on the 80 GB A100 main stack |
| Oracle (B7) | GPT-6 Luna (OpenAI) | Optional external-answer ceiling; not a Core arm. 3 samples/question: mean single-call accuracy + bootstrap CI + 3-sample majority (ties → first sample). ~$3–9 per 1,000; 1.05M context / 128K output |
| Reranker | ms-marco-MiniLM cross-encoder | Standard, fast, validated for passage reranking |
| Inference engine | vLLM | Free, fast, standard for local multi-model serving |
| Orchestration | LangGraph | State-machine model fits round-based debate with injection control |
| Retrieval APIs | PubMed, arXiv, Semantic Scholar | Domain coverage matches scientific QA target; free academic access |
The Core matrix uses three datasets. GPQA-Main and MMLU-Pro STEM are the primary evidence-grounded benchmarks; HLE is the abstention control, never pooled with the primary tests. SciClaimBench is the controlled gold-evidence benchmark, not a Core dataset: it is frozen at 180 items in v1.2. The dataset audit found that BrokenMath (451/451) and BrokenArXiv (87/87) are proof-only with 0% answer-type retention, so they are appendix-only and are not used for CCR or evidence grounding.
| Dataset | Role | Source | Known Limitations |
|---|---|---|---|
| GPQA-Main | Primary evidence-grounded benchmark | Idavidrein/gpqa (HuggingFace, official; gated) | 448 total, 400 cap in the matrix; options shuffled; not adversarial |
| MMLU-Pro STEM | Primary scale/robustness benchmark | TIGER-Lab/MMLU-Pro (HuggingFace, official) | STEM-stratified, 400 cap in the matrix; broad, non-adversarial |
| SciClaimBench | Controlled benchmark release (v1.2 freeze); not a Core dataset | Original release (planned; CC BY 4.0) | Annotation-intensive; 180 floor, 300 target, 600 stretch, 800 no-go |
| HLE | Abstention / ceiling control | cais/hle (HuggingFace, official; gated) | Retrieval-resistant; 200 cap; never pooled with the primary tests |
| BrokenMath | Appendix-only premise rejection | INSAIT-Institute/BrokenMath (HuggingFace, official) | 451/451 proof-only; 0% answer-type retention — not usable for CCR |
| BrokenArXiv | Appendix-only qualitative | MathArena/brokenarxiv (HuggingFace, official) | 87/87 proof-only; 0% answer-type retention; cite the exact monthly snapshot |
The design is two seeds for the main matrix (three for the pilot), with a 24-item calibration before the pilot. The executable Core matrix is 4 arms × GPQA-Main 400 / MMLU-Pro STEM 400 / HLE 200 × 2 seeds = 24 runs, budgeted at 1,280 GPU-h. B8-oracle is not a Core arm, and SciClaimBench is not in Core v1.0. The six-arm target adds B8-placebo and B8-oracle through WP-18, and the pre-registration records both the target and the executable subset. The benchmark ladder is 180 items as the floor (SciClaimBench v1.2 freeze), 300 as the target, 600 as a stretch, and 800 as a no-go unless 2–3 dedicated annotators are committed.
The injection protocol pre-filters questions where the three agents produce divergent initial answers, so every evaluated debate presents a genuine disagreement. Results on the injected sets are framed as stress-test performance, not average-case deployment. All datasets are official author releases from HuggingFace. GPQA and HLE require a free click-through agreement before download. Dataset papers to cite: GPQA (COLM 2024), MMLU-Pro (NeurIPS 2024 D&B), HLE (Nature 2026), BrokenMath (arXiv 2025), BrokenArXiv / MathArena (arXiv 2026).
We design a controlled injection protocol that simulates social-pressure collapse. The protocol operates in six steps: (1) question selection and filtering; (2) independent initial answer collection; (3) baseline (no-injection) debate run; (4) injection — a fabricated system message stating that two of three agents agree on an answer (the wrong one), inserted between Rounds 1 and 2; (5) post-injection debate continuation; (6) measurement of whether the correct minority agent abandons its position. The injection is only applied to questions where the correct answer is initially held by a minority (1 of 3). The protocol achieves an inter-rater reliability target of κ > 0.75.
Three primary metrics operationalise the evaluation. Before the pilot, the verifier audit checks 150–200 claim–evidence pairs; the lexical baseline scores 0.58 balanced accuracy on the synthetic audit, against a target of ≥0.75 balanced accuracy and Cohen's κ ≥0.60. Collapse Rate (CCR) measures the proportion of correct minority agents that abandon their position under injected pressure:
where C is the set of correct-minority agents across all debates. Minority Preservation Rate (MPR) measures whether a correct minority survives to appear in the final output:
where M is the total number of debates with a correct minority.Evidence Calibration Rate (ECR) measures how well trust scores track actual correctness probability:
In addition to these primary metrics, we report standard accuracy on GPQA and MMLU-Pro as a sanity check, conduct a human evaluation (n = 60, Likert-scale ratings with κ validation), and compute Cohen's d effect sizes for all pairwise condition comparisons. Statistical significance is assessed via paired bootstrap with 95% confidence intervals across two random seeds in the main matrix (three seeds for the pilot). The unit of analysis is the question; seed 2 is a pre-registered replication, not a second independent study.
The full comparison set is B1–B10. The pre-registered Core matrix runs four executable arms: B3 (vanilla MAD), B4-matched (retrieved evidence with the same partition and prompts as B8, majority aggregation), B8-no-trust (same evidence and prompts, majority aggregation; primary control), and B8 (trust-weighted treatment). The six-arm target adds B8-placebo (verdicts shuffled within debate) and B8-oracle (gold evidence), pending WP-18. The primary Full matrix is 32 rows with the gold-evidence oracle on SciClaimBench; the 36-row alternate uses gold-answer trust (1,440 GPU-h, $235). WP-13 is the landed B8-no-trust prompt split, not the placebo/oracle slice.
B5, B6, and B9 preempt the “why not self-consistency / MoA / iMAD” objections, and B10 covers the closest published mitigation. B9 (iMAD) carries the highest implementation difficulty and is budgeted at approximately 10 working days. The appendix grid is a thesis plan, not part of the pre-registered primary matrix; no claim is made that B8 beats all B1–B10 comparison definitions.
Four planned ablation conditions isolate the contribution of each architectural component: (A1) trust calibration disabled (fallback to majority vote); (A2) progressive retrieval disabled (single-shot retrieval only); (A3) source partitioning disabled (all agents retrieve from the same corpus); (A4) adaptive confidence gating disabled (debate runs on every question). An N-scaling ablation (N ∈ {2, 3, 5}) on a 100-question subset probes sensitivity to panel size. Every ablation preserves all other components, enabling clean attribution of any performance delta.
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Trust signal does not behaviourally change output | Medium | Critical | 24-item calibration + pilot (Dec 2026) before the matrix; if negative, report as still-publishable finding |
| Retrieval noise degrades evidence quality | Medium | Moderate | Cross-encoder reranker + citation count filter (>10 citations) |
| Novelty erosion from published mitigations | Medium | Moderate | Sharpened differentiation: evidence-grounded, numeric, bounded trust; B10 comparison |
| Single 80 GB A100 compute ceiling | Medium | Moderate | 4-bit 24–27B main stack (~53.3 GB) on one 80 GB A100; Core is 24 runs / 1,280 GPU-h; Full primary is 32 rows / 1,280 GPU-h (36-row alternate / 1,440 GPU-h) |
| Dataset mismatch (proof-only benchmarks) | High | Moderate | BrokenMath/BrokenArXiv appendix-only; GPQA-Main + MMLU-Pro STEM primary; HLE abstention control; SciClaimBench controlled set |
| iMAD reimplementation divergence | Medium | Moderate | Explicit divergence documentation; use published numbers for easy conditions where comparable |
| HLE access delayed | Low–Medium | Low–Moderate | HLE is an abstention control; GPQA-Main + MMLU-Pro STEM remain primary |
| Retrieval API downtime | Low | Low | Local caching; OpenAlex fallback API |
The highest-risk assumption is that trust-weight changes behaviourally alter the final output rather than being ignored by the aggregation step. A 24-item calibration and pilot in Dec 2026 tests this before the matrix, so a failed premise is discovered early and cheaply. All risks are documented transparently with pre-specified fallback plans rather than assumed away.
P0 was released in Nov 2026. The N1 96 GB smoke + FP8 precision replication, WP-18 placebo/oracle, 24-item calibration and verifier audit precede the G2 pilot in Dec 2026. The G3 Core run follows in Jan 2027, and the G4/G5 Full run plus the benchmark ladder in Feb 2027. Core and Full are separate matrices on the same 4-bit main stack; do not assume Core cells are reusable. The plan totals $500–600 with a $600 hard stop; the 96 GB FP8 stack is limited to the N1 smoke and a small GPQA precision replication. If no 80 GB A100 is available, the fallback is $1,050–1,150 and requires re-approval. Human evaluation and the SDT analysis follow in Mar, results freeze in Apr, thesis chapters in May, thesis submission in Jun, and the defence in Jun/Jul 2027. The paper is written and submitted after the defence (Aug–Dec 2027). No conference deadline drives the thesis clock.
| Period | Phase | Key Objectives | Deliverable / Gate |
|---|---|---|---|
| Jul–Sep 2026 | Phase 0 | Literature freeze, environment setup, reproduce vanilla MAD | Gate 0: base debate loop reproduces — achieved 16 Sep 2026 (10/10 debates, 3/3 rounds) |
| Oct–Nov 2026 | P0 | Engineering hardening: options, parser/scorer, trust projection, verifier interface, claim-domain routing, endpoints, manifest/resume | 514 tests; P0 gate 13/13 passed |
| Dec 2026 | Pilot run | N1 96 GB smoke + FP8 precision replication ($25–40), WP-18 placebo/oracle, 24-item calibration, verifier audit (150–200 pairs; target ≥0.75 balanced accuracy, κ ≥0.60), 50-question Month-1 pilot, then the 100–150-question G2 pilot | G2: κ > 0.75 and measured behavioural effect; if not, reframe honestly |
| Jan 2027 | Core run | G3: 4 arms × GPQA-Main 400 / MMLU-Pro 400 / HLE 200 × 2 seeds = 24 runs, 1,280 GPU-h; B8-oracle is not a Core arm; SciClaimBench is not in Core v1.0 | Core matrix + pre-registered causal contrast (B8 vs B8-no-trust) |
| Feb 2027 | Full run + benchmark | G4/G5: 32 rows at 300 items/dataset with the gold-evidence oracle on SciClaimBench (1,280 GPU-h; $210); WP-18 adds B8-placebo/B8-oracle. The 36-row gold-answer alternate is 1,440 GPU-h ($235). SciClaimBench v1.2 freezes at 180 items | Full run + benchmark ladder: 180 floor, 300 target, 600 stretch, 800 no-go |
| Mar 2027 | Human eval + SDT | Human evaluation / SDT, benchmark annotation, verifier-audit follow-up | Human-eval + benchmark package |
| Apr 2027 | Results freeze | Freeze results, CIs, and effect sizes; no post-hoc arm changes | Frozen results |
| May 2027 | Thesis chapters | Write thesis chapters against the frozen results | Thesis draft |
| Jun 2027 | Thesis submission | Submit the FYDP thesis | Submitted thesis |
| Jun–Jul 2027 | Defence | Defend the thesis | Defence slides |
| Aug–Dec 2027 | Paper | Write and submit the paper after the defence | TACL rolling submission; EMNLP 2028 fallback |
The Go/No-Go logic is unchanged in spirit: Gate 0 validated the base loop (achieved 16 Sep 2026, 10/10 debates, mean 370.8 s/debate); the G2 pilot requires the injection protocol to reach inter-rater reliability κ > 0.75 and a measured behavioural effect; G3 runs the Core matrix; G4/G5 run the Full target. The Core matrix is only interpreted against the pre-registered endpoints (B8 vs B8-no-trust). If a gate is not met, the next phase is restructured to address the failure rather than letting the schedule dictate the claim.
Primary contribution (C1): a trust-calibrated, evidence-grounded aggregation mechanism for multi-agent debate, formally specified with boundedness, no-silencing, and no-domination properties, and planned for evaluation against the four executable arms and the six-arm target. The pre-registered targets are a 20–30% CCR reduction versus standard MAD, an ECR above 0.80 on the primary evaluation set, and non-inferior accuracy on GPQA-Main and MMLU-Pro STEM. These are success criteria, not measured results; the P0 gate is engineering readiness only.
Co-primary contribution (C2): an open, reproducible sycophancy-stress evaluation harness comprising the controlled injection protocol, three standardised metrics (CCR, MPR, ECR), and a statistical testing toolkit. The harness is designed to be independently reusable by other researchers regardless of whether they adopt our mechanism.
Publication target: TACL (rolling submission) is the primary target, written after the thesis defence. EMNLP 2028 is the fallback; a workshop is the safe floor. The completed FYDP thesis and a fully reproducible codebase will be released regardless of publication outcome.
Novelty statement: to our knowledge, no existing published or preprint work combines in-session, evidence-grounded, dynamically-updating trust with boundedness properties of this form. The closest published mitigation (ConsensAgent) operates via textual prompt refinement; the closest uncertainty-based method (DebUnc) relies on self-reported signals; and the closest theoretical framework (Estornell & Liu) does not produce a deployable system. Our mechanism occupies the intersection of external evidence grounding, dynamic in-session updating, and bounded design properties — a combination we have not found in the existing literature.
Chan, C.-M., et al. (2024). ChatEval: Towards better LLM-based evaluation through multi-agent debate. ICLR 2024.
Du, Y., et al. (2024). Improving factuality and reasoning in language models through multiagent debate. ICML 2024. arXiv:2305.14325.
Estornell, A., & Liu, Y. (2024). Multi-LLM debate: Framework, principals, and interventions. NeurIPS 2024.
Fan, J., et al. (2026). iMAD: Intelligent multi-agent debate for efficient LLM reasoning. AAAI 2026 (Oral).
He, J., et al. (2026). Minority Sentinel: A post-hoc classifier for minority-answer suppression in multi-agent debate. AgentSearch Workshop @ SIGIR 2026.
Pitre, H., Ramakrishnan, V., & Wang, W. Y. (2025). ConsensAgent: Multi-agent debate with prompt refinement for mitigating sycophancy.Findings of ACL 2025, pp. 22112–22133.
Wang, J., et al. (2025). Mixture-of-Agents: Leveraging diverse LLMs for improved reasoning. ICLR 2025 (Poster).
Yao, B., et al. (2025). Peacemaker or troublemaker: How sycophancy shapes multi-agent debate. arXiv:2509.23055 (preprint; ICLR 2026 submission withdrawn).
Yoffe, L., Amayuelas, A., & Wang, W. Y. (2025). DebUnc: Improving large language model agent communication with uncertainty metrics. Findings of EMNLP 2025, 23299–23315.