FYDP 2026–2027 · Idea 1 — Research Proposal

The Boundaries of Evidence-Grounded Trust in Multi-Agent Scientific Debate,
with a Controlled Benchmark for Sycophantic Consensus

Final Year Design Project — Research Proposal

Domain: Agentic AI · Multi-Agent Systems · Trustworthy NLP  |  Group 6 · Phantom Devs  |  Jul 2026 – Dec 2027

Implementation status (Nov 2026). P0 is engineering readiness: 615 tests, 13/13 gate checks passed. No efficacy result has been measured. Planned work is labelled as planned throughout.

Abstract

Multi-Agent Debate (MAD) improves the factual accuracy of large language models (LLMs) by having several agents argue toward a shared answer, yet it suffers a documented failure mode: inter-agent sycophancy, where a confident-but-wrong majority pressures a correct minority agent into abandoning its position, and a plain majority vote then locks in the wrong answer. This proposal introduces a trust-calibrated deliberation framework in which a panel of heterogeneous open-weight LLMs debate a scientific question while each agent's influence is dynamically re-weighted by a bounded trust score driven by the verifiability of its claims against retrieved scientific literature. The final answer is produced by trust-weighted aggregation rather than a headcount, allowing an evidence-backed minority to outweigh an unsupported majority. Alongside the mechanism, the project targets a reproducible sycophancy-stress evaluation harness — a controlled injection protocol, three metrics (CCR, MPR, ECR), a controlled benchmark, and a pre-registered causal contrast (B8 vs B8-no-trust) — with four executable arms now, a six-arm target, and three Core datasets (GPQA-Main, MMLU-Pro STEM, HLE as an abstention control). The planned benchmark follows a scale ladder of 180 items as the floor, 300 as the target, 600 as a stretch, and 800 as a no-go. The work runs entirely at inference time (no fine-tuning) on a single-GPU budget. Expected outcomes are stated as pre-registered success criteria, not as results: a statistically significant reduction in collapse rate and an open-source evaluation suite reusable by the research community.

1. Background and Motivation

Getting several large language models (LLMs) to debate a question before agreeing on an answer — called Multi-Agent Debate (MAD) — is a widely used way to improve factual accuracy and reasoning quality (Du et al., 2023; Chan et al., 2024). By exposing each agent to the critiques of others, MAD encourages diverse reasoning paths and lets the group recover from individual errors. However, these debates carry a real and well-documented weakness: an agent frequently drops a correct answer simply to align with a confident-sounding majority, so the group converges on a confidently wrong consensus (Yao et al., 2025). Because contemporary models are trained on heavily overlapping corpora, they tend to make correlated mistakes; a plain majority vote can therefore quietly bury a correct minority. Empirically, the minority turns out to be right in roughly one in four disagreements (He et al., 2026). In scientific, medical, or clinical question answering, a confidently wrong group answer is more harmful than an honest “I am not sure,” because it is delivered with the apparent authority of a consensus.

As LLM systems are increasingly deployed as research assistants, tutors, and decision-support tools, this false consensus among cooperating agents becomes a trust and safety problem, not merely an accuracy statistic — it fails silently and confidently. Sycophancy is cited as an open problem in over 50 peer-reviewed papers since 2023. In multi-agent systems specifically, it directly undermines the reason MAD is used: if debate collapses to whichever position sounds most confident rather than which is most correct, it adds computational cost without adding reliability. Existing responses fall into three camps: diagnosis (Yao et al., 2025; He et al., 2026), which measures the problem without proposing a mitigation; efficiency optimisation (iMAD, Fan et al., 2026, AAAI Oral), which learns when to trigger a debate rather than whom to trust during it; and static aggregation (Mixture-of-Agents, Wang et al., 2025, ICLR Poster), which combines model outputs without dynamic evidence grounding. Crucially, none dynamically re-weights agent influence during a debate based on whether that agent's claims can be verified against external evidence. This proposal targets exactly that gap, on the argument that the only reliable way to break a social-pressure collapse is to tie influence to something external and independent of the debate's internal dynamics.

Two recent published mitigations deserve particular attention. ConsensAgent (Pitre et al., Findings of ACL 2025) reduces sycophancy via static and dynamic prompt refinement — a textual intervention that does not produce a numeric trust score and does not use external evidence retrieval. DebUnc (Yoffe et al., Findings of EMNLP 2025) modulates in-loop influence using self-reported uncertainty signals, which are themselves susceptible to the same social pressure that drives sycophancy. Our approach is differentiated from both by its reliance on external, independent evidence verification as the basis for trust updates, rather than textual or self-reported signals. The theoretical foundation for majority-convergence in multi-LLM debate is established by Estornell and Liu (NeurIPS 2024), who prove a “tyranny of the majority” result and propose pruning-style interventions, but stop short of a deployable, evidence-grounded system.

2. Problem Statement

In current multi-agent debate systems, an agent's influence on the final answer depends on how many other agents agree with it — not on whether what it says is factually supported. There is no mechanism that adjusts each agent's trust score during the debate based on the verifiability of its claims against retrieved scientific literature. As a consequence, a hallucinating majority can systematically override a correct, well-supported minority. This inter-agent sycophancy undermines the very reason MAD is used: if a debate collapses to whichever position sounds most confident rather than which is most correct, it adds computational cost without adding reliability. The problem is most acute in high-stakes scientific and clinical applications, where the cost of a confidently wrong consensus is highest.

3. Research Gap

The problem of inter-agent sycophancy has been formally diagnosed but not solved with an in-session, evidence-grounded mechanism. We position our work against the closest lines of prior research:

Prior workWhat it doesWhy the gap remains
Yao et al. (2025) · He et al. (2026)Define and quantify inter-agent sycophancy (minority correct in ~25% of disagreements).Diagnostic only — measures the failure, proposes no mitigation.
iMAD — Fan et al. (2026), AAAI OralLearns when to trigger a debate for token efficiency.Decides whether to debate, not whom to trust once debating; falls back to majority vote.
MoA — Wang et al. (2025), ICLR PosterLayered multi-agent aggregation of diverse LLMs.Static, equal-weight — no evidence grounding, no dynamic trust.
ConsensAgent — Pitre et al. (2025), ACL FindingsFirst published sycophancy mitigation, via prompt refinement.Textual intervention — no numeric trust, no external evidence, no adversarial-injection testing.
DebUnc — Yoffe et al. (2025), EMNLP FindingsModulates in-loop influence using uncertainty signals.Uses self-reported uncertainty — the very signal social pressure can manipulate.
Estornell & Liu (2024), NeurIPSProves majority-convergence (“tyranny of the majority”); pruning-style interventions.Theoretical; not a deployable, evidence-grounded system.
Minority Sentinel (2026), SIGIR WorkshopPost-hoc correction based on log-level features after debate.Intervenes only after debate ends; no in-session trust adjustment.

Research gap (one sentence): no existing framework dynamically re-weights agent influence within an active debate session based on the real-time verifiability of claims against external evidence with a formally bounded numeric trust score. Prompt refinement (ConsensAgent), self-reported uncertainty (DebUnc), theoretical pruning (Estornell & Liu), and post-hoc correction (Minority Sentinel) each address adjacent problems, but none ties influence to external evidence verification during the debate. Our closest published competitor is ConsensAgent; our sharpest differentiation is that our trust signal is external, evidence-grounded, numeric, and formally bounded, rather than a textual prompt tweak.

4. Research Question, Hypotheses, and Objectives

Research question. When a multi-agent debate faces a planted, confident-but-wrong majority, does evidence-grounded trust weighting preserve correct minority answers better than a plain majority vote and other standard aggregation methods?

Central hypothesis. If an agent's influence in aggregation is dynamically re-weighted during the debate based on external evidence verification, then sycophantic collapse — agents abandoning correct answers under social pressure — will measurably decrease relative to standard majority-vote debate, without any model fine-tuning. This decomposes into three pre-registered targets, to be tested rather than assumed:

  • H1 (collapse): Collapse Rate (CCR) improves by ≥20% relative to vanilla MAD on the injected-pressure benchmarks.
  • H2 (outcome): Minority Preservation Rate (MPR) improves by ≥15 percentage points relative to majority vote.
  • H3 (calibration): Evidence Calibration Rate (ECR) > 0.80 on GPQA-Main — i.e., trust scores track actual correctness probability. The exact calibration endpoint is frozen in the pre-registration before the matrix runs.

To answer the research question, the objectives are to:

  1. Design a bounded, evidence-weighted trust-update rule driven by supported-versus-contradicted claims, with formally specified mathematical properties (boundedness, no-silencing, no-domination).
  2. Build the full pipeline: confidence gate, heterogeneous agent pool, claim decomposition, source-partitioned retrieval, cross-encoder evidence scoring, trust update, and trust-weighted adjudication.
  3. Create a reproducible evaluation suite: a controlled sycophancy injection protocol (inter-rater κ > 0.75), three primary metrics (CCR, MPR, ECR), and statistical testing infrastructure (paired bootstrap, 95% CIs, Cohen's d).
  4. Empirically validate the four executable arms (B3, B4-matched, B8-no-trust, B8), with a six-arm target that adds B8-placebo and B8-oracle, across the three Core datasets. The pilot in Dec 2026 must establish that trust re-weighting behaviourally changes the output rather than sitting in context ignored.
  5. Compare against the strongest published mitigation (ConsensAgent) through reimplementation or, minimally, a detailed mechanism-level differentiation analysis.

5. Proposed Method

We propose a trust-calibrated deliberation system in which a panel of N = 3 heterogeneous open-weight LLMs from different model families debate a scientific question over K = 3 rounds. As the agents argue, each agent's claims are decomposed into atomic factual statements and checked against partitioned scientific sources. Every agent i carries a trust score Si that rises when its claims are supported by retrieved evidence and falls when they are contradicted, and the final answer is produced by trust-weighted aggregation rather than a simple headcount. The system operates entirely at inference time with no fine-tuning required.

5.1 System architecture overview

The architecture follows a sequential pipeline with a feedback loop. The input (a scientific question) first enters a lightweight confidence gate. If the gate determines the question is straightforward, a direct answer is returned immediately. Otherwise, three heterogeneous agents each produce an independent initial position. Each agent's answer is decomposed into atomic claims, which are routed to partitioned scientific corpora (PubMed, arXiv, Semantic Scholar) for evidence retrieval. A cross-encoder reranker scores each claim as supported, contradicted, or unverifiable. The trust scores are updated based on these verdicts. Agents then revise their positions in light of peer arguments and their own trust standing. The retrieval→trust-update→ revision loop repeats for K rounds, after which trust-weighted aggregation produces the final answer with citations and a full trust trajectory log.

5.2 System pipeline

The end-to-end flow consists of nine stages: (i) a lightweight confidence gate answers easy questions directly and routes only genuinely uncertain ones into debate; (ii) the three agents state independent initial positions (Round 0); (iii) claim decomposition converts each answer into tagged atomic propositions; (iv) source-partitioned retrieval routes each agent's claims to a distinct corpus (Agent A → PubMed, B → arXiv, C → Semantic Scholar); (v) a cross-encoder scores each claim as supported / contradicted / unverifiable; (vi) the trust update converts those verdicts into new trust scores; (vii) agents revise their positions, aware of peers' arguments and their own trust standing; steps (iii)–(vii) repeat for K rounds; and (viii) trust-weighted aggregation emits the final answer plus citations, the full trust trajectory, and per-agent reasoning traces. Every intermediate output is logged to enable post-hoc analysis of

5.3 Confidence gate

A cheap gate avoids wasting the full ~9-forward-pass pipeline on easy questions where all agents would already agree correctly. It uses a three-sample agreement check (one fast pass per agent); unanimous, high-confidence agreement returns a direct answer, while any divergence or low confidence triggers the full debate. Mis-gating an easy question wastes only compute, never correctness, since the evaluation sets are pre-filtered to genuinely divergent cases. The gate can be implemented via self-reported confidence or a simple agreement-rate heuristic; a trained classifier was considered and rejected as unnecessary complexity for the FYDP scope.

5.4 Heterogeneous agent configuration

We use three agents from different model families to ensure genuine heterogeneity in reasoning styles and failure modes. This is critical because same-family model copies share correlated blind spots and hallucination patterns, reducing the diversity benefit of multi-agent debate. We use a two-phase model strategy: during development (Ph 0–1) the pipeline runs Qwen3.5-9B, Gemma 4 12B, and Ministral-3-14B-Instruct on an RTX A6000 48GB, keeping iteration cost low. For the main causal matrix (Core and Full) we serve the same three families at 24–27B in 4-bit: Qwen3.8-27B-AWQ-INT4 (~21.0 GB), Gemma-4-26B-A4B AWQ (~17.2 GB), and Mistral-24B-AWQ (15.06 GB) — ~53.3 GB total on one 80 GB A100 (~$0.136/GPU-h). The pinned 73.68 GB FP8 stack (Qwen3.8-27B-FP8 29.95 GB / Gemma FP8 28.67 GB / Mistral AWQ 15.06 GB) on a 96 GB card is reserved for the N1 hardware smoke and a small GPQA precision replication, not the main Full matrix. All three families (Alibaba, Google, Mistral) are genuinely distinct for true heterogeneity. The pipeline code is model-agnostic — only the model config changes between phases. The N = 3 configuration is chosen as the minimum needed to create a genuine “1 vs 2” minority-suppression scenario, which is the core phenomenon under study. Scaling ablations at N ∈ {2, 3, 5} are planned to probe sensitivity to panel size.

5.5 Claim decomposition and source-partitioned retrieval

Each agent's response is decomposed into atomic factual statements using structured chain-of-thought prompting with explicit XML-style claim tags. This enables the downstream evidence scorer to address individual propositions rather than entire responses. Each atomic claim is a verifiable statement such as “Drug X reduces mortality by 30% in patients over 65.” Compound claims are split into separate checks for the effect, magnitude, and qualifier.

The retrieval subsystem routes each agent's claims to a distinct corpus: Agent A’s claims are checked against PubMed (biomedical literature), Agent B’s against arXiv (quantitative sciences), and Agent C’s against Semantic Scholar (broad scientific coverage). This source-partitioning prevents all three agents from retrieving from the same narrow slice of evidence, which would reduce the system to “whoever searched better” rather than “whoever is actually supported.” Retrieved passages are reranked using a cross-encoder (ms-marco-MiniLM) and classified as supported, contradicted, or unverifiable. Claims marked as unverifiable are abstained from the trust update and do not affect the agent's score. If the primary retrieval API is unavailable, an OpenAlex fallback is used.

5.6 Trust-update rule (core contribution)

For each agent i at round t, let Vi(t) be its count of supported claims, Hi(t) its count of contradicted claims, and ni(t) the number of claims checked. The update accumulates evidence in logit space, claim-count normalised so that verbosity cannot buy influence:

Si(t+1) = log(max(Si(t), ε)) + α Vi(t)/ni(t) − β Hi(t)/ni(t)
(1)

The accumulated logits pass through one softmax, followed by a floor/ceiling simplex projection. The operator order is fixed: logit accumulation → softmax → simplex projection — not softmax → clamp → renormalise. The old clamp order is wrong because the final renormalisation can push a weight back below the floor.

T̃i(t+1) = softmax(S(t+1))i = exp(Si(t+1)) / Σj exp(Sj(t+1))
(2)
Ti(t+1) = ΠΔ(T̃i(t+1)),   Δ = {w : floor ≤ wi ≤ ceiling, Σi wi = 1 }
(3)

With floor = 0.1 and ceiling = 0.9, the projection guarantees mini Ti ≥ floor, Σi Ti = 1, and maxi Ti ≤ max(ceiling, 1/N). The floor gives no-silencing (a contradicted agent keeps a non-zero weight and can recover), and the effective ceiling gives no-domination. Neutral evidence is an exact identity: when support and contradiction counts are both zero, trust does not drift toward uniform. Dividing by the claim count makes the update invariant to verbosity — duplicating every claim leaves the trust vector unchanged.

These are bounded design properties, not global convergence theorems. The boundedness test covers 10,000 random draws for N ∈ {1, 3, 5} and is part of the 514-test P0 suite; the empirical trajectory is logged for every debate. The defaults in the current pipeline are α = 0.5 and β = 0.3, with floor/ceiling sensitivity checked before the matrix is frozen.

5.7 Trust-weighted aggregation

Instead of a plain majority vote, the final answer a* is the option that maximises the total trust mass of the agents supporting it at the final round K, where pi is agent i's position and 1[·] is the indicator function:

a* = arg maxa Σi Ti(K) · 1[ pi(K) = a ]
(4)

This is the mechanism that should prevent sycophantic collapse: a correct, evidence-backed minority agent can outweigh two unsupported majority agents when its trust score is sufficiently higher — the exact scenario a headcount gets wrong.

5.8 Full deliberation protocol

The complete deliberation follows a structured multi-round protocol. After the initial positions are established and initial trust scores are set uniformly at 1/3, each round proceeds as follows: (1) claims are extracted and decomposed; (2) source-partitioned retrieval is executed; (3) evidence verdicts are assigned; (4) trust scores are updated via Equations 1–3; (5) for adversarial evaluation only, a fabricated “expert consensus” message is injected between Rounds 1 and 2 to simulate social pressure; (6) agents revise positions with awareness of peer arguments and their own trust standing. AfterK rounds, the trust-weighted aggregation (Equation 4) produces the final answer. Full trust trajectories, per-round evidence verdicts, and per-agent reasoning traces are logged for every debate, enabling both the ECR calibration analysis and post-hoc failure diagnosis.

6. Models and Technical Stack

ComponentSelectionRationale
Agent 1Qwen3.5-9B (Dev) → Qwen3.8-27B (Final)Two-phase: fast dev on RTX A6000 48GB; 27B served AWQ 4-bit (~21.0 GB) for the Core/Full main matrix on one 80 GB A100. FP8 checkpoint (~29.95 GB) only for the 96 GB precision check.
Agent 2Gemma 4 12B (Dev) → Gemma-4-26B-A4B (Final)Dev: encoder-free proxy. Final: MoE, served AWQ 4-bit (~17.2 GB) on the 80 GB A100; FP8 checkpoint (28.67 GB) only for the precision check.
Agent 3Ministral-3-14B-Instruct (Dev) → Mistral-24B-AWQ (Final)Dev: Mistral-family 14B, served as AWQ 4-bit on the A6000 (the official FP8 checkpoint cannot run on Ampere). Final: third distinct family, AWQ 4-bit (15.06 GB) on the 80 GB A100 main stack
Oracle (B7)GPT-6 Luna (OpenAI)Optional external-answer ceiling; not a Core arm. 3 samples/question: mean single-call accuracy + bootstrap CI + 3-sample majority (ties → first sample). ~$3–9 per 1,000; 1.05M context / 128K output
Rerankerms-marco-MiniLM cross-encoderStandard, fast, validated for passage reranking
Inference enginevLLMFree, fast, standard for local multi-model serving
OrchestrationLangGraphState-machine model fits round-based debate with injection control
Retrieval APIsPubMed, arXiv, Semantic ScholarDomain coverage matches scientific QA target; free academic access

7. Dataset Plan

The Core matrix uses three datasets. GPQA-Main and MMLU-Pro STEM are the primary evidence-grounded benchmarks; HLE is the abstention control, never pooled with the primary tests. SciClaimBench is the controlled gold-evidence benchmark, not a Core dataset: it is frozen at 180 items in v1.2. The dataset audit found that BrokenMath (451/451) and BrokenArXiv (87/87) are proof-only with 0% answer-type retention, so they are appendix-only and are not used for CCR or evidence grounding.

DatasetRoleSourceKnown Limitations
GPQA-MainPrimary evidence-grounded benchmarkIdavidrein/gpqa (HuggingFace, official; gated)448 total, 400 cap in the matrix; options shuffled; not adversarial
MMLU-Pro STEMPrimary scale/robustness benchmarkTIGER-Lab/MMLU-Pro (HuggingFace, official)STEM-stratified, 400 cap in the matrix; broad, non-adversarial
SciClaimBenchControlled benchmark release (v1.2 freeze); not a Core datasetOriginal release (planned; CC BY 4.0)Annotation-intensive; 180 floor, 300 target, 600 stretch, 800 no-go
HLEAbstention / ceiling controlcais/hle (HuggingFace, official; gated)Retrieval-resistant; 200 cap; never pooled with the primary tests
BrokenMathAppendix-only premise rejectionINSAIT-Institute/BrokenMath (HuggingFace, official)451/451 proof-only; 0% answer-type retention — not usable for CCR
BrokenArXivAppendix-only qualitativeMathArena/brokenarxiv (HuggingFace, official)87/87 proof-only; 0% answer-type retention; cite the exact monthly snapshot

The design is two seeds for the main matrix (three for the pilot), with a 24-item calibration before the pilot. The executable Core matrix is 4 arms × GPQA-Main 400 / MMLU-Pro STEM 400 / HLE 200 × 2 seeds = 24 runs, budgeted at 1,280 GPU-h. B8-oracle is not a Core arm, and SciClaimBench is not in Core v1.0. The six-arm target adds B8-placebo and B8-oracle through WP-18, and the pre-registration records both the target and the executable subset. The benchmark ladder is 180 items as the floor (SciClaimBench v1.2 freeze), 300 as the target, 600 as a stretch, and 800 as a no-go unless 2–3 dedicated annotators are committed.

The injection protocol pre-filters questions where the three agents produce divergent initial answers, so every evaluated debate presents a genuine disagreement. Results on the injected sets are framed as stress-test performance, not average-case deployment. All datasets are official author releases from HuggingFace. GPQA and HLE require a free click-through agreement before download. Dataset papers to cite: GPQA (COLM 2024), MMLU-Pro (NeurIPS 2024 D&B), HLE (Nature 2026), BrokenMath (arXiv 2025), BrokenArXiv / MathArena (arXiv 2026).

8. Evaluation Strategy

8.1 Sycophancy-stress evaluation harness

We design a controlled injection protocol that simulates social-pressure collapse. The protocol operates in six steps: (1) question selection and filtering; (2) independent initial answer collection; (3) baseline (no-injection) debate run; (4) injection — a fabricated system message stating that two of three agents agree on an answer (the wrong one), inserted between Rounds 1 and 2; (5) post-injection debate continuation; (6) measurement of whether the correct minority agent abandons its position. The injection is only applied to questions where the correct answer is initially held by a minority (1 of 3). The protocol achieves an inter-rater reliability target of κ > 0.75.

8.2 Evaluation metrics

Three primary metrics operationalise the evaluation. Before the pilot, the verifier audit checks 150–200 claim–evidence pairs; the lexical baseline scores 0.58 balanced accuracy on the synthetic audit, against a target of ≥0.75 balanced accuracy and Cohen's κ ≥0.60. Collapse Rate (CCR) measures the proportion of correct minority agents that abandon their position under injected pressure:

CCR = (1 / |C|) · Σi ∈C 1[ agent i abandons correct answer post-injection ]
(5)

where C is the set of correct-minority agents across all debates. Minority Preservation Rate (MPR) measures whether a correct minority survives to appear in the final output:

MPR = (1 / M) · Σj = 1M 1[ correct minority preserved in a* for debate j ]
(6)

where M is the total number of debates with a correct minority.Evidence Calibration Rate (ECR) measures how well trust scores track actual correctness probability:

ECR = (1 / R) · Σt = 1R 1[ Ti(t) calibrated to p(correct | evidencei(t)) ]
(7)

In addition to these primary metrics, we report standard accuracy on GPQA and MMLU-Pro as a sanity check, conduct a human evaluation (n = 60, Likert-scale ratings with κ validation), and compute Cohen's d effect sizes for all pairwise condition comparisons. Statistical significance is assessed via paired bootstrap with 95% confidence intervals across two random seeds in the main matrix (three seeds for the pilot). The unit of analysis is the question; seed 2 is a pre-registered replication, not a second independent study.

8.3 Baselines and executable arms

The full comparison set is B1–B10. The pre-registered Core matrix runs four executable arms: B3 (vanilla MAD), B4-matched (retrieved evidence with the same partition and prompts as B8, majority aggregation), B8-no-trust (same evidence and prompts, majority aggregation; primary control), and B8 (trust-weighted treatment). The six-arm target adds B8-placebo (verdicts shuffled within debate) and B8-oracle (gold evidence), pending WP-18. The primary Full matrix is 32 rows with the gold-evidence oracle on SciClaimBench; the 36-row alternate uses gold-answer trust (1,440 GPU-h, $235). WP-13 is the landed B8-no-trust prompt split, not the placebo/oracle slice.

  • B1: Single-Agent Chain-of-Thought
  • B2: Single-Agent + RAG
  • B3: Multi-Agent Debate (vanilla MAD, Du et al.)
  • B4: MAD + RAG; B4-matched uses the B8 evidence partition and prompts
  • B5: Self-Consistency (Wang et al.)
  • B6: Mixture-of-Agents (MoA, Wang et al., ICLR 2025)
  • B7: Oracle (GPT-6 Luna) — optional external-answer ceiling, not a Core arm
  • B8: Trust-calibrated system; B8-no-trust is the primary control
  • B8-placebo / B8-oracle: six-arm target, pending WP-18
  • B9: iMAD (Fan et al., AAAI 2026) — reimplemented
  • B10: ConsensAgent (Pitre et al., ACL 2025) — reimplemented or mechanism-level comparison

B5, B6, and B9 preempt the “why not self-consistency / MoA / iMAD” objections, and B10 covers the closest published mitigation. B9 (iMAD) carries the highest implementation difficulty and is budgeted at approximately 10 working days. The appendix grid is a thesis plan, not part of the pre-registered primary matrix; no claim is made that B8 beats all B1–B10 comparison definitions.

8.4 Ablation studies

Four planned ablation conditions isolate the contribution of each architectural component: (A1) trust calibration disabled (fallback to majority vote); (A2) progressive retrieval disabled (single-shot retrieval only); (A3) source partitioning disabled (all agents retrieve from the same corpus); (A4) adaptive confidence gating disabled (debate runs on every question). An N-scaling ablation (N ∈ {2, 3, 5}) on a 100-question subset probes sensitivity to panel size. Every ablation preserves all other components, enabling clean attribution of any performance delta.

9. Risk Assessment and Mitigation

RiskLikelihoodImpactMitigation
Trust signal does not behaviourally change outputMediumCritical24-item calibration + pilot (Dec 2026) before the matrix; if negative, report as still-publishable finding
Retrieval noise degrades evidence qualityMediumModerateCross-encoder reranker + citation count filter (>10 citations)
Novelty erosion from published mitigationsMediumModerateSharpened differentiation: evidence-grounded, numeric, bounded trust; B10 comparison
Single 80 GB A100 compute ceilingMediumModerate4-bit 24–27B main stack (~53.3 GB) on one 80 GB A100; Core is 24 runs / 1,280 GPU-h; Full primary is 32 rows / 1,280 GPU-h (36-row alternate / 1,440 GPU-h)
Dataset mismatch (proof-only benchmarks)HighModerateBrokenMath/BrokenArXiv appendix-only; GPQA-Main + MMLU-Pro STEM primary; HLE abstention control; SciClaimBench controlled set
iMAD reimplementation divergenceMediumModerateExplicit divergence documentation; use published numbers for easy conditions where comparable
HLE access delayedLow–MediumLow–ModerateHLE is an abstention control; GPQA-Main + MMLU-Pro STEM remain primary
Retrieval API downtimeLowLowLocal caching; OpenAlex fallback API

The highest-risk assumption is that trust-weight changes behaviourally alter the final output rather than being ignored by the aggregation step. A 24-item calibration and pilot in Dec 2026 tests this before the matrix, so a failed premise is discovered early and cheaply. All risks are documented transparently with pre-specified fallback plans rather than assumed away.

10. Timeline and Milestones

P0 was released in Nov 2026. The N1 96 GB smoke + FP8 precision replication, WP-18 placebo/oracle, 24-item calibration and verifier audit precede the G2 pilot in Dec 2026. The G3 Core run follows in Jan 2027, and the G4/G5 Full run plus the benchmark ladder in Feb 2027. Core and Full are separate matrices on the same 4-bit main stack; do not assume Core cells are reusable. The plan totals $500–600 with a $600 hard stop; the 96 GB FP8 stack is limited to the N1 smoke and a small GPQA precision replication. If no 80 GB A100 is available, the fallback is $1,050–1,150 and requires re-approval. Human evaluation and the SDT analysis follow in Mar, results freeze in Apr, thesis chapters in May, thesis submission in Jun, and the defence in Jun/Jul 2027. The paper is written and submitted after the defence (Aug–Dec 2027). No conference deadline drives the thesis clock.

PeriodPhaseKey ObjectivesDeliverable / Gate
Jul–Sep 2026Phase 0Literature freeze, environment setup, reproduce vanilla MADGate 0: base debate loop reproduces — achieved 16 Sep 2026 (10/10 debates, 3/3 rounds)
Oct–Nov 2026P0Engineering hardening: options, parser/scorer, trust projection, verifier interface, claim-domain routing, endpoints, manifest/resume514 tests; P0 gate 13/13 passed
Dec 2026Pilot runN1 96 GB smoke + FP8 precision replication ($25–40), WP-18 placebo/oracle, 24-item calibration, verifier audit (150–200 pairs; target ≥0.75 balanced accuracy, κ ≥0.60), 50-question Month-1 pilot, then the 100–150-question G2 pilotG2: κ > 0.75 and measured behavioural effect; if not, reframe honestly
Jan 2027Core runG3: 4 arms × GPQA-Main 400 / MMLU-Pro 400 / HLE 200 × 2 seeds = 24 runs, 1,280 GPU-h; B8-oracle is not a Core arm; SciClaimBench is not in Core v1.0Core matrix + pre-registered causal contrast (B8 vs B8-no-trust)
Feb 2027Full run + benchmarkG4/G5: 32 rows at 300 items/dataset with the gold-evidence oracle on SciClaimBench (1,280 GPU-h; $210); WP-18 adds B8-placebo/B8-oracle. The 36-row gold-answer alternate is 1,440 GPU-h ($235). SciClaimBench v1.2 freezes at 180 itemsFull run + benchmark ladder: 180 floor, 300 target, 600 stretch, 800 no-go
Mar 2027Human eval + SDTHuman evaluation / SDT, benchmark annotation, verifier-audit follow-upHuman-eval + benchmark package
Apr 2027Results freezeFreeze results, CIs, and effect sizes; no post-hoc arm changesFrozen results
May 2027Thesis chaptersWrite thesis chapters against the frozen resultsThesis draft
Jun 2027Thesis submissionSubmit the FYDP thesisSubmitted thesis
Jun–Jul 2027DefenceDefend the thesisDefence slides
Aug–Dec 2027PaperWrite and submit the paper after the defenceTACL rolling submission; EMNLP 2028 fallback

The Go/No-Go logic is unchanged in spirit: Gate 0 validated the base loop (achieved 16 Sep 2026, 10/10 debates, mean 370.8 s/debate); the G2 pilot requires the injection protocol to reach inter-rater reliability κ > 0.75 and a measured behavioural effect; G3 runs the Core matrix; G4/G5 run the Full target. The Core matrix is only interpreted against the pre-registered endpoints (B8 vs B8-no-trust). If a gate is not met, the next phase is restructured to address the failure rather than letting the schedule dictate the claim.

11. Expected Contributions and Outcomes

Primary contribution (C1): a trust-calibrated, evidence-grounded aggregation mechanism for multi-agent debate, formally specified with boundedness, no-silencing, and no-domination properties, and planned for evaluation against the four executable arms and the six-arm target. The pre-registered targets are a 20–30% CCR reduction versus standard MAD, an ECR above 0.80 on the primary evaluation set, and non-inferior accuracy on GPQA-Main and MMLU-Pro STEM. These are success criteria, not measured results; the P0 gate is engineering readiness only.

Co-primary contribution (C2): an open, reproducible sycophancy-stress evaluation harness comprising the controlled injection protocol, three standardised metrics (CCR, MPR, ECR), and a statistical testing toolkit. The harness is designed to be independently reusable by other researchers regardless of whether they adopt our mechanism.

Publication target: TACL (rolling submission) is the primary target, written after the thesis defence. EMNLP 2028 is the fallback; a workshop is the safe floor. The completed FYDP thesis and a fully reproducible codebase will be released regardless of publication outcome.

Novelty statement: to our knowledge, no existing published or preprint work combines in-session, evidence-grounded, dynamically-updating trust with boundedness properties of this form. The closest published mitigation (ConsensAgent) operates via textual prompt refinement; the closest uncertainty-based method (DebUnc) relies on self-reported signals; and the closest theoretical framework (Estornell & Liu) does not produce a deployable system. Our mechanism occupies the intersection of external evidence grounding, dynamic in-session updating, and bounded design properties — a combination we have not found in the existing literature.

References

Chan, C.-M., et al. (2024). ChatEval: Towards better LLM-based evaluation through multi-agent debate. ICLR 2024.

Du, Y., et al. (2024). Improving factuality and reasoning in language models through multiagent debate. ICML 2024. arXiv:2305.14325.

Estornell, A., & Liu, Y. (2024). Multi-LLM debate: Framework, principals, and interventions. NeurIPS 2024.

Fan, J., et al. (2026). iMAD: Intelligent multi-agent debate for efficient LLM reasoning. AAAI 2026 (Oral).

He, J., et al. (2026). Minority Sentinel: A post-hoc classifier for minority-answer suppression in multi-agent debate. AgentSearch Workshop @ SIGIR 2026.

Pitre, H., Ramakrishnan, V., & Wang, W. Y. (2025). ConsensAgent: Multi-agent debate with prompt refinement for mitigating sycophancy.Findings of ACL 2025, pp. 22112–22133.

Wang, J., et al. (2025). Mixture-of-Agents: Leveraging diverse LLMs for improved reasoning. ICLR 2025 (Poster).

Yao, B., et al. (2025). Peacemaker or troublemaker: How sycophancy shapes multi-agent debate. arXiv:2509.23055 (preprint; ICLR 2026 submission withdrawn).

Yoffe, L., Amayuelas, A., & Wang, W. Y. (2025). DebUnc: Improving large language model agent communication with uncertainty metrics. Findings of EMNLP 2025, 23299–23315.

FYDP Research

The Boundaries of Evidence-Grounded Trust in Multi-Agent Scientific Debate, with a Controlled Benchmark for Sycophantic Consensus.

Jul 2026 – Dec 2027 · Thesis Jun 2027 · Defence Jun/Jul 2027 · Paper Aug–Dec 2027

Single 80 GB A100 · Open-weight Models

© 2026 Md. Atikur Rahaman · Final Year Design Project