FYDP Learning & Execution Roadmap · Idea 1

Idea 1 Learning & Execution Roadmap

The Boundaries of Evidence-Grounded Trust in Multi-Agent Scientific Debate, with a Controlled Benchmark for Sycophantic Consensus · Nov 2026 – Dec 2027

TimelineP0 release → Paper (Nov 2026 – Dec 2027)
Weekly Time15–20 Hours
Target VenueTACL (rolling) · EMNLP 2028 fallback
Research FocusAgentic AI · Anti-Sycophancy · Trust Calibration

A. Execution Plan (Blueprint §12)

P0 was released in Nov 2026. The N1 96 GB smoke + FP8 precision replication, WP-18 placebo/oracle, 24-item calibration and verifier audit precede the G2 pilot in Dec 2026; the G3 Core run follows in Jan 2027; the G4/G5 Full run plus the benchmark ladder follows in Feb; human eval/SDT in Mar; results freeze in Apr; thesis chapters in May; thesis submission in Jun; defence in Jun/Jul; paper after the defence (Aug–Dec 2027). Gates mark explicit go/no-go decisions. Core and Full are separate matrices on the same 4-bit main stack; do not assume Core cells are reusable. The plan totals $500–600 with a $600 hard stop; the main matrix runs the 4-bit 24–27B stack (~53.3 GB) on one 80 GB A100 at $0.136/GPU-h, and the 96 GB FP8 stack is limited to the N1 smoke and a small GPQA precision replication. If no 80 GB A100 is available, the fallback is $1,050–1,150 and requires re-approval.

PhaseDurationKey DeliverablesProgress
Ph 0: Literature & SetupJul–Sep 2026Literature freeze, vLLM 3-model serving on the A6000, Gate 0 MAD reproduction — passed 16 Sep 2026 (10/10 debates × 3 rounds)
Ph 1: P0 EngineeringOct–Nov 2026Core wired: options in prompts, parser/scorer, trust simplex projection, verifier interface, claim-domain routing, endpoints, manifest/resume. 514 tests; P0 gate 13/13 passed.
Ph 2: Pilot runDec 2026N1 96 GB smoke + FP8 precision replication ($25–40) → WP-18 placebo/oracle → 24-item calibration → verifier audit (150–200 pairs; lexical 0.58 balanced accuracy; target ≥0.75, κ ≥0.60) → 50-question Month-1 pilot → 100–150-question G2 pilot. 3 seeds for the pilot; 2 for the main matrix. G2 decides whether the Core run proceeds.
Ph 3: Core runJan 2027G3 Core run: 4 executable arms (B3, B4-matched, B8-no-trust, B8) × GPQA-Main 400 / MMLU-Pro 400 / HLE 200 × 2 seeds = 24 runs, 1,280 GPU-h. B8-oracle is not a Core arm; SciClaimBench is not in Core v1.0 (180 items at the v1.2 benchmark freeze). PRE_REGISTRATION.md v1.0.
Ph 4: Full run + benchmarkFeb 2027G4/G5 Full: 32 rows at 300 items/dataset with the gold-evidence oracle on SciClaimBench (1,280 GPU-h; $210), plus a 36-row gold-answer alternate (1,440 GPU-h; $235). Adds B8-placebo/B8-oracle via WP-18. SciClaimBench ladder: 180 floor, 300 target, 600 stretch, 800 no-go.
Ph 5: Human Eval + SDTMar 2027Human evaluation / SDT, benchmark annotation, and verifier-audit follow-up.
Ph 6: Results FreezeApr 2027Freeze results, CIs, and effect sizes. No new runs or arm changes after the freeze.
Ph 7: Thesis ChaptersMay 2027Write the thesis chapters against the frozen results.
Ph 8: Thesis Submit + DefenceJun–Jul 2027Thesis submitted June 2027; defence June/July 2027.
Ph 9: PaperAug–Dec 2027Paper writing and submission after the defence (TACL rolling; EMNLP 2028 fallback).
  • Gate 0 — achieved 16 Sep 2026 (Jul–Sep 2026): Vanilla MAD reproduction (Du et al. 2023) on a GPQA slice: 10/10 debates completed all 3 rounds, no empty positions, mean 370.8 s/debate. Full report in experiments/gate0/.
    Working MAD reproduction — done
  • P0 engineering gate (Nov 2026): 615 tests; P0 gate 13/13 passed and released (tags p0-gate, p0-release). Current report evidence: 70-pp report, 19-pp journal, 0 undefined references. Core wiring is in place — this is engineering readiness, not an efficacy result.
    P0 gate report + runnable matrix — 13/13
  • Pilot run (Dec 2026): N1 96 GB smoke + FP8 precision replication, WP-18 placebo/oracle, a 24-item calibration, and the verifier audit precede the pilot. The 50-question Month-1 pilot is followed by the 100–150-question G2 pilot. G2 requires baseline CCR ≥0.30, trust AUROC ≥0.65, ≥65% correct-direction changes, and injection κ ≥0.75; if it fails, the project reframes to report the negative result honestly.
    G2 pilot-run report
  • Core run (Jan 2027): G3 Core run: 4 executable arms × GPQA-Main 400 / MMLU-Pro 400 / HLE 200 × 2 seeds = 24 runs, budgeted at 1,280 GPU-h. B8-oracle is not a Core arm, and SciClaimBench is not in Core v1.0. No efficacy claim is made before the pre-registered analysis.
    G3 Core matrix + pre-registered causal contrast (B8 vs B8-no-trust)
  • Full run + benchmark (Feb 2027): G4/G5 Full: 32 rows at 300 items/dataset with the gold-evidence oracle on SciClaimBench (1,280 GPU-h), adding B8-placebo/B8-oracle through WP-18; the 36-row gold-answer alternate is 1,440 GPU-h. SciClaimBench benchmark freeze at 180 items (v1.2); ladder 180 floor, 300 target, 600 stretch, 800 no-go.
    Full-run matrix + SciClaimBench v1.2
  • Human eval + SDT (Mar 2027): Human evaluation and SDT, benchmark annotation, and the verifier-audit follow-up.
    Human-eval + benchmark package
  • Results freeze (Apr 2027): Freeze all results, confidence intervals, and effect sizes. No post-hoc arm or endpoint changes.
    Frozen results
  • Thesis chapters (May 2027): Write the thesis chapters against the frozen results.
    Thesis draft
  • Thesis submission (Jun 2027): Submit the FYDP thesis in June 2027.
    Submitted thesis
  • Defence (Jun–Jul 2027): Defend the thesis in June/July 2027.
    Defence slides
  • Paper after defence (Aug–Dec 2027): Write and submit the paper version after the defence; TACL rolling submission, EMNLP 2028 fallback.
    Paper submission

B. Implementation Order (Blueprint §13)

Exact build sequence with dependency reasoning. Each step depends on the previous.

1
vLLM multi-model serving setup
Nothing else can be tested without this. Done — three-model serving verified 16 Sep 2026.
2
Vanilla MAD reproduction (Du et al. 2023)
Gate 0 — validates base loop independent of our additions. Done — 10/10 debates, 3/3 rounds.
3
Injection protocol (§5.4)
Must exist before trust mechanism — trust calibration needs a working stress test. Protocol and B8 injection wiring are in place; GPU pilot next.
4
N1 smoke + FP8 precision check, calibration, verifier audit, then pilot (Nov–Dec 2026)
N1 96 GB smoke + FP8 precision replication ($25–40), WP-18 placebo/oracle, 24-item calibration, verifier audit (150–200 pairs; lexical baseline 0.58 balanced accuracy; target ≥0.75, κ ≥0.60), then the 50-question Month-1 pilot and 100–150-question G2 pilot.
5
Claim decomposition + source-partitioned retrieval
Trust formula inputs are outputs of retrieval. Claim decomposition, domain routing, verifier interface, and endpoints are wired at P0.
6
Trust update function
Unit-test the operator order in isolation: logit accumulation → softmax → floor/ceiling simplex projection. Clamp-then-renormalise is wrong.
7
Full debate loop (LangGraph)
Core loop, manifest, and resume are wired; WP-13 (B8-no-trust prompt split) landed, and WP-18 remains for the placebo/oracle arms before the Full run.
8
Baselines B1–B4
Built and mock-verified; B3 and B4-matched are in the executable matrix.
9
Baselines B5–B6 (Self-Consistency, MoA)
Independent of core mechanism. Can build in parallel with step 10.
10
Baseline B9 (iMAD reimplementation)
Highest difficulty (~10 days). Sequenced last.
11
Core run → Full run
G3 Core: 24 runs at 1,280 GPU-h. G4/G5 Full primary: 32 rows at 300 items/dataset with the gold-evidence oracle on SciClaimBench (1,280 GPU-h); the 36-row gold-answer alternate is 1,440 GPU-h. B8-placebo/B8-oracle are WP-18.
12
Human evaluation + failure analysis
Last — requires completed experimental results to annotate against.
🌿 Repository workflow

Two long-lived branches only: main (public mirror) and dev (development; main is the release deploy). Snapshots are git tags, not long-lived phase branches: p0-gate and p0-release exist now; Pilot, Core and Full tags are planned. Feature work lands on short-lived slices and merges back with --no-ff.

⚠ Critical Ordering Constraints
  • Never build trust update before retrieval exists.
  • Never build B9 (iMAD) before the core system (steps 1–7).
  • Never run the Core run before the Dec 2026 calibration + pilot confirms the core assumption.

C. Month-by-Month Learning Plan (Honest)

What you actually need to learn, when, and how much — mapped to the blueprint's build order. The rule: ~30% watching, ~70% building. Every month ends with a running script, not notes.

MonthPhaseEffortLearn (only this)Build / Exit checkLinks
Jul–Sep 2026Ph 0 — complete~15 hr/wkOnly the Transformer / attention / quantization units from the ML/DL playlists (skip what you know); structured prompting for tagged output; vLLM + HF Hub basics.Repo skeleton; 3-model vLLM serving on the A6000; Gate 0 MAD reproduction passed 16 Sep 2026 (10/10 debates × 3 rounds, mean 370.8 s/debate).
Oct–Nov 2026P0 — released Nov 2026~20 hr/wkLangGraph StateGraph subset — the ~5-API core, not the whole library (CampusX playlist, videos 1–6 only); trust projection math from the P0 interfaces.Core wired: options in prompts, parser/scorer, trust simplex projection, verifier interface, claim-domain routing, endpoints, manifest/resume. 514 tests; P0 gate 13/13.
Dec 2026Pilot run~15 hr/wkOnly the stats you will actually run: paired bootstrap, McNemar, Cohen's d, Cohen's κ — short scipy walkthroughs, no full stats course.N1 96 GB smoke + FP8 precision replication ($25–40); WP-18 placebo/oracle; 24-item calibration; verifier audit (150–200 pairs; lexical 0.58 balanced accuracy; target ≥0.75, κ ≥0.60); 50-question Month-1 pilot, then the 100–150-question G2 pilot. G2 decides whether the Core run proceeds.
Jan 2027Core run~20 hr/wkNothing new — debugging discipline and log inspection only. If you need a concept, look it up on demand.G3 Core run: 4 arms × GPQA-Main 400 / MMLU-Pro 400 / HLE 200 × 2 seeds = 24 runs, 1,280 GPU-h. B8-oracle is not a Core arm; SciClaimBench is not in Core v1.0. No efficacy claim until the pre-registered analysis.—
Feb 2027Full run + benchmark~20 hr/wkBenchmark annotation and verifier-audit protocol; no new ML theory.WP-18 adds B8-placebo/B8-oracle; G4/G5 Full primary is 32 rows at 300 items/dataset with the gold-evidence oracle on SciClaimBench (1,280 GPU-h; $210); the 36-row gold-answer alternate is 1,440 GPU-h ($235). SciClaimBench: 180 floor, 300 target, 600 stretch, 800 no-go.—
Mar–Apr 2027Human eval + freeze~15 hr/wkInter-annotator agreement (κ/ICC) for the human evaluation.Human evaluation / SDT, benchmark annotation, verifier-audit follow-up, then results freeze with CIs and effect sizes. No post-hoc arm changes.—
May–Jul 2027Thesis + defence~15 hr/wkVenue/thesis formatting only; no new technical material.Thesis chapters (May), submission (Jun), defence (Jun/Jul).—
Aug–Dec 2027Paper~10 hr/wkTACL/EMNLP writing guides and reviewer-response practice.Write and submit the paper after the defence. TACL rolling; EMNLP 2028 fallback.—
How not to waste the month
  • Do not binge-watch playlists. Binge-watching is the #1 way to spend a month learning nothing you can reproduce. Watch the one video for the concept you are about to implement, then implement it.
  • Do not take a full stats or ML course. This project needs ~6 statistical tests and ~4 architecture concepts. Learn them on demand.
  • Skip if you already know it: Python (skip 100 Days of Python entirely), basic ML concepts, and any LangChain content — you need LangGraph only, and only its StateGraph core.
  • If you fall behind: cut the reranker to a simple overlap score and catch up in the Pilot run — never skip the Gate 0 MAD reproduction or the Dec 2026 calibration + pilot.

D. Learning Resources (Verified Links)

A short shelf, not a library. Prefer the official docs for everything you build; use the playlists for concepts only. All links verified.

E. Learning Level Guide

LevelMeaning
🔵 BeginnerBasic conceptual understanding only
🟢 IntermediateComfortable using and modifying existing implementations
🟡 AdvancedAble to implement, debug, and explain independently
🔴 ExpertDeep enough understanding to contribute novel research ideas

Core ML & Deep Learning

#TopicLevel NeededImportanceWhy NeededLearn
1Neural Networks🟢 IntermediateHighUnderstand model structure and inference100 Days of DL (CampusX) ↗
2Transformer Architecture🟡 AdvancedCriticalFoundation of all LLM agents100 Days of DL — Transformer units ↗
3Attention Mechanism🟡 AdvancedCriticalUnderstanding trust influence and context handling100 Days of DL (CampusX) ↗
4Embeddings & Vector Representations🟢 IntermediateCriticalCore of RAG and retrieval pipelines100 Days of ML (CampusX) ↗
5Model Quantization (GPTQ / AWQ)🟢 IntermediateHighEfficient 4-bit inference on the 80 GB A100 main stackvLLM quantization docs ↗
6Inference vs Fine-Tuning🟢 IntermediateMediumYour work focuses on inference orchestrationHF docs ↗

Large Language Models

#TopicLevel NeededImportanceWhy NeededLearn
1LLM Text Generation🟡 AdvancedCriticalUnderstand how models reason and failHF docs ↗
2Chain-of-Thought (CoT)🟡 AdvancedCriticalMain baseline and reasoning styleWei et al. 2022 ↗
3Self-Consistency Decoding🟡 AdvancedCriticalMajor experimental baseline (B5)Wang et al. 2022 ↗
4Prompt Engineering🟢 IntermediateHighAgent orchestration and structured promptingOpenAI prompt guide ↗
5Temperature & Sampling🟢 IntermediateMediumGenerating agent diversityHF docs ↗
6Hallucination in LLMs🔴 ExpertCriticalYour main failure modeYour literature review ↗
7Sycophancy in LLMs🔴 ExpertCriticalCore research problemSharma et al. 2023 ↗
8Model Calibration (ECE)🔴 ExpertCriticalFoundation of trust calibrationGuo et al. 2017 ↗
9Two-Phase Model Strategy🟢 IntermediateHighDev stack (Qwen3.5-9B / Gemma 4 12B / Ministral-3-14B) → 4-bit main stack (Qwen3.8-27B-AWQ-INT4 ~21.0 GB / Gemma 4 26B-A4B AWQ ~17.2 GB / Mistral-24B-AWQ 15.06 GB; ~53.3 GB, one 80 GB A100); FP8 check stack (73.68 GB) is N1 smoke + GPQA precision replicationProposal §7 ↗
10Sycophancy Injection Protocol Design🔴 ExpertCriticalCCR operationalization; κ > 0.75 pilot check (Phase 1–2)Proposal §5.4 ↗
11LLM Uncertainty & Confidence Estimation🟡 AdvancedHighLightweight confidence gating (Phase 2)Your literature review ↗

Multi-Agent Systems

#TopicLevel NeededImportanceWhy NeededLearn
1Multi-Agent Debate (MAD)🔴 ExpertCriticalMain paradigm your work extendsDu et al. 2023 ↗
2Mixture of Agents (MoA)🟡 AdvancedHighImportant baseline (B6)Wang et al. 2024 ↗
3iMAD Framework🟡 AdvancedCriticalClosest competitor baselineYour iMAD review ↗
4Consensus Mechanisms🟡 AdvancedHighUnderstanding majority-vote limitations—
5Trust & Reputation Systems🔴 ExpertCriticalFoundation of your contributionYour iMAD review ↗
6Agentic AI Paradigm🟢 IntermediateMediumBroader research positioningLangGraph playlist (CampusX) ↗
7Debate Prompt Design🟡 AdvancedHighStructuring adversarial reasoning (Phase 1–2)LangGraph playlist (CampusX) ↗
8Adaptive Triggering / Confidence Gating🟡 AdvancedHighGate full debate pipeline (Phase 2)Proposal §5.2 ↗
9Heterogeneous Multi-Model Agent Design🟡 AdvancedHighReducing correlated hallucinationsProposal §5.4 ↗

Retrieval-Augmented Generation (RAG)

#TopicLevel NeededImportanceWhy NeededLearn
1Dense Retrieval (DPR, Contriever)🟡 AdvancedCriticalAtomic claim retrieval backbonesentence-transformers ↗
2Cross-Encoder Reranking🟡 AdvancedCriticalEvidence scoring per claim (Phase 2)sentence-transformers ↗
3FAISS Vector Index🟢 IntermediateHighEfficient similarity search at scaleFAISS (GitHub) ↗
4Source-Partitioned Retrieval🔴 ExpertCriticalYour novel retrieval strategy (Phase 2–3)Proposal §5.5 ↗
5Atomic Claim Decomposition🔴 ExpertCriticalBreaking agent utterances into verifiable propositions (Phase 2)Proposal §5.5 ↗
6RAG Evaluation Metrics🟡 AdvancedHighMeasuring retrieval quality (Phase 3–4)lm-eval-harness ↗

Evaluation & Statistics

#TopicLevel NeededImportanceWhy NeededLearn
1Statistical Significance Testing🟡 AdvancedCriticalPaired bootstrap, McNemar tests for all main resultsSciPy docs ↗
2Effect Size Reporting🟡 AdvancedCriticalCohen's d, Cliff's delta for sycophancy reduction claims—
3Inter-rater Agreement (κ)🟡 AdvancedHighValidating sycophancy injection protocol (Phase 1 pilot)—
4Ablation Study Design🔴 ExpertCriticalIsolating each component contribution; required for Q1 submissionProposal §11 ↗
5Calibration Metrics (ECE, ECR)🔴 ExpertCriticalCalibration endpoint for the trust signal (H3)Guo et al. 2017 ↗
6Benchmark Evaluation Harness🟡 AdvancedHighReproducible multi-benchmark evaluationlm-eval-harness ↗

FYDP Research

The Boundaries of Evidence-Grounded Trust in Multi-Agent Scientific Debate, with a Controlled Benchmark for Sycophantic Consensus.

Jul 2026 – Dec 2027 · Thesis Jun 2027 · Defence Jun/Jul 2027 · Paper Aug–Dec 2027

Single 80 GB A100 · Open-weight Models

© 2026 Md. Atikur Rahaman · Final Year Design Project