Conflict_Checker/.hermes/plans/2026-08-10_100946-evidence-verification-cross-sheet-correlation.md

Evidence Verification + Cross-Sheet Correlation Implementation Plan

For Hermes: Use subagent-driven-development skill to implement this plan task-by-task.

Goal: Stop vision-extraction misreads (e.g. "(2) 2x6 STUD PACK" vs the actual "(5) 2x6") from becoming confident downstream findings, and correlate the same physical element across sheets (S101/S205/S401) so no stage reasons from one sheet's text in isolation.

Architecture: Three independently shippable phases on the agent pipeline (backend/agents/runner.py):

1. Disputed-value detection — deterministic post-link pass that flags contradictory extracted values inside a cluster and surfaces them to the critic/specialist prompts.
2. Cross-sheet xref linking — linker gains detail-reference/tag buckets that join assertions across levels (today (level, family) bucketing splits S101/S205/S401 apart).
3. Evidence verification wave (5b) — a bounded vision fact-check agent re-reads the cited sheet images for high-severity / disputed findings before the Brain merge, annotates or suppresses findings built on phantom text.

Tech Stack: Python 3.14, pytest (tests/), existing call_json LLM wrapper (supports images_b64, reasoning_effort, reasoning_max_tokens).

---

Current context / root cause (from job 959e16407573)

Finding validated_issues[3] ("FRONT PERSPECTIVE detail on Sheet S401", HSS16x4 on "(2) 2x6 STUD PACK", severity critical) is a false positive built on a wave-1 vision misread. The sheet actually shows a (5) 2x6 stud pack (matching S205/S101). Chain of failure:

1. Wave 1 (SheetExtractorAgent) froze the misread into text. From then on it is "ground truth".
2. Wave 3 linker (backend/agents/linker.py:33 build_link_scopes) buckets by (level, family). S101 (foundation), S205 (details), S401 (sections) get different level values, so assertions about the same front-wall header never share a link scope or cluster. No cross-sheet corroboration happened.
3. Wave 5 ConstructabilityAgent (backend/agents/construct_agent.py:53) calls call_json with no images — in this job 120/120 constructability calls were +0img. It reasoned arithmetically from the misread text ("2 x 1.5in = 3in < 4in -> unbuildable"). It even held the "(5) 2x6 STUD PACK" assertion in the same scope but labeled it "Ambiguous column size specification" instead of arbitrating.
4. Nothing between wave 5 and the report ever looks at a sheet image again. Only the wave-4 conflict critic receives images, and only for its own cluster's pages.

Also confirmed in this log (separate known bug, fixed in Task 7 while we're here): wave-4 conflict critic truncates on Gemini thinking tokens because conflict_critic.py:59 passes max_tokens=config.REASON_MAX_TOKENS (4096) with no reasoning budget — 13/121 calls hit finish_reason=length.

Assumptions

• Assertions carry id, attribute, value, source_text, location_key (room/grid/detail_reference/tag/level) — see linker._payload and _serialize.slim_assertion.
• Extractor assertions already carry a confidence field (per test fixtures).
• validate_issue in backend/pipeline/_stage.py guarantees each finding an issue_id.
• Test convention: unittest.mock.patch("backend.agents.<module>.call_json", ...) — see tests/agents/test_sheet_extractor_fallback.py. Run tests with .venv/bin/python -m pytest tests/ -x -q.
• AgentResult.error defaults to "" (not None) in assertions.

---

Phase 1 — Disputed-value detection + prompt hardening

Task 1: find_disputes pure function (TDD)

Objective: Detect "same attribute, different values" inside one cluster's assertions.

Files:

• Create: backend/agents/disputes.py
• Test: tests/agents/test_disputes.py

Step 1: Write failing test

# tests/agents/test_disputes.py
from backend.agents.disputes import annotate_clusters, find_disputes


def _a(id_, attribute, value):
    return {"id": id_, "attribute": attribute, "value": value,
            "source_text": value}


def test_find_disputes_flags_same_attribute_different_values():
    assertions = [
        _a("a1", "stud_pack_size", "(2) 2x6 STUD PACK"),
        _a("a2", "stud_pack_size", "(5) 2x6 STUD PACK"),
        _a("a3", "beam_size", "HSS16X4X5/8"),
    ]
    disputes = find_disputes(assertions)
    assert len(disputes) == 1
    assert disputes[0]["attribute"] == "stud_pack_size"
    assert disputes[0]["values"] == ["(2) 2x6 STUD PACK", "(5) 2x6 STUD PACK"]
    assert disputes[0]["assertion_ids"] == ["a1", "a2"]


def test_find_disputes_ignores_agreeing_values_and_blanks():
    assertions = [
        _a("a1", "beam_size", "HSS16X4X5/8"),
        _a("a2", "beam_size", "  hss16x4x5/8 "),   # same after normalize
        _a("a3", "", "orphan"),                     # no attribute -> skipped
        _a("a4", "beam_size", ""),                  # no value -> skipped
    ]
    assert find_disputes(assertions) == []


def test_annotate_clusters_writes_disputed_attributes():
    clusters = [
        {"key": "c1", "assertions": [
            _a("a1", "stud_pack_size", "(2) 2x6"),
            _a("a2", "stud_pack_size", "(5) 2x6"),
        ]},
        {"key": "c2", "assertions": [_a("a3", "x", "1"), _a("a4", "x", "1")]},
    ]
    assert annotate_clusters(clusters) == 1
    assert clusters[0]["disputed_attributes"][0]["attribute"] == "stud_pack_size"
    assert "disputed_attributes" not in clusters[1]

Step 2: Run test to verify failure

Run: .venv/bin/python -m pytest tests/agents/test_disputes.py -v Expected: FAIL — ModuleNotFoundError: backend.agents.disputes

Step 3: Implement

# backend/agents/disputes.py
"""Deterministic detection of contradictory extracted values within a cluster.

Extraction is a vision pass: quantities and sizes can be misread ("(2) 2x6" vs
"(5) 2x6"). Cluster members are supposed to describe the same real-world
element, so two members asserting different values for the same attribute are
a probable misread. Flag these so downstream text-only stages treat the value
as unverified instead of reasoning from one reading.
"""

import re
from typing import Dict, List


def _norm(value) -> str:
    return re.sub(r"\s+", " ", str(value or "").strip().lower())


def find_disputes(assertions: List[Dict]) -> List[Dict]:
    """Same attribute with >= 2 distinct normalized values = disputed."""
    groups: Dict[str, Dict[str, set]] = {}
    for assertion in assertions:
        attribute = _norm(assertion.get("attribute"))
        value = _norm(assertion.get("value"))
        if not attribute or not value:
            continue
        groups.setdefault(attribute, {}).setdefault(value, set()).add(
            assertion.get("id")
        )
    disputes = []
    for attribute, values in sorted(groups.items()):
        if len(values) < 2:
            continue
        disputes.append({
            "attribute": attribute,
            "values": sorted(values),
            "assertion_ids": sorted(
                aid for ids in values.values() for aid in ids if aid
            ),
        })
    return disputes


def annotate_clusters(clusters: List[Dict]) -> int:
    """Attach disputed_attributes to each cluster that has any. Returns count."""
    annotated = 0
    for cluster in clusters:
        disputes = find_disputes(cluster.get("assertions") or [])
        if disputes:
            cluster["disputed_attributes"] = disputes
            annotated += 1
    return annotated

Step 4: Run test to verify pass

Run: .venv/bin/python -m pytest tests/agents/test_disputes.py -v Expected: 3 passed

Step 5: Commit

git add backend/agents/disputes.py tests/agents/test_disputes.py
git commit -m "feat: deterministic disputed-value detection for cluster assertions"

---

Task 2: Wire annotate_clusters into the runner + serialization

Objective: Disputes must be visible to the wave-4 critic and wave-5 constructability prompts.

Files:

• Modify: backend/agents/runner.py (after memory.replace("clusters", clusters), ~line 116)
• Modify: backend/pipeline/_serialize.py (slim_clusters, line 46)
• Test: tests/agents/test_disputes.py (append)

Step 1: Write failing test

def test_slim_clusters_preserves_disputed_attributes():
    from backend.pipeline._serialize import slim_clusters
    cluster = {"key": "c1", "assertions": [],
               "disputed_attributes": [{"attribute": "a", "values": ["1", "2"],
                                         "assertion_ids": ["x", "y"]}]}
    slim = slim_clusters([cluster])[0]
    assert slim["disputed_attributes"][0]["values"] == ["1", "2"]

Step 2: Run test to verify failure

Run: .venv/bin/python -m pytest tests/agents/test_disputes.py::test_slim_clusters_preserves_disputed_attributes -v Expected: FAIL — KeyError: 'disputed_attributes'

Step 3: Implement

In backend/pipeline/_serialize.py slim_clusters, add the key:

def slim_clusters(clusters: List[Dict]) -> List[Dict]:
    return [
        {
            "key": c.get("key"),
            "location": c.get("location"),
            "disciplines": c.get("disciplines"),
            "kind": c.get("kind"),
            **({"disputed_attributes": c["disputed_attributes"]}
               if c.get("disputed_attributes") else {}),
            "assertions": [slim_assertion(a) for a in c.get("assertions", [])],
        }
        for c in clusters
    ]

In backend/agents/runner.py, right after clusters = [...] / object_graph = build_object_graph(clusters) (before memory.replace("clusters", clusters)):

from backend.agents.disputes import annotate_clusters
...
    object_graph = build_object_graph(clusters)
    disputed_count = annotate_clusters(clusters)
    if disputed_count:
        orchestrator.log(
            f"[Link] {disputed_count} clusters carry disputed extracted values"
        )

(Check Orchestrator for the actual log method name — orchestrator.stage(...) exists; if no .log, use the module's existing logging/print convention. Adjust to match.)

Step 4: Run tests

Run: .venv/bin/python -m pytest tests/agents/ -v Expected: all pass (including existing test_runner_review_gate.py)

Step 5: Commit

git add backend/agents/runner.py backend/pipeline/_serialize.py tests/agents/test_disputes.py
git commit -m "feat: surface disputed extracted values to critic and specialist prompts"

---

Task 3: Prompt hardening — extracted text is fallible

Objective: Tell text-only specialists how to handle disputed/unverified values so they stop asserting buildability conclusions from a single (possibly misread) number.

Files:

• Modify: backend/prompts.py CONSTRUCTABILITY_SYSTEM_PROMPT (line 533) and CONSTRUCTABILITY_USER_INSTRUCTION (line 551)

Step 1: Edit prompts

Append to CONSTRUCTABILITY_SYSTEM_PROMPT Rules list (after line 547, before "Use plain ASCII"):

- Assertions are machine-extracted from sheet images and may contain misread values,
  especially quantities and member sizes (e.g. "(2) 2x6" vs "(5) 2x6").
- When the cluster lists disputed_attributes, or two evidence items disagree on a
  numeric value, do NOT assert a buildability conclusion from one reading. Report the
  ambiguity itself (category "detail_gap", confidence "low") and state that the value
  needs verification against the sheet.

Append to CONSTRUCTABILITY_USER_INSTRUCTION after the Cross-discipline conflicts already found: {conflicts} line:

Disputed extracted values in this cluster (possible vision misreads - treat as unverified): {disputes}

Step 2: Wire the {disputes} placeholder in construct_agent.py

In backend/agents/construct_agent.py run(), extend the substitutions dict:

            substitutions = {
                "assertions": dumps(cluster["assertions"]),
                "clusters": dumps(slim_clusters([cluster])),
                "conflicts": dumps(scope.payload.get("conflicts") or []),
                "disputes": dumps(cluster.get("disputed_attributes") or []),
            }

Step 3: Run full test suite (prompt edits can break runner tests that snapshot prompts)

Run: .venv/bin/python -m pytest tests/ -q Expected: all pass

Step 4: Commit

git add backend/prompts.py backend/agents/construct_agent.py
git commit -m "feat: constructability prompt treats disputed extracted values as unverified"

---

Phase 2 — Cross-sheet xref linking

Task 4: detail-reference / tag xref buckets in the linker (TDD)

Objective: Assertions sharing a detail_reference or a member tag get linked across levels, so S101/S205/S401 details of the same physical element land in one scope.

Files:

• Modify: backend/agents/linker.py (build_link_scopes, line 33)
• Test: tests/agents/test_linker_xref.py

Step 1: Write failing test

# tests/agents/test_linker_xref.py
from backend.agents.base import AgentScope
from backend.agents.linker import build_link_scopes


def _sheet(number, page, level, assertions):
    return {"sheet_number": number, "page_number": page,
            "discipline": "Structural", "level": level,
            "assertions": assertions}


def _assertion(id_, ref=None, tag=None, level=None):
    return {"id": id_, "attribute": "stud_pack_size", "value": "(5) 2x6",
            "source_text": "(5) 2x6 STUD PACK",
            "location_key": {"detail_reference": ref, "tag": tag,
                             "level": level}}


def test_xref_scope_joins_same_detail_reference_across_levels():
    sheets = [
        _sheet("S101", 10, "foundation", [_assertion("a1", ref="A/S205")]),
        _sheet("S205", 20, "roof", [_assertion("a2", ref="A/S205")]),
        _sheet("S401", 30, "roof", [_assertion("a3", ref="A/S205")]),
    ]
    scopes = build_link_scopes(sheets)
    xref = [s for s in scopes if s.scope_id.startswith("xref:")]
    assert xref, "expected a cross-level detail-reference scope"
    ids = {a["id"] for s in xref for a in s.payload["assertions"]}
    assert ids == {"a1", "a2", "a3"}


def test_xref_scope_requires_two_distinct_sheets():
    sheets = [
        _sheet("S401", 30, "roof", [_assertion("a1", ref="A/S205"),
                                    _assertion("a2", ref="A/S205")]),
    ]
    scopes = build_link_scopes(sheets)
    assert not [s for s in scopes if s.scope_id.startswith("xref:")]


def test_xref_scope_joins_shared_member_tag():
    sheets = [
        _sheet("S102", 5, "roof", [_assertion("a1", tag="HSS16X4X5/8")]),
        _sheet("S401", 30, "unknown", [_assertion("a2", tag="HSS16X4X5/8")]),
    ]
    scopes = build_link_scopes(sheets)
    xref = [s for s in scopes if s.scope_id.startswith("xref:")]
    assert xref

Step 2: Run test to verify failure

Run: .venv/bin/python -m pytest tests/agents/test_linker_xref.py -v Expected: FAIL — no xref: scopes produced

Step 3: Implement

Rewrite build_link_scopes in backend/agents/linker.py (keep the existing (level, family) bucketing, add the xref pass):

def _xref_keys(assertion: Dict) -> List[str]:
    """Cross-level join keys: detail references and member tags."""
    location = assertion.get("location_key") or {}
    keys = []
    ref = re.sub(r"\s+", "", str(location.get("detail_reference") or "")).upper()
    if ref:
        keys.append(f"detail:{ref}")
    tag = re.sub(r"\s+", "", str(location.get("tag") or "")).upper()
    if re.match(r"^[A-Z]{2,}\d", tag):  # member marks: HSS16X4X5/8, W12X26, ...
        keys.append(f"tag:{tag}")
    return keys


def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]:
    """Partition facts by level and object/tag family, then enforce a hard cap.

    A second pass joins assertions that share a detail_reference or member tag
    ACROSS levels, so plan/detail/section sheets describing the same physical
    element are linked together even though their levels differ.
    """
    buckets: Dict[Tuple[str, str], List[Dict]] = defaultdict(list)
    xref: Dict[str, List[Dict]] = defaultdict(list)
    for sheet in sheets:
        for assertion in sheet.get("assertions", []):
            enriched = {
                **assertion,
                "discipline": sheet.get("discipline") or "Unknown",
                "sheet_number": sheet.get("sheet_number"),
                "page_number": sheet.get("page_number"),
            }
            level = str((assertion.get("location_key") or {}).get("level")
                        or sheet.get("level") or "unknown").lower()
            buckets[(level, _family(assertion))].append(enriched)
            for key in _xref_keys(assertion):
                xref[key].append(enriched)

    scopes: List[AgentScope] = []
    cap = max(2, config.AGENT_LINK_MAX_ASSERTIONS)
    for (level, family), assertions in sorted(buckets.items()):
        for offset in range(0, len(assertions), cap):
            chunk = assertions[offset:offset + cap]
            if len(chunk) < 2:
                continue
            scopes.append(AgentScope(
                scope_id=f"{level}:{family}:{offset // cap + 1}",
                payload={"assertions": chunk, "level": level, "family": family},
            ))
    for key, assertions in sorted(xref.items()):
        sheets_present = {a.get("sheet_number") for a in assertions}
        if len(assertions) < 2 or len(sheets_present) < 2:
            continue
        scopes.append(AgentScope(
            scope_id=f"xref:{key}",
            payload={"assertions": assertions[:cap],
                     "level": "xref", "family": key},
        ))
    return scopes

Step 4: Run tests

Run: .venv/bin/python -m pytest tests/agents/test_linker_xref.py tests/agents/ -v Expected: all pass (watch existing runner tests for scope-count coupling)

Step 5: Commit

git add backend/agents/linker.py tests/agents/test_linker_xref.py
git commit -m "feat: cross-level xref link scopes via detail_reference and member tag"

---

Phase 3 — Evidence verification wave (5b)

Task 5: Config knobs + verify prompts

Objective: Add the tuning surface and prompts for the vision fact-check agent.

Files:

• Modify: backend/config.py (near line 44, with the other AGENT_* knobs)
• Modify: backend/prompts.py (append near the CONFLICT prompts, ~line 460)
• Modify: backend/.env.example

Step 1: Add config knobs to backend/config.py

AGENT_VERIFY_MODEL         = os.getenv("AGENT_VERIFY_MODEL", "") or MODEL
AGENT_VERIFY_CONCURRENCY   = int(os.getenv("AGENT_VERIFY_CONCURRENCY", "4"))
AGENT_VERIFY_MAX_CHECKS    = int(os.getenv("AGENT_VERIFY_MAX_CHECKS", "20"))
AGENT_VERIFY_SEVERITIES    = {
    s.strip().lower()
    for s in os.getenv("AGENT_VERIFY_SEVERITIES", "critical,high").split(",")
    if s.strip()
}
AGENT_VERIFY_REASONING_EFFORT = os.getenv("AGENT_VERIFY_REASONING_EFFORT", "low").strip()
VERIFY_MAX_TOKENS          = int(os.getenv("VERIFY_MAX_TOKENS", "8192"))

Append to backend/.env.example:

# Wave 5b evidence verification (vision fact-check of cited sheet text)
AGENT_VERIFY_MAX_CHECKS=20
AGENT_VERIFY_SEVERITIES=critical,high
AGENT_VERIFY_REASONING_EFFORT=low
VERIFY_MAX_TOKENS=8192

Step 2: Add prompts to backend/prompts.py

VERIFY_SYSTEM_PROMPT = """You are a meticulous construction document checker verifying machine-extracted evidence against the actual drawing sheet images.
For each evidence item you are given the sheet it was extracted from and the verbatim text the extractor claims appears there.
Judge each item against the images:
- confirmed: the text (or an obvious equivalent) appears on the cited sheet and means what the finding claims.
- corrected: the sheet shows a DIFFERENT value than the extracted text. Give the actual verbatim text.
- not_found: nothing like the extracted text appears on the cited sheet.
Be strict about numbers, quantities, and member sizes: "(2) 2x6" and "(5) 2x6" are different values. HSS16x4 and HSS16x16 are different values.
Use plain ASCII only.
Respond only with valid JSON."""

VERIFY_USER_INSTRUCTION = """Verify this finding's evidence against the attached sheet images.
Respond ONLY with a valid JSON object - no markdown fences, no explanation:
{ "verdicts": [ { "sheet": "string", "source_text": "the evidence text judged", "verdict": "confirmed | corrected | not_found", "actual_text": "verbatim sheet text when corrected, else null", "notes": "string or null" } ] }
Finding: {finding}"""

Step 3: Sanity check

Run: .venv/bin/python -c "from backend import config, prompts; print(config.AGENT_VERIFY_MAX_CHECKS, config.AGENT_VERIFY_SEVERITIES); print(prompts.VERIFY_SYSTEM_PROMPT[:40])" Expected: 20 {'critical', 'high'} and prompt text

Step 4: Commit

git add backend/config.py backend/prompts.py backend/.env.example
git commit -m "feat: config knobs and prompts for evidence verification wave"

---

Task 6: EvidenceVerifierAgent + runner wave 5b (TDD)

Objective: Re-read cited sheet images for selected findings; annotate verified findings, suppress refuted ones before the Brain merge.

Files:

• Create: backend/agents/verifier.py
• Modify: backend/agents/runner.py (new wave between wave 5 and wave 6, ~line 167)
• Modify: backend/agents/construct_agent.py line 67 (stamp cluster_key for dispute-based selection)
• Test: tests/agents/test_verifier.py

Step 1: Write failing test

# tests/agents/test_verifier.py
from unittest.mock import patch

from backend.agents.base import AgentScope, AgentUsage
from backend.agents.verifier import (
    EvidenceVerifierAgent, apply_verdicts, select_findings,
)


def _finding(sev="critical", issue_id="i1", sheets=("S401",), cluster_key=None):
    f = {"issue_id": issue_id, "severity": sev, "confidence": "high",
         "source_stage": "constructability", "sheets": list(sheets),
         "description": "HSS16x4 on (2) 2x6 STUD PACK is unbuildable",
         "evidence": [{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
                       "asserted_value": "3-inch width"}]}
    if cluster_key:
        f["cluster_key"] = cluster_key
    return f


def test_select_findings_by_severity_and_dispute():
    findings = [_finding("critical"), _finding("low", "i2"),
                _finding("medium", "i3", cluster_key="c9")]
    clusters = [{"key": "c9", "disputed_attributes": [{"attribute": "a"}]}]
    selected = select_findings(findings, clusters, max_checks=20,
                               severities={"critical", "high"})
    assert [f["issue_id"] for f in selected] == ["i1", "i3"]


def test_select_findings_respects_cap():
    findings = [_finding("critical", f"i{n}") for n in range(30)]
    selected = select_findings(findings, [], max_checks=5,
                               severities={"critical"})
    assert len(selected) == 5


def test_run_attaches_verdicts_and_marks_refuted():
    agent = EvidenceVerifierAgent(usage=AgentUsage())
    scope = AgentScope(scope_id="verify:0", payload={
        "finding_index": 0,
        "finding": _finding(),
        "images_b64": ["QUJD"],
    })
    verdicts = {"verdicts": [
        {"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
         "verdict": "corrected", "actual_text": "(5) 2x6 STUD PACK",
         "notes": "callout reads (5)"},
    ]}
    with patch("backend.agents.verifier.call_json", return_value=verdicts):
        result = agent.run(scope)
    assert not result.error
    artifact = result.artifacts[0]
    assert artifact["finding_index"] == 0
    assert artifact["status"] == "refuted"      # no evidence confirmed
    assert artifact["verdicts"][0]["actual_text"] == "(5) 2x6 STUD PACK"


def test_apply_verdicts_annotates_and_suppresses():
    findings = [_finding("critical", "i1"), _finding("high", "i2")]
    from backend.agents.base import AgentResult
    results = [AgentResult(scope_id="verify:0", artifacts=[
        {"finding_index": 0, "status": "refuted", "verdicts": []},
        {"finding_index": 1, "status": "confirmed", "verdicts": []},
    ])]
    suppressed = apply_verdicts(findings, results)
    assert suppressed == [findings[0]]
    assert findings[0]["verification"]["status"] == "refuted"
    assert findings[1]["verification"]["status"] == "confirmed"

Step 2: Run test to verify failure

Run: .venv/bin/python -m pytest tests/agents/test_verifier.py -v Expected: FAIL — ModuleNotFoundError: backend.agents.verifier

Step 3: Implement backend/agents/verifier.py

"""Wave 5b: vision fact-check of extracted evidence against cited sheet images.

Downstream specialists are text-only; a wave-1 vision misread ("(2) 2x6" vs
"(5) 2x6") otherwise becomes immutable ground truth. For high-severity or
dispute-linked findings, re-read the cited sheets and adjudicate each evidence
item: confirmed / corrected / not_found. Findings whose evidence is entirely
unconfirmed are suppressed before the Brain merge.
"""

from typing import Dict, List, Optional, Set

from backend import config
from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure
from backend.llm import call_json
from backend.pipeline._serialize import dumps
from backend.pipeline._stage import collect_list, render
from backend.prompts import VERIFY_SYSTEM_PROMPT, VERIFY_USER_INSTRUCTION

_SEVERITY_RANK = {"critical": 0, "high": 1, "medium": 2, "low": 3}
_VERDICTS = ("confirmed", "corrected", "not_found")


def select_findings(
    findings: List[Dict],
    clusters: List[Dict],
    max_checks: int,
    severities: Set[str],
) -> List[Dict]:
    """Severity-gated selection plus any finding tied to a disputed cluster."""
    disputed_keys = {
        cluster.get("key") for cluster in clusters
        if cluster.get("disputed_attributes")
    }
    selected = [
        finding for finding in findings
        if str(finding.get("severity") or "").lower() in severities
        or finding.get("cluster_key") in disputed_keys
    ]
    selected.sort(key=lambda f: _SEVERITY_RANK.get(
        str(f.get("severity") or "").lower(), 9))
    return selected[:max_checks]


def _valid_verdict(item: Dict) -> Optional[Dict]:
    if not isinstance(item, dict):
        return None
    verdict = str(item.get("verdict") or "").lower()
    if verdict not in _VERDICTS:
        return None
    return {
        "sheet": item.get("sheet") or "",
        "source_text": item.get("source_text") or "",
        "verdict": verdict,
        "actual_text": item.get("actual_text"),
        "notes": item.get("notes"),
    }


def _status(verdicts: List[Dict]) -> str:
    if not verdicts:
        return "unverified"
    confirmed = sum(1 for v in verdicts if v["verdict"] == "confirmed")
    if confirmed == len(verdicts):
        return "confirmed"
    if confirmed == 0:
        return "refuted"
    return "mixed"


class EvidenceVerifierAgent:
    name = "verify"

    def __init__(self, usage: AgentUsage) -> None:
        self.usage = usage

    def run(self, scope: AgentScope) -> AgentResult:
        try:
            finding = scope.payload["finding"]
            instruction = render(
                VERIFY_USER_INSTRUCTION, {"finding": dumps(finding)}
            )
            parsed = call_json(
                system_prompt=VERIFY_SYSTEM_PROMPT,
                user_text=instruction,
                images_b64=scope.payload.get("images_b64") or [],
                max_tokens=config.VERIFY_MAX_TOKENS,
                model=config.AGENT_VERIFY_MODEL,
                reasoning_effort=config.AGENT_VERIFY_REASONING_EFFORT or None,
                usage_tracker=self.usage,
                usage_stage="agent.verify",
            )
            verdicts = collect_list(parsed, "verdicts", _valid_verdict)
            return AgentResult(scope_id=scope.scope_id, artifacts=[{
                "finding_index": scope.payload["finding_index"],
                "status": _status(verdicts),
                "verdicts": verdicts,
            }])
        except Exception as exc:
            return failure(scope, exc)


def apply_verdicts(
    findings: List[Dict], verify_results: List[AgentResult]
) -> List[Dict]:
    """Annotate findings with verification; return refuted ones to suppress."""
    by_index: Dict[int, Dict] = {}
    for result in verify_results:
        for artifact in result.artifacts:
            by_index[artifact["finding_index"]] = artifact
    suppressed = []
    for index, finding in enumerate(findings):
        artifact = by_index.get(index)
        if not artifact:
            continue
        finding["verification"] = {
            "status": artifact["status"],
            "verdicts": artifact["verdicts"],
        }
        if artifact["status"] == "refuted":
            finding["confidence"] = "low"
            suppressed.append(finding)
    return suppressed

Step 4: Stamp cluster_key on constructability findings

In backend/agents/construct_agent.py line 66-67, change:

            for finding in findings:
                finding.update(agent=self.name, scope_id=scope.scope_id)

to:

            for finding in findings:
                finding.update(agent=self.name, scope_id=scope.scope_id,
                               cluster_key=cluster.get("key"))

Step 5: Wire wave 5b into backend/agents/runner.py

After memory.extend("findings", specialist_findings) (line 167) and before gap_findings / wave 6:

    orchestrator.stage("Agent wave 5b: evidence verification")
    sheet_to_page = {
        sheet.get("sheet_number"): sheet.get("page_number") for sheet in sheets
    }
    verify_targets = select_findings(
        specialist_findings, clusters,
        max_checks=config.AGENT_VERIFY_MAX_CHECKS,
        severities=config.AGENT_VERIFY_SEVERITIES,
    )
    target_indexes = {id(f): i for i, f in enumerate(specialist_findings)}
    verify_scopes = [
        AgentScope(
            scope_id=f"verify:{target_indexes[id(finding)]}",
            payload={
                "finding_index": target_indexes[id(finding)],
                "finding": finding,
                "images_b64": [
                    page_to_b64[sheet_to_page[name]]
                    for name in (finding.get("sheets") or [])
                                [:config.AGENT_CONFLICT_MAX_IMAGES]
                    if sheet_to_page.get(name) in page_to_b64
                ],
            },
        )
        for finding in verify_targets
    ]
    verify_results = orchestrator.run_scopes(
        EvidenceVerifierAgent(usage), verify_scopes,
        config.AGENT_VERIFY_CONCURRENCY,
    )
    suppressed = apply_verdicts(specialist_findings, verify_results)
    if suppressed:
        suppressed_ids = {id(f) for f in suppressed}
        specialist_findings = [
            f for f in specialist_findings if id(f) not in suppressed_ids
        ]
        memory.replace("suppressed", suppressed)
        memory.extend("findings", specialist_findings)  # see note below

NOTE for implementer: memory.extend("findings", ...) already ran with the un-suppressed list. Adjust ordering so verification happens BEFORE memory.extend("findings", specialist_findings) — i.e. move the extend to after wave 5b — so the Brain never sees refuted findings. Keep gap_findings logic unchanged. Also add imports at top of runner.py:

from backend.agents.verifier import (
    EvidenceVerifierAgent, apply_verdicts, select_findings,
)

And in the report dicts (both the require_review branch ~line 217-225 and the wave-7 branch ~line 293-300), populate suppressed issues:

        "suppressed_issues": memory.snapshot().get("suppressed") or [],

Finally, verify results cost shows up as agent.verify in summary.cost_by_stage automatically via usage_stage="agent.verify".

Step 6: Update existing runner tests

tests/agents/test_runner_review_gate.py monkeypatches BrainAgent and convert_pdf_to_images but lets waves 1-5 run against... check how LLM calls are stubbed there (likely call_json returns None -> empty artifacts, which is fine). The new wave must no-op cleanly when select_findings returns [] (zero scopes -> run_scopes returns [] per orchestrator.py:60). Verify by running the suite; if a runner test now fails because verification selects a stubbed finding, monkeypatch select_findings to lambda *a, **k: [] in that test file's _patch_brain helper.

Step 7: Run full suite

Run: .venv/bin/python -m pytest tests/ -q Expected: all pass

Step 8: Commit

git add backend/agents/verifier.py backend/agents/runner.py backend/agents/construct_agent.py tests/agents/test_verifier.py tests/agents/test_runner_review_gate.py
git commit -m "feat: wave 5b evidence verification - vision fact-check before Brain merge"

---

Task 7 (small, related): reasoning budget for the conflict critic

Objective: Fix the wave-4 truncation found in this same job (13/121 calls hit finish_reason=length at the 4096 cap with ~3.7k thinking tokens).

Files:

• Modify: backend/agents/conflict_critic.py:55-63
• Modify: backend/pipeline/conflict_checker.py:85 (same pattern, classic path)

Step 1: Apply the extractor's reasoning-knob pattern

            parsed = call_json(
                system_prompt=CONFLICT_SYSTEM_PROMPT,
                user_text=instruction,
                images_b64=images,
                max_tokens=config.REASON_MAX_TOKENS,
                model=config.AGENT_CONFLICT_MODEL,
                reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
                reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
                usage_tracker=self.usage,
                usage_stage="agent.conflict",
            )

(Match the exact kwarg names SheetExtractorAgent uses — check backend/agents/extractors.py for whether it passes None when the budget is 0, and mirror that guard.)

Step 2: Run tests

Run: .venv/bin/python -m pytest tests/ -q Expected: all pass

Step 3: Commit

git add backend/agents/conflict_critic.py backend/pipeline/conflict_checker.py
git commit -m "fix: reasoning budget for conflict critic (wave-4 max_tokens truncation)"

---

Tests / validation

1. .venv/bin/python -m pytest tests/ -q — full suite green.
2. Targeted repro of the original failure: pull the S401 page image from job 959e16407573's output dir on sits-docker (or re-render the PDF page), then run one EvidenceVerifierAgent scope locally against the finding JSON from validated_issues[3]. Expected: verdict corrected, actual_text: "(5) 2x6 STUD PACK", status refuted.
3. End-to-end: rerun the same Cypress TX PDF through the pipeline (local with LLM_CACHE/LLM_RAW_DUMP per the conflict-checker skill). Expected:
   • log shows Agent wave 5b: evidence verification with a bounded number of calls;
   • the S401 stud-pack finding is either absent from validated_issues and present in suppressed_issues with verification verdicts, or downgraded to low confidence;
   • agent.verify appears in summary.cost_by_stage;
   • zero finish_reason=length lines in wave 4 (Task 7).
4. Cost check: wave 5b adds at most AGENT_VERIFY_MAX_CHECKS (20) vision calls — for this job's profile that is well under $1.

Risks, tradeoffs, open questions

• Dispute false positives: cluster members with legitimately different values (e.g. two doors in one door cluster) will produce disputed_attributes. Mitigation: prompts treat disputes as "unverified", not "wrong"; only severity-gated findings burn verification calls. Tune later by restricting find_disputes to numeric-ish values if noise is high.
• Verifier can also misread. It is one model checking another with the same eyes. Mitigation: verdict requires actual_text verbatim evidence for corrected, and only fully-unconfirmed findings are suppressed (mixed keeps the finding with a note).
• xref cost: extra link scopes. Bounded by the >= 2 distinct sheets gate and the existing assertion cap; expect a handful of extra scopes per set.
• Suppression in review mode: refuted findings land in suppressed_issues — the review UI/finalizer must tolerate that list being non-empty (it is currently always [] in agent mode). Open question: surface suppressed items in the human review queue as informational, or keep them report-only?
• Open question: should wave-4 conflict findings (which already saw images) also be verification-eligible? Plan says no (they had the pixels); revisit if critics show the same misread pattern.