# Evidence Verification + Cross-Sheet Correlation Implementation Plan

> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.

**Goal:** Stop vision-extraction misreads (e.g. "(2) 2x6 STUD PACK" vs the actual "(5) 2x6") from becoming confident downstream findings, and correlate the same physical element across sheets (S101/S205/S401) so no stage reasons from one sheet's text in isolation.

**Architecture:** Three independently shippable phases on the agent pipeline (`backend/agents/runner.py`):
1. **Disputed-value detection** — deterministic post-link pass that flags contradictory extracted values inside a cluster and surfaces them to the critic/specialist prompts.
2. **Cross-sheet xref linking** — linker gains detail-reference/tag buckets that join assertions across levels (today `(level, family)` bucketing splits S101/S205/S401 apart).
3. **Evidence verification wave (5b)** — a bounded vision fact-check agent re-reads the cited sheet images for high-severity / disputed findings before the Brain merge, annotates or suppresses findings built on phantom text.

**Tech Stack:** Python 3.14, pytest (`tests/`), existing `call_json` LLM wrapper (supports `images_b64`, `reasoning_effort`, `reasoning_max_tokens`).

---

## Current context / root cause (from job 959e16407573)

Finding `validated_issues[3]` ("FRONT PERSPECTIVE detail on Sheet S401", HSS16x4 on
"(2) 2x6 STUD PACK", severity critical) is a **false positive built on a wave-1 vision
misread**. The sheet actually shows a (5) 2x6 stud pack (matching S205/S101). Chain of failure:

1. Wave 1 (`SheetExtractorAgent`) froze the misread into text. From then on it is "ground truth".
2. Wave 3 linker (`backend/agents/linker.py:33` `build_link_scopes`) buckets by
   `(level, family)`. S101 (foundation), S205 (details), S401 (sections) get different
   `level` values, so assertions about the same front-wall header never share a link scope
   or cluster. No cross-sheet corroboration happened.
3. Wave 5 `ConstructabilityAgent` (`backend/agents/construct_agent.py:53`) calls
   `call_json` **with no images** — in this job 120/120 constructability calls were `+0img`.
   It reasoned arithmetically from the misread text ("2 x 1.5in = 3in < 4in -> unbuildable").
   It even held the "(5) 2x6 STUD PACK" assertion in the same scope but labeled it
   "Ambiguous column size specification" instead of arbitrating.
4. Nothing between wave 5 and the report ever looks at a sheet image again. Only the wave-4
   conflict critic receives images, and only for its own cluster's pages.

Also confirmed in this log (separate known bug, fixed in Task 7 while we're here): wave-4
conflict critic truncates on Gemini thinking tokens because `conflict_critic.py:59` passes
`max_tokens=config.REASON_MAX_TOKENS` (4096) with no reasoning budget — 13/121 calls hit
`finish_reason=length`.

## Assumptions

- Assertions carry `id`, `attribute`, `value`, `source_text`, `location_key`
  (`room`/`grid`/`detail_reference`/`tag`/`level`) — see `linker._payload` and
  `_serialize.slim_assertion`.
- Extractor assertions already carry a `confidence` field (per test fixtures).
- `validate_issue` in `backend/pipeline/_stage.py` guarantees each finding an `issue_id`.
- Test convention: `unittest.mock.patch("backend.agents.<module>.call_json", ...)` —
  see `tests/agents/test_sheet_extractor_fallback.py`. Run tests with
  `.venv/bin/python -m pytest tests/ -x -q`.
- `AgentResult.error` defaults to `""` (not None) in assertions.

---

## Phase 1 — Disputed-value detection + prompt hardening

### Task 1: `find_disputes` pure function (TDD)

**Objective:** Detect "same attribute, different values" inside one cluster's assertions.

**Files:**
- Create: `backend/agents/disputes.py`
- Test: `tests/agents/test_disputes.py`

**Step 1: Write failing test**

```python
# tests/agents/test_disputes.py
from backend.agents.disputes import annotate_clusters, find_disputes


def _a(id_, attribute, value):
    return {"id": id_, "attribute": attribute, "value": value,
            "source_text": value}


def test_find_disputes_flags_same_attribute_different_values():
    assertions = [
        _a("a1", "stud_pack_size", "(2) 2x6 STUD PACK"),
        _a("a2", "stud_pack_size", "(5) 2x6 STUD PACK"),
        _a("a3", "beam_size", "HSS16X4X5/8"),
    ]
    disputes = find_disputes(assertions)
    assert len(disputes) == 1
    assert disputes[0]["attribute"] == "stud_pack_size"
    assert disputes[0]["values"] == ["(2) 2x6 STUD PACK", "(5) 2x6 STUD PACK"]
    assert disputes[0]["assertion_ids"] == ["a1", "a2"]


def test_find_disputes_ignores_agreeing_values_and_blanks():
    assertions = [
        _a("a1", "beam_size", "HSS16X4X5/8"),
        _a("a2", "beam_size", "  hss16x4x5/8 "),   # same after normalize
        _a("a3", "", "orphan"),                     # no attribute -> skipped
        _a("a4", "beam_size", ""),                  # no value -> skipped
    ]
    assert find_disputes(assertions) == []


def test_annotate_clusters_writes_disputed_attributes():
    clusters = [
        {"key": "c1", "assertions": [
            _a("a1", "stud_pack_size", "(2) 2x6"),
            _a("a2", "stud_pack_size", "(5) 2x6"),
        ]},
        {"key": "c2", "assertions": [_a("a3", "x", "1"), _a("a4", "x", "1")]},
    ]
    assert annotate_clusters(clusters) == 1
    assert clusters[0]["disputed_attributes"][0]["attribute"] == "stud_pack_size"
    assert "disputed_attributes" not in clusters[1]
```

**Step 2: Run test to verify failure**

Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py -v`
Expected: FAIL — `ModuleNotFoundError: backend.agents.disputes`

**Step 3: Implement**

```python
# backend/agents/disputes.py
"""Deterministic detection of contradictory extracted values within a cluster.

Extraction is a vision pass: quantities and sizes can be misread ("(2) 2x6" vs
"(5) 2x6"). Cluster members are supposed to describe the same real-world
element, so two members asserting different values for the same attribute are
a probable misread. Flag these so downstream text-only stages treat the value
as unverified instead of reasoning from one reading.
"""

import re
from typing import Dict, List


def _norm(value) -> str:
    return re.sub(r"\s+", " ", str(value or "").strip().lower())


def find_disputes(assertions: List[Dict]) -> List[Dict]:
    """Same attribute with >= 2 distinct normalized values = disputed."""
    groups: Dict[str, Dict[str, set]] = {}
    for assertion in assertions:
        attribute = _norm(assertion.get("attribute"))
        value = _norm(assertion.get("value"))
        if not attribute or not value:
            continue
        groups.setdefault(attribute, {}).setdefault(value, set()).add(
            assertion.get("id")
        )
    disputes = []
    for attribute, values in sorted(groups.items()):
        if len(values) < 2:
            continue
        disputes.append({
            "attribute": attribute,
            "values": sorted(values),
            "assertion_ids": sorted(
                aid for ids in values.values() for aid in ids if aid
            ),
        })
    return disputes


def annotate_clusters(clusters: List[Dict]) -> int:
    """Attach disputed_attributes to each cluster that has any. Returns count."""
    annotated = 0
    for cluster in clusters:
        disputes = find_disputes(cluster.get("assertions") or [])
        if disputes:
            cluster["disputed_attributes"] = disputes
            annotated += 1
    return annotated
```

**Step 4: Run test to verify pass**

Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py -v`
Expected: 3 passed

**Step 5: Commit**

```bash
git add backend/agents/disputes.py tests/agents/test_disputes.py
git commit -m "feat: deterministic disputed-value detection for cluster assertions"
```

---

### Task 2: Wire `annotate_clusters` into the runner + serialization

**Objective:** Disputes must be visible to the wave-4 critic and wave-5 constructability prompts.

**Files:**
- Modify: `backend/agents/runner.py` (after `memory.replace("clusters", clusters)`, ~line 116)
- Modify: `backend/pipeline/_serialize.py` (`slim_clusters`, line 46)
- Test: `tests/agents/test_disputes.py` (append)

**Step 1: Write failing test**

```python
def test_slim_clusters_preserves_disputed_attributes():
    from backend.pipeline._serialize import slim_clusters
    cluster = {"key": "c1", "assertions": [],
               "disputed_attributes": [{"attribute": "a", "values": ["1", "2"],
                                         "assertion_ids": ["x", "y"]}]}
    slim = slim_clusters([cluster])[0]
    assert slim["disputed_attributes"][0]["values"] == ["1", "2"]
```

**Step 2: Run test to verify failure**

Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py::test_slim_clusters_preserves_disputed_attributes -v`
Expected: FAIL — `KeyError: 'disputed_attributes'`

**Step 3: Implement**

In `backend/pipeline/_serialize.py` `slim_clusters`, add the key:

```python
def slim_clusters(clusters: List[Dict]) -> List[Dict]:
    return [
        {
            "key": c.get("key"),
            "location": c.get("location"),
            "disciplines": c.get("disciplines"),
            "kind": c.get("kind"),
            **({"disputed_attributes": c["disputed_attributes"]}
               if c.get("disputed_attributes") else {}),
            "assertions": [slim_assertion(a) for a in c.get("assertions", [])],
        }
        for c in clusters
    ]
```

In `backend/agents/runner.py`, right after `clusters = [...]` / `object_graph = build_object_graph(clusters)` (before `memory.replace("clusters", clusters)`):

```python
from backend.agents.disputes import annotate_clusters
...
    object_graph = build_object_graph(clusters)
    disputed_count = annotate_clusters(clusters)
    if disputed_count:
        orchestrator.log(
            f"[Link] {disputed_count} clusters carry disputed extracted values"
        )
```

(Check `Orchestrator` for the actual log method name — `orchestrator.stage(...)` exists;
if no `.log`, use the module's existing logging/print convention. Adjust to match.)

**Step 4: Run tests**

Run: `.venv/bin/python -m pytest tests/agents/ -v`
Expected: all pass (including existing `test_runner_review_gate.py`)

**Step 5: Commit**

```bash
git add backend/agents/runner.py backend/pipeline/_serialize.py tests/agents/test_disputes.py
git commit -m "feat: surface disputed extracted values to critic and specialist prompts"
```

---

### Task 3: Prompt hardening — extracted text is fallible

**Objective:** Tell text-only specialists how to handle disputed/unverified values so they stop asserting buildability conclusions from a single (possibly misread) number.

**Files:**
- Modify: `backend/prompts.py` `CONSTRUCTABILITY_SYSTEM_PROMPT` (line 533) and `CONSTRUCTABILITY_USER_INSTRUCTION` (line 551)

**Step 1: Edit prompts**

Append to `CONSTRUCTABILITY_SYSTEM_PROMPT` Rules list (after line 547, before "Use plain ASCII"):

```
- Assertions are machine-extracted from sheet images and may contain misread values,
  especially quantities and member sizes (e.g. "(2) 2x6" vs "(5) 2x6").
- When the cluster lists disputed_attributes, or two evidence items disagree on a
  numeric value, do NOT assert a buildability conclusion from one reading. Report the
  ambiguity itself (category "detail_gap", confidence "low") and state that the value
  needs verification against the sheet.
```

Append to `CONSTRUCTABILITY_USER_INSTRUCTION` after the `Cross-discipline conflicts already found: {conflicts}` line:

```
Disputed extracted values in this cluster (possible vision misreads - treat as unverified): {disputes}
```

**Step 2: Wire the `{disputes}` placeholder in `construct_agent.py`**

In `backend/agents/construct_agent.py` `run()`, extend the `substitutions` dict:

```python
            substitutions = {
                "assertions": dumps(cluster["assertions"]),
                "clusters": dumps(slim_clusters([cluster])),
                "conflicts": dumps(scope.payload.get("conflicts") or []),
                "disputes": dumps(cluster.get("disputed_attributes") or []),
            }
```

**Step 3: Run full test suite (prompt edits can break runner tests that snapshot prompts)**

Run: `.venv/bin/python -m pytest tests/ -q`
Expected: all pass

**Step 4: Commit**

```bash
git add backend/prompts.py backend/agents/construct_agent.py
git commit -m "feat: constructability prompt treats disputed extracted values as unverified"
```

---

## Phase 2 — Cross-sheet xref linking

### Task 4: detail-reference / tag xref buckets in the linker (TDD)

**Objective:** Assertions sharing a `detail_reference` or a member `tag` get linked across levels, so S101/S205/S401 details of the same physical element land in one scope.

**Files:**
- Modify: `backend/agents/linker.py` (`build_link_scopes`, line 33)
- Test: `tests/agents/test_linker_xref.py`

**Step 1: Write failing test**

```python
# tests/agents/test_linker_xref.py
from backend.agents.base import AgentScope
from backend.agents.linker import build_link_scopes


def _sheet(number, page, level, assertions):
    return {"sheet_number": number, "page_number": page,
            "discipline": "Structural", "level": level,
            "assertions": assertions}


def _assertion(id_, ref=None, tag=None, level=None):
    return {"id": id_, "attribute": "stud_pack_size", "value": "(5) 2x6",
            "source_text": "(5) 2x6 STUD PACK",
            "location_key": {"detail_reference": ref, "tag": tag,
                             "level": level}}


def test_xref_scope_joins_same_detail_reference_across_levels():
    sheets = [
        _sheet("S101", 10, "foundation", [_assertion("a1", ref="A/S205")]),
        _sheet("S205", 20, "roof", [_assertion("a2", ref="A/S205")]),
        _sheet("S401", 30, "roof", [_assertion("a3", ref="A/S205")]),
    ]
    scopes = build_link_scopes(sheets)
    xref = [s for s in scopes if s.scope_id.startswith("xref:")]
    assert xref, "expected a cross-level detail-reference scope"
    ids = {a["id"] for s in xref for a in s.payload["assertions"]}
    assert ids == {"a1", "a2", "a3"}


def test_xref_scope_requires_two_distinct_sheets():
    sheets = [
        _sheet("S401", 30, "roof", [_assertion("a1", ref="A/S205"),
                                    _assertion("a2", ref="A/S205")]),
    ]
    scopes = build_link_scopes(sheets)
    assert not [s for s in scopes if s.scope_id.startswith("xref:")]


def test_xref_scope_joins_shared_member_tag():
    sheets = [
        _sheet("S102", 5, "roof", [_assertion("a1", tag="HSS16X4X5/8")]),
        _sheet("S401", 30, "unknown", [_assertion("a2", tag="HSS16X4X5/8")]),
    ]
    scopes = build_link_scopes(sheets)
    xref = [s for s in scopes if s.scope_id.startswith("xref:")]
    assert xref
```

**Step 2: Run test to verify failure**

Run: `.venv/bin/python -m pytest tests/agents/test_linker_xref.py -v`
Expected: FAIL — no `xref:` scopes produced

**Step 3: Implement**

Rewrite `build_link_scopes` in `backend/agents/linker.py` (keep the existing
`(level, family)` bucketing, add the xref pass):

```python
def _xref_keys(assertion: Dict) -> List[str]:
    """Cross-level join keys: detail references and member tags."""
    location = assertion.get("location_key") or {}
    keys = []
    ref = re.sub(r"\s+", "", str(location.get("detail_reference") or "")).upper()
    if ref:
        keys.append(f"detail:{ref}")
    tag = re.sub(r"\s+", "", str(location.get("tag") or "")).upper()
    if re.match(r"^[A-Z]{2,}\d", tag):  # member marks: HSS16X4X5/8, W12X26, ...
        keys.append(f"tag:{tag}")
    return keys


def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]:
    """Partition facts by level and object/tag family, then enforce a hard cap.

    A second pass joins assertions that share a detail_reference or member tag
    ACROSS levels, so plan/detail/section sheets describing the same physical
    element are linked together even though their levels differ.
    """
    buckets: Dict[Tuple[str, str], List[Dict]] = defaultdict(list)
    xref: Dict[str, List[Dict]] = defaultdict(list)
    for sheet in sheets:
        for assertion in sheet.get("assertions", []):
            enriched = {
                **assertion,
                "discipline": sheet.get("discipline") or "Unknown",
                "sheet_number": sheet.get("sheet_number"),
                "page_number": sheet.get("page_number"),
            }
            level = str((assertion.get("location_key") or {}).get("level")
                        or sheet.get("level") or "unknown").lower()
            buckets[(level, _family(assertion))].append(enriched)
            for key in _xref_keys(assertion):
                xref[key].append(enriched)

    scopes: List[AgentScope] = []
    cap = max(2, config.AGENT_LINK_MAX_ASSERTIONS)
    for (level, family), assertions in sorted(buckets.items()):
        for offset in range(0, len(assertions), cap):
            chunk = assertions[offset:offset + cap]
            if len(chunk) < 2:
                continue
            scopes.append(AgentScope(
                scope_id=f"{level}:{family}:{offset // cap + 1}",
                payload={"assertions": chunk, "level": level, "family": family},
            ))
    for key, assertions in sorted(xref.items()):
        sheets_present = {a.get("sheet_number") for a in assertions}
        if len(assertions) < 2 or len(sheets_present) < 2:
            continue
        scopes.append(AgentScope(
            scope_id=f"xref:{key}",
            payload={"assertions": assertions[:cap],
                     "level": "xref", "family": key},
        ))
    return scopes
```

**Step 4: Run tests**

Run: `.venv/bin/python -m pytest tests/agents/test_linker_xref.py tests/agents/ -v`
Expected: all pass (watch existing runner tests for scope-count coupling)

**Step 5: Commit**

```bash
git add backend/agents/linker.py tests/agents/test_linker_xref.py
git commit -m "feat: cross-level xref link scopes via detail_reference and member tag"
```

---

## Phase 3 — Evidence verification wave (5b)

### Task 5: Config knobs + verify prompts

**Objective:** Add the tuning surface and prompts for the vision fact-check agent.

**Files:**
- Modify: `backend/config.py` (near line 44, with the other AGENT_* knobs)
- Modify: `backend/prompts.py` (append near the CONFLICT prompts, ~line 460)
- Modify: `backend/.env.example`

**Step 1: Add config knobs to `backend/config.py`**

```python
AGENT_VERIFY_MODEL         = os.getenv("AGENT_VERIFY_MODEL", "") or MODEL
AGENT_VERIFY_CONCURRENCY   = int(os.getenv("AGENT_VERIFY_CONCURRENCY", "4"))
AGENT_VERIFY_MAX_CHECKS    = int(os.getenv("AGENT_VERIFY_MAX_CHECKS", "20"))
AGENT_VERIFY_SEVERITIES    = {
    s.strip().lower()
    for s in os.getenv("AGENT_VERIFY_SEVERITIES", "critical,high").split(",")
    if s.strip()
}
AGENT_VERIFY_REASONING_EFFORT = os.getenv("AGENT_VERIFY_REASONING_EFFORT", "low").strip()
VERIFY_MAX_TOKENS          = int(os.getenv("VERIFY_MAX_TOKENS", "8192"))
```

Append to `backend/.env.example`:

```
# Wave 5b evidence verification (vision fact-check of cited sheet text)
AGENT_VERIFY_MAX_CHECKS=20
AGENT_VERIFY_SEVERITIES=critical,high
AGENT_VERIFY_REASONING_EFFORT=low
VERIFY_MAX_TOKENS=8192
```

**Step 2: Add prompts to `backend/prompts.py`**

```python
VERIFY_SYSTEM_PROMPT = """You are a meticulous construction document checker verifying machine-extracted evidence against the actual drawing sheet images.
For each evidence item you are given the sheet it was extracted from and the verbatim text the extractor claims appears there.
Judge each item against the images:
- confirmed: the text (or an obvious equivalent) appears on the cited sheet and means what the finding claims.
- corrected: the sheet shows a DIFFERENT value than the extracted text. Give the actual verbatim text.
- not_found: nothing like the extracted text appears on the cited sheet.
Be strict about numbers, quantities, and member sizes: "(2) 2x6" and "(5) 2x6" are different values. HSS16x4 and HSS16x16 are different values.
Use plain ASCII only.
Respond only with valid JSON."""

VERIFY_USER_INSTRUCTION = """Verify this finding's evidence against the attached sheet images.
Respond ONLY with a valid JSON object - no markdown fences, no explanation:
{ "verdicts": [ { "sheet": "string", "source_text": "the evidence text judged", "verdict": "confirmed | corrected | not_found", "actual_text": "verbatim sheet text when corrected, else null", "notes": "string or null" } ] }
Finding: {finding}"""
```

**Step 3: Sanity check**

Run: `.venv/bin/python -c "from backend import config, prompts; print(config.AGENT_VERIFY_MAX_CHECKS, config.AGENT_VERIFY_SEVERITIES); print(prompts.VERIFY_SYSTEM_PROMPT[:40])"`
Expected: `20 {'critical', 'high'}` and prompt text

**Step 4: Commit**

```bash
git add backend/config.py backend/prompts.py backend/.env.example
git commit -m "feat: config knobs and prompts for evidence verification wave"
```

---

### Task 6: `EvidenceVerifierAgent` + runner wave 5b (TDD)

**Objective:** Re-read cited sheet images for selected findings; annotate verified findings, suppress refuted ones before the Brain merge.

**Files:**
- Create: `backend/agents/verifier.py`
- Modify: `backend/agents/runner.py` (new wave between wave 5 and wave 6, ~line 167)
- Modify: `backend/agents/construct_agent.py` line 67 (stamp `cluster_key` for dispute-based selection)
- Test: `tests/agents/test_verifier.py`

**Step 1: Write failing test**

```python
# tests/agents/test_verifier.py
from unittest.mock import patch

from backend.agents.base import AgentScope, AgentUsage
from backend.agents.verifier import (
    EvidenceVerifierAgent, apply_verdicts, select_findings,
)


def _finding(sev="critical", issue_id="i1", sheets=("S401",), cluster_key=None):
    f = {"issue_id": issue_id, "severity": sev, "confidence": "high",
         "source_stage": "constructability", "sheets": list(sheets),
         "description": "HSS16x4 on (2) 2x6 STUD PACK is unbuildable",
         "evidence": [{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
                       "asserted_value": "3-inch width"}]}
    if cluster_key:
        f["cluster_key"] = cluster_key
    return f


def test_select_findings_by_severity_and_dispute():
    findings = [_finding("critical"), _finding("low", "i2"),
                _finding("medium", "i3", cluster_key="c9")]
    clusters = [{"key": "c9", "disputed_attributes": [{"attribute": "a"}]}]
    selected = select_findings(findings, clusters, max_checks=20,
                               severities={"critical", "high"})
    assert [f["issue_id"] for f in selected] == ["i1", "i3"]


def test_select_findings_respects_cap():
    findings = [_finding("critical", f"i{n}") for n in range(30)]
    selected = select_findings(findings, [], max_checks=5,
                               severities={"critical"})
    assert len(selected) == 5


def test_run_attaches_verdicts_and_marks_refuted():
    agent = EvidenceVerifierAgent(usage=AgentUsage())
    scope = AgentScope(scope_id="verify:0", payload={
        "finding_index": 0,
        "finding": _finding(),
        "images_b64": ["QUJD"],
    })
    verdicts = {"verdicts": [
        {"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
         "verdict": "corrected", "actual_text": "(5) 2x6 STUD PACK",
         "notes": "callout reads (5)"},
    ]}
    with patch("backend.agents.verifier.call_json", return_value=verdicts):
        result = agent.run(scope)
    assert not result.error
    artifact = result.artifacts[0]
    assert artifact["finding_index"] == 0
    assert artifact["status"] == "refuted"      # no evidence confirmed
    assert artifact["verdicts"][0]["actual_text"] == "(5) 2x6 STUD PACK"


def test_apply_verdicts_annotates_and_suppresses():
    findings = [_finding("critical", "i1"), _finding("high", "i2")]
    from backend.agents.base import AgentResult
    results = [AgentResult(scope_id="verify:0", artifacts=[
        {"finding_index": 0, "status": "refuted", "verdicts": []},
        {"finding_index": 1, "status": "confirmed", "verdicts": []},
    ])]
    suppressed = apply_verdicts(findings, results)
    assert suppressed == [findings[0]]
    assert findings[0]["verification"]["status"] == "refuted"
    assert findings[1]["verification"]["status"] == "confirmed"
```

**Step 2: Run test to verify failure**

Run: `.venv/bin/python -m pytest tests/agents/test_verifier.py -v`
Expected: FAIL — `ModuleNotFoundError: backend.agents.verifier`

**Step 3: Implement `backend/agents/verifier.py`**

```python
"""Wave 5b: vision fact-check of extracted evidence against cited sheet images.

Downstream specialists are text-only; a wave-1 vision misread ("(2) 2x6" vs
"(5) 2x6") otherwise becomes immutable ground truth. For high-severity or
dispute-linked findings, re-read the cited sheets and adjudicate each evidence
item: confirmed / corrected / not_found. Findings whose evidence is entirely
unconfirmed are suppressed before the Brain merge.
"""

from typing import Dict, List, Optional, Set

from backend import config
from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure
from backend.llm import call_json
from backend.pipeline._serialize import dumps
from backend.pipeline._stage import collect_list, render
from backend.prompts import VERIFY_SYSTEM_PROMPT, VERIFY_USER_INSTRUCTION

_SEVERITY_RANK = {"critical": 0, "high": 1, "medium": 2, "low": 3}
_VERDICTS = ("confirmed", "corrected", "not_found")


def select_findings(
    findings: List[Dict],
    clusters: List[Dict],
    max_checks: int,
    severities: Set[str],
) -> List[Dict]:
    """Severity-gated selection plus any finding tied to a disputed cluster."""
    disputed_keys = {
        cluster.get("key") for cluster in clusters
        if cluster.get("disputed_attributes")
    }
    selected = [
        finding for finding in findings
        if str(finding.get("severity") or "").lower() in severities
        or finding.get("cluster_key") in disputed_keys
    ]
    selected.sort(key=lambda f: _SEVERITY_RANK.get(
        str(f.get("severity") or "").lower(), 9))
    return selected[:max_checks]


def _valid_verdict(item: Dict) -> Optional[Dict]:
    if not isinstance(item, dict):
        return None
    verdict = str(item.get("verdict") or "").lower()
    if verdict not in _VERDICTS:
        return None
    return {
        "sheet": item.get("sheet") or "",
        "source_text": item.get("source_text") or "",
        "verdict": verdict,
        "actual_text": item.get("actual_text"),
        "notes": item.get("notes"),
    }


def _status(verdicts: List[Dict]) -> str:
    if not verdicts:
        return "unverified"
    confirmed = sum(1 for v in verdicts if v["verdict"] == "confirmed")
    if confirmed == len(verdicts):
        return "confirmed"
    if confirmed == 0:
        return "refuted"
    return "mixed"


class EvidenceVerifierAgent:
    name = "verify"

    def __init__(self, usage: AgentUsage) -> None:
        self.usage = usage

    def run(self, scope: AgentScope) -> AgentResult:
        try:
            finding = scope.payload["finding"]
            instruction = render(
                VERIFY_USER_INSTRUCTION, {"finding": dumps(finding)}
            )
            parsed = call_json(
                system_prompt=VERIFY_SYSTEM_PROMPT,
                user_text=instruction,
                images_b64=scope.payload.get("images_b64") or [],
                max_tokens=config.VERIFY_MAX_TOKENS,
                model=config.AGENT_VERIFY_MODEL,
                reasoning_effort=config.AGENT_VERIFY_REASONING_EFFORT or None,
                usage_tracker=self.usage,
                usage_stage="agent.verify",
            )
            verdicts = collect_list(parsed, "verdicts", _valid_verdict)
            return AgentResult(scope_id=scope.scope_id, artifacts=[{
                "finding_index": scope.payload["finding_index"],
                "status": _status(verdicts),
                "verdicts": verdicts,
            }])
        except Exception as exc:
            return failure(scope, exc)


def apply_verdicts(
    findings: List[Dict], verify_results: List[AgentResult]
) -> List[Dict]:
    """Annotate findings with verification; return refuted ones to suppress."""
    by_index: Dict[int, Dict] = {}
    for result in verify_results:
        for artifact in result.artifacts:
            by_index[artifact["finding_index"]] = artifact
    suppressed = []
    for index, finding in enumerate(findings):
        artifact = by_index.get(index)
        if not artifact:
            continue
        finding["verification"] = {
            "status": artifact["status"],
            "verdicts": artifact["verdicts"],
        }
        if artifact["status"] == "refuted":
            finding["confidence"] = "low"
            suppressed.append(finding)
    return suppressed
```

**Step 4: Stamp `cluster_key` on constructability findings**

In `backend/agents/construct_agent.py` line 66-67, change:

```python
            for finding in findings:
                finding.update(agent=self.name, scope_id=scope.scope_id)
```

to:

```python
            for finding in findings:
                finding.update(agent=self.name, scope_id=scope.scope_id,
                               cluster_key=cluster.get("key"))
```

**Step 5: Wire wave 5b into `backend/agents/runner.py`**

After `memory.extend("findings", specialist_findings)` (line 167) and before
`gap_findings` / wave 6:

```python
    orchestrator.stage("Agent wave 5b: evidence verification")
    sheet_to_page = {
        sheet.get("sheet_number"): sheet.get("page_number") for sheet in sheets
    }
    verify_targets = select_findings(
        specialist_findings, clusters,
        max_checks=config.AGENT_VERIFY_MAX_CHECKS,
        severities=config.AGENT_VERIFY_SEVERITIES,
    )
    target_indexes = {id(f): i for i, f in enumerate(specialist_findings)}
    verify_scopes = [
        AgentScope(
            scope_id=f"verify:{target_indexes[id(finding)]}",
            payload={
                "finding_index": target_indexes[id(finding)],
                "finding": finding,
                "images_b64": [
                    page_to_b64[sheet_to_page[name]]
                    for name in (finding.get("sheets") or [])
                                [:config.AGENT_CONFLICT_MAX_IMAGES]
                    if sheet_to_page.get(name) in page_to_b64
                ],
            },
        )
        for finding in verify_targets
    ]
    verify_results = orchestrator.run_scopes(
        EvidenceVerifierAgent(usage), verify_scopes,
        config.AGENT_VERIFY_CONCURRENCY,
    )
    suppressed = apply_verdicts(specialist_findings, verify_results)
    if suppressed:
        suppressed_ids = {id(f) for f in suppressed}
        specialist_findings = [
            f for f in specialist_findings if id(f) not in suppressed_ids
        ]
        memory.replace("suppressed", suppressed)
        memory.extend("findings", specialist_findings)  # see note below
```

NOTE for implementer: `memory.extend("findings", ...)` already ran with the
un-suppressed list. Adjust ordering so verification happens BEFORE
`memory.extend("findings", specialist_findings)` — i.e. move the extend to after
wave 5b — so the Brain never sees refuted findings. Keep `gap_findings` logic
unchanged. Also add imports at top of runner.py:

```python
from backend.agents.verifier import (
    EvidenceVerifierAgent, apply_verdicts, select_findings,
)
```

And in the report dicts (both the `require_review` branch ~line 217-225 and the
wave-7 branch ~line 293-300), populate suppressed issues:

```python
        "suppressed_issues": memory.snapshot().get("suppressed") or [],
```

Finally, `verify` results cost shows up as `agent.verify` in
`summary.cost_by_stage` automatically via `usage_stage="agent.verify"`.

**Step 6: Update existing runner tests**

`tests/agents/test_runner_review_gate.py` monkeypatches `BrainAgent` and
`convert_pdf_to_images` but lets waves 1-5 run against... check how LLM calls
are stubbed there (likely `call_json` returns None -> empty artifacts, which is
fine). The new wave must no-op cleanly when `select_findings` returns `[]`
(zero scopes -> `run_scopes` returns `[]` per orchestrator.py:60). Verify by
running the suite; if a runner test now fails because verification selects a
stubbed finding, monkeypatch `select_findings` to `lambda *a, **k: []` in that
test file's `_patch_brain` helper.

**Step 7: Run full suite**

Run: `.venv/bin/python -m pytest tests/ -q`
Expected: all pass

**Step 8: Commit**

```bash
git add backend/agents/verifier.py backend/agents/runner.py backend/agents/construct_agent.py tests/agents/test_verifier.py tests/agents/test_runner_review_gate.py
git commit -m "feat: wave 5b evidence verification - vision fact-check before Brain merge"
```

---

### Task 7 (small, related): reasoning budget for the conflict critic

**Objective:** Fix the wave-4 truncation found in this same job (13/121 calls hit `finish_reason=length` at the 4096 cap with ~3.7k thinking tokens).

**Files:**
- Modify: `backend/agents/conflict_critic.py:55-63`
- Modify: `backend/pipeline/conflict_checker.py:85` (same pattern, classic path)

**Step 1: Apply the extractor's reasoning-knob pattern**

```python
            parsed = call_json(
                system_prompt=CONFLICT_SYSTEM_PROMPT,
                user_text=instruction,
                images_b64=images,
                max_tokens=config.REASON_MAX_TOKENS,
                model=config.AGENT_CONFLICT_MODEL,
                reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
                reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
                usage_tracker=self.usage,
                usage_stage="agent.conflict",
            )
```

(Match the exact kwarg names `SheetExtractorAgent` uses — check
`backend/agents/extractors.py` for whether it passes `None` when the budget is
0, and mirror that guard.)

**Step 2: Run tests**

Run: `.venv/bin/python -m pytest tests/ -q`
Expected: all pass

**Step 3: Commit**

```bash
git add backend/agents/conflict_critic.py backend/pipeline/conflict_checker.py
git commit -m "fix: reasoning budget for conflict critic (wave-4 max_tokens truncation)"
```

---

## Tests / validation

1. `.venv/bin/python -m pytest tests/ -q` — full suite green.
2. **Targeted repro of the original failure:** pull the S401 page image from job
   959e16407573's output dir on sits-docker (or re-render the PDF page), then run
   one `EvidenceVerifierAgent` scope locally against the finding JSON from
   `validated_issues[3]`. Expected: verdict `corrected`,
   `actual_text: "(5) 2x6 STUD PACK"`, status `refuted`.
3. **End-to-end:** rerun the same Cypress TX PDF through the pipeline (local with
   `LLM_CACHE`/`LLM_RAW_DUMP` per the conflict-checker skill). Expected:
   - log shows `Agent wave 5b: evidence verification` with a bounded number of calls;
   - the S401 stud-pack finding is either absent from `validated_issues` and present
     in `suppressed_issues` with verification verdicts, or downgraded to low confidence;
   - `agent.verify` appears in `summary.cost_by_stage`;
   - zero `finish_reason=length` lines in wave 4 (Task 7).
4. Cost check: wave 5b adds at most `AGENT_VERIFY_MAX_CHECKS` (20) vision calls —
   for this job's profile that is well under $1.

## Risks, tradeoffs, open questions

- **Dispute false positives:** cluster members with legitimately different values
  (e.g. two doors in one door cluster) will produce `disputed_attributes`. Mitigation:
  prompts treat disputes as "unverified", not "wrong"; only severity-gated findings
  burn verification calls. Tune later by restricting `find_disputes` to numeric-ish
  values if noise is high.
- **Verifier can also misread.** It is one model checking another with the same eyes.
  Mitigation: verdict requires `actual_text` verbatim evidence for `corrected`, and
  only fully-unconfirmed findings are suppressed (mixed keeps the finding with a note).
- **xref cost:** extra link scopes. Bounded by the >= 2 distinct sheets gate and the
  existing assertion cap; expect a handful of extra scopes per set.
- **Suppression in review mode:** refuted findings land in `suppressed_issues` — the
  review UI/finalizer must tolerate that list being non-empty (it is currently always
  `[]` in agent mode). Open question: surface suppressed items in the human review
  queue as informational, or keep them report-only?
- **Open question:** should wave-4 conflict findings (which already saw images) also be
  verification-eligible? Plan says no (they had the pixels); revisit if critics show
  the same misread pattern.