diff --git a/.gitea/workflows/docker-release.yml b/.gitea/workflows/docker-release.yml index 522493e..73eca50 100644 --- a/.gitea/workflows/docker-release.yml +++ b/.gitea/workflows/docker-release.yml @@ -2,7 +2,7 @@ name: Docker Release on: push: - branches: [main] + branches: [main, agent-mode] tags: ["v*"] env: @@ -25,12 +25,19 @@ jobs: { echo "tags<> "$GITHUB_OUTPUT" @@ -56,6 +63,8 @@ jobs: context: . push: true tags: ${{ steps.meta.outputs.tags }} + build-args: | + APP_BUILD=sha-${{ steps.meta.outputs.short_sha }} release: needs: build-and-push diff --git a/.hermes/plans/2026-08-10_100946-evidence-verification-cross-sheet-correlation.md b/.hermes/plans/2026-08-10_100946-evidence-verification-cross-sheet-correlation.md new file mode 100644 index 0000000..f5d6e1e --- /dev/null +++ b/.hermes/plans/2026-08-10_100946-evidence-verification-cross-sheet-correlation.md @@ -0,0 +1,936 @@ +# Evidence Verification + Cross-Sheet Correlation Implementation Plan + +> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task. + +**Goal:** Stop vision-extraction misreads (e.g. "(2) 2x6 STUD PACK" vs the actual "(5) 2x6") from becoming confident downstream findings, and correlate the same physical element across sheets (S101/S205/S401) so no stage reasons from one sheet's text in isolation. + +**Architecture:** Three independently shippable phases on the agent pipeline (`backend/agents/runner.py`): +1. **Disputed-value detection** — deterministic post-link pass that flags contradictory extracted values inside a cluster and surfaces them to the critic/specialist prompts. +2. **Cross-sheet xref linking** — linker gains detail-reference/tag buckets that join assertions across levels (today `(level, family)` bucketing splits S101/S205/S401 apart). +3. **Evidence verification wave (5b)** — a bounded vision fact-check agent re-reads the cited sheet images for high-severity / disputed findings before the Brain merge, annotates or suppresses findings built on phantom text. + +**Tech Stack:** Python 3.14, pytest (`tests/`), existing `call_json` LLM wrapper (supports `images_b64`, `reasoning_effort`, `reasoning_max_tokens`). + +--- + +## Current context / root cause (from job 959e16407573) + +Finding `validated_issues[3]` ("FRONT PERSPECTIVE detail on Sheet S401", HSS16x4 on +"(2) 2x6 STUD PACK", severity critical) is a **false positive built on a wave-1 vision +misread**. The sheet actually shows a (5) 2x6 stud pack (matching S205/S101). Chain of failure: + +1. Wave 1 (`SheetExtractorAgent`) froze the misread into text. From then on it is "ground truth". +2. Wave 3 linker (`backend/agents/linker.py:33` `build_link_scopes`) buckets by + `(level, family)`. S101 (foundation), S205 (details), S401 (sections) get different + `level` values, so assertions about the same front-wall header never share a link scope + or cluster. No cross-sheet corroboration happened. +3. Wave 5 `ConstructabilityAgent` (`backend/agents/construct_agent.py:53`) calls + `call_json` **with no images** — in this job 120/120 constructability calls were `+0img`. + It reasoned arithmetically from the misread text ("2 x 1.5in = 3in < 4in -> unbuildable"). + It even held the "(5) 2x6 STUD PACK" assertion in the same scope but labeled it + "Ambiguous column size specification" instead of arbitrating. +4. Nothing between wave 5 and the report ever looks at a sheet image again. Only the wave-4 + conflict critic receives images, and only for its own cluster's pages. + +Also confirmed in this log (separate known bug, fixed in Task 7 while we're here): wave-4 +conflict critic truncates on Gemini thinking tokens because `conflict_critic.py:59` passes +`max_tokens=config.REASON_MAX_TOKENS` (4096) with no reasoning budget — 13/121 calls hit +`finish_reason=length`. + +## Assumptions + +- Assertions carry `id`, `attribute`, `value`, `source_text`, `location_key` + (`room`/`grid`/`detail_reference`/`tag`/`level`) — see `linker._payload` and + `_serialize.slim_assertion`. +- Extractor assertions already carry a `confidence` field (per test fixtures). +- `validate_issue` in `backend/pipeline/_stage.py` guarantees each finding an `issue_id`. +- Test convention: `unittest.mock.patch("backend.agents..call_json", ...)` — + see `tests/agents/test_sheet_extractor_fallback.py`. Run tests with + `.venv/bin/python -m pytest tests/ -x -q`. +- `AgentResult.error` defaults to `""` (not None) in assertions. + +--- + +## Phase 1 — Disputed-value detection + prompt hardening + +### Task 1: `find_disputes` pure function (TDD) + +**Objective:** Detect "same attribute, different values" inside one cluster's assertions. + +**Files:** +- Create: `backend/agents/disputes.py` +- Test: `tests/agents/test_disputes.py` + +**Step 1: Write failing test** + +```python +# tests/agents/test_disputes.py +from backend.agents.disputes import annotate_clusters, find_disputes + + +def _a(id_, attribute, value): + return {"id": id_, "attribute": attribute, "value": value, + "source_text": value} + + +def test_find_disputes_flags_same_attribute_different_values(): + assertions = [ + _a("a1", "stud_pack_size", "(2) 2x6 STUD PACK"), + _a("a2", "stud_pack_size", "(5) 2x6 STUD PACK"), + _a("a3", "beam_size", "HSS16X4X5/8"), + ] + disputes = find_disputes(assertions) + assert len(disputes) == 1 + assert disputes[0]["attribute"] == "stud_pack_size" + assert disputes[0]["values"] == ["(2) 2x6 STUD PACK", "(5) 2x6 STUD PACK"] + assert disputes[0]["assertion_ids"] == ["a1", "a2"] + + +def test_find_disputes_ignores_agreeing_values_and_blanks(): + assertions = [ + _a("a1", "beam_size", "HSS16X4X5/8"), + _a("a2", "beam_size", " hss16x4x5/8 "), # same after normalize + _a("a3", "", "orphan"), # no attribute -> skipped + _a("a4", "beam_size", ""), # no value -> skipped + ] + assert find_disputes(assertions) == [] + + +def test_annotate_clusters_writes_disputed_attributes(): + clusters = [ + {"key": "c1", "assertions": [ + _a("a1", "stud_pack_size", "(2) 2x6"), + _a("a2", "stud_pack_size", "(5) 2x6"), + ]}, + {"key": "c2", "assertions": [_a("a3", "x", "1"), _a("a4", "x", "1")]}, + ] + assert annotate_clusters(clusters) == 1 + assert clusters[0]["disputed_attributes"][0]["attribute"] == "stud_pack_size" + assert "disputed_attributes" not in clusters[1] +``` + +**Step 2: Run test to verify failure** + +Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py -v` +Expected: FAIL — `ModuleNotFoundError: backend.agents.disputes` + +**Step 3: Implement** + +```python +# backend/agents/disputes.py +"""Deterministic detection of contradictory extracted values within a cluster. + +Extraction is a vision pass: quantities and sizes can be misread ("(2) 2x6" vs +"(5) 2x6"). Cluster members are supposed to describe the same real-world +element, so two members asserting different values for the same attribute are +a probable misread. Flag these so downstream text-only stages treat the value +as unverified instead of reasoning from one reading. +""" + +import re +from typing import Dict, List + + +def _norm(value) -> str: + return re.sub(r"\s+", " ", str(value or "").strip().lower()) + + +def find_disputes(assertions: List[Dict]) -> List[Dict]: + """Same attribute with >= 2 distinct normalized values = disputed.""" + groups: Dict[str, Dict[str, set]] = {} + for assertion in assertions: + attribute = _norm(assertion.get("attribute")) + value = _norm(assertion.get("value")) + if not attribute or not value: + continue + groups.setdefault(attribute, {}).setdefault(value, set()).add( + assertion.get("id") + ) + disputes = [] + for attribute, values in sorted(groups.items()): + if len(values) < 2: + continue + disputes.append({ + "attribute": attribute, + "values": sorted(values), + "assertion_ids": sorted( + aid for ids in values.values() for aid in ids if aid + ), + }) + return disputes + + +def annotate_clusters(clusters: List[Dict]) -> int: + """Attach disputed_attributes to each cluster that has any. Returns count.""" + annotated = 0 + for cluster in clusters: + disputes = find_disputes(cluster.get("assertions") or []) + if disputes: + cluster["disputed_attributes"] = disputes + annotated += 1 + return annotated +``` + +**Step 4: Run test to verify pass** + +Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py -v` +Expected: 3 passed + +**Step 5: Commit** + +```bash +git add backend/agents/disputes.py tests/agents/test_disputes.py +git commit -m "feat: deterministic disputed-value detection for cluster assertions" +``` + +--- + +### Task 2: Wire `annotate_clusters` into the runner + serialization + +**Objective:** Disputes must be visible to the wave-4 critic and wave-5 constructability prompts. + +**Files:** +- Modify: `backend/agents/runner.py` (after `memory.replace("clusters", clusters)`, ~line 116) +- Modify: `backend/pipeline/_serialize.py` (`slim_clusters`, line 46) +- Test: `tests/agents/test_disputes.py` (append) + +**Step 1: Write failing test** + +```python +def test_slim_clusters_preserves_disputed_attributes(): + from backend.pipeline._serialize import slim_clusters + cluster = {"key": "c1", "assertions": [], + "disputed_attributes": [{"attribute": "a", "values": ["1", "2"], + "assertion_ids": ["x", "y"]}]} + slim = slim_clusters([cluster])[0] + assert slim["disputed_attributes"][0]["values"] == ["1", "2"] +``` + +**Step 2: Run test to verify failure** + +Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py::test_slim_clusters_preserves_disputed_attributes -v` +Expected: FAIL — `KeyError: 'disputed_attributes'` + +**Step 3: Implement** + +In `backend/pipeline/_serialize.py` `slim_clusters`, add the key: + +```python +def slim_clusters(clusters: List[Dict]) -> List[Dict]: + return [ + { + "key": c.get("key"), + "location": c.get("location"), + "disciplines": c.get("disciplines"), + "kind": c.get("kind"), + **({"disputed_attributes": c["disputed_attributes"]} + if c.get("disputed_attributes") else {}), + "assertions": [slim_assertion(a) for a in c.get("assertions", [])], + } + for c in clusters + ] +``` + +In `backend/agents/runner.py`, right after `clusters = [...]` / `object_graph = build_object_graph(clusters)` (before `memory.replace("clusters", clusters)`): + +```python +from backend.agents.disputes import annotate_clusters +... + object_graph = build_object_graph(clusters) + disputed_count = annotate_clusters(clusters) + if disputed_count: + orchestrator.log( + f"[Link] {disputed_count} clusters carry disputed extracted values" + ) +``` + +(Check `Orchestrator` for the actual log method name — `orchestrator.stage(...)` exists; +if no `.log`, use the module's existing logging/print convention. Adjust to match.) + +**Step 4: Run tests** + +Run: `.venv/bin/python -m pytest tests/agents/ -v` +Expected: all pass (including existing `test_runner_review_gate.py`) + +**Step 5: Commit** + +```bash +git add backend/agents/runner.py backend/pipeline/_serialize.py tests/agents/test_disputes.py +git commit -m "feat: surface disputed extracted values to critic and specialist prompts" +``` + +--- + +### Task 3: Prompt hardening — extracted text is fallible + +**Objective:** Tell text-only specialists how to handle disputed/unverified values so they stop asserting buildability conclusions from a single (possibly misread) number. + +**Files:** +- Modify: `backend/prompts.py` `CONSTRUCTABILITY_SYSTEM_PROMPT` (line 533) and `CONSTRUCTABILITY_USER_INSTRUCTION` (line 551) + +**Step 1: Edit prompts** + +Append to `CONSTRUCTABILITY_SYSTEM_PROMPT` Rules list (after line 547, before "Use plain ASCII"): + +``` +- Assertions are machine-extracted from sheet images and may contain misread values, + especially quantities and member sizes (e.g. "(2) 2x6" vs "(5) 2x6"). +- When the cluster lists disputed_attributes, or two evidence items disagree on a + numeric value, do NOT assert a buildability conclusion from one reading. Report the + ambiguity itself (category "detail_gap", confidence "low") and state that the value + needs verification against the sheet. +``` + +Append to `CONSTRUCTABILITY_USER_INSTRUCTION` after the `Cross-discipline conflicts already found: {conflicts}` line: + +``` +Disputed extracted values in this cluster (possible vision misreads - treat as unverified): {disputes} +``` + +**Step 2: Wire the `{disputes}` placeholder in `construct_agent.py`** + +In `backend/agents/construct_agent.py` `run()`, extend the `substitutions` dict: + +```python + substitutions = { + "assertions": dumps(cluster["assertions"]), + "clusters": dumps(slim_clusters([cluster])), + "conflicts": dumps(scope.payload.get("conflicts") or []), + "disputes": dumps(cluster.get("disputed_attributes") or []), + } +``` + +**Step 3: Run full test suite (prompt edits can break runner tests that snapshot prompts)** + +Run: `.venv/bin/python -m pytest tests/ -q` +Expected: all pass + +**Step 4: Commit** + +```bash +git add backend/prompts.py backend/agents/construct_agent.py +git commit -m "feat: constructability prompt treats disputed extracted values as unverified" +``` + +--- + +## Phase 2 — Cross-sheet xref linking + +### Task 4: detail-reference / tag xref buckets in the linker (TDD) + +**Objective:** Assertions sharing a `detail_reference` or a member `tag` get linked across levels, so S101/S205/S401 details of the same physical element land in one scope. + +**Files:** +- Modify: `backend/agents/linker.py` (`build_link_scopes`, line 33) +- Test: `tests/agents/test_linker_xref.py` + +**Step 1: Write failing test** + +```python +# tests/agents/test_linker_xref.py +from backend.agents.base import AgentScope +from backend.agents.linker import build_link_scopes + + +def _sheet(number, page, level, assertions): + return {"sheet_number": number, "page_number": page, + "discipline": "Structural", "level": level, + "assertions": assertions} + + +def _assertion(id_, ref=None, tag=None, level=None): + return {"id": id_, "attribute": "stud_pack_size", "value": "(5) 2x6", + "source_text": "(5) 2x6 STUD PACK", + "location_key": {"detail_reference": ref, "tag": tag, + "level": level}} + + +def test_xref_scope_joins_same_detail_reference_across_levels(): + sheets = [ + _sheet("S101", 10, "foundation", [_assertion("a1", ref="A/S205")]), + _sheet("S205", 20, "roof", [_assertion("a2", ref="A/S205")]), + _sheet("S401", 30, "roof", [_assertion("a3", ref="A/S205")]), + ] + scopes = build_link_scopes(sheets) + xref = [s for s in scopes if s.scope_id.startswith("xref:")] + assert xref, "expected a cross-level detail-reference scope" + ids = {a["id"] for s in xref for a in s.payload["assertions"]} + assert ids == {"a1", "a2", "a3"} + + +def test_xref_scope_requires_two_distinct_sheets(): + sheets = [ + _sheet("S401", 30, "roof", [_assertion("a1", ref="A/S205"), + _assertion("a2", ref="A/S205")]), + ] + scopes = build_link_scopes(sheets) + assert not [s for s in scopes if s.scope_id.startswith("xref:")] + + +def test_xref_scope_joins_shared_member_tag(): + sheets = [ + _sheet("S102", 5, "roof", [_assertion("a1", tag="HSS16X4X5/8")]), + _sheet("S401", 30, "unknown", [_assertion("a2", tag="HSS16X4X5/8")]), + ] + scopes = build_link_scopes(sheets) + xref = [s for s in scopes if s.scope_id.startswith("xref:")] + assert xref +``` + +**Step 2: Run test to verify failure** + +Run: `.venv/bin/python -m pytest tests/agents/test_linker_xref.py -v` +Expected: FAIL — no `xref:` scopes produced + +**Step 3: Implement** + +Rewrite `build_link_scopes` in `backend/agents/linker.py` (keep the existing +`(level, family)` bucketing, add the xref pass): + +```python +def _xref_keys(assertion: Dict) -> List[str]: + """Cross-level join keys: detail references and member tags.""" + location = assertion.get("location_key") or {} + keys = [] + ref = re.sub(r"\s+", "", str(location.get("detail_reference") or "")).upper() + if ref: + keys.append(f"detail:{ref}") + tag = re.sub(r"\s+", "", str(location.get("tag") or "")).upper() + if re.match(r"^[A-Z]{2,}\d", tag): # member marks: HSS16X4X5/8, W12X26, ... + keys.append(f"tag:{tag}") + return keys + + +def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]: + """Partition facts by level and object/tag family, then enforce a hard cap. + + A second pass joins assertions that share a detail_reference or member tag + ACROSS levels, so plan/detail/section sheets describing the same physical + element are linked together even though their levels differ. + """ + buckets: Dict[Tuple[str, str], List[Dict]] = defaultdict(list) + xref: Dict[str, List[Dict]] = defaultdict(list) + for sheet in sheets: + for assertion in sheet.get("assertions", []): + enriched = { + **assertion, + "discipline": sheet.get("discipline") or "Unknown", + "sheet_number": sheet.get("sheet_number"), + "page_number": sheet.get("page_number"), + } + level = str((assertion.get("location_key") or {}).get("level") + or sheet.get("level") or "unknown").lower() + buckets[(level, _family(assertion))].append(enriched) + for key in _xref_keys(assertion): + xref[key].append(enriched) + + scopes: List[AgentScope] = [] + cap = max(2, config.AGENT_LINK_MAX_ASSERTIONS) + for (level, family), assertions in sorted(buckets.items()): + for offset in range(0, len(assertions), cap): + chunk = assertions[offset:offset + cap] + if len(chunk) < 2: + continue + scopes.append(AgentScope( + scope_id=f"{level}:{family}:{offset // cap + 1}", + payload={"assertions": chunk, "level": level, "family": family}, + )) + for key, assertions in sorted(xref.items()): + sheets_present = {a.get("sheet_number") for a in assertions} + if len(assertions) < 2 or len(sheets_present) < 2: + continue + scopes.append(AgentScope( + scope_id=f"xref:{key}", + payload={"assertions": assertions[:cap], + "level": "xref", "family": key}, + )) + return scopes +``` + +**Step 4: Run tests** + +Run: `.venv/bin/python -m pytest tests/agents/test_linker_xref.py tests/agents/ -v` +Expected: all pass (watch existing runner tests for scope-count coupling) + +**Step 5: Commit** + +```bash +git add backend/agents/linker.py tests/agents/test_linker_xref.py +git commit -m "feat: cross-level xref link scopes via detail_reference and member tag" +``` + +--- + +## Phase 3 — Evidence verification wave (5b) + +### Task 5: Config knobs + verify prompts + +**Objective:** Add the tuning surface and prompts for the vision fact-check agent. + +**Files:** +- Modify: `backend/config.py` (near line 44, with the other AGENT_* knobs) +- Modify: `backend/prompts.py` (append near the CONFLICT prompts, ~line 460) +- Modify: `backend/.env.example` + +**Step 1: Add config knobs to `backend/config.py`** + +```python +AGENT_VERIFY_MODEL = os.getenv("AGENT_VERIFY_MODEL", "") or MODEL +AGENT_VERIFY_CONCURRENCY = int(os.getenv("AGENT_VERIFY_CONCURRENCY", "4")) +AGENT_VERIFY_MAX_CHECKS = int(os.getenv("AGENT_VERIFY_MAX_CHECKS", "20")) +AGENT_VERIFY_SEVERITIES = { + s.strip().lower() + for s in os.getenv("AGENT_VERIFY_SEVERITIES", "critical,high").split(",") + if s.strip() +} +AGENT_VERIFY_REASONING_EFFORT = os.getenv("AGENT_VERIFY_REASONING_EFFORT", "low").strip() +VERIFY_MAX_TOKENS = int(os.getenv("VERIFY_MAX_TOKENS", "8192")) +``` + +Append to `backend/.env.example`: + +``` +# Wave 5b evidence verification (vision fact-check of cited sheet text) +AGENT_VERIFY_MAX_CHECKS=20 +AGENT_VERIFY_SEVERITIES=critical,high +AGENT_VERIFY_REASONING_EFFORT=low +VERIFY_MAX_TOKENS=8192 +``` + +**Step 2: Add prompts to `backend/prompts.py`** + +```python +VERIFY_SYSTEM_PROMPT = """You are a meticulous construction document checker verifying machine-extracted evidence against the actual drawing sheet images. +For each evidence item you are given the sheet it was extracted from and the verbatim text the extractor claims appears there. +Judge each item against the images: +- confirmed: the text (or an obvious equivalent) appears on the cited sheet and means what the finding claims. +- corrected: the sheet shows a DIFFERENT value than the extracted text. Give the actual verbatim text. +- not_found: nothing like the extracted text appears on the cited sheet. +Be strict about numbers, quantities, and member sizes: "(2) 2x6" and "(5) 2x6" are different values. HSS16x4 and HSS16x16 are different values. +Use plain ASCII only. +Respond only with valid JSON.""" + +VERIFY_USER_INSTRUCTION = """Verify this finding's evidence against the attached sheet images. +Respond ONLY with a valid JSON object - no markdown fences, no explanation: +{ "verdicts": [ { "sheet": "string", "source_text": "the evidence text judged", "verdict": "confirmed | corrected | not_found", "actual_text": "verbatim sheet text when corrected, else null", "notes": "string or null" } ] } +Finding: {finding}""" +``` + +**Step 3: Sanity check** + +Run: `.venv/bin/python -c "from backend import config, prompts; print(config.AGENT_VERIFY_MAX_CHECKS, config.AGENT_VERIFY_SEVERITIES); print(prompts.VERIFY_SYSTEM_PROMPT[:40])"` +Expected: `20 {'critical', 'high'}` and prompt text + +**Step 4: Commit** + +```bash +git add backend/config.py backend/prompts.py backend/.env.example +git commit -m "feat: config knobs and prompts for evidence verification wave" +``` + +--- + +### Task 6: `EvidenceVerifierAgent` + runner wave 5b (TDD) + +**Objective:** Re-read cited sheet images for selected findings; annotate verified findings, suppress refuted ones before the Brain merge. + +**Files:** +- Create: `backend/agents/verifier.py` +- Modify: `backend/agents/runner.py` (new wave between wave 5 and wave 6, ~line 167) +- Modify: `backend/agents/construct_agent.py` line 67 (stamp `cluster_key` for dispute-based selection) +- Test: `tests/agents/test_verifier.py` + +**Step 1: Write failing test** + +```python +# tests/agents/test_verifier.py +from unittest.mock import patch + +from backend.agents.base import AgentScope, AgentUsage +from backend.agents.verifier import ( + EvidenceVerifierAgent, apply_verdicts, select_findings, +) + + +def _finding(sev="critical", issue_id="i1", sheets=("S401",), cluster_key=None): + f = {"issue_id": issue_id, "severity": sev, "confidence": "high", + "source_stage": "constructability", "sheets": list(sheets), + "description": "HSS16x4 on (2) 2x6 STUD PACK is unbuildable", + "evidence": [{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK", + "asserted_value": "3-inch width"}]} + if cluster_key: + f["cluster_key"] = cluster_key + return f + + +def test_select_findings_by_severity_and_dispute(): + findings = [_finding("critical"), _finding("low", "i2"), + _finding("medium", "i3", cluster_key="c9")] + clusters = [{"key": "c9", "disputed_attributes": [{"attribute": "a"}]}] + selected = select_findings(findings, clusters, max_checks=20, + severities={"critical", "high"}) + assert [f["issue_id"] for f in selected] == ["i1", "i3"] + + +def test_select_findings_respects_cap(): + findings = [_finding("critical", f"i{n}") for n in range(30)] + selected = select_findings(findings, [], max_checks=5, + severities={"critical"}) + assert len(selected) == 5 + + +def test_run_attaches_verdicts_and_marks_refuted(): + agent = EvidenceVerifierAgent(usage=AgentUsage()) + scope = AgentScope(scope_id="verify:0", payload={ + "finding_index": 0, + "finding": _finding(), + "images_b64": ["QUJD"], + }) + verdicts = {"verdicts": [ + {"sheet": "S401", "source_text": "(2) 2x6 STUD PACK", + "verdict": "corrected", "actual_text": "(5) 2x6 STUD PACK", + "notes": "callout reads (5)"}, + ]} + with patch("backend.agents.verifier.call_json", return_value=verdicts): + result = agent.run(scope) + assert not result.error + artifact = result.artifacts[0] + assert artifact["finding_index"] == 0 + assert artifact["status"] == "refuted" # no evidence confirmed + assert artifact["verdicts"][0]["actual_text"] == "(5) 2x6 STUD PACK" + + +def test_apply_verdicts_annotates_and_suppresses(): + findings = [_finding("critical", "i1"), _finding("high", "i2")] + from backend.agents.base import AgentResult + results = [AgentResult(scope_id="verify:0", artifacts=[ + {"finding_index": 0, "status": "refuted", "verdicts": []}, + {"finding_index": 1, "status": "confirmed", "verdicts": []}, + ])] + suppressed = apply_verdicts(findings, results) + assert suppressed == [findings[0]] + assert findings[0]["verification"]["status"] == "refuted" + assert findings[1]["verification"]["status"] == "confirmed" +``` + +**Step 2: Run test to verify failure** + +Run: `.venv/bin/python -m pytest tests/agents/test_verifier.py -v` +Expected: FAIL — `ModuleNotFoundError: backend.agents.verifier` + +**Step 3: Implement `backend/agents/verifier.py`** + +```python +"""Wave 5b: vision fact-check of extracted evidence against cited sheet images. + +Downstream specialists are text-only; a wave-1 vision misread ("(2) 2x6" vs +"(5) 2x6") otherwise becomes immutable ground truth. For high-severity or +dispute-linked findings, re-read the cited sheets and adjudicate each evidence +item: confirmed / corrected / not_found. Findings whose evidence is entirely +unconfirmed are suppressed before the Brain merge. +""" + +from typing import Dict, List, Optional, Set + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline._serialize import dumps +from backend.pipeline._stage import collect_list, render +from backend.prompts import VERIFY_SYSTEM_PROMPT, VERIFY_USER_INSTRUCTION + +_SEVERITY_RANK = {"critical": 0, "high": 1, "medium": 2, "low": 3} +_VERDICTS = ("confirmed", "corrected", "not_found") + + +def select_findings( + findings: List[Dict], + clusters: List[Dict], + max_checks: int, + severities: Set[str], +) -> List[Dict]: + """Severity-gated selection plus any finding tied to a disputed cluster.""" + disputed_keys = { + cluster.get("key") for cluster in clusters + if cluster.get("disputed_attributes") + } + selected = [ + finding for finding in findings + if str(finding.get("severity") or "").lower() in severities + or finding.get("cluster_key") in disputed_keys + ] + selected.sort(key=lambda f: _SEVERITY_RANK.get( + str(f.get("severity") or "").lower(), 9)) + return selected[:max_checks] + + +def _valid_verdict(item: Dict) -> Optional[Dict]: + if not isinstance(item, dict): + return None + verdict = str(item.get("verdict") or "").lower() + if verdict not in _VERDICTS: + return None + return { + "sheet": item.get("sheet") or "", + "source_text": item.get("source_text") or "", + "verdict": verdict, + "actual_text": item.get("actual_text"), + "notes": item.get("notes"), + } + + +def _status(verdicts: List[Dict]) -> str: + if not verdicts: + return "unverified" + confirmed = sum(1 for v in verdicts if v["verdict"] == "confirmed") + if confirmed == len(verdicts): + return "confirmed" + if confirmed == 0: + return "refuted" + return "mixed" + + +class EvidenceVerifierAgent: + name = "verify" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + finding = scope.payload["finding"] + instruction = render( + VERIFY_USER_INSTRUCTION, {"finding": dumps(finding)} + ) + parsed = call_json( + system_prompt=VERIFY_SYSTEM_PROMPT, + user_text=instruction, + images_b64=scope.payload.get("images_b64") or [], + max_tokens=config.VERIFY_MAX_TOKENS, + model=config.AGENT_VERIFY_MODEL, + reasoning_effort=config.AGENT_VERIFY_REASONING_EFFORT or None, + usage_tracker=self.usage, + usage_stage="agent.verify", + ) + verdicts = collect_list(parsed, "verdicts", _valid_verdict) + return AgentResult(scope_id=scope.scope_id, artifacts=[{ + "finding_index": scope.payload["finding_index"], + "status": _status(verdicts), + "verdicts": verdicts, + }]) + except Exception as exc: + return failure(scope, exc) + + +def apply_verdicts( + findings: List[Dict], verify_results: List[AgentResult] +) -> List[Dict]: + """Annotate findings with verification; return refuted ones to suppress.""" + by_index: Dict[int, Dict] = {} + for result in verify_results: + for artifact in result.artifacts: + by_index[artifact["finding_index"]] = artifact + suppressed = [] + for index, finding in enumerate(findings): + artifact = by_index.get(index) + if not artifact: + continue + finding["verification"] = { + "status": artifact["status"], + "verdicts": artifact["verdicts"], + } + if artifact["status"] == "refuted": + finding["confidence"] = "low" + suppressed.append(finding) + return suppressed +``` + +**Step 4: Stamp `cluster_key` on constructability findings** + +In `backend/agents/construct_agent.py` line 66-67, change: + +```python + for finding in findings: + finding.update(agent=self.name, scope_id=scope.scope_id) +``` + +to: + +```python + for finding in findings: + finding.update(agent=self.name, scope_id=scope.scope_id, + cluster_key=cluster.get("key")) +``` + +**Step 5: Wire wave 5b into `backend/agents/runner.py`** + +After `memory.extend("findings", specialist_findings)` (line 167) and before +`gap_findings` / wave 6: + +```python + orchestrator.stage("Agent wave 5b: evidence verification") + sheet_to_page = { + sheet.get("sheet_number"): sheet.get("page_number") for sheet in sheets + } + verify_targets = select_findings( + specialist_findings, clusters, + max_checks=config.AGENT_VERIFY_MAX_CHECKS, + severities=config.AGENT_VERIFY_SEVERITIES, + ) + target_indexes = {id(f): i for i, f in enumerate(specialist_findings)} + verify_scopes = [ + AgentScope( + scope_id=f"verify:{target_indexes[id(finding)]}", + payload={ + "finding_index": target_indexes[id(finding)], + "finding": finding, + "images_b64": [ + page_to_b64[sheet_to_page[name]] + for name in (finding.get("sheets") or []) + [:config.AGENT_CONFLICT_MAX_IMAGES] + if sheet_to_page.get(name) in page_to_b64 + ], + }, + ) + for finding in verify_targets + ] + verify_results = orchestrator.run_scopes( + EvidenceVerifierAgent(usage), verify_scopes, + config.AGENT_VERIFY_CONCURRENCY, + ) + suppressed = apply_verdicts(specialist_findings, verify_results) + if suppressed: + suppressed_ids = {id(f) for f in suppressed} + specialist_findings = [ + f for f in specialist_findings if id(f) not in suppressed_ids + ] + memory.replace("suppressed", suppressed) + memory.extend("findings", specialist_findings) # see note below +``` + +NOTE for implementer: `memory.extend("findings", ...)` already ran with the +un-suppressed list. Adjust ordering so verification happens BEFORE +`memory.extend("findings", specialist_findings)` — i.e. move the extend to after +wave 5b — so the Brain never sees refuted findings. Keep `gap_findings` logic +unchanged. Also add imports at top of runner.py: + +```python +from backend.agents.verifier import ( + EvidenceVerifierAgent, apply_verdicts, select_findings, +) +``` + +And in the report dicts (both the `require_review` branch ~line 217-225 and the +wave-7 branch ~line 293-300), populate suppressed issues: + +```python + "suppressed_issues": memory.snapshot().get("suppressed") or [], +``` + +Finally, `verify` results cost shows up as `agent.verify` in +`summary.cost_by_stage` automatically via `usage_stage="agent.verify"`. + +**Step 6: Update existing runner tests** + +`tests/agents/test_runner_review_gate.py` monkeypatches `BrainAgent` and +`convert_pdf_to_images` but lets waves 1-5 run against... check how LLM calls +are stubbed there (likely `call_json` returns None -> empty artifacts, which is +fine). The new wave must no-op cleanly when `select_findings` returns `[]` +(zero scopes -> `run_scopes` returns `[]` per orchestrator.py:60). Verify by +running the suite; if a runner test now fails because verification selects a +stubbed finding, monkeypatch `select_findings` to `lambda *a, **k: []` in that +test file's `_patch_brain` helper. + +**Step 7: Run full suite** + +Run: `.venv/bin/python -m pytest tests/ -q` +Expected: all pass + +**Step 8: Commit** + +```bash +git add backend/agents/verifier.py backend/agents/runner.py backend/agents/construct_agent.py tests/agents/test_verifier.py tests/agents/test_runner_review_gate.py +git commit -m "feat: wave 5b evidence verification - vision fact-check before Brain merge" +``` + +--- + +### Task 7 (small, related): reasoning budget for the conflict critic + +**Objective:** Fix the wave-4 truncation found in this same job (13/121 calls hit `finish_reason=length` at the 4096 cap with ~3.7k thinking tokens). + +**Files:** +- Modify: `backend/agents/conflict_critic.py:55-63` +- Modify: `backend/pipeline/conflict_checker.py:85` (same pattern, classic path) + +**Step 1: Apply the extractor's reasoning-knob pattern** + +```python + parsed = call_json( + system_prompt=CONFLICT_SYSTEM_PROMPT, + user_text=instruction, + images_b64=images, + max_tokens=config.REASON_MAX_TOKENS, + model=config.AGENT_CONFLICT_MODEL, + reasoning_effort=config.EXTRACT_REASONING_EFFORT or None, + reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None, + usage_tracker=self.usage, + usage_stage="agent.conflict", + ) +``` + +(Match the exact kwarg names `SheetExtractorAgent` uses — check +`backend/agents/extractors.py` for whether it passes `None` when the budget is +0, and mirror that guard.) + +**Step 2: Run tests** + +Run: `.venv/bin/python -m pytest tests/ -q` +Expected: all pass + +**Step 3: Commit** + +```bash +git add backend/agents/conflict_critic.py backend/pipeline/conflict_checker.py +git commit -m "fix: reasoning budget for conflict critic (wave-4 max_tokens truncation)" +``` + +--- + +## Tests / validation + +1. `.venv/bin/python -m pytest tests/ -q` — full suite green. +2. **Targeted repro of the original failure:** pull the S401 page image from job + 959e16407573's output dir on sits-docker (or re-render the PDF page), then run + one `EvidenceVerifierAgent` scope locally against the finding JSON from + `validated_issues[3]`. Expected: verdict `corrected`, + `actual_text: "(5) 2x6 STUD PACK"`, status `refuted`. +3. **End-to-end:** rerun the same Cypress TX PDF through the pipeline (local with + `LLM_CACHE`/`LLM_RAW_DUMP` per the conflict-checker skill). Expected: + - log shows `Agent wave 5b: evidence verification` with a bounded number of calls; + - the S401 stud-pack finding is either absent from `validated_issues` and present + in `suppressed_issues` with verification verdicts, or downgraded to low confidence; + - `agent.verify` appears in `summary.cost_by_stage`; + - zero `finish_reason=length` lines in wave 4 (Task 7). +4. Cost check: wave 5b adds at most `AGENT_VERIFY_MAX_CHECKS` (20) vision calls — + for this job's profile that is well under $1. + +## Risks, tradeoffs, open questions + +- **Dispute false positives:** cluster members with legitimately different values + (e.g. two doors in one door cluster) will produce `disputed_attributes`. Mitigation: + prompts treat disputes as "unverified", not "wrong"; only severity-gated findings + burn verification calls. Tune later by restricting `find_disputes` to numeric-ish + values if noise is high. +- **Verifier can also misread.** It is one model checking another with the same eyes. + Mitigation: verdict requires `actual_text` verbatim evidence for `corrected`, and + only fully-unconfirmed findings are suppressed (mixed keeps the finding with a note). +- **xref cost:** extra link scopes. Bounded by the >= 2 distinct sheets gate and the + existing assertion cap; expect a handful of extra scopes per set. +- **Suppression in review mode:** refuted findings land in `suppressed_issues` — the + review UI/finalizer must tolerate that list being non-empty (it is currently always + `[]` in agent mode). Open question: surface suppressed items in the human review + queue as informational, or keep them report-only? +- **Open question:** should wave-4 conflict findings (which already saw images) also be + verification-eligible? Plan says no (they had the pixels); revisit if critics show + the same misread pattern. diff --git a/Dockerfile b/Dockerfile index e87d122..37ec1e5 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,5 +1,8 @@ FROM python:3.12-slim-bookworm +LABEL org.opencontainers.image.title="Conflict Checker" \ + org.opencontainers.image.description="Classic and experimental scoped Agent pipelines" + RUN apt-get update \ && apt-get install -y --no-install-recommends poppler-utils \ && rm -rf /var/lib/apt/lists/* @@ -16,6 +19,10 @@ COPY cli cli RUN mkdir -p backend/uploads backend/outputs backend/.llm_cache ENV PYTHONUNBUFFERED=1 +# Build identifier baked in by CI (sha-, matches the image tag); +# defaults to "dev" for local builds. Surfaced in /health and the site header. +ARG APP_BUILD=dev +ENV APP_BUILD=${APP_BUILD} EXPOSE 8099 HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ diff --git a/NOTES.md b/NOTES.md index 329c19c..0e453e3 100644 --- a/NOTES.md +++ b/NOTES.md @@ -2,7 +2,7 @@ Orientation for a new coding session. Setup and Docker details live in [README.md](README.md). This file tracks what the code actually does and what tends to waste time. -**Last updated:** 2026-07-31 · tip `a6b0c8f` on Gitea `main` +**Last updated:** 2026-08-02 · `agent-mode` branch (merged `main` tip `bf508bf`) ## What this is @@ -18,11 +18,13 @@ Source of truth: Scout IT Gitea — `gitea.scoutitsystems.com/woogi/Conflict_Che |-------|--------| | API | Python 3.12, FastAPI, Uvicorn ([backend/main.py](backend/main.py)) | | UI | Single static file [frontend/index.html](frontend/index.html), served by FastAPI | -| Pipeline | Shared by web + CLI: [backend/pipeline/runner.py](backend/pipeline/runner.py) | -| LLM | OpenRouter via `openai` SDK; default `google/gemini-2.5-pro`. Vision always OpenRouter; text stages can use local vLLM | +| Pipelines | **Classic:** [backend/pipeline/runner.py](backend/pipeline/runner.py) (shared by web + CLI). **Agent:** [backend/agents/runner.py](backend/agents/runner.py) — experimental scoped specialist agents, selected per job (`pipeline_mode`) | +| Review gate | Agent jobs stop at `needs_review` for human decisions before the report emails ([backend/review/](backend/review/)) | +| LLM | OpenRouter via `openai` SDK; default `google/gemini-2.5-pro`. Vision always OpenRouter; text stages can use local vLLM (classic/hybrid only) | | PDF | `pdf2image` + system `poppler-utils` → JPEG page images | | Jobs | In-memory threads ([backend/jobs.py](backend/jobs.py)) — no Redis/DB | -| Deploy | Docker Compose; app on port **8099** | +| Tests | `pytest tests/` (~82 tests; see Quick start) | +| Deploy | Docker Compose; app on port **8099**; public URL `https://conchecker.scoutitsystems.com` | ## Live pipeline (authoritative) @@ -53,15 +55,20 @@ PDF → images → extract → sheet index → jurisdiction Design notes for Stage 2/3 engines also live under `Changes/*.docx`. +**Agent pipeline** (`pipeline_mode=agent`): scoped specialist agents in [backend/agents/](backend/agents/) (extractors, linker, brain, critics, RFI writer) run through `Orchestrator` + `ProjectMemory`; artifacts under `outputs//agent/`. Agent mode is OpenRouter-only (no hybrid) and, when `AGENT_REQUIRE_REVIEW=true`, stops at `needs_review` until a human saves decisions and finalizes via the review endpoints. Design docs: `docs/superpowers/`. + ## HTTP API (current) | Method | Path | Purpose | |--------|------|---------| -| GET | `/health` | Liveness + default `model` / `text_model` + key/email flags | -| GET | `/models` | OpenRouter catalog split into `vision[]` / `text[]` + `defaults` (cached ~1h) | +| GET | `/health` | Liveness + `model` / `text_model` + `version` / `build` + key/email flags | +| GET | `/models` | OpenRouter catalog split into `vision[]` / `text[]` + `defaults`, with per-1M-token pricing (cached ~1h; **502** when OpenRouter is unreachable) | | POST | `/check` | Upload PDF; returns `{job_id}` immediately | -| GET | `/jobs/{id}` | Status poll. Running: `stage` + `log_tail`. Done/error: `report` and/or `error` + full `log` | -| GET | `/jobs/{id}/log` | Full run log JSON (`lines`, `text`); `?plain=1` for text/plain | +| GET | `/jobs/{id}` | Status poll. Running: `stage` + `log_tail`. Done/error/needs_review/finalization_error: `report` and/or `error` + full `log` | +| GET | `/jobs/{id}/log` | Full run log as `text/plain` (404 when no log file) | +| GET | `/jobs/{id}/review` | Review queue + progress + saved decisions (agent jobs) | +| POST | `/jobs/{id}/review-decisions` | Save reviewer decisions (409 outside needs_review/reviewing) | +| POST | `/jobs/{id}/finalize-review` | Background finalize + send report (409 unless review gate passed) | | GET | `/jobs/{id}/sheet-image/{page}` | JPEG of source PDF page for the sheet viewer | | GET | `/` | Serves `frontend/index.html` | @@ -69,7 +76,8 @@ Design notes for Stage 2/3 engines also live under `Changes/*.docx`. - Required: `file` (PDF) - Optional: `notification_email`, `project_name`, `address`, `occupancy`, `work_type` -- Compute: `text_local` (`true` = hybrid local text) +- Pipeline: `pipeline_mode` (`classic` default, or `agent`) +- Compute: `text_local` (`true` = hybrid local text; forced off for agent mode) - Models: `vision_model`, `text_model` (OpenRouter ids; blank = config defaults) ## Job logs @@ -77,10 +85,15 @@ Design notes for Stage 2/3 engines also live under `Changes/*.docx`. Pipeline `print()` is teed for the job thread ([backend/job_log.py](backend/job_log.py)): - Live: `GET /jobs/{id}` → `log_tail` (last 80 lines) -- Done/error: same payload includes full `log` +- Done/error/needs_review/finalization_error: same payload includes full `log` - Disk: `backend/outputs//job.log` (survives restart; status registry does not) -- API: `GET /jobs/{id}/log` or `?plain=1` -- UI: “Run log” panel updates while running; stays visible after finish/fail +- API: `GET /jobs/{id}/log` → `text/plain` +- UI: “Run log” panel updates while running; stays visible after finish/fail/review +- Failures append the **full traceback** to the log; each run starts with a header line (job id, mode, models, start time) +- `outputs//job.json` (written at start) carries email/mode/models so the disk fallback can rebuild a job after restart +- **Verbose LLM observability** (`LLM_VERBOSE=true`, default on): one `[LLM] #NNNN stage | model (backend) | in/out sizes | cost | parsed-item counts` line per call in `job.log` — list-valued keys show counts (`conflicts[3]`) so empty (miss) or invented (hallucination) results stand out +- **Raw dumps** (`LLM_RAW_DUMP=true`, default on): full prompt + raw response per call in `outputs//llm_raw/NNNN_stage_model.json` (base64 images excluded, `n_images` recorded). This is the source for tracing *why* the model missed/invented an item +- **End-of-log cost**: every run ends with an `=== Estimated LLM cost (this run) ===` block (total, per-stage, models); failed runs append a cost-so-far line. Local/hybrid calls have no $ accounting — total covers OpenRouter only ## Vision vs text models @@ -91,7 +104,7 @@ Two models, not one: | Vision | `MODEL` | Extract, conflict reason (images) | Always OpenRouter | | Text | `TEXT_MODEL` (falls back to `MODEL`) | Sheet index, jurisdiction, normalize, cluster(LLM), QAQC, code, construct, validate, risk, RFI | OpenRouter, or local when hybrid | -UI: two dropdowns filled from `GET /models` ([backend/models_catalog.py](backend/models_catalog.py)). Per-run picks go through `set_model_overrides()` in [backend/llm.py](backend/llm.py); runner clears them in `finally`. Hybrid: text dropdown also names the local model override and OpenRouter fallback. +UI: two dropdowns (with per-1M pricing) filled from `GET /models` ([backend/models.py](backend/models.py)), shown only for OpenRouter compute. Per-run picks go through `set_model_overrides(vision, text)` in [backend/llm.py](backend/llm.py): classic runs pass them as `run_pipeline` kwargs (runner clears in `finally`); agent runs set them module-level around `run_agent_pipeline`. UI picks beat per-call agent `AGENT_*_MODEL` args but **never name the hybrid local model** — local stays on `LOCAL_TEXT_MODEL`; the text pick only covers the cloud fallback. ## Where to change what @@ -103,7 +116,9 @@ UI: two dropdowns filled from `GET /models` ([backend/models_catalog.py](backend | UI (upload, models, live log, results) | [frontend/index.html](frontend/index.html) | | CLI tuning loop | [cli/run_check.py](cli/run_check.py) | | LLM client, cache, cost, model overrides | [backend/llm.py](backend/llm.py) | -| OpenRouter vision/text model lists | [backend/models_catalog.py](backend/models_catalog.py) | +| OpenRouter catalog + pricing + vision/text split | [backend/models.py](backend/models.py) | +| Agent-mode pipeline | [backend/agents/](backend/agents/) (`runner.py` entry; agents call `llm.call_json` with per-call model args) | +| Human review gate (queue, decisions, finalize) | [backend/review/](backend/review/) | | Job registry + stdout tee log | [backend/jobs.py](backend/jobs.py), [backend/job_log.py](backend/job_log.py) | | Stage helpers (prompt render, issue validate) | [backend/pipeline/_stage.py](backend/pipeline/_stage.py) | | Code text corpus (Stage 7) | [backend/code_corpus/](backend/code_corpus/) | @@ -115,7 +130,7 @@ Older prompt snapshot: `backend/prompts.py.v1`. 1. **Prompt placeholders** — Use `str.replace` via `_stage.render`, never `str.format`. Prompts contain literal `{` JSON braces. 2. **Clustering** — Default is LLM (`CLUSTERER=llm`); empty LLM result falls back to deterministic. Deterministic clusters need ≥2 disciplines (or schedule-vs-plan); single-discipline “missing” gaps are a known limit. -3. **Jobs are in-memory** — Process restart clears job status; `outputs//` (report + `job.log` + `source.pdf`) still reload via disk fallback. +3. **Jobs are in-memory** — Process restart clears job status; `outputs//` (report + `job.log` + `job.json` + `source.pdf`) still reload via disk fallback, including `needs_review` recovery. 4. **Dependency pin** — `httpx==0.27.2` with `openai==1.51.0`. httpx ≥0.28 breaks openai’s `proxies=` kwarg. 5. **Code corpus licensing** — Only `ada_2010.txt` is shipped. Do not paste IBC/IFC/IECC without a license (see `backend/code_corpus/README.md`). 6. **Dual assertion schema** — Newer `{sheet, objects[]}` is mapped to legacy `{assertions[]}` with `attribute`/`value` for older stages. @@ -124,23 +139,24 @@ Older prompt snapshot: `backend/prompts.py.v1`. 9. **Samples** — `samples/*.pdf` are gitignored; drop PDFs locally for CLI runs. 10. **README drift** — Treat README for setup/CI; treat this file + `runner.py` for pipeline truth. `prompts.py` header may still say some prompts are unwired — they are wired through the runner. 11. **Git identity** — This box has no `user.name` / `user.email`; commits need `GIT_AUTHOR_*` / `GIT_COMMITTER_*` env vars (do not `git config`). Remote push to Gitea works. -12. **No local Python deps on host** — App is meant to run in Docker; bare `python3` imports may miss `dotenv` / `httpx`. Prefer `docker compose`. +12. **No local Python deps on host** — App is meant to run in Docker; bare `python3` imports may miss `dotenv` / `httpx`. Prefer `docker compose`. For tests on this box: venv + requirements, but unpin Pillow (`Pillow>=11`) — 10.4.0 doesn't build on Python 3.14 (Docker uses 3.12, where the pin is fine). +13. **Agent mode constraints** — OpenRouter-only (hybrid disabled in UI and forced off server-side); review gate statuses are `needs_review → reviewing → finalizing → done` (`finalization_error` on finalize failure); only terminal states include the full `log` in polls. ## Quick start pointers - Full setup: [README.md](README.md) (`docker compose up -d --build` → http://localhost:8099). - Local CLI: `python cli/run_check.py samples/your_set.pdf --out out/your_set`. - Prompt iteration: set `LLM_CACHE=true` in `backend/.env` so unchanged stages replay for free; clear with `rm -rf backend/.llm_cache`. -- Artifacts per job: `assertions.json`, `clusters.json`, per-stage JSON, `conflicts.json`, `report.md`, `job.log`, `source.pdf` under `backend/outputs//`. -- No automated test suite; validate via CLI dumps and golden-set diffs (described in README). +- Artifacts per job: `assertions.json`, `clusters.json`, per-stage JSON, `conflicts.json`, `report.md`, `job.log`, `job.json`, `source.pdf` under `backend/outputs//`. +- Tests: `python -m pytest tests/` (needs the deps from `requirements.txt` + `pytest`; on this box use a venv, see gotcha #12). -## Recent work (2026-07-31) +## Recent work (2026-08-02, agent-mode) -Shipped on `main` as `a6b0c8f`: +Merged `main` tip (`a6b0c8f` + `bf508bf`) into `agent-mode`, reconciling with this branch's own earlier implementations: -- Per-job run log (tee + disk + API + UI) -- Separate vision/text model dropdowns backed by OpenRouter `/models` -- Session notes file (this doc) +- **Two model dropdowns** — main's vision/text split ported onto this branch's priced catalog (`models.py`); pickers stay OpenRouter-compute-only, and UI picks never override `LOCAL_TEXT_MODEL` (main's hybrid footgun avoided). +- **Better run logs** — main's timestamped line-splitting tee, `log_tail` polls, terminal-state full log, and log-only disk recovery merged with this branch's header line, `job.json` metadata, and review-gate states. Failed runs now also append the traceback to `job.log`. +- `backend/models_catalog.py` (main's unpriced catalog) intentionally dropped in favor of `models.py`. ## Conflict categories (taxonomy) diff --git a/README.md b/README.md index 54e2f1c..7ea2b0a 100644 --- a/README.md +++ b/README.md @@ -147,6 +147,76 @@ python cli/run_check.py samples/your_set.pdf --out out/your_set # -> out/your_set/report.md + conflicts.json ``` +The Classic pipeline remains the recommended default. The experimental Agent +fork runs in the same image and can be selected in the web UI or from the CLI: + +```bash +python cli/run_check.py samples/your_set.pdf --mode agent --out out/agent-run +``` + +Agent mode uses OpenRouter for every model call and runs bounded specialist +waves: one-sheet extraction, sheet-index/jurisdiction orientation, semantic +linkers partitioned by level and object family, per-cluster conflict critics, +batched code review, cluster-scoped constructability, summary-only completeness, +central Brain consolidation, and one-finding RFI writers. It returns the same +`conflicts`, `validated_issues`, `rfis`, and `summary` fields as Classic. + +Agent artifacts are also written under `/agent/`, including wave +snapshots and the final Project Memory. `summary.agent_stats`, +`summary.cost_by_stage`, and `summary.models_used` are job-local, so concurrent +Agent jobs do not share accounting. + +Optional `AGENT_*_MODEL` variables select an OpenRouter model per specialist. +The `AGENT_*_CONCURRENCY` and scope-cap variables in `backend/.env.example` +bound fan-out and prompt size. Agent mode intentionally ignores the hybrid/local +text option in v1. + +### Agent mode: required human review + +By default (`AGENT_REQUIRE_REVIEW=true`) an Agent run **stops after the Brain +consolidation wave** and waits for a human before anything ships: + +``` +Brain merge -> needs_review -> review UI (/?job=) -> finalize -> final report +``` + +The job lifecycle adds review states: `needs_review` (queue built, waiting), +`reviewing` (decisions submitted), `finalizing` (targeted reruns + RFI writers +running), then `done` — or `finalization_error` if finalization fails. Open the +job in the web UI to work the queue: blocking items (high/critical severity, +low confidence, sensitive categories) must be decided; clean-cluster items are +non-blocking spot-checks. + +Email is **two-phase**: a "review required" notice goes out when the job enters +`needs_review` (with a link to the review UI); the final conflict report email +is only sent after finalization completes. The unreviewed report never leaves +the server. + +**Privacy boundary:** all review artifacts (queue, decisions, final report) are +job-local under `outputs//review/`. Cross-job review-feedback +aggregation, when built, excludes verbatim `source_text`, images, and comments +unless `REVIEW_AGGREGATE_INCLUDE_TEXT=true`. + +Config knobs (see `backend/.env.example`): + +| Key | Default | Effect | +|-----|---------|--------| +| `AGENT_REQUIRE_REVIEW` | `true` | `false` = Agent jobs skip the gate entirely (old behavior: RFIs, final report, one email) | +| `AGENT_REVIEW_AUDIT_SAMPLE` | `5` | Max clean clusters added to the queue as spot-checks | +| `REVIEW_AGGREGATE_INCLUDE_TEXT` | `false` | Allow future aggregate feedback to include source text/images/comments | + +From the CLI, `--no-review` bypasses the gate for that run (it overrides +`AGENT_REQUIRE_REVIEW=true`): + +```bash +python cli/run_check.py samples/your_set.pdf --mode agent --no-review --out out/agent-run +``` + +**Deployment note:** the review endpoints (`/jobs/{id}/review-decisions`, +`/jobs/{id}/finalize-review`) are **state-changing and sensitive** — they accept +human decisions that alter the final report. Do **not** expose the UI/API +publicly without reverse-proxy auth or a shared access token in front of it. + Web UI (upload + view): ```bash @@ -155,6 +225,11 @@ uvicorn backend.main:app --reload --port 8099 # open http://127.0.0.1:8099 Or use Docker: `docker compose up -d` (see **Setup** above). +The standard Docker image contains both pipelines; no additional queue, +database, or model service is required. Set `AI_API_KEY` in `backend/.env` and +choose Agent mode per request. Treat Agent output as experimental and compare it +against a reviewed golden set before using it for issuance decisions. + ## Conflict categories `dimensional_disagreement`, `elevation_disagreement`, `location_mismatch`, diff --git a/backend/.env.example b/backend/.env.example index 705c680..bf6b637 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -4,18 +4,62 @@ AI_BASE_URL=https://openrouter.ai/api/v1 AI_API_KEY=sk-or-... MODEL=google/gemini-2.5-pro +# Optional Agent-mode OpenRouter model overrides (inherit MODEL/TEXT_MODEL when blank) +AGENT_EXTRACT_MODEL= +AGENT_INDEX_MODEL= +AGENT_JURISDICTION_MODEL= +AGENT_LINKER_MODEL= +AGENT_CONFLICT_MODEL= +AGENT_CODE_MODEL= +AGENT_CONSTRUCT_MODEL= +AGENT_COMPLETENESS_MODEL= +AGENT_BRAIN_MODEL= +AGENT_RFI_MODEL= + +# Agent-mode hard scope limits / concurrency +AGENT_LINK_MAX_ASSERTIONS=60 +AGENT_CLUSTER_MAX_ASSERTIONS=24 +AGENT_CONFLICT_MAX_IMAGES=6 +AGENT_CODE_BATCH_SIZE=60 +AGENT_BRAIN_MAX_TOKENS=16384 +AGENT_LINK_CONCURRENCY=4 +AGENT_CONFLICT_CONCURRENCY=4 +AGENT_SPECIALIST_CONCURRENCY=4 +AGENT_RFI_CONCURRENCY=4 + +# Agent-mode human-review gate (pipeline stops after Brain until a human reviews) +AGENT_REQUIRE_REVIEW=true +# Max clean clusters added to the review queue as non-blocking spot-checks +AGENT_REVIEW_AUDIT_SAMPLE=5 +# Allow future cross-job review-feedback aggregation to include source_text/images/comments +REVIEW_AGGREGATE_INCLUDE_TEXT=false + # Pipeline tuning PDF_DPI=100 MAX_PAGES=60 MAX_DIMENSION=2400 LLM_TIMEOUT=180 -EXTRACT_MAX_TOKENS=8192 +EXTRACT_MAX_TOKENS=65536 +# Reasoning effort for per-sheet extraction (low keeps Gemini thinking tokens +# from eating the output budget). Blank = don't send the parameter. +EXTRACT_REASONING_EFFORT=low +# Hard thinking-token budget for extraction (OpenRouter reasoning max_tokens / +# Gemini thinking_budget). Stronger than effort; 0 = fall back to effort only. +EXTRACT_REASONING_MAX_TOKENS=2048 REASON_MAX_TOKENS=4096 EXTRACT_CONCURRENCY=4 REASON_CONCURRENCY=4 # Public URL users reach this server on (used for the link in result emails) -APP_BASE_URL=http://localhost:8099 +APP_BASE_URL=https://conchecker.scoutitsystems.com +# APP_BUILD is set by CI at image build time (sha-) - do not set manually. + +# LLM observability (job-log verbosity + raw request/response dumps) +# LLM_VERBOSE: one line per LLM call in job.log (model, sizes, item counts, cost) +# LLM_RAW_DUMP: full prompt+response per call in outputs//llm_raw/ +# (base64 images excluded). Both default on; set false to quiet down. +LLM_VERBOSE=true +LLM_RAW_DUMP=true # Email notifications (optional). Leave SMTP_HOST blank to disable. # Examples: @@ -28,3 +72,24 @@ SMTP_PASSWORD= SMTP_FROM= SMTP_USE_TLS=true SMTP_USE_SSL=false + +# Wave 5b evidence verification (vision fact-check of cited sheet text) +AGENT_VERIFY_MAX_CHECKS=20 +AGENT_VERIFY_SEVERITIES=critical,high +AGENT_VERIFY_REASONING_EFFORT=low +VERIFY_MAX_TOKENS=8192 + +# Text-layer grounding (deterministic PDF text layer via PyMuPDF) +# TEXT_LAYER_ENABLED: master switch for text-layer extraction/grounding +# TEXT_LAYER_MIN_CHARS: below this per page the sheet stays vision-only +# TEXT_LAYER_MAX_CHARS: cap of text layer injected into the extractor prompt +# VERIFY_TEXT_MAX_CHARS: cap of the text-layer excerpt in verify scopes +# VERIFY_HI_DPI_CROPS: evidence-located high-DPI crops in the verifier +# VERIFY_CROP_DPI / VERIFY_CROP_MARGIN_PTS: crop render DPI / padding (PDF points) +TEXT_LAYER_ENABLED=true +TEXT_LAYER_MIN_CHARS=20 +TEXT_LAYER_MAX_CHARS=12000 +VERIFY_TEXT_MAX_CHARS=8000 +VERIFY_HI_DPI_CROPS=true +VERIFY_CROP_DPI=300 +VERIFY_CROP_MARGIN_PTS=36 diff --git a/backend/agents/__init__.py b/backend/agents/__init__.py new file mode 100644 index 0000000..cb1355f --- /dev/null +++ b/backend/agents/__init__.py @@ -0,0 +1,10 @@ +"""Parallel, specialist-agent pipeline isolated from the Classic runner.""" + + +def run_agent_pipeline(*args, **kwargs): + """Lazy package-level entry point that avoids importing optional runtime deps.""" + from backend.agents.runner import run_agent_pipeline as _run + + return _run(*args, **kwargs) + +__all__ = ["run_agent_pipeline"] diff --git a/backend/agents/base.py b/backend/agents/base.py new file mode 100644 index 0000000..d4e464c --- /dev/null +++ b/backend/agents/base.py @@ -0,0 +1,89 @@ +"""Shared contracts and job-local accounting for Agent-mode workers.""" + +import threading +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Protocol + + +@dataclass(frozen=True) +class AgentScope: + """A bounded work package passed to exactly one specialist agent.""" + + scope_id: str + payload: Dict[str, Any] = field(default_factory=dict) + + +@dataclass +class AgentResult: + """Artifacts returned by a specialist for collection by the orchestrator.""" + + scope_id: str + artifacts: List[Dict[str, Any]] = field(default_factory=list) + error: str = "" + + +@dataclass +class AgentUsage: + """Thread-safe usage accounting owned by one Agent pipeline run.""" + + usd: float = 0.0 + calls: int = 0 + cached: int = 0 + by_stage: Dict[str, Dict[str, Any]] = field(default_factory=dict) + models: Dict[str, set] = field(default_factory=lambda: { + "vision": set(), + "text_cloud": set(), + }) + _lock: threading.Lock = field(default_factory=threading.Lock, repr=False) + + def record( + self, + stage: str, + model: str, + usd: float = 0.0, + cached: bool = False, + has_images: bool = False, + ) -> None: + with self._lock: + bucket = self.by_stage.setdefault( + stage, {"usd": 0.0, "calls": 0, "cached": 0} + ) + if cached: + self.cached += 1 + bucket["cached"] += 1 + else: + self.calls += 1 + self.usd += usd + bucket["calls"] += 1 + bucket["usd"] += usd + family = "vision" if has_images else "text_cloud" + self.models[family].add(model) + + def snapshot(self) -> Dict[str, Any]: + with self._lock: + return { + "usd": self.usd, + "calls": self.calls, + "cached": self.cached, + "by_stage": {k: dict(v) for k, v in self.by_stage.items()}, + "models": { + "vision": sorted(self.models["vision"]), + "text_local": [], + "text_cloud": sorted(self.models["text_cloud"]), + "fallback_count": 0, + }, + } + + +class ScopedAgent(Protocol): + """Protocol implemented by each future specialist agent.""" + + name: str + + def run(self, scope: AgentScope) -> AgentResult: + ... + + +def failure(scope: AgentScope, error: Exception) -> AgentResult: + """Convert a worker exception into a non-fatal scoped result.""" + return AgentResult(scope_id=scope.scope_id, error=str(error)) diff --git a/backend/agents/brain.py b/backend/agents/brain.py new file mode 100644 index 0000000..5aef589 --- /dev/null +++ b/backend/agents/brain.py @@ -0,0 +1,129 @@ +"""Central merge, judge, and prioritization agent.""" + +import json +import re +from typing import Dict, List, Tuple + +from backend import config +from backend.agents.base import AgentUsage +from backend.agents.prompts import BRAIN_SYSTEM_PROMPT, BRAIN_USER_PROMPT +from backend.llm import call_json +from backend.pipeline._stage import collect_list, validate_issue + + +def _finding_ref(finding: Dict, index: int) -> str: + return ( + finding.get("issue_id") + or f"{finding.get('agent', 'agent')}:{finding.get('scope_id', '?')}:{index + 1}" + ) + + +def _signature(finding: Dict) -> Tuple[str, str, str]: + norm = lambda value: re.sub(r"[^a-z0-9]+", " ", str(value).lower()).strip() + description = " ".join(norm(finding.get("description")).split()[:12]) + return ( + norm(finding.get("category")), + norm(finding.get("location")), + description, + ) + + +def _fallback(findings: List[Dict]) -> Tuple[List[Dict], List[Dict]]: + """Conservative local consolidation when the Brain call fails.""" + kept: Dict[Tuple[str, str, str], Dict] = {} + refs: Dict[Tuple[str, str, str], List[str]] = {} + decisions: List[Dict] = [] + severity_rank = {"critical": 4, "high": 3, "medium": 2, "low": 1} + for index, finding in enumerate(findings): + ref = _finding_ref(finding, index) + supported = bool(finding.get("evidence")) or finding.get("agent") == "completeness" + if not supported or not finding.get("description"): + decisions.append({ + "finding_refs": [ref], + "action": "dropped", + "reason": "missing actionable support", + "kept_issue_id": None, + }) + continue + signature = _signature(finding) + if signature not in kept: + kept[signature] = dict(finding) + refs[signature] = [ref] + else: + refs[signature].append(ref) + existing = kept[signature] + if severity_rank.get(finding.get("severity"), 2) > severity_rank.get( + existing.get("severity"), 2 + ): + existing["severity"] = finding.get("severity") + existing["evidence"] = ( + existing.get("evidence") or [] + ) + (finding.get("evidence") or []) + + issues = list(kept.values()) + for index, (signature, issue) in enumerate(kept.items()): + issue["issue_id"] = issue.get("issue_id") or f"AGENT-{index + 1:04d}" + issue["risk_score"] = { + "critical": 95, "high": 75, "medium": 50, "low": 25 + }.get(issue.get("severity"), 50) + issue["recommended_priority"] = { + "critical": "immediate", + "high": "before_bid", + "medium": "before_construction", + "low": "track_only", + }.get(issue.get("severity"), "before_construction") + decisions.append({ + "finding_refs": refs[signature], + "action": "merged" if len(refs[signature]) > 1 else "kept", + "reason": "conservative deterministic fallback", + "kept_issue_id": issue["issue_id"], + }) + issues.sort(key=lambda item: -int(item.get("risk_score") or 0)) + return issues, decisions + + +class BrainAgent: + name = "brain" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run( + self, + findings: List[Dict], + sheet_index: Dict, + jurisdiction: Dict, + ) -> Tuple[List[Dict], List[Dict]]: + instruction = BRAIN_USER_PROMPT + for key, value in { + "sheet_index": sheet_index, + "jurisdiction": jurisdiction, + "findings": findings, + }.items(): + instruction = instruction.replace( + "{" + key + "}", json.dumps(value, ensure_ascii=True) + ) + parsed = call_json( + system_prompt=BRAIN_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.AGENT_BRAIN_MAX_TOKENS, + model=config.AGENT_BRAIN_MODEL, + usage_tracker=self.usage, + usage_stage="agent.brain", + ) + issues = collect_list( + parsed, "issues", lambda item: validate_issue(item, item.get("source_stage", "")) + ) + if not issues: + return _fallback(findings) + raw_issues = parsed.get("issues") if isinstance(parsed, dict) else [] + for index, issue in enumerate(issues): + raw = raw_issues[index] if index < len(raw_issues) else {} + issue["issue_id"] = issue.get("issue_id") or f"AGENT-{index + 1:04d}" + issue["risk_score"] = raw.get("risk_score") or issue.get("risk_score") or 50 + issue["recommended_priority"] = ( + raw.get("recommended_priority") or "before_construction" + ) + issues.sort(key=lambda item: -int(item.get("risk_score") or 0)) + decisions = parsed.get("decisions") or [] + return issues, [item for item in decisions if isinstance(item, dict)] diff --git a/backend/agents/code_agent.py b/backend/agents/code_agent.py new file mode 100644 index 0000000..83ff495 --- /dev/null +++ b/backend/agents/code_agent.py @@ -0,0 +1,95 @@ +"""Scoped code/accessibility review agents.""" + +from typing import Dict, List + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline import code_refs +from backend.pipeline._serialize import dumps, slim_sheets +from backend.pipeline._stage import collect_list, validate_issue +from backend.pipeline.jurisdiction import active_review_paths +from backend.prompts import CODE_REVIEW_SYSTEM_PROMPT, CODE_REVIEW_USER_INSTRUCTION + + +def build_code_scopes( + sheets: List[Dict], jurisdiction: Dict, sheet_index: Dict +) -> List[AgentScope]: + cap = max(1, config.AGENT_CODE_BATCH_SIZE) + fragments: List[Dict] = [] + for sheet in sheets: + assertions = sheet.get("assertions") or [] + if not assertions: + continue + for offset in range(0, len(assertions), cap): + fragments.append({ + **sheet, + "assertions": assertions[offset:offset + cap], + }) + batches: List[List[Dict]] = [] + current: List[Dict] = [] + count = 0 + for fragment in fragments: + size = len(fragment["assertions"]) + if current and count + size > cap: + batches.append(current) + current, count = [], 0 + current.append(fragment) + count += size + if current: + batches.append(current) + return [ + AgentScope( + scope_id=f"code:{index + 1}", + payload={ + "sheets": batch, + "jurisdiction": jurisdiction, + "sheet_index": sheet_index, + }, + ) + for index, batch in enumerate(batches) + ] + + +class CodeAgent: + name = "code_reviewer" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + sheets = scope.payload.get("sheets") or [] + jurisdiction = scope.payload.get("jurisdiction") or {} + sheet_index = scope.payload.get("sheet_index") or {} + assertions = [ + assertion for sheet in sheets + for assertion in sheet.get("assertions", []) + ] + excerpts = code_refs.retrieve( + active_review_paths(jurisdiction), assertions + ) + instruction = CODE_REVIEW_USER_INSTRUCTION + for key, value in { + "jurisdiction": dumps(jurisdiction), + "sheet_index": dumps(sheet_index), + "assertions": dumps(slim_sheets(sheets)), + "code_references": code_refs.format_excerpts(excerpts), + }.items(): + instruction = instruction.replace("{" + key + "}", value) + parsed = call_json( + system_prompt=CODE_REVIEW_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.CODE_MAX_TOKENS, + model=config.AGENT_CODE_MODEL, + usage_tracker=self.usage, + usage_stage="agent.code", + ) + findings = collect_list( + parsed, "issues", lambda item: validate_issue(item, "code") + ) + for finding in findings: + finding.update(agent=self.name, scope_id=scope.scope_id) + return AgentResult(scope_id=scope.scope_id, artifacts=findings) + except Exception as exc: + return failure(scope, exc) diff --git a/backend/agents/completeness.py b/backend/agents/completeness.py new file mode 100644 index 0000000..bb09e12 --- /dev/null +++ b/backend/agents/completeness.py @@ -0,0 +1,55 @@ +"""Summary-only drawing-set completeness agent.""" + +import json +from typing import Dict, List + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.agents.prompts import COMPLETENESS_SYSTEM_PROMPT, COMPLETENESS_USER_PROMPT +from backend.llm import call_json +from backend.pipeline._stage import collect_list, validate_issue + + +def build_sheet_summaries(sheets: List[Dict]) -> List[Dict]: + """Return counts and classifications only; never raw assertions.""" + return [{ + "sheet_number": sheet.get("sheet_number"), + "sheet_title": sheet.get("sheet_title"), + "discipline": sheet.get("discipline"), + "drawing_type": sheet.get("drawing_type"), + "level": sheet.get("level"), + "assertion_count": len(sheet.get("assertions") or []), + "unresolved_count": len(sheet.get("unresolved_items") or []), + } for sheet in sheets] + + +class CompletenessAgent: + name = "completeness" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + instruction = COMPLETENESS_USER_PROMPT + for key in ("sheet_index", "sheet_summaries", "cluster_summary"): + instruction = instruction.replace( + "{" + key + "}", + json.dumps(scope.payload.get(key) or {}, ensure_ascii=True), + ) + parsed = call_json( + system_prompt=COMPLETENESS_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.QAQC_MAX_TOKENS, + model=config.AGENT_COMPLETENESS_MODEL, + usage_tracker=self.usage, + usage_stage="agent.completeness", + ) + findings = collect_list( + parsed, "issues", lambda item: validate_issue(item, "qaqc") + ) + for finding in findings: + finding.update(agent=self.name, scope_id=scope.scope_id) + return AgentResult(scope_id=scope.scope_id, artifacts=findings) + except Exception as exc: + return failure(scope, exc) diff --git a/backend/agents/conflict_critic.py b/backend/agents/conflict_critic.py new file mode 100644 index 0000000..f0f5f8b --- /dev/null +++ b/backend/agents/conflict_critic.py @@ -0,0 +1,76 @@ +"""Per-cluster conflict critics with hard evidence and image caps.""" + +from typing import Dict, List + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline.conflict_checker import _evidence_block, _valid_conflict +from backend.prompts import CONFLICT_SYSTEM_PROMPT, CONFLICT_USER_INSTRUCTION + + +def _as_finding(conflict: Dict, scope_id: str) -> Dict: + return { + "issue_id": conflict.get("conflict_id") or "", + "source_stage": "conflict", + "category": conflict.get("category") or "uncategorized", + "severity": conflict.get("severity") or "medium", + "confidence": conflict.get("confidence") or "medium", + "location": conflict.get("location") or "", + "disciplines": conflict.get("disciplines") or [], + "sheets": conflict.get("sheets") or [], + "description": conflict.get("description") or "", + "evidence": conflict.get("evidence") or [], + "recommended_resolution": conflict.get("recommended_resolution") or "", + "code_reference": None, + "agent": "conflict_critic", + "scope_id": scope_id, + } + + +class ConflictCriticAgent: + name = "conflict_critic" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + cluster = dict(scope.payload["cluster"]) + cluster["assertions"] = ( + cluster.get("assertions") or [] + )[:config.AGENT_CLUSTER_MAX_ASSERTIONS] + page_to_b64: Dict[int, str] = scope.payload.get("page_to_b64") or {} + images: List[str] = [] + for page_number in ( + cluster.get("page_numbers") or [] + )[:config.AGENT_CONFLICT_MAX_IMAGES]: + if page_to_b64.get(page_number): + images.append(page_to_b64[page_number]) + instruction = ( + CONFLICT_USER_INSTRUCTION + .replace("{location}", cluster.get("location") or "") + .replace("{evidence}", _evidence_block(cluster)) + ) + parsed = call_json( + system_prompt=CONFLICT_SYSTEM_PROMPT, + user_text=instruction, + images_b64=images, + max_tokens=config.REASON_MAX_TOKENS, + model=config.AGENT_CONFLICT_MODEL, + usage_tracker=self.usage, + usage_stage="agent.conflict", + reasoning_effort=config.EXTRACT_REASONING_EFFORT or None, + reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None, + ) + candidates = parsed if isinstance(parsed, list) else ( + parsed.get("conflicts") if isinstance(parsed, dict) else [] + ) + findings = [] + for candidate in candidates or []: + conflict = _valid_conflict(candidate, cluster) + if conflict: + findings.append(_as_finding(conflict, scope.scope_id)) + return AgentResult(scope_id=scope.scope_id, artifacts=findings) + except Exception as exc: + return failure(scope, exc) diff --git a/backend/agents/construct_agent.py b/backend/agents/construct_agent.py new file mode 100644 index 0000000..5d3487d --- /dev/null +++ b/backend/agents/construct_agent.py @@ -0,0 +1,72 @@ +"""Zone/cluster-scoped constructability agents.""" + +from typing import Dict, List + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline._serialize import dumps, slim_clusters +from backend.pipeline._stage import collect_list, validate_issue +from backend.prompts import ( + CONSTRUCTABILITY_SYSTEM_PROMPT, + CONSTRUCTABILITY_USER_INSTRUCTION, +) + + +def build_construct_scopes( + clusters: List[Dict], conflict_findings: List[Dict] +) -> List[AgentScope]: + scopes = [] + for index, cluster in enumerate(clusters): + related = [ + finding for finding in conflict_findings + if finding.get("scope_id") == f"conflict:{cluster.get('key')}" + or finding.get("location") == cluster.get("location") + ] + scopes.append(AgentScope( + scope_id=f"construct:{index + 1}", + payload={"cluster": cluster, "conflicts": related}, + )) + return scopes + + +class ConstructabilityAgent: + name = "constructability" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + cluster = dict(scope.payload["cluster"]) + cluster["assertions"] = ( + cluster.get("assertions") or [] + )[:config.AGENT_CLUSTER_MAX_ASSERTIONS] + instruction = CONSTRUCTABILITY_USER_INSTRUCTION + substitutions = { + "assertions": dumps(cluster["assertions"]), + "clusters": dumps(slim_clusters([cluster])), + "conflicts": dumps(scope.payload.get("conflicts") or []), + "disputes": dumps(cluster.get("disputed_attributes") or []), + } + for key, value in substitutions.items(): + instruction = instruction.replace("{" + key + "}", value) + parsed = call_json( + system_prompt=CONSTRUCTABILITY_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.CONSTRUCT_MAX_TOKENS, + model=config.AGENT_CONSTRUCT_MODEL, + usage_tracker=self.usage, + usage_stage="agent.constructability", + ) + findings = collect_list( + parsed, + "issues", + lambda item: validate_issue(item, "constructability"), + ) + for finding in findings: + finding.update(agent=self.name, scope_id=scope.scope_id, + cluster_key=cluster.get("key")) + return AgentResult(scope_id=scope.scope_id, artifacts=findings) + except Exception as exc: + return failure(scope, exc) diff --git a/backend/agents/disputes.py b/backend/agents/disputes.py new file mode 100644 index 0000000..56dd24b --- /dev/null +++ b/backend/agents/disputes.py @@ -0,0 +1,55 @@ +"""Deterministic detection of contradictory extracted values within a cluster. + +Extraction is a vision pass: quantities and sizes can be misread ("(2) 2x6" vs +"(5) 2x6"). Cluster members are supposed to describe the same real-world +element, so two members asserting different values for the same attribute are +a probable misread. Flag these so downstream text-only stages treat the value +as unverified instead of reasoning from one reading. +""" + +import re +from typing import Dict, List + + +def _norm(value) -> str: + return re.sub(r"\s+", " ", ("" if value is None else str(value)).strip().lower()) + + +def find_disputes(assertions: List[Dict]) -> List[Dict]: + """Same attribute with >= 2 distinct normalized values = disputed.""" + groups: Dict[str, Dict[str, Dict]] = {} + for assertion in assertions: + attribute = _norm(assertion.get("attribute")) + value = _norm(assertion.get("value")) + if not attribute or not value: + continue + # Group on the normalized value, but keep the original (whitespace- + # collapsed) text so disputes read like the sheet, not a lowercase munge. + original = re.sub(r"\s+", " ", str(assertion.get("value")).strip()) + bucket = groups.setdefault(attribute, {}).setdefault( + value, {"original": original, "ids": set()} + ) + bucket["ids"].add(assertion.get("id")) + disputes = [] + for attribute, values in sorted(groups.items()): + if len(values) < 2: + continue + disputes.append({ + "attribute": attribute, + "values": sorted(v["original"] for v in values.values()), + "assertion_ids": sorted( + aid for v in values.values() for aid in v["ids"] if aid + ), + }) + return disputes + + +def annotate_clusters(clusters: List[Dict]) -> int: + """Attach disputed_attributes to each cluster that has any. Returns count.""" + annotated = 0 + for cluster in clusters: + disputes = find_disputes(cluster.get("assertions") or []) + if disputes: + cluster["disputed_attributes"] = disputes + annotated += 1 + return annotated diff --git a/backend/agents/extractors.py b/backend/agents/extractors.py new file mode 100644 index 0000000..f6706ab --- /dev/null +++ b/backend/agents/extractors.py @@ -0,0 +1,145 @@ +"""Scoped extraction and orientation agents.""" + +import json +from typing import Dict + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline.extractor import _normalize_sheet, _text_layer_block +from backend.pipeline.sheet_index import _index_input +from backend.prompts import ( + EXTRACTOR_SYSTEM_PROMPT, + EXTRACTOR_USER_INSTRUCTION, + JURISDICTION_SYSTEM_PROMPT, + JURISDICTION_USER_INSTRUCTION, + SHEET_INDEX_SYSTEM_PROMPT, + SHEET_INDEX_USER_INSTRUCTION, +) + + +# Appended to the extractor instruction on the second-chance retry. Dense plan +# sheets blow the output budget on the full schema; compact mode trades +# per-object verbosity for actually finishing the page. +_COMPACT_RETRY_SUFFIX = """ +IMPORTANT - COMPACT RETRY: the first pass did not complete. Keep the SAME JSON +schema, but extract at most 40 objects, prioritizing coordination-relevant +items (equipment, fixtures, devices, keynotes, dimensions, markers/callouts, +schedule rows). Keep descriptions/attributes short; skip review_uses entries +you are unsure about. Finish the JSON - a smaller complete answer beats a +larger truncated one.""" + + +def _wrap_bare_list(parsed, page_number: int): + """Models sometimes skip the {sheet, objects} wrapper and return a bare + objects array (especially after truncation repair). Accept it - the sheet + header falls back to title-block deduction downstream.""" + if isinstance(parsed, list): + print(f"[Extract] Page {page_number}: wrapping bare objects array " + f"({len(parsed)} items, no sheet header)") + return {"sheet": {}, "objects": parsed} + return parsed + + +class SheetExtractorAgent: + name = "sheet_extractor" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def _call(self, instruction: str, page: Dict): + return call_json( + system_prompt=EXTRACTOR_SYSTEM_PROMPT, + user_text=instruction, + images_b64=[page["base64"]], + max_tokens=config.EXTRACT_MAX_TOKENS, + model=config.AGENT_EXTRACT_MODEL, + usage_tracker=self.usage, + usage_stage="agent.extract", + reasoning_effort=config.EXTRACT_REASONING_EFFORT or None, + reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None, + ) + + def run(self, scope: AgentScope) -> AgentResult: + try: + page = scope.payload["page"] + instruction = EXTRACTOR_USER_INSTRUCTION.replace( + "{sheet_hint}", str(scope.payload.get("sheet_hint") or "") + ) + _text_layer_block(page) + parsed = _wrap_bare_list(self._call(instruction, page), + page["page_number"]) + if not isinstance(parsed, dict): + # Second chance: same page, compact instructions. Runs only + # when the full-schema pass returned nothing usable. + print(f"[Extract] Page {page['page_number']}: full extraction " + f"failed, retrying compact") + parsed = _wrap_bare_list( + self._call(instruction + _COMPACT_RETRY_SUFFIX, page), + page["page_number"], + ) + if not isinstance(parsed, dict): + raise ValueError("no structured extraction returned") + sheet = _normalize_sheet(parsed, page["page_number"], + page_text=page.get("text_layer")) + return AgentResult(scope_id=scope.scope_id, artifacts=[sheet]) + except Exception as exc: + return failure(scope, exc) + + +class SheetIndexAgent: + name = "sheet_index" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + sheets = scope.payload.get("sheets") or [] + instruction = SHEET_INDEX_USER_INSTRUCTION.replace( + "{sheet_index_input}", + json.dumps(_index_input(sheets), ensure_ascii=True), + ) + parsed = call_json( + system_prompt=SHEET_INDEX_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.SHEET_INDEX_MAX_TOKENS, + model=config.AGENT_INDEX_MODEL, + usage_tracker=self.usage, + usage_stage="agent.sheet_index", + ) + if isinstance(parsed, list): + parsed = {"sheet_index": parsed, "missing_expected_sheets": []} + if not isinstance(parsed, dict): + raise ValueError("no sheet index returned") + return AgentResult(scope_id=scope.scope_id, artifacts=[parsed]) + except Exception as exc: + return failure(scope, exc) + + +class JurisdictionAgent: + name = "jurisdiction" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + project_input: Dict = scope.payload.get("project_input") or {} + instruction = JURISDICTION_USER_INSTRUCTION.replace( + "{project_input}", json.dumps(project_input, ensure_ascii=True) + ) + parsed = call_json( + system_prompt=JURISDICTION_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.JURISDICTION_MAX_TOKENS, + model=config.AGENT_JURISDICTION_MODEL, + usage_tracker=self.usage, + usage_stage="agent.jurisdiction", + ) + if not isinstance(parsed, dict): + raise ValueError("no jurisdiction profile returned") + profile = parsed.get("project_code_profile") + artifact = profile if isinstance(profile, dict) else parsed + return AgentResult(scope_id=scope.scope_id, artifacts=[artifact]) + except Exception as exc: + return failure(scope, exc) diff --git a/backend/agents/linker.py b/backend/agents/linker.py new file mode 100644 index 0000000..2b25a7b --- /dev/null +++ b/backend/agents/linker.py @@ -0,0 +1,221 @@ +"""Bounded semantic linkers that build coordination clusters.""" + +import json +import re +from collections import defaultdict +from typing import Dict, Iterable, List, Tuple + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline.clusterer import cluster_by_location +from backend.pipeline.llm_clusterer import _location +from backend.prompts import CLUSTER_SYSTEM_PROMPT, CLUSTER_USER_INSTRUCTION + + +def _family(assertion: Dict) -> str: + location = assertion.get("location_key") or {} + if location.get("room"): + return "room" + if location.get("grid"): + return "grid" + if location.get("detail_reference"): + return "detail" + tag = str(location.get("tag") or "") + match = re.match(r"[A-Za-z]+", tag) + return ( + (match.group(0).lower() if match else "") + or (assertion.get("object_type") or "").lower() + or "general" + ) + + +def _xref_keys(assertion: Dict) -> List[str]: + """Cross-level join keys: detail references and member tags.""" + location = assertion.get("location_key") or {} + keys = [] + ref = re.sub(r"\s+", "", str(location.get("detail_reference") or "")).upper() + if ref: + keys.append(f"detail:{ref}") + tag = re.sub(r"\s+", "", str(location.get("tag") or "")).upper() + if re.match(r"^[A-Z]+\d", tag): # member marks: W12X26, HSS16X4X5/8, ... + keys.append(f"tag:{tag}") + return keys + + +def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]: + """Partition facts by level and object/tag family, then enforce a hard cap.""" + buckets: Dict[Tuple[str, str], List[Dict]] = defaultdict(list) + xref: Dict[str, List[Dict]] = defaultdict(list) + for sheet in sheets: + for assertion in sheet.get("assertions", []): + enriched = { + **assertion, + "discipline": sheet.get("discipline") or "Unknown", + "sheet_number": sheet.get("sheet_number"), + "page_number": sheet.get("page_number"), + } + level = str((assertion.get("location_key") or {}).get("level") + or sheet.get("level") or "unknown").lower() + buckets[(level, _family(assertion))].append(enriched) + for key in _xref_keys(assertion): + xref[key].append(enriched) + + scopes: List[AgentScope] = [] + cap = max(2, config.AGENT_LINK_MAX_ASSERTIONS) + for (level, family), assertions in sorted(buckets.items()): + for offset in range(0, len(assertions), cap): + chunk = assertions[offset:offset + cap] + if len(chunk) < 2: + continue + scopes.append(AgentScope( + scope_id=f"{level}:{family}:{offset // cap + 1}", + payload={"assertions": chunk, "level": level, "family": family}, + )) + for key, assertions in sorted(xref.items()): + sheets_present = {a.get("sheet_number") for a in assertions} + if len(assertions) < 2 or len(sheets_present) < 2: + continue + chunk = assertions[:cap] + if len({a.get("sheet_number") for a in chunk}) < 2: + continue # cap landed on a single sheet — xref adds nothing + scopes.append(AgentScope( + scope_id=f"xref:{key}", + payload={"assertions": chunk, + "level": "xref", "family": key}, + )) + return scopes + + +def _payload(assertions: Iterable[Dict]) -> List[Dict]: + return [{ + "assertion_id": item.get("id"), + "discipline": item.get("discipline"), + "sheet_number": item.get("sheet_number"), + "attribute": item.get("attribute"), + "value": item.get("value"), + "location_key": item.get("location_key"), + "source_text": item.get("source_text"), + } for item in assertions] + + +def _fallback(assertions: List[Dict]) -> List[Dict]: + """Use the deterministic linker within this scope when semantic linking fails.""" + by_sheet: Dict[Tuple, Dict] = {} + for item in assertions: + key = (item.get("sheet_number"), item.get("page_number")) + sheet = by_sheet.setdefault(key, { + "sheet_number": item.get("sheet_number"), + "page_number": item.get("page_number"), + "discipline": item.get("discipline"), + "assertions": [], + }) + sheet["assertions"].append(item) + return cluster_by_location(list(by_sheet.values())) + + +class LinkerAgent: + name = "linker" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + assertions = scope.payload.get("assertions") or [] + by_id = {item.get("id"): item for item in assertions if item.get("id")} + instruction = CLUSTER_USER_INSTRUCTION.replace( + "{normalized_assertions}", + json.dumps(_payload(assertions), ensure_ascii=True), + ) + parsed = call_json( + system_prompt=CLUSTER_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.CLUSTER_MAX_TOKENS, + model=config.AGENT_LINKER_MODEL, + usage_tracker=self.usage, + usage_stage="agent.link", + ) + raw = parsed if isinstance(parsed, list) else ( + parsed.get("clusters") if isinstance(parsed, dict) else [] + ) + clusters: List[Dict] = [] + for candidate in raw or []: + if not isinstance(candidate, dict): + continue + members = [ + by_id[item_id] + for item_id in candidate.get("assertion_ids") or [] + if item_id in by_id + ] + if len(members) < 2: + continue + allowed_sheets = [] + for member in members: + sheet = member.get("sheet_number") + if sheet not in allowed_sheets: + allowed_sheets.append(sheet) + allowed_sheets = allowed_sheets[:config.AGENT_CONFLICT_MAX_IMAGES] + members = [ + member for member in members + if member.get("sheet_number") in allowed_sheets + ][:config.AGENT_CLUSTER_MAX_ASSERTIONS] + primary = candidate.get("primary_location_key") or {} + clusters.append({ + "key": f"{scope.scope_id}:{candidate.get('cluster_id') or len(clusters) + 1}", + "location": _location(primary), + "disciplines": sorted({ + member.get("discipline") or "Unknown" for member in members + }), + "page_numbers": sorted({ + member["page_number"] for member in members + if member.get("page_number") + }), + "sheets": sorted({ + member["sheet_number"] for member in members + if member.get("sheet_number") + }), + "assertions": members, + "kind": "agent_semantic", + "scope_id": scope.scope_id, + }) + if not clusters: + clusters = _fallback(assertions) + for cluster in clusters: + cluster["scope_id"] = scope.scope_id + cluster["kind"] = "agent_deterministic" + return AgentResult(scope_id=scope.scope_id, artifacts=clusters) + except Exception as exc: + return failure(scope, exc) + + +def build_object_graph(clusters: List[Dict]) -> Dict: + """Build a deterministic graph view from linker output.""" + nodes = [] + edges = [] + seen = set() + for cluster in clusters: + cluster_id = cluster.get("key") + nodes.append({ + "id": cluster_id, + "type": "cluster", + "location": cluster.get("location"), + "sheets": cluster.get("sheets") or [], + }) + for assertion in cluster.get("assertions") or []: + assertion_id = assertion.get("id") + if not assertion_id: + continue + if assertion_id not in seen: + seen.add(assertion_id) + nodes.append({ + "id": assertion_id, + "type": assertion.get("object_type") or "assertion", + "sheet": assertion.get("sheet_number"), + }) + edges.append({ + "source": assertion_id, + "target": cluster_id, + "relationship": "member_of", + }) + return {"nodes": nodes, "edges": edges} diff --git a/backend/agents/memory.py b/backend/agents/memory.py new file mode 100644 index 0000000..369d400 --- /dev/null +++ b/backend/agents/memory.py @@ -0,0 +1,68 @@ +"""Thread-safe per-job blackboard for the Agent pipeline.""" + +import copy +import json +import os +import threading +from typing import Any, Dict, Iterable, Optional + + +_COLLECTION_KEYS = {"sheets", "clusters", "findings", "decisions", "rfis", + "suppressed"} +_MAPPING_KEYS = {"sheet_index", "jurisdiction", "object_graph"} +_MEMORY_KEYS = _COLLECTION_KEYS | _MAPPING_KEYS + + +class ProjectMemory: + """Owns intermediate Agent-mode state and optional debug snapshots.""" + + def __init__(self, artifact_dir: Optional[str] = None) -> None: + self.artifact_dir = artifact_dir + self._lock = threading.RLock() + self._data: Dict[str, Any] = { + **{key: [] for key in _COLLECTION_KEYS}, + **{key: {} for key in _MAPPING_KEYS}, + } + if artifact_dir: + os.makedirs(artifact_dir, exist_ok=True) + + def replace(self, key: str, value: Any) -> None: + """Replace one named memory section.""" + self._validate_key(key) + with self._lock: + self._data[key] = copy.deepcopy(value) + + def append(self, key: str, value: Dict[str, Any]) -> None: + """Append one artifact to a list-backed memory section.""" + if key not in _COLLECTION_KEYS: + raise KeyError(f"{key!r} is not an appendable memory section") + with self._lock: + self._data[key].append(copy.deepcopy(value)) + + def extend(self, key: str, values: Iterable[Dict[str, Any]]) -> None: + """Append several artifacts under one lock.""" + if key not in _COLLECTION_KEYS: + raise KeyError(f"{key!r} is not an appendable memory section") + with self._lock: + self._data[key].extend(copy.deepcopy(list(values))) + + def snapshot(self) -> Dict[str, Any]: + """Return a detached, JSON-serializable view of current state.""" + with self._lock: + return copy.deepcopy(self._data) + + def dump(self, filename: str = "memory.json") -> Optional[str]: + """Persist a snapshot when this job has an artifact directory.""" + if not self.artifact_dir: + return None + path = os.path.join(self.artifact_dir, filename) + temp_path = f"{path}.tmp" + with open(temp_path, "w", encoding="utf-8") as f: + json.dump(self.snapshot(), f, indent=2) + os.replace(temp_path, path) + return path + + @staticmethod + def _validate_key(key: str) -> None: + if key not in _MEMORY_KEYS: + raise KeyError(f"Unknown project memory section: {key!r}") diff --git a/backend/agents/orchestrator.py b/backend/agents/orchestrator.py new file mode 100644 index 0000000..0efeb4b --- /dev/null +++ b/backend/agents/orchestrator.py @@ -0,0 +1,78 @@ +"""Wave scheduler for the Agent pipeline.""" + +from concurrent.futures import ThreadPoolExecutor, as_completed +from dataclasses import dataclass, field +from typing import Callable, Dict, Iterable, List, Optional + +from backend.agents.base import AgentResult, AgentScope, ScopedAgent +from backend.agents.memory import ProjectMemory + + +@dataclass +class AgentStats: + """Job-local accounting; never shared across concurrent jobs.""" + + calls: int = 0 + scopes: int = 0 + merges: int = 0 + failed_scopes: List[str] = field(default_factory=list) + + def as_dict(self) -> Dict: + return { + "calls": self.calls, + "scopes": self.scopes, + "merges": self.merges, + "failed_scopes": list(self.failed_scopes), + } + + +class Orchestrator: + """Coordinates bounded fan-out/fan-in waves against one ProjectMemory.""" + + def __init__( + self, + memory: ProjectMemory, + on_stage: Optional[Callable[[str], None]] = None, + ) -> None: + self.memory = memory + self.on_stage = on_stage + self.stats = AgentStats() + + def stage(self, name: str) -> None: + print(f"\n=== {name} ===") + if self.on_stage: + self.on_stage(name) + + def initialize(self) -> Dict: + """Initialize the job-local artifact store.""" + self.stage("Initialize agent pipeline") + self.memory.dump() + return self.stats.as_dict() + + def run_scopes( + self, + agent: ScopedAgent, + scopes: Iterable[AgentScope], + concurrency: int, + ) -> List[AgentResult]: + """Run independent scopes; one failure never aborts the wave.""" + scope_list = list(scopes) + if not scope_list: + return [] + results: List[AgentResult] = [] + with ThreadPoolExecutor(max_workers=max(1, concurrency)) as pool: + futures = {pool.submit(agent.run, scope): scope for scope in scope_list} + for future in as_completed(futures): + scope = futures[future] + self.stats.scopes += 1 + try: + result = future.result() + except Exception as exc: + result = AgentResult(scope_id=scope.scope_id, error=str(exc)) + if result.error: + self.stats.failed_scopes.append( + f"{agent.name}:{scope.scope_id}: {result.error}" + ) + results.append(result) + results.sort(key=lambda result: result.scope_id) + return results diff --git a/backend/agents/prompts.py b/backend/agents/prompts.py new file mode 100644 index 0000000..c40ab53 --- /dev/null +++ b/backend/agents/prompts.py @@ -0,0 +1,25 @@ +"""Prompts unique to the scoped Agent pipeline.""" + +COMPLETENESS_SYSTEM_PROMPT = """You are a senior construction-document completeness reviewer. +Review only the supplied sheet index and aggregate counts. Identify missing sheets, +schedules, details, or clearly incomplete coverage. Do not infer drawing facts and do +not report direct design conflicts. A missing-information finding may use the sheet +index itself as evidence. Respond only with valid JSON.""" + +COMPLETENESS_USER_PROMPT = """Review this summarized drawing set for completeness. +Return {"issues":[{"issue_id":"string","source_stage":"qaqc","category":"missing_sheet | missing_schedule | missing_detail | missing_information | bid_readiness | permit_readiness | other","severity":"critical | high | medium | low","confidence":"high | medium | low","location":"sheet or drawing set","disciplines":["string"],"sheets":["string"],"description":"string","evidence":[],"recommended_resolution":"string","code_reference":null}]}. +Sheet index: {sheet_index} +Aggregate sheet summaries: {sheet_summaries} +Cluster summary: {cluster_summary}""" + +BRAIN_SYSTEM_PROMPT = """You are the central decision layer for a construction drawing +review. Merge duplicate specialist findings, reject vague or unsupported findings, +preserve verbatim evidence, and prioritize the kept issues. Do not create new issues. +Conflicts need drawing evidence; completeness findings may instead cite an explicit +missing item from the sheet index. Return only valid JSON.""" + +BRAIN_USER_PROMPT = """Judge and consolidate these scoped specialist findings. +Return {"issues":[{"issue_id":"string","source_stage":"conflict | qaqc | code | constructability","category":"string","severity":"critical | high | medium | low","confidence":"high | medium | low","location":"string","disciplines":["string"],"sheets":["string"],"description":"string","evidence":[{"discipline":"string","sheet":"string","source_text":"string","asserted_value":"string"}],"recommended_resolution":"string","code_reference":"string or null","risk_score":1,"recommended_priority":"immediate | before_bid | before_permit | before_construction | track_only"}],"decisions":[{"finding_refs":["string"],"action":"kept | merged | dropped","reason":"string","kept_issue_id":"string or null"}]}. +Sheet index: {sheet_index} +Jurisdiction summary: {jurisdiction} +Specialist findings: {findings}""" diff --git a/backend/agents/rfi_writer.py b/backend/agents/rfi_writer.py new file mode 100644 index 0000000..93fc352 --- /dev/null +++ b/backend/agents/rfi_writer.py @@ -0,0 +1,42 @@ +"""One-finding-per-call RFI writers.""" + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline._serialize import dumps +from backend.pipeline.rfi import _valid_rfi +from backend.prompts import RFI_SYSTEM_PROMPT, RFI_USER_INSTRUCTION + + +class RFIWriterAgent: + name = "rfi_writer" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + finding = scope.payload["finding"] + instruction = RFI_USER_INSTRUCTION.replace( + "{prioritized_issues}", dumps([finding]) + ) + parsed = call_json( + system_prompt=RFI_SYSTEM_PROMPT, + user_text=instruction, + max_tokens=config.RFI_MAX_TOKENS, + model=config.AGENT_RFI_MODEL, + usage_tracker=self.usage, + usage_stage="agent.rfi", + ) + candidates = parsed if isinstance(parsed, list) else ( + parsed.get("rfi_comments") if isinstance(parsed, dict) else [] + ) + rfis = [] + for candidate in candidates or []: + rfi = _valid_rfi(candidate) + if rfi: + rfi["issue_id"] = rfi.get("issue_id") or finding.get("issue_id") + rfis.append(rfi) + return AgentResult(scope_id=scope.scope_id, artifacts=rfis[:1]) + except Exception as exc: + return failure(scope, exc) diff --git a/backend/agents/runner.py b/backend/agents/runner.py new file mode 100644 index 0000000..7a5716a --- /dev/null +++ b/backend/agents/runner.py @@ -0,0 +1,496 @@ +"""Public entry point for the scoped Agent-mode pipeline.""" + +import base64 +import json +import os +from typing import Callable, Dict, Optional + +from backend import config +from backend.agents.base import AgentScope, AgentUsage +from backend.agents.brain import BrainAgent +from backend.agents.code_agent import CodeAgent, build_code_scopes +from backend.agents.completeness import CompletenessAgent, build_sheet_summaries +from backend.agents.conflict_critic import ConflictCriticAgent +from backend.agents.construct_agent import ConstructabilityAgent, build_construct_scopes +from backend.agents.disputes import annotate_clusters +from backend.agents.extractors import ( + JurisdictionAgent, + SheetExtractorAgent, + SheetIndexAgent, +) +from backend.agents.linker import LinkerAgent, build_link_scopes, build_object_graph +from backend.agents.memory import ProjectMemory +from backend.agents.orchestrator import Orchestrator +from backend.agents.rfi_writer import RFIWriterAgent +from backend.agents.verifier import ( + EvidenceVerifierAgent, apply_verdicts, select_findings, +) +from backend.llm import reset_cost +from backend.pipeline.pdf_processor import convert_pdf_to_images +from backend.pipeline.report import build_report, to_markdown +from backend.pipeline.sheet_index import derive_project_meta_from_cover +from backend.review.gate import build_review_queue +from backend.review.store import ReviewStore +from backend.text_layer import ( + attach_text_layers, coverage_gaps, find_evidence_bbox, render_crop, +) + + +def run_agent_pipeline( + pdf_path: str, + out_dir: Optional[str] = None, + on_stage: Optional[Callable[[str], None]] = None, + project_input: Optional[Dict] = None, + source_name: Optional[str] = None, + require_review: bool = True, +) -> Dict: + """Run all scoped specialist waves and return a Classic-compatible report.""" + if not os.path.isfile(pdf_path): + raise FileNotFoundError(pdf_path) + + # Keep the llm module's counters job-local (matches Classic): the job + # log's failure-path cost estimate in jobs.py reads llm.get_cost(). + reset_cost() + + agent_dir = os.path.join(out_dir, "agent") if out_dir else None + memory = ProjectMemory(artifact_dir=agent_dir) + orchestrator = Orchestrator(memory=memory, on_stage=on_stage) + usage = AgentUsage() + orchestrator.initialize() + + orchestrator.stage("Agent ingest: PDF -> images") + pages = convert_pdf_to_images(pdf_path) + page_to_b64 = {page["page_number"]: page["base64"] for page in pages} + text_dir = os.path.join(agent_dir, "text") if agent_dir else None + page_words = attach_text_layers(pdf_path, pages, text_dir=text_dir) + page_to_text = {page["page_number"]: page.get("text_layer") for page in pages} + + orchestrator.stage("Agent wave 1: extract sheets") + extract_scopes = [ + AgentScope( + scope_id=f"sheet:{page['page_number']}", + payload={"page": page}, + ) + for page in pages + ] + extract_results = orchestrator.run_scopes( + SheetExtractorAgent(usage), extract_scopes, config.EXTRACT_CONCURRENCY + ) + sheets = [ + artifact + for result in extract_results + for artifact in result.artifacts + ] + sheets.sort(key=lambda sheet: sheet.get("page_number") or 0) + memory.replace("sheets", sheets) + memory.dump("01-extract.json") + # Coverage signal: text layer present but extraction failed/empty reuses + # the failed-scopes gap-finding path (finding built below wave 6). + for gap_page in coverage_gaps(pages, sheets): + orchestrator.stats.failed_scopes.append( + f"sheet_extractor:sheet:{gap_page}: extraction gap " + f"(text layer present, no objects extracted)" + ) + + cover_meta = derive_project_meta_from_cover( + sheets, source_name or os.path.basename(pdf_path) + ) + merged_input = {**cover_meta, **(project_input or {})} + orchestrator.stage("Agent wave 2: sheet index and jurisdiction") + index_results = orchestrator.run_scopes( + SheetIndexAgent(usage), + [AgentScope("sheet-index", {"sheets": sheets})], + 1, + ) + jurisdiction_results = orchestrator.run_scopes( + JurisdictionAgent(usage), + [AgentScope("jurisdiction", {"project_input": merged_input})], + 1, + ) + sheet_index = ( + index_results[0].artifacts[0] + if index_results and index_results[0].artifacts else {} + ) + jurisdiction = ( + jurisdiction_results[0].artifacts[0] + if jurisdiction_results and jurisdiction_results[0].artifacts else {} + ) + memory.replace("sheet_index", sheet_index) + memory.replace("jurisdiction", jurisdiction) + memory.dump("02-orient.json") + + orchestrator.stage("Agent wave 3: scoped semantic linking") + link_results = orchestrator.run_scopes( + LinkerAgent(usage), + build_link_scopes(sheets), + config.AGENT_LINK_CONCURRENCY, + ) + clusters = [ + artifact + for result in link_results + for artifact in result.artifacts + ][:config.CLUSTER_MAX] + object_graph = build_object_graph(clusters) + disputed_count = annotate_clusters(clusters) + if disputed_count: + orchestrator.stage(f"[Link] {disputed_count} clusters carry disputed extracted values") + memory.replace("clusters", clusters) + memory.replace("object_graph", object_graph) + memory.dump("03-link.json") + + orchestrator.stage("Agent wave 4: per-cluster conflict critics") + conflict_scopes = [ + AgentScope( + scope_id=f"conflict:{cluster.get('key')}", + payload={"cluster": cluster, "page_to_b64": page_to_b64}, + ) + for cluster in clusters + ] + conflict_results = orchestrator.run_scopes( + ConflictCriticAgent(usage), + conflict_scopes, + config.AGENT_CONFLICT_CONCURRENCY, + ) + conflict_findings = [ + artifact + for result in conflict_results + for artifact in result.artifacts + ] + memory.extend("findings", conflict_findings) + + orchestrator.stage("Agent wave 5: scoped specialists") + code_results = orchestrator.run_scopes( + CodeAgent(usage), + build_code_scopes(sheets, jurisdiction, sheet_index), + config.AGENT_SPECIALIST_CONCURRENCY, + ) + construct_results = orchestrator.run_scopes( + ConstructabilityAgent(usage), + build_construct_scopes(clusters, conflict_findings), + config.AGENT_SPECIALIST_CONCURRENCY, + ) + completeness_scope = AgentScope("completeness", { + "sheet_index": sheet_index, + "sheet_summaries": build_sheet_summaries(sheets), + "cluster_summary": { + "count": len(clusters), + "by_kind": _counts(clusters, "kind"), + }, + }) + completeness_results = orchestrator.run_scopes( + CompletenessAgent(usage), [completeness_scope], 1 + ) + specialist_findings = [ + artifact + for result in code_results + construct_results + completeness_results + for artifact in result.artifacts + ] + + orchestrator.stage("Agent wave 5b: evidence verification") + sheet_to_page = {str(s.get("sheet_number")): s.get("page_number") + for s in sheets} + verify_targets = select_findings( + specialist_findings, clusters, + max_checks=config.AGENT_VERIFY_MAX_CHECKS, + severities=config.AGENT_VERIFY_SEVERITIES, + ) + target_indexes = {id(f): i for i, f in enumerate(specialist_findings)} + verify_scopes = [] + for finding in verify_targets: + cited_pages = [ + sheet_to_page[str(name)] + for name in (finding.get("sheets") or []) + if sheet_to_page.get(str(name)) in page_to_b64 + ] + images = [ + page_to_b64[p] + for p in cited_pages[:config.AGENT_CONFLICT_MAX_IMAGES] + ] + if not images: + continue # never judge evidence against images we could not load + # Text oracle: concatenated text layer of the cited sheets, capped. + excerpt = "\n\n".join( + f"--- Page {p} ---\n{page_to_text[p]}" + for p in cited_pages + if page_to_text.get(p) + )[:config.VERIFY_TEXT_MAX_CHARS] + if config.VERIFY_HI_DPI_CROPS: + images = _evidence_crops(finding, cited_pages, sheet_to_page, + page_words, page_to_b64, pdf_path, + fallback=images) + verify_scopes.append(AgentScope( + scope_id=f"verify:{target_indexes[id(finding)]}", + payload={ + "finding_index": target_indexes[id(finding)], + "finding": finding, + "images_b64": images, + "text_layer_excerpt": excerpt, + }, + )) + verify_results = orchestrator.run_scopes( + EvidenceVerifierAgent(usage), verify_scopes, config.AGENT_VERIFY_CONCURRENCY) + suppressed = apply_verdicts(specialist_findings, verify_results) + if suppressed: + suppressed_ids = {id(f) for f in suppressed} + specialist_findings = [f for f in specialist_findings if id(f) not in suppressed_ids] + memory.replace("suppressed", suppressed) + + memory.extend("findings", specialist_findings) + gap_findings = [ + { + "issue_id": f"AGENT-GAP-{index + 1:03d}", + "source_stage": "qaqc", + "category": "analysis_gap", + "severity": "low", + "confidence": "high", + "location": failed_scope.split(":", 2)[1] if ":" in failed_scope else "", + "disciplines": [], + "sheets": [], + "description": f"Agent analysis scope did not complete: {failed_scope}", + "evidence": [], + "recommended_resolution": "Review this scope manually or rerun the job.", + "code_reference": None, + "agent": "completeness", + "scope_id": "failed-scopes", + } + for index, failed_scope in enumerate(orchestrator.stats.failed_scopes) + ] + memory.extend("findings", gap_findings) + memory.dump("05-specialists.json") + + orchestrator.stage("Agent wave 6: Brain merge, judge, prioritize") + all_findings = memory.snapshot()["findings"] + if all_findings: + prioritized, decisions = BrainAgent(usage).run( + all_findings, sheet_index, jurisdiction + ) + else: + prioritized, decisions = [], [] + memory.extend("decisions", decisions) + orchestrator.stats.merges = sum( + 1 for decision in decisions if decision.get("action") == "merged" + ) + + if require_review: + orchestrator.stage("Agent review gate: build human-review queue") + memory_snapshot = memory.snapshot() + queue = build_review_queue(memory_snapshot, prioritized, decisions, + limit=config.AGENT_REVIEW_AUDIT_SAMPLE) + store = ReviewStore(out_dir) + store.write_queue(queue) + candidate_conflicts = [_finding_as_conflict(item) for item in conflict_findings] + report = build_report( + conflicts=candidate_conflicts, + sheets=sheets, + clusters=clusters, + source=source_name or os.path.basename(pdf_path), + ) + report.update({ + "project_input": merged_input, + "jurisdiction": jurisdiction, + "sheet_index": sheet_index, + "project_intelligence": object_graph, + "validated_issues": prioritized, + "rfis": [], + "suppressed_issues": memory.snapshot().get("suppressed") or [], + }) + progress = store.progress(queue) + # Same usage/stats summary block as the wave-7 path (rfis: 0 — they + # are drafted only after human review finalizes the run). + cost = usage.snapshot() + orchestrator.stats.calls = cost["calls"] + stats = orchestrator.stats.as_dict() + report["summary"].update({ + "pipeline_mode": "agent", + "agent_status": "needs_review", + "review": progress, + "agent_stats": stats, + "by_stage": { + "conflicts": len(conflict_findings), + "qaqc": sum( + 1 for item in specialist_findings + if item.get("source_stage") == "qaqc" + ), + "code": sum( + 1 for item in specialist_findings + if item.get("source_stage") == "code" + ), + "constructability": sum( + 1 for item in specialist_findings + if item.get("source_stage") == "constructability" + ), + "validated": len(prioritized), + "rfis": 0, + }, + "cost_usd": round(cost["usd"], 4), + "llm_calls": cost["calls"], + "cached_calls": cost["cached"], + "cost_by_stage": cost["by_stage"], + "text_backend": "openrouter", + "models_used": cost["models"], + }) + if out_dir: + _dump(out_dir, "conflicts.json", report) + _dump(out_dir, "validated_issues.json", prioritized) + # Snapshot for the review finalizer's targeted clarification reruns. + memory.dump("memory.json") + return report + + orchestrator.stage("Agent wave 7: per-finding RFI writers") + rfi_scopes = [ + AgentScope( + scope_id=f"rfi:{finding.get('issue_id') or index + 1}", + payload={"finding": finding}, + ) + for index, finding in enumerate(prioritized) + ] + rfi_results = orchestrator.run_scopes( + RFIWriterAgent(usage), rfi_scopes, config.AGENT_RFI_CONCURRENCY + ) + rfis = [ + artifact for result in rfi_results for artifact in result.artifacts + ] + memory.extend("rfis", rfis) + memory.dump("memory.json") + + orchestrator.stage("Build agent report") + conflicts = [_finding_as_conflict(item) for item in conflict_findings] + report = build_report( + conflicts=conflicts, + sheets=sheets, + clusters=clusters, + source=source_name or os.path.basename(pdf_path), + ) + report.update({ + "project_input": merged_input, + "jurisdiction": jurisdiction, + "sheet_index": sheet_index, + "project_intelligence": object_graph, + "validated_issues": prioritized, + "rfis": rfis, + "suppressed_issues": memory.snapshot().get("suppressed") or [], + }) + cost = usage.snapshot() + orchestrator.stats.calls = cost["calls"] + stats = orchestrator.stats.as_dict() + report["summary"].update({ + "pipeline_mode": "agent", + "agent_status": "complete", + "agent_stats": stats, + "by_stage": { + "conflicts": len(conflict_findings), + "qaqc": sum( + 1 for item in specialist_findings + if item.get("source_stage") == "qaqc" + ), + "code": sum( + 1 for item in specialist_findings + if item.get("source_stage") == "code" + ), + "constructability": sum( + 1 for item in specialist_findings + if item.get("source_stage") == "constructability" + ), + "validated": len(prioritized), + "rfis": len(rfis), + }, + "cost_usd": round(cost["usd"], 4), + "llm_calls": cost["calls"], + "cached_calls": cost["cached"], + "cost_by_stage": cost["by_stage"], + "text_backend": "openrouter", + "models_used": cost["models"], + }) + + if out_dir: + os.makedirs(out_dir, exist_ok=True) + _dump(out_dir, "assertions.json", sheets) + _dump(out_dir, "clusters.json", [_without_base64(item) for item in clusters]) + _dump(out_dir, "sheet_index.json", sheet_index) + _dump(out_dir, "jurisdiction.json", jurisdiction) + _dump(out_dir, "project_intelligence.json", object_graph) + _dump(out_dir, "validated_issues.json", prioritized) + _dump(out_dir, "rfis.json", rfis) + _dump(out_dir, "conflicts.json", report) + with open(os.path.join(out_dir, "report.md"), "w", encoding="utf-8") as f: + f.write(to_markdown(report)) + + return report + + +def _dump(out_dir: str, name: str, value) -> None: + with open(os.path.join(out_dir, name), "w", encoding="utf-8") as f: + json.dump(value, f, indent=2) + + +def _evidence_crops( + finding: Dict, + cited_pages: list, + sheet_to_page: Dict, + page_words: Dict, + page_to_b64: Dict, + pdf_path: str, + fallback: list, +) -> list: + """High-DPI crops around each evidence item's source_text, located via the + page text layer. Crops REPLACE full-page images when at least one evidence + location resolves confidently; otherwise the full-page fallback is kept. + Never returns an empty list when fallback is non-empty (I2 guard).""" + crops: list = [] + for item in finding.get("evidence") or []: + if len(crops) >= config.AGENT_CONFLICT_MAX_IMAGES: + break + if not isinstance(item, dict): + continue + source_text = item.get("source_text") or "" + if not source_text: + continue + # Prefer the page named on the evidence item, then any cited page. + candidates = [] + named_page = sheet_to_page.get(str(item.get("sheet") or "")) + if named_page in cited_pages: + candidates.append(named_page) + candidates.extend(p for p in cited_pages if p not in candidates) + for page in candidates: + bbox = find_evidence_bbox(page_words.get(page) or [], source_text) + if bbox is None: + continue + crop = render_crop(pdf_path, page, bbox) + if not crop: + continue + crops.append(base64.b64encode(crop).decode("utf-8")) + break + return crops or fallback + + +def _counts(items, key: str) -> Dict[str, int]: + counts: Dict[str, int] = {} + for item in items: + value = str(item.get(key) or "unknown") + counts[value] = counts.get(value, 0) + 1 + return counts + + +def _finding_as_conflict(finding: Dict) -> Dict: + return { + "category": finding.get("category") or "uncategorized", + "severity": finding.get("severity") or "medium", + "disciplines": finding.get("disciplines") or [], + "location": finding.get("location") or "", + "sheets": finding.get("sheets") or [], + "description": finding.get("description") or "", + "evidence": finding.get("evidence") or [], + "recommended_resolution": finding.get("recommended_resolution") or "", + "confidence": finding.get("confidence") or "medium", + "cluster_key": finding.get("scope_id"), + } + + +def _without_base64(cluster: Dict) -> Dict: + return { + **cluster, + "assertions": [ + {key: value for key, value in assertion.items() if key != "base64"} + for assertion in cluster.get("assertions") or [] + ], + } diff --git a/backend/agents/verifier.py b/backend/agents/verifier.py new file mode 100644 index 0000000..3d55f63 --- /dev/null +++ b/backend/agents/verifier.py @@ -0,0 +1,99 @@ +"""Wave 5b: vision fact-check of extracted evidence against cited sheet images.""" + +from backend import config +from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure +from backend.llm import call_json +from backend.pipeline._serialize import dumps +from backend.pipeline._stage import collect_list, render +from backend.prompts import VERIFY_SYSTEM_PROMPT, VERIFY_USER_INSTRUCTION + +_SEVERITY_RANK = {"critical": 0, "high": 1, "medium": 2, "low": 3} +_VERDICTS = ("confirmed", "corrected", "not_found") + + +def select_findings(findings, clusters, max_checks, severities): + """Severity-gated selection plus any finding tied to a disputed cluster.""" + disputed_keys = {c.get("key") for c in clusters if c.get("disputed_attributes")} + selected = [f for f in findings + if str(f.get("severity") or "").lower() in severities + or f.get("cluster_key") in disputed_keys] + selected.sort(key=lambda f: _SEVERITY_RANK.get( + str(f.get("severity") or "").lower(), 9)) + return selected[:max_checks] + + +def _valid_verdict(item): + if not isinstance(item, dict): + return None + verdict = str(item.get("verdict") or "").lower() + if verdict not in _VERDICTS: + return None + return {"sheet": item.get("sheet") or "", + "source_text": item.get("source_text") or "", + "verdict": verdict, + "actual_text": item.get("actual_text"), + "notes": item.get("notes")} + + +def _status(verdicts): + if not verdicts: + return "unverified" + confirmed = sum(1 for v in verdicts if v["verdict"] == "confirmed") + if confirmed == len(verdicts): + return "confirmed" + if confirmed == 0: + return "refuted" + return "mixed" + + +class EvidenceVerifierAgent: + name = "verify" + + def __init__(self, usage: AgentUsage) -> None: + self.usage = usage + + def run(self, scope: AgentScope) -> AgentResult: + try: + finding = scope.payload["finding"] + instruction = render(VERIFY_USER_INSTRUCTION, { + "finding": dumps(finding), + "text_layer": scope.payload.get("text_layer_excerpt") + or "(no text layer available for the cited sheets)", + }) + parsed = call_json( + system_prompt=VERIFY_SYSTEM_PROMPT, + user_text=instruction, + images_b64=scope.payload.get("images_b64") or [], + max_tokens=config.VERIFY_MAX_TOKENS, + model=config.AGENT_VERIFY_MODEL, + reasoning_effort=config.AGENT_VERIFY_REASONING_EFFORT or None, + usage_tracker=self.usage, + usage_stage="agent.verify", + ) + verdicts = collect_list(parsed, "verdicts", _valid_verdict) + return AgentResult(scope_id=scope.scope_id, artifacts=[{ + "finding_index": scope.payload["finding_index"], + "status": _status(verdicts), + "verdicts": verdicts, + }]) + except Exception as exc: + return failure(scope, exc) + + +def apply_verdicts(findings, verify_results): + """Annotate findings with verification; return refuted ones to suppress.""" + by_index = {} + for result in verify_results: + for artifact in result.artifacts: + by_index[artifact["finding_index"]] = artifact + suppressed = [] + for index, finding in enumerate(findings): + artifact = by_index.get(index) + if not artifact: + continue + finding["verification"] = {"status": artifact["status"], + "verdicts": artifact["verdicts"]} + if artifact["status"] == "refuted": + finding["confidence"] = "low" + suppressed.append(finding) + return suppressed diff --git a/backend/config.py b/backend/config.py index 5d7009d..83d0ef7 100644 --- a/backend/config.py +++ b/backend/config.py @@ -22,6 +22,66 @@ MODEL = os.getenv("MODEL", "google/gemini-2.5-pro") # MODEL when unset. TEXT_MODEL = os.getenv("TEXT_MODEL", "") or MODEL +# Agent-mode model overrides (OpenRouter IDs). Empty values inherit the +# matching general-purpose model so the skeleton requires no extra config. +AGENT_EXTRACT_MODEL = os.getenv("AGENT_EXTRACT_MODEL", "") or MODEL +AGENT_INDEX_MODEL = os.getenv("AGENT_INDEX_MODEL", "") or TEXT_MODEL +AGENT_JURISDICTION_MODEL = os.getenv("AGENT_JURISDICTION_MODEL", "") or TEXT_MODEL +AGENT_LINKER_MODEL = os.getenv("AGENT_LINKER_MODEL", "") or TEXT_MODEL +AGENT_CONFLICT_MODEL = os.getenv("AGENT_CONFLICT_MODEL", "") or MODEL +AGENT_CODE_MODEL = os.getenv("AGENT_CODE_MODEL", "") or TEXT_MODEL +AGENT_CONSTRUCT_MODEL = os.getenv("AGENT_CONSTRUCT_MODEL", "") or TEXT_MODEL +AGENT_COMPLETENESS_MODEL = os.getenv("AGENT_COMPLETENESS_MODEL", "") or TEXT_MODEL +AGENT_BRAIN_MODEL = os.getenv("AGENT_BRAIN_MODEL", "") or TEXT_MODEL +AGENT_RFI_MODEL = os.getenv("AGENT_RFI_MODEL", "") or TEXT_MODEL + +# Agent-mode hard scope limits. These are intentionally independent of Classic +# batching so Agent workers can never grow into whole-set reasoning calls. +AGENT_LINK_MAX_ASSERTIONS = int(os.getenv("AGENT_LINK_MAX_ASSERTIONS", "60")) +AGENT_CLUSTER_MAX_ASSERTIONS = int(os.getenv("AGENT_CLUSTER_MAX_ASSERTIONS", "24")) +AGENT_CONFLICT_MAX_IMAGES = int(os.getenv("AGENT_CONFLICT_MAX_IMAGES", "6")) +AGENT_CODE_BATCH_SIZE = int(os.getenv("AGENT_CODE_BATCH_SIZE", "60")) +AGENT_BRAIN_MAX_TOKENS = int(os.getenv("AGENT_BRAIN_MAX_TOKENS", "16384")) +AGENT_LINK_CONCURRENCY = int(os.getenv("AGENT_LINK_CONCURRENCY", "4")) +AGENT_CONFLICT_CONCURRENCY = int(os.getenv("AGENT_CONFLICT_CONCURRENCY", "4")) +AGENT_SPECIALIST_CONCURRENCY = int(os.getenv("AGENT_SPECIALIST_CONCURRENCY", "4")) +AGENT_RFI_CONCURRENCY = int(os.getenv("AGENT_RFI_CONCURRENCY", "4")) + +# Wave 5b evidence verification (vision fact-check of cited sheet text) +AGENT_VERIFY_MODEL = os.getenv("AGENT_VERIFY_MODEL", "") or MODEL +AGENT_VERIFY_CONCURRENCY = int(os.getenv("AGENT_VERIFY_CONCURRENCY", "4")) +AGENT_VERIFY_MAX_CHECKS = int(os.getenv("AGENT_VERIFY_MAX_CHECKS", "20")) +AGENT_VERIFY_SEVERITIES = { + s.strip().lower() + for s in os.getenv("AGENT_VERIFY_SEVERITIES", "critical,high").split(",") + if s.strip() +} +AGENT_VERIFY_REASONING_EFFORT = os.getenv("AGENT_VERIFY_REASONING_EFFORT", "low").strip() +VERIFY_MAX_TOKENS = int(os.getenv("VERIFY_MAX_TOKENS", "8192")) + +# -- Text-layer grounding (deterministic PDF text layer via PyMuPDF) ---- +# The vector text layer is extracted once per job and grounds the extractor, +# rescues misquoted-but-real values in the grounding guard, and serves the +# wave-5b verifier as a text oracle plus high-DPI evidence crops. +TEXT_LAYER_ENABLED = os.getenv("TEXT_LAYER_ENABLED", "true").strip().lower() in ("1", "true", "yes") +TEXT_LAYER_MIN_CHARS = int(os.getenv("TEXT_LAYER_MIN_CHARS", "20")) # below this per page -> no text layer +TEXT_LAYER_MAX_CHARS = int(os.getenv("TEXT_LAYER_MAX_CHARS", "12000")) # cap per sheet in extractor prompt +VERIFY_TEXT_MAX_CHARS = int(os.getenv("VERIFY_TEXT_MAX_CHARS", "8000"))# cap of excerpt in verify scope +VERIFY_HI_DPI_CROPS = os.getenv("VERIFY_HI_DPI_CROPS", "true").strip().lower() in ("1", "true", "yes") +VERIFY_CROP_DPI = int(os.getenv("VERIFY_CROP_DPI", "300")) +VERIFY_CROP_MARGIN_PTS = int(os.getenv("VERIFY_CROP_MARGIN_PTS", "36"))# padding around evidence bbox (PDF points) + +# Agent-mode human-review gate. When on (default), Agent runs stop after the +# Brain merge and wait for human decisions before RFIs/final report/email go +# out. AGENT_REVIEW_AUDIT_SAMPLE caps how many clean clusters get added to the +# queue as non-blocking spot-checks. REVIEW_AGGREGATE_INCLUDE_TEXT controls +# whether future cross-job review feedback aggregation may include verbatim +# source_text/images/comments (off by default = privacy-preserving). +AGENT_REQUIRE_REVIEW = os.getenv("AGENT_REQUIRE_REVIEW", "true").strip().lower() in ("1", "true", "yes") +AGENT_REVIEW_AUDIT_SAMPLE = int(os.getenv("AGENT_REVIEW_AUDIT_SAMPLE", "5")) +# NOTE: currently unwired - reserved for future cross-job aggregation tooling. +REVIEW_AGGREGATE_INCLUDE_TEXT = os.getenv("REVIEW_AGGREGATE_INCLUDE_TEXT", "false").strip().lower() in ("1", "true", "yes") + # -- Hybrid (local text LLM) ---------------------------------------- # Optional OpenAI-compatible local endpoint (e.g. a vLLM box) for the text-only # QAQC stages. Vision stages ALWAYS use OpenRouter. The user picks hybrid per @@ -37,7 +97,21 @@ PDF_DPI = int(os.getenv("PDF_DPI", "100")) MAX_PAGES = int(os.getenv("MAX_PAGES", "60")) MAX_DIMENSION = int(os.getenv("MAX_DIMENSION", "2400")) # px cap on the long edge LLM_TIMEOUT = int(os.getenv("LLM_TIMEOUT", "180")) # seconds per call -EXTRACT_MAX_TOKENS = int(os.getenv("EXTRACT_MAX_TOKENS", "16384")) +# Gemini 2.5 Pro counts thinking tokens against max_tokens, so the visible +# JSON budget is well under this number on dense sheets. 65536 is the model's +# output ceiling - give thinking all the room it wants so visible JSON never +# truncates; the thinking budget itself is capped separately below. +EXTRACT_MAX_TOKENS = int(os.getenv("EXTRACT_MAX_TOKENS", "65536")) +# Reasoning effort for the per-sheet extractor (OpenRouter reasoning knob). +# Extraction is perceptive, not deliberative - "low" keeps thinking tokens +# from eating the output budget. Empty string disables the parameter. +EXTRACT_REASONING_EFFORT = os.getenv("EXTRACT_REASONING_EFFORT", "low").strip() +# Hard thinking-token budget for the extractor (OpenRouter reasoning +# max_tokens -> Gemini thinking_budget). "low" effort alone still let Gemini +# burn ~25k thinking tokens per sheet (job 98194fa8d215); a hard cap forces +# the budget into visible output. 0 disables -> falls back to the effort knob. +# Mutually exclusive with effort when set (OpenRouter rejects both together). +EXTRACT_REASONING_MAX_TOKENS = int(os.getenv("EXTRACT_REASONING_MAX_TOKENS", "2048")) REASON_MAX_TOKENS = int(os.getenv("REASON_MAX_TOKENS", "4096")) # -- QAQC stage knobs (Stages 0-1, 3, 6-11) ------------------------- @@ -66,6 +140,14 @@ CLUSTER_MAX = int(os.getenv("CLUSTER_MAX", "120")) LLM_CACHE = os.getenv("LLM_CACHE", "false").strip().lower() in ("1", "true", "yes") LLM_CACHE_DIR = os.getenv("LLM_CACHE_DIR", os.path.join(_BASE_DIR, ".llm_cache")) +# Verbose LLM observability. Per call, one line lands in the job log (model, +# backend, prompt size, response size, parsed-item counts, per-call cost) and +# the full request/response is dumped to /llm_raw/ (base64 image +# payloads excluded; image count recorded instead) so missed or hallucinated +# items can be traced back to exactly what the model saw and returned. +LLM_VERBOSE = os.getenv("LLM_VERBOSE", "true").strip().lower() in ("1", "true", "yes") +LLM_RAW_DUMP = os.getenv("LLM_RAW_DUMP", "true").strip().lower() in ("1", "true", "yes") + # Parallelism (ThreadPoolExecutor workers) EXTRACT_CONCURRENCY = int(os.getenv("EXTRACT_CONCURRENCY", "4")) REASON_CONCURRENCY = int(os.getenv("REASON_CONCURRENCY", "4")) @@ -87,7 +169,10 @@ APP_VERSION = "0.1.0" # Public base URL used to build the "view results" link in notification # emails. Set to whatever address users reach this server on (e.g. the # Tailscale/LAN URL) so the link in the email actually resolves. -APP_BASE_URL = os.getenv("APP_BASE_URL", "http://localhost:8099") +APP_BASE_URL = os.getenv("APP_BASE_URL", "https://conchecker.scoutitsystems.com") +# Build identifier baked into the Docker image by CI (sha-, matching +# the image tag). Shown in the site header and /health. "dev" for local runs. +APP_BUILD = os.getenv("APP_BUILD", "dev") # -- Email / SMTP (optional notification on completion) ------------- # If unset, the app still works; it just logs "SMTP not configured" and diff --git a/backend/email_sender.py b/backend/email_sender.py index 4dfb36b..cbd31b2 100644 --- a/backend/email_sender.py +++ b/backend/email_sender.py @@ -38,6 +38,22 @@ def _send(msg: EmailMessage) -> bool: return False +def send_review_required(recipient_email: str, report: dict, review_url: str) -> bool: + if not recipient_email or not _smtp_ready(): + return False + msg = EmailMessage() + msg["Subject"] = f"Conflict Checker - review required - {report.get('source', 'drawing set')}" + msg["From"] = config.SMTP_FROM or config.SMTP_USER + msg["To"] = recipient_email + review = report.get("summary", {}).get("review", {}) + msg.set_content( + "Agent analysis is complete and waiting for human review.\n\n" + f"Required review items: {review.get('required', 0)}\n" + f"Review URL: {review_url}\n" + ) + return _send(msg) + + def send_conflict_report( recipient_email: str, report: Dict, diff --git a/backend/jobs.py b/backend/jobs.py index 7262623..93d572d 100644 --- a/backend/jobs.py +++ b/backend/jobs.py @@ -14,21 +14,29 @@ A teed stdout/stderr log is kept in memory and written to outputs//job.l so failed or suspicious runs can be reviewed after the fact. """ +import json import os import time +import traceback import uuid import shutil import threading -from typing import Dict, List, Optional +from contextlib import contextmanager +from typing import Dict, Iterator, List, Optional from backend import config +from backend import llm from backend.job_log import capture_stdio, read_log_file, stamp_line +from backend.agents.runner import run_agent_pipeline from backend.pipeline.runner import run_pipeline -from backend.email_sender import send_conflict_report +from backend.email_sender import send_conflict_report, send_review_required _jobs: Dict[str, Dict] = {} _lock = threading.Lock() _LOG_TAIL = 80 +PIPELINE_MODES = {"classic", "agent"} +# States where the job will produce no more log output; polls get the full log. +_TERMINAL_STATES = {"done", "error", "needs_review", "finalization_error"} def _set(job_id: str, **fields) -> None: @@ -59,19 +67,26 @@ def create_job( email: Optional[str] = None, project_input: Optional[Dict] = None, text_local: bool = False, + pipeline_mode: str = "classic", vision_model: Optional[str] = None, text_model: Optional[str] = None, ) -> str: """Register a job and kick off its background thread. Returns the job_id.""" + pipeline_mode = pipeline_mode.strip().lower() + if pipeline_mode not in PIPELINE_MODES: + raise ValueError(f"Unsupported pipeline mode: {pipeline_mode!r}") + # Agent mode v1 is OpenRouter-only. + text_local = bool(text_local and pipeline_mode == "classic") job_id = uuid.uuid4().hex[:12] with _lock: _jobs[job_id] = { "job_id": job_id, - "status": "queued", # queued -> running -> done | error + "status": "queued", # queued -> running -> done | needs_review | error "source": source_filename, "email": email or None, "project_input": project_input or {}, "text_local": text_local, + "pipeline_mode": pipeline_mode, "vision_model": (vision_model or "").strip() or None, "text_model": (text_model or "").strip() or None, "stage": None, @@ -83,7 +98,8 @@ def create_job( } threading.Thread( target=_run, - args=(job_id, pdf_path, project_input, text_local, vision_model, text_model), + args=(job_id, pdf_path, project_input, text_local, pipeline_mode, + vision_model, text_model), daemon=True, ).start() return job_id @@ -94,6 +110,7 @@ def _run( pdf_path: str, project_input: Optional[Dict] = None, text_local: bool = False, + pipeline_mode: str = "classic", vision_model: Optional[str] = None, text_model: Optional[str] = None, ) -> None: @@ -101,33 +118,34 @@ def _run( log_path = os.path.join(out_dir, "job.log") try: _set(job_id, status="running") - # Keep a copy of the source PDF so its sheets can be viewed later. os.makedirs(out_dir, exist_ok=True) # Truncate any leftover log if job_id somehow collided (shouldn't). with open(log_path, "w", encoding="utf-8"): pass - shutil.copy2(pdf_path, os.path.join(out_dir, "source.pdf")) + header = (f"=== Job {job_id} | {pipeline_mode} | {_jobs[job_id].get('source')} | " + f"vision={vision_model or 'default'} text={text_model or 'default'} | " + f"started {time.strftime('%Y-%m-%d %H:%M:%S %Z', time.gmtime())} UTC ===") + _append_log(job_id, header, log_path) def on_line(raw: str) -> None: _append_log(job_id, raw, log_path) with capture_stdio(on_line): - report = run_pipeline( - pdf_path, - out_dir=out_dir, - on_stage=lambda name: _set(job_id, stage=name), - project_input=project_input, - source_name=_jobs[job_id].get("source"), - text_local=text_local, - vision_model=vision_model, - text_model=text_model, - ) - _set(job_id, status="done", report=report, finished_at=time.time(), stage=None) - _notify(job_id, report, out_dir) + _run_pipeline(job_id, pdf_path, out_dir, project_input, text_local, + pipeline_mode, vision_model, text_model) except Exception as e: - # Also land in the job log via print under the tee when possible. + # Land the failure AND its traceback in the job log so failed runs can + # be diagnosed from the log alone (the stdio tee is already torn down). try: _append_log(job_id, f"[Jobs] Job {job_id} failed: {e}", log_path) + for ln in traceback.format_exc().rstrip().splitlines(): + _append_log(job_id, ln, log_path) + # Cost-so-far for the failed run (counters are reset per job). + cost = llm.get_cost() + _append_log(job_id, + f"[Jobs] Estimated LLM cost before failure: " + f"${cost['usd']:.4f} over {cost['calls']} live calls " + f"({cost.get('cached', 0)} cached)", log_path) except Exception: pass print(f"[Jobs] Job {job_id} failed: {e}") @@ -140,6 +158,117 @@ def _run( pass +def _run_pipeline(job_id: str, pdf_path: str, out_dir: str, + project_input: Optional[Dict], text_local: bool, + pipeline_mode: str, vision_model: Optional[str], + text_model: Optional[str]) -> None: + """The body of a job run; executes inside the job's tee'd log capture.""" + # Persist minimal job metadata so the disk fallback in get_job can + # recover the recipient email / pipeline mode after a server restart + # (plain json.dump, matching the _dump style used elsewhere). + with open(os.path.join(out_dir, "job.json"), "w", encoding="utf-8") as f: + json.dump({ + "job_id": job_id, + "email": _jobs[job_id].get("email"), + "pipeline_mode": pipeline_mode, + "source": _jobs[job_id].get("source"), + "vision_model": vision_model, + "text_model": text_model, + }, f, indent=2) + # Keep a copy of the source PDF so its sheets can be viewed later. + shutil.copy2(pdf_path, os.path.join(out_dir, "source.pdf")) + # Raw per-call LLM request/response dumps land in /llm_raw/. + if config.LLM_RAW_DUMP: + llm.set_raw_dump_dir(os.path.join(out_dir, "llm_raw")) + runner = run_agent_pipeline if pipeline_mode == "agent" else run_pipeline + runner_kwargs = { + "out_dir": out_dir, + "on_stage": lambda name: _set(job_id, stage=name), + "project_input": project_input, + "source_name": _jobs[job_id].get("source"), + } + if pipeline_mode == "classic": + # run_pipeline takes the picks as params and clears them in finally. + runner_kwargs["text_local"] = text_local + runner_kwargs["vision_model"] = vision_model + runner_kwargs["text_model"] = text_model + else: + runner_kwargs["require_review"] = config.AGENT_REQUIRE_REVIEW + # The agent runner has no override params; set them module-level. + if vision_model or text_model: + print(f"[Jobs] Model overrides for this run: " + f"vision={vision_model or '(default)'} text={text_model or '(default)'}") + llm.set_model_overrides(vision_model, text_model) + try: + report = runner(pdf_path, **runner_kwargs) + finally: + llm.set_raw_dump_dir(None) + if pipeline_mode == "agent": + llm.set_model_overrides(None, None) + report.setdefault("summary", {})["pipeline_mode"] = pipeline_mode + _log_cost_summary(report.get("summary", {})) + if report["summary"].get("agent_status") == "needs_review": + # Human-review gate: hold the job, don't email the unreviewed report. + _set(job_id, status="needs_review", report=report, + finished_at=time.time(), stage=None) + email = _jobs[job_id].get("email") + if email: + review_url = f"{config.APP_BASE_URL.rstrip('/')}/?job={job_id}" + send_review_required(email, report, review_url) + else: + _set(job_id, status="done", report=report, finished_at=time.time(), stage=None) + _notify(job_id, report, out_dir) + + +def _log_cost_summary(summary: Dict, label: str = "this run") -> None: + """ + End-of-log estimated LLM cost block, printed inside the job's stdio tee so + it lands at the tail of job.log. Both runners populate the same summary + fields (cost_usd / llm_calls / cached_calls / cost_by_stage / models_used). + Hybrid note: local text calls carry no usage accounting, so the dollar + figure covers OpenRouter calls only (local call counts still appear). + """ + if summary.get("cost_usd") is None and not summary.get("llm_calls"): + return + print(f"\n=== Estimated LLM cost ({label}) ===") + print(f" Total: ${summary.get('cost_usd', 0.0):.4f} across " + f"{summary.get('llm_calls', 0)} live calls " + f"({summary.get('cached_calls', 0)} cached at $0)") + for name, bucket in (summary.get("cost_by_stage") or {}).items(): + print(f" {name}: ${bucket.get('usd', 0.0):.4f} " + f"({bucket.get('calls', 0)} live, {bucket.get('cached', 0)} cached)") + mu = summary.get("models_used") or {} + if mu.get("vision"): + print(f" Vision models: {', '.join(mu['vision'])}") + if mu.get("text_local"): + print(f" Text models (local): {', '.join(mu['text_local'])}") + if mu.get("text_cloud"): + print(f" Text models (cloud): {', '.join(mu['text_cloud'])}") + if mu.get("fallback_count"): + print(f" Local->cloud fallbacks: {mu['fallback_count']}") + if mu.get("text_local"): + print(" Note: local calls have no cost accounting; " + "the dollar total covers OpenRouter usage only.") + + +@contextmanager +def capture_job_output(job_id: str, out_dir: str) -> Iterator[None]: + """ + Re-open the stdio tee + raw LLM dump dir for post-run work that still + belongs to this job (review finalization): lines append to job.log and + the in-memory log, raw dumps resume under /llm_raw/. The tee is + process-global — same overlapping-job caveat as the main run. + """ + log_path = os.path.join(out_dir, "job.log") + if config.LLM_RAW_DUMP: + llm.set_raw_dump_dir(os.path.join(out_dir, "llm_raw")) + try: + with capture_stdio(lambda raw: _append_log(job_id, raw, log_path)): + yield + finally: + llm.set_raw_dump_dir(None) + + def _notify(job_id: str, report: Dict, out_dir: str) -> None: email = _jobs[job_id].get("email") if not email: @@ -159,7 +288,6 @@ def _notify_error(job_id: str) -> None: email = job.get("email") if not email: return - # Reuse the report mailer with a minimal error-shaped payload. try: from backend.email_sender import _smtp_ready, _send from email.message import EmailMessage @@ -204,8 +332,8 @@ def get_job_log(job_id: str) -> Optional[List[str]]: def get_job(job_id: str) -> Optional[Dict]: """Public job view. Includes the full report only when done. - Falls back to the on-disk conflicts.json when the job isn't in the - in-memory registry (e.g. after a server restart). + Falls back to the on-disk artifacts (conflicts.json / job.log) when the + job isn't in the in-memory registry (e.g. after a server restart). """ with _lock: job = _jobs.get(job_id) @@ -215,7 +343,7 @@ def get_job(job_id: str) -> Optional[Dict]: out["log_tail"] = log[-_LOG_TAIL:] # Full log on terminal states so the UI can show it without a # second fetch; keep polls light while running. - if out.get("status") in ("done", "error"): + if out.get("status") in _TERMINAL_STATES: out["log"] = log else: out.pop("log", None) @@ -227,30 +355,53 @@ def get_job(job_id: str) -> Optional[Dict]: if not os.path.isfile(report_path) and not log: return None try: - import json report = None if os.path.isfile(report_path): with open(report_path, encoding="utf-8") as f: report = json.load(f) + summary = (report or {}).get("summary", {}) + if report is None: + # Crashed before writing a report; the log is the only artifact. + status = "error" + else: + # Recover the job's real state: a job that stopped at the review gate + # must come back as needs_review (not done) or it can never finalize. + status = "needs_review" if summary.get("agent_status") == "needs_review" else "done" + # job.json (written at job start) carries the recipient email and + # pipeline mode so the final notification still fires after a restart. + # Missing/corrupt job.json degrades to the previous derivations. + meta: Dict = {} + meta_path = os.path.join(config.OUTPUT_DIR, job_id, "job.json") + try: + with open(meta_path, encoding="utf-8") as f: + loaded = json.load(f) + if isinstance(loaded, dict): + meta = loaded + except (OSError, json.JSONDecodeError): + pass source_pdf = os.path.join(config.OUTPUT_DIR, job_id, "source.pdf") - status = "done" if report is not None else "error" - return { - "job_id": job_id, - "status": status, - "source": (report or {}).get("source", os.path.basename(report_path)), - "email": None, + job = { + "job_id": job_id, + "status": status, + "source": meta.get("source") or (report or {}).get("source", os.path.basename(report_path)), + "email": meta.get("email"), "project_input": (report or {}).get("project_input", {}), - "text_local": (report or {}).get("summary", {}).get("text_backend") == "local", - "vision_model": None, - "text_model": None, - "stage": None, - "created_at": os.path.getmtime(source_pdf) if os.path.isfile(source_pdf) else None, - "finished_at": os.path.getmtime(report_path) if os.path.isfile(report_path) else None, - "report": report, - "error": None if report is not None else "Report missing; see job log", - "log": log, - "log_tail": log[-_LOG_TAIL:], + "text_local": summary.get("text_backend") == "local", + "pipeline_mode": meta.get("pipeline_mode") or summary.get("pipeline_mode", "classic"), + "vision_model": meta.get("vision_model"), + "text_model": meta.get("text_model"), + "stage": None, + "created_at": os.path.getmtime(source_pdf) if os.path.isfile(source_pdf) else None, + "finished_at": os.path.getmtime(report_path) if os.path.isfile(report_path) else None, + "report": report, + "error": None if report is not None else "Report missing; see job log", + "log": log, + "log_tail": log[-_LOG_TAIL:], } + # Hydrate the in-memory registry so _set(...) transitions (reviewing, + # finalizing, done) work for restart-recovered jobs. + with _lock: + return dict(_jobs.setdefault(job_id, job)) except Exception as e: print(f"[Jobs] Failed to load job {job_id} from disk: {e}") return None diff --git a/backend/llm.py b/backend/llm.py index 14537bb..1380817 100644 --- a/backend/llm.py +++ b/backend/llm.py @@ -8,6 +8,7 @@ with an image) and conflict-reasoning (Stage 3, with images) calls. """ import os +import re import json import hashlib import threading @@ -23,10 +24,116 @@ _clients: Dict[str, OpenAI] = {} # call_json when routing a no-image (text) call. Module-global mirrors the # set_stage/cost pattern (single-user tool). _text_local = False -# Optional per-run model overrides from the UI (empty = use config defaults). +# Per-job model overrides (user picked models in the UI). Same module-global +# pattern: set by the job runner before the pipeline starts, cleared after. +# Vision applies to image calls, text to no-image calls on OpenRouter (and to +# the local->cloud fallback). The LOCAL endpoint's model name is never taken +# from these overrides - hybrid local keeps LOCAL_TEXT_MODEL. _vision_model_override: Optional[str] = None _text_model_override: Optional[str] = None +# Per-job raw request/response dumps (missed/hallucinated-item debugging). +# Set by the job runner to /llm_raw at job start, cleared after. +# Same module-global pattern as the model overrides (single-user tool). +_raw_dump_dir: Optional[str] = None +_seq_lock = threading.Lock() +_call_seq = 0 + + +def set_raw_dump_dir(path: Optional[str]) -> None: + """Point raw LLM request/response dumps at a directory. None disables.""" + global _raw_dump_dir, _call_seq + with _seq_lock: + _raw_dump_dir = path + _call_seq = 0 + + +def _next_seq() -> int: + global _call_seq + with _seq_lock: + _call_seq += 1 + return _call_seq + + +def _summarize_parsed(parsed: Any) -> str: + """ + Compact digest of a parsed response for the job log. List values become + item counts (e.g. conflicts[3]) so a stage that returned nothing (miss) + or invented items (hallucination) is visible without opening the raw dump. + """ + if isinstance(parsed, list): + return f"list[{len(parsed)}]" + if not isinstance(parsed, dict): + return type(parsed).__name__ + parts = [] + for k, v in parsed.items(): + if isinstance(v, list): + parts.append(f"{k}[{len(v)}]") + elif isinstance(v, dict): + parts.append(f"{k}{{{len(v)}}}") + else: + s = str(v) + parts.append(f"{k}={s[:40]!r}{'...' if len(s) > 40 else ''}") + out = ", ".join(parts) + return out[:300] + ("..." if len(out) > 300 else "") + + +def _dump_raw(seq: int, be: Dict[str, Any], usage_stage: str, + system_prompt: str, user_text: str, + images_b64: Optional[List[str]], max_tokens: int, + raw: str, parsed: Any, + finish_reason: Optional[str] = None) -> None: + """Write the full request/response for one call to the job's llm_raw dir.""" + if not _raw_dump_dir: + return + try: + os.makedirs(_raw_dump_dir, exist_ok=True) + safe_stage = re.sub(r"[^A-Za-z0-9_.-]+", "_", usage_stage)[:40] + safe_model = re.sub(r"[^A-Za-z0-9_.-]+", "_", be["model"]) + payload = { + "seq": seq, + "stage": usage_stage, + "model": be["model"], + "backend": "local" if be.get("local") else "cloud", + "max_tokens": max_tokens, + # base64 image payloads deliberately excluded (multi-MB each); + # the count + source.pdf in the job dir identify what was sent. + "n_images": len(images_b64 or []), + "system_prompt": system_prompt, + "user_text": user_text, + "finish_reason": finish_reason, + "raw_response": raw, + "parsed": parsed, + } + path = os.path.join(_raw_dump_dir, f"{seq:04d}_{safe_stage}_{safe_model}.json") + tmp = f"{path}.{threading.get_ident()}.tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2) + os.replace(tmp, path) # atomic so thread-pooled stages can't tear it + except OSError as e: + print(f"[LLM] raw dump failed: {e}") + + +def _log_call(seq: int, be: Dict[str, Any], usage_stage: str, + user_text: str, images_b64: Optional[List[str]], + raw: str, parsed: Any, usd: Optional[float], + cached: bool = False, + reasoning_tokens: Optional[int] = None) -> None: + """One verbose per-call line for the job log (tee'd by job_log.py).""" + if not config.LLM_VERBOSE: + return + backend = "local" if be.get("local") else "cloud" + if cached: + cost_str = "cache hit" + elif usd is not None: + cost_str = f"${usd:.4f}" + else: + cost_str = "cost n/a" + think_str = f"think {reasoning_tokens}tk | " if reasoning_tokens is not None else "" + print(f"[LLM] #{seq:04d} {usage_stage} | {be['model']} ({backend}) | " + f"in {len(user_text)}ch+{len(images_b64 or [])}img | " + f"out {len(raw)}ch | {think_str}{cost_str} | {_summarize_parsed(parsed)}") + def set_text_backend(local: bool) -> None: """Choose whether text (no-image) calls go to the local endpoint this run.""" @@ -35,7 +142,7 @@ def set_text_backend(local: bool) -> None: def set_model_overrides(vision: Optional[str] = None, text: Optional[str] = None) -> None: - """Per-run OpenRouter model picks. None/blank clears back to config defaults.""" + """Per-run OpenRouter vision/text model picks. None/blank clears to defaults.""" global _vision_model_override, _text_model_override _vision_model_override = (vision or "").strip() or None _text_model_override = (text or "").strip() or None @@ -125,9 +232,11 @@ def _add_cached() -> None: def _cache_key(model: str, system_prompt: str, user_text: str, images_b64: Optional[List[str]], max_tokens: int, - json_mode: bool) -> str: + json_mode: bool, reasoning_effort: Optional[str] = None, + reasoning_max_tokens: Optional[int] = None) -> str: h = hashlib.sha256() - parts = [model, str(max_tokens), str(json_mode), system_prompt, user_text] + parts = [model, str(max_tokens), str(json_mode), str(reasoning_effort), + str(reasoning_max_tokens), system_prompt, user_text] for b in (images_b64 or []): parts.append(b) for p in parts: @@ -177,20 +286,22 @@ def _resolve_backend(has_images: bool, model_override: Optional[str]) -> Dict[st return { "base_url": config.LOCAL_BASE_URL, "api_key": config.LOCAL_API_KEY, - "model": (model_override or _text_model_override - or config.LOCAL_TEXT_MODEL or config.TEXT_MODEL), + # Local model name comes from per-call args or LOCAL_TEXT_MODEL — + # never the UI's OpenRouter picks, which a local server won't serve. + "model": model_override or config.LOCAL_TEXT_MODEL or config.TEXT_MODEL, "usage": False, # local has no OpenRouter usage accounting "local": True, } - # Vision, or text-on-OpenRouter (default / fallback). + # Vision, or text-on-OpenRouter (default / fallback). A per-job override + # (user's UI model pick) wins over per-call and env defaults. if has_images: - default_model = _vision_model_override or config.MODEL + model = _vision_model_override or model_override or config.MODEL else: - default_model = _text_model_override or config.TEXT_MODEL + model = _text_model_override or model_override or config.TEXT_MODEL return { "base_url": config.AI_BASE_URL, "api_key": config.AI_API_KEY, - "model": model_override or default_model, + "model": model, "usage": True, "local": False, } @@ -251,18 +362,32 @@ def _repair_truncated(raw: str) -> Optional[Dict[str, Any]]: return None -def _record_cost(response) -> None: +def _response_cost(response) -> Optional[float]: """Pull OpenRouter's per-call USD cost out of the usage object, if present.""" try: dump = response.model_dump() except Exception: - return + return None usage = dump.get("usage") or {} cost = usage.get("cost") if cost is None: cost = (usage.get("cost_details") or {}).get("upstream_inference_cost") - if isinstance(cost, (int, float)): - _add_cost(float(cost)) + return float(cost) if isinstance(cost, (int, float)) else None + + +def _reasoning_tokens(response) -> Optional[int]: + """Hidden thinking tokens for this call (OpenRouter usage details). + + Direct evidence of whether the reasoning knob is working - without it, + thinking burn can only be inferred from char counts vs the token cap.""" + try: + dump = response.model_dump() + except Exception: + return None + usage = dump.get("usage") or {} + details = usage.get("completion_tokens_details") or {} + n = details.get("reasoning_tokens") + return int(n) if isinstance(n, (int, float)) else None def _parse(raw: str) -> Optional[Dict[str, Any]]: @@ -282,6 +407,10 @@ def call_json( images_b64: Optional[List[str]] = None, max_tokens: int = 4096, model: Optional[str] = None, + usage_tracker: Optional[Any] = None, + usage_stage: str = "?", + reasoning_effort: Optional[str] = None, + reasoning_max_tokens: Optional[int] = None, ) -> Optional[Dict[str, Any]]: """ Send one chat completion expecting a JSON object back. @@ -289,18 +418,32 @@ def call_json( Uses the provider's JSON mode (response_format) so the model returns a bare JSON object instead of prose/empty text, and recovers from max_tokens truncation. images_b64: optional base64 JPEGs attached as high-detail image - parts. Returns the parsed dict, or None on a hard failure (caller degrades). + parts. reasoning_effort: optional OpenRouter reasoning knob ("low"/"medium"/ + "high") - keeps thinking models from burning the output budget on hidden + reasoning. reasoning_max_tokens: optional hard thinking-token budget + (OpenRouter reasoning max_tokens -> Gemini thinking_budget); stronger than + effort, and takes precedence when both are given. Returns the parsed dict, + or None on a hard failure (caller degrades). """ has_images = bool(images_b64) be = _resolve_backend(has_images, model) + seq = _next_seq() cache_key = None if config.LLM_CACHE: cache_key = _cache_key(be["model"], system_prompt, user_text, - images_b64, max_tokens, json_mode=True) + images_b64, max_tokens, json_mode=True, + reasoning_effort=reasoning_effort, + reasoning_max_tokens=reasoning_max_tokens) hit = _cache_get(cache_key) if hit is not None: _add_cached() + if usage_tracker: + usage_tracker.record( + usage_stage, be["model"], cached=True, has_images=has_images + ) + _log_call(seq, be, usage_stage, user_text, images_b64, + "", hit, None, cached=True) return hit content: List[Dict[str, Any]] = [] @@ -321,20 +464,53 @@ def call_json( for attempt in range(2): try: client = get_client(be["base_url"], be["api_key"]) - kwargs = dict(model=be["model"], messages=messages, - max_tokens=max_tokens, timeout=config.LLM_TIMEOUT) + kwargs: Dict[str, Any] = dict(model=be["model"], messages=messages, + max_tokens=max_tokens, timeout=config.LLM_TIMEOUT) + extra_body: Dict[str, Any] = {} if be["usage"]: - kwargs["extra_body"] = {"usage": {"include": True}} + extra_body["usage"] = {"include": True} + # OpenRouter reasoning knob; only sent to cloud backends (local + # servers reject unknown fields). A hard thinking budget wins over + # the vaguer effort tier - OpenRouter treats them as exclusive. + if not be.get("local"): + if reasoning_max_tokens: + extra_body["reasoning"] = {"max_tokens": int(reasoning_max_tokens)} + elif reasoning_effort: + extra_body["reasoning"] = {"effort": reasoning_effort} + if extra_body: + kwargs["extra_body"] = extra_body if use_json_mode: kwargs["response_format"] = {"type": "json_object"} response = client.chat.completions.create(**kwargs) - _record_cost(response) - raw = _strip_fences(response.choices[0].message.content or "") + usd = _response_cost(response) + if usd is not None: + _add_cost(usd) + if usage_tracker: + usage_tracker.record( + usage_stage, be["model"], usd=usd or 0.0, has_images=has_images + ) + choice = response.choices[0] + finish_reason = getattr(choice, "finish_reason", None) + raw = _strip_fences(choice.message.content or "") + reasoning_tokens = _reasoning_tokens(response) + if finish_reason == "length": + # Hit max_tokens (thinking tokens included on reasoning models). + # Logged explicitly so silent truncation isn't mistaken for a + # parse problem; _parse below still salvages what it can. + print(f"[LLM] output hit max_tokens (finish_reason=length, " + f"{len(raw)}ch returned, " + f"thinking={reasoning_tokens}tk)") parsed = _parse(raw) if parsed is not None: _record_model(be, has_images, fell_back) if cache_key: _cache_set(cache_key, parsed) + _log_call(seq, be, usage_stage, user_text, images_b64, + raw, parsed, usd, reasoning_tokens=reasoning_tokens) + if config.LLM_RAW_DUMP: + _dump_raw(seq, be, usage_stage, system_prompt, user_text, + images_b64, max_tokens, raw, parsed, + finish_reason=finish_reason) return parsed if attempt == 0: print("[LLM] JSON parse error (retrying)") diff --git a/backend/main.py b/backend/main.py index 25251e2..8145c44 100644 --- a/backend/main.py +++ b/backend/main.py @@ -11,16 +11,21 @@ ever needs concurrency. import os import tempfile +import threading +import time from typing import Optional from fastapi import FastAPI, UploadFile, File, Form, HTTPException -from fastapi.responses import HTMLResponse, JSONResponse, Response, PlainTextResponse +from fastapi.responses import HTMLResponse, JSONResponse, Response from fastapi.staticfiles import StaticFiles -from backend import config -from backend.jobs import create_job, get_job, get_job_log -from backend.models_catalog import list_models +import backend.jobs +from backend import config, llm +from backend.jobs import PIPELINE_MODES, create_job, get_job, _set from backend.pipeline.pdf_processor import render_page_jpeg +from backend.review.feedback import decision_to_label, write_label +from backend.review.finalizer import finalize_review +from backend.review.store import ReviewStore app = FastAPI(title=config.APP_TITLE, version=config.APP_VERSION) @@ -31,14 +36,33 @@ _FRONTEND_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", " def health(): return {"status": "ok", "model": config.MODEL, "text_model": config.TEXT_MODEL, + "version": config.APP_VERSION, + "build": config.APP_BUILD, "key_configured": bool(config.AI_API_KEY), "email_configured": bool(config.SMTP_HOST and config.SMTP_USER and config.SMTP_PASSWORD)} @app.get("/models") -def models(): - """Vision vs text OpenRouter model lists for the UI dropdowns.""" - return JSONResponse(list_models()) +def list_models(): + """Vision/text OpenRouter model lists with pricing for the UI dropdowns.""" + from backend.models import fetch_models, split_vision_text + models = fetch_models() + if models is None: + raise HTTPException(status_code=502, + detail="Could not fetch the model list from OpenRouter") + vision, text = split_vision_text(models) + return {"vision": vision, "text": text, + "defaults": {"vision": config.MODEL, "text": config.TEXT_MODEL}} + + +@app.get("/jobs/{job_id}/log") +def job_log(job_id: str): + """The full captured stdout/stderr log of a job run (persists on disk).""" + path = os.path.join(config.OUTPUT_DIR, job_id, "job.log") + if not os.path.isfile(path): + raise HTTPException(status_code=404, detail="Log not found for this job") + with open(path, encoding="utf-8", errors="replace") as f: + return Response(content=f.read(), media_type="text/plain") @app.post("/check") @@ -50,6 +74,7 @@ async def check( occupancy: Optional[str] = Form(None), work_type: Optional[str] = Form(None), text_local: bool = Form(False), + pipeline_mode: str = Form("classic"), vision_model: Optional[str] = Form(None), text_model: Optional[str] = Form(None), ): @@ -66,6 +91,12 @@ async def check( """ if not file.filename.lower().endswith(".pdf"): raise HTTPException(status_code=400, detail="Please upload a PDF.") + pipeline_mode = pipeline_mode.strip().lower() + if pipeline_mode not in PIPELINE_MODES: + raise HTTPException( + status_code=400, + detail=f"pipeline_mode must be one of: {', '.join(sorted(PIPELINE_MODES))}", + ) os.makedirs(config.UPLOAD_DIR, exist_ok=True) suffix = "_" + os.path.basename(file.filename) @@ -84,16 +115,16 @@ async def check( } v_model = (vision_model or "").strip() or None t_model = (text_model or "").strip() or None - job_id = create_job( - tmp_path, - source_filename=file.filename, - email=email, - project_input=project_input, - text_local=text_local, - vision_model=v_model, - text_model=t_model, - ) - return JSONResponse({"job_id": job_id, "status": "queued", "email": email}) + job_id = create_job(tmp_path, source_filename=file.filename, email=email, + project_input=project_input, text_local=text_local, + pipeline_mode=pipeline_mode, vision_model=v_model, + text_model=t_model) + return JSONResponse({ + "job_id": job_id, + "status": "queued", + "email": email, + "pipeline_mode": pipeline_mode, + }) @app.get("/jobs/{job_id}") @@ -104,15 +135,121 @@ def job_status(job_id: str): return JSONResponse(job) -@app.get("/jobs/{job_id}/log") -def job_log(job_id: str, plain: bool = False): - """Full captured run log (also on disk as outputs//job.log).""" - lines = get_job_log(job_id) - if lines is None: +@app.get("/jobs/{job_id}/review") +def review_queue(job_id: str): + job = get_job(job_id) + if not job: raise HTTPException(status_code=404, detail="Job not found") - if plain: - return PlainTextResponse("\n".join(lines) + ("\n" if lines else "")) - return JSONResponse({"job_id": job_id, "lines": lines, "text": "\n".join(lines)}) + out_dir = job.get("out_dir") or os.path.join(config.OUTPUT_DIR, job_id) + # Read-only endpoint: don't create review/ dirs just by looking at them + # (readers already degrade to empty on missing files). + store = ReviewStore(out_dir, create=False) + queue = store.read_queue() + return {"queue": queue, "progress": store.progress(queue), + "decisions": store.read_decisions()} + + +@app.post("/jobs/{job_id}/review-decisions") +def save_review_decisions(job_id: str, payload: dict): + job = get_job(job_id) + if not job: + raise HTTPException(status_code=404, detail="Job not found") + if job.get("status") not in ("needs_review", "reviewing"): + # Positive state guard, mirroring the finalize endpoint: only jobs + # sitting at (or working through) the review gate accept decisions. + raise HTTPException(status_code=409, detail={ + "detail": f"cannot save review decisions for a job in status {job.get('status')}", + }) + out_dir = job.get("out_dir") or os.path.join(config.OUTPUT_DIR, job_id) + store = ReviewStore(out_dir) + queue = store.read_queue() + items_by_id = {item.get("review_item_id"): item for item in queue} + saved = 0 + try: + for decision in payload.get("decisions") or []: + store.append_decision(decision) + queue_item = items_by_id.get(decision.get("review_item_id")) + if queue_item is not None: + write_label(out_dir, decision_to_label(queue_item, decision, job)) + saved += 1 + except ValueError as e: + raise HTTPException(status_code=422, detail=str(e)) + progress = store.progress(queue) + if job.get("status") == "needs_review" and saved > 0 and progress["remaining"] > 0: + try: + _set(job_id, status="reviewing") + except KeyError: + pass # job not in the in-memory registry (e.g. loaded from disk) + return {"progress": progress} + + +def _finalize_job(job_id: str, out_dir: str) -> None: + """Background finalization: the ONE place the final report email may fire.""" + try: + # Re-open the job's log tee + raw dump dir so the finalization LLM + # calls (clarification reruns, RFI drafting) land in job.log / llm_raw. + with backend.jobs.capture_job_output(job_id, out_dir): + print("\n=== Review finalization ===") + llm.reset_cost() # finalization-only cost attribution + report = finalize_review(job_id, out_dir) + cost = llm.get_cost() + backend.jobs._log_cost_summary({ + "cost_usd": round(cost["usd"], 4), + "llm_calls": cost["calls"], + "cached_calls": cost.get("cached", 0), + "cost_by_stage": cost.get("by_stage", {}), + "models_used": cost.get("models", {}), + }, label="finalization") + except Exception as e: + try: + _set(job_id, status="finalization_error", error=str(e), + finished_at=time.time(), stage=None) + except KeyError: + pass # job not in the in-memory registry + return + try: + _set(job_id, status="done", report=report, + finished_at=time.time(), stage=None) + except KeyError: + pass + try: + backend.jobs._notify(job_id, report, out_dir) + except Exception as e: + print(f"[Jobs] Final notification for {job_id} failed: {e}") + + +@app.post("/jobs/{job_id}/finalize-review") +def finalize_review_endpoint(job_id: str): + job = get_job(job_id) + if not job: + raise HTTPException(status_code=404, detail="Job not found") + if job.get("status") in ("done", "finalizing"): + raise HTTPException(status_code=409, detail={ + "detail": f"job is already {job['status']}", + }) + if job.get("status") not in ("needs_review", "reviewing", "finalization_error"): + # Positive state-machine guard: finalization (and the final email) is + # only reachable after the job has passed through the review gate. + raise HTTPException(status_code=409, detail={ + "detail": f"cannot finalize a job in status {job.get('status')}", + }) + out_dir = job.get("out_dir") or os.path.join(config.OUTPUT_DIR, job_id) + store = ReviewStore(out_dir) + queue = store.read_queue() + decisions = store.read_decisions() + if any(item.get("blocking") and item.get("review_item_id") not in decisions + for item in queue): + # 409 detail shape: {"detail": , "progress": } + raise HTTPException(status_code=409, detail={ + "detail": "incomplete review", + "progress": store.progress(queue), + }) + try: + _set(job_id, status="finalizing") + except KeyError: + pass # job not in the in-memory registry (e.g. loaded from disk) + threading.Thread(target=_finalize_job, args=(job_id, out_dir), daemon=True).start() + return {"status": "finalizing"} @app.get("/jobs/{job_id}/sheet-image/{page}") diff --git a/backend/models.py b/backend/models.py new file mode 100644 index 0000000..596710b --- /dev/null +++ b/backend/models.py @@ -0,0 +1,90 @@ +""" +models.py - Fetch the available OpenRouter model list with pricing (cached). + +The /models endpoint is public (no API key needed). Results are normalized to +per-1M-token USD costs for display and cached in memory for an hour; callers +degrade gracefully when OpenRouter is unreachable. Each entry also carries a +vision flag (accepts image input) so the UI can offer separate vision/text +model dropdowns. +""" + +import time +from typing import List, Optional, Tuple + +import httpx + +from backend import config + +_CACHE_TTL_SECONDS = 3600 +_cache = {"at": 0.0, "models": None} + + +def _per_mtok(rate) -> float: + """OpenRouter pricing is USD per token (as a string); display is per 1M.""" + try: + return round(float(rate) * 1_000_000, 4) + except (TypeError, ValueError): + return 0.0 + + +def _is_vision(item: dict) -> bool: + """True when the model accepts image input and produces text output.""" + arch = item.get("architecture") or {} + inputs = arch.get("input_modalities") or [] + outputs = arch.get("output_modalities") or [] + # Legacy string form: "text+image->text" + modality = (arch.get("modality") or "").lower() + has_image_in = ("image" in inputs) or ("image" in modality.split("->")[0]) + has_text_out = ("text" in outputs) or ("->text" in modality) or (not outputs and not modality) + return has_image_in and has_text_out + + +def _fetch_openrouter_models() -> Optional[List[dict]]: + """Raw GET of the OpenRouter model list; None on any failure.""" + try: + response = httpx.get(f"{config.AI_BASE_URL.rstrip('/')}/models", timeout=10) + response.raise_for_status() + data = response.json().get("data") + return data if isinstance(data, list) else None + except Exception as e: + print(f"[Models] OpenRouter /models fetch failed: {e}") + return None + + +def fetch_models(force: bool = False) -> Optional[List[dict]]: + """Normalized model list for the UI picker, or None when unavailable.""" + if ( + not force + and _cache["models"] is not None + and time.time() - _cache["at"] < _CACHE_TTL_SECONDS + ): + return _cache["models"] + data = _fetch_openrouter_models() + if data is None: + return None + models = [ + { + "id": item.get("id") or "", + "name": item.get("name") or item.get("id") or "", + "prompt_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("prompt")), + "completion_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("completion")), + "context_length": item.get("context_length"), + "vision": _is_vision(item), + } + for item in data + if item.get("id") + ] + models.sort(key=lambda m: m["id"]) + _cache["models"] = models + _cache["at"] = time.time() + return models + + +def split_vision_text(models: List[dict]) -> Tuple[List[dict], List[dict]]: + """Partition the normalized catalog into (vision, text) lists for the UI. + + Every catalog model takes text in/out, so vision models appear in both + lists (same dicts, pricing included). + """ + vision = [m for m in models if m.get("vision")] + return vision, list(models) diff --git a/backend/models_catalog.py b/backend/models_catalog.py deleted file mode 100644 index 0f7d811..0000000 --- a/backend/models_catalog.py +++ /dev/null @@ -1,101 +0,0 @@ -""" -models_catalog.py - OpenRouter model list for the UI dropdowns. - -Fetches https://openrouter.ai/api/v1/models (cached ~1h) and splits into: - - vision: accepts image input and returns text - - text: chat models that return text (may also be multimodal) -""" - -import time -from typing import Any, Dict, List - -from backend import config - -_TTL_SEC = 3600 -_cache: Dict[str, Any] = {"at": 0.0, "payload": None} - - -def _entry(m: Dict[str, Any]) -> Dict[str, str]: - return { - "id": m.get("id") or "", - "name": m.get("name") or m.get("id") or "", - } - - -def _ensure_default(items: List[Dict[str, str]], model_id: str) -> List[Dict[str, str]]: - """Prepend the configured default if OpenRouter didn't return it.""" - if not model_id: - return items - if any(x["id"] == model_id for x in items): - return items - return [{"id": model_id, "name": model_id}] + items - - -def _fetch_raw() -> List[Dict[str, Any]]: - import httpx # local import so the app can start without httpx in odd envs - headers = {"Accept": "application/json"} - if config.AI_API_KEY: - headers["Authorization"] = f"Bearer {config.AI_API_KEY}" - url = f"{config.AI_BASE_URL.rstrip('/')}/models" - # Ask for text-output chat models (includes multimodal). "all" is huge. - with httpx.Client(timeout=30.0) as client: - r = client.get(url, headers=headers, params={"output_modalities": "text"}) - r.raise_for_status() - data = r.json() - return data.get("data") or [] - - -def list_models() -> Dict[str, Any]: - """Return {vision, text, defaults} for the frontend selects.""" - now = time.time() - if _cache["payload"] and (now - _cache["at"]) < _TTL_SEC: - return _cache["payload"] - - try: - raw = _fetch_raw() - except Exception as e: - # Degrade to configured defaults so the UI still works offline. - print(f"[Models] OpenRouter catalog fetch failed: {e}") - vision = _ensure_default([], config.MODEL) - text = _ensure_default([], config.TEXT_MODEL) - payload = { - "vision": vision, - "text": text, - "defaults": {"vision": config.MODEL, "text": config.TEXT_MODEL}, - "error": str(e), - } - return payload - - vision: List[Dict[str, str]] = [] - text: List[Dict[str, str]] = [] - for m in raw: - mid = m.get("id") or "" - if not mid: - continue - arch = m.get("architecture") or {} - inputs = arch.get("input_modalities") or [] - outputs = arch.get("output_modalities") or [] - # Legacy string form: "text+image->text" - modality = (arch.get("modality") or "").lower() - has_image_in = ("image" in inputs) or ("image" in modality.split("->")[0]) - has_text_out = ("text" in outputs) or ("->text" in modality) or (not outputs and not modality) - has_text_in = ("text" in inputs) or ("text" in modality) or not inputs - - if has_image_in and has_text_out: - vision.append(_entry(m)) - if has_text_in and has_text_out: - text.append(_entry(m)) - - vision.sort(key=lambda x: x["name"].lower()) - text.sort(key=lambda x: x["name"].lower()) - vision = _ensure_default(vision, config.MODEL) - text = _ensure_default(text, config.TEXT_MODEL) - - payload = { - "vision": vision, - "text": text, - "defaults": {"vision": config.MODEL, "text": config.TEXT_MODEL}, - } - _cache["at"] = now - _cache["payload"] = payload - return payload diff --git a/backend/pipeline/_serialize.py b/backend/pipeline/_serialize.py index 815ad41..371a111 100644 --- a/backend/pipeline/_serialize.py +++ b/backend/pipeline/_serialize.py @@ -54,6 +54,8 @@ def slim_clusters(clusters: List[Dict]) -> List[Dict]: "location": c.get("location"), "disciplines": c.get("disciplines"), "kind": c.get("kind"), + **({"disputed_attributes": c["disputed_attributes"]} + if c.get("disputed_attributes") else {}), "assertions": [slim_assertion(a) for a in c.get("assertions", [])], } for c in clusters diff --git a/backend/pipeline/conflict_checker.py b/backend/pipeline/conflict_checker.py index fc0bdde..61c1c0b 100644 --- a/backend/pipeline/conflict_checker.py +++ b/backend/pipeline/conflict_checker.py @@ -32,6 +32,16 @@ def _evidence_block(cluster: Dict) -> str: f"{a.get('attribute','')} = {a.get('value','')} | " f"\"{a.get('source_text','')}\"" ) + disputes = cluster.get("disputed_attributes") or [] + if disputes: + lines.append("") + for d in disputes: + lines.append( + "DISPUTED VALUE (possible extraction misread): " + f"attribute={d.get('attribute','')} " + f"values={' | '.join(d.get('values') or [])} " + f"(assertions {', '.join(d.get('assertion_ids') or [])})" + ) return "\n".join(lines) @@ -83,6 +93,8 @@ def _check_one(cluster: Dict, page_to_b64: Dict[int, str]) -> List[Dict]: user_text=user_text, images_b64=_images_for(cluster, page_to_b64), max_tokens=config.REASON_MAX_TOKENS, + reasoning_effort=config.EXTRACT_REASONING_EFFORT or None, + reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None, ) if isinstance(parsed, list): candidates = parsed diff --git a/backend/pipeline/constructability.py b/backend/pipeline/constructability.py index ed1c735..e8265e1 100644 --- a/backend/pipeline/constructability.py +++ b/backend/pipeline/constructability.py @@ -27,6 +27,10 @@ def constructability_review(sheets: List[Dict], clusters: List[Dict], "assertions": dumps(slim_sheets(sheets)), "clusters": dumps(slim_clusters(clusters)), "conflicts": dumps(conflicts), + "disputes": dumps([ + d for cluster in clusters + for d in (cluster.get("disputed_attributes") or []) + ]), }, max_tokens=config.CONSTRUCT_MAX_TOKENS, ) diff --git a/backend/pipeline/extractor.py b/backend/pipeline/extractor.py index f75dac8..491942f 100644 --- a/backend/pipeline/extractor.py +++ b/backend/pipeline/extractor.py @@ -64,7 +64,8 @@ def discipline_from_sheet_number(sheet_number: Optional[str]) -> Optional[str]: return None -def _is_grounded(value: str, source_text: str, graphical_basis: str = "") -> bool: +def _is_grounded(value: str, source_text: str, graphical_basis: str = "", + page_text: Optional[str] = None) -> bool: """ Keep an object only if its primary value is supported by its source_text, OR it is a graphical object (has graphical_basis with no text to quote). @@ -72,6 +73,9 @@ def _is_grounded(value: str, source_text: str, graphical_basis: str = "") -> boo - If graphical_basis is set and source_text is absent, the object is valid. - If the value contains digits, every distinct digit-run must appear in source_text (catches invented dimensions/counts/elevations). + - Rescue tier: when page_text (the deterministic text layer) is given, + digit-runs absent from source_text but present in the page text are + still grounded - vision quoted imperfectly but the value is real. - If the value has no digits, require some alphabetic-token overlap. """ # Graphical objects (no readable text on sheet) are always allowed through. @@ -85,7 +89,11 @@ def _is_grounded(value: str, source_text: str, graphical_basis: str = "") -> boo val_digits = set(_DIGITS_RE.findall(value)) if val_digits: src_digits = set(_DIGITS_RE.findall(source_text)) - return val_digits.issubset(src_digits) + if val_digits.issubset(src_digits): + return True + if page_text: + return val_digits.issubset(set(_DIGITS_RE.findall(page_text))) + return False # No digits: text-based grounding. val_norm = re.sub(r"[^a-z0-9]+", " ", value.lower()).strip() @@ -109,7 +117,24 @@ def _primary_value(obj: Dict) -> str: or obj.get("name") or obj.get("tag") or "") -def _normalize_sheet(parsed: Dict, page_number: int) -> Dict: +def _grounding_stamp(value: str, source_text: str, + page_text: Optional[str]) -> Optional[str]: + """\"text_layer\" when the object survived only via the text-layer rescue + tier (digits absent from source_text but present in the page text).""" + if not page_text: + return None + val_digits = set(_DIGITS_RE.findall(str(value))) + if not val_digits: + return None + if val_digits.issubset(set(_DIGITS_RE.findall(source_text))): + return None + if val_digits.issubset(set(_DIGITS_RE.findall(page_text))): + return "text_layer" + return None + + +def _normalize_sheet(parsed: Dict, page_number: int, + page_text: Optional[str] = None) -> Dict: """ Validate + clean one parsed sheet result, attaching page_number and ids. @@ -138,6 +163,7 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict: raw_objects = parsed.get("objects") or parsed.get("assertions") or [] clean: List[Dict] = [] dropped = 0 + rescued = 0 for idx, obj in enumerate(raw_objects): if not isinstance(obj, dict): @@ -149,9 +175,13 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict: # Derive a primary value for the grounding check primary_val = _primary_value(obj) - if not _is_grounded(primary_val, source_text, graphical_basis): + if not _is_grounded(primary_val, source_text, graphical_basis, + page_text=page_text): dropped += 1 continue + grounding = _grounding_stamp(primary_val, source_text, page_text) + if grounding: + rescued += 1 # --- location_key: new schema is richer; map to legacy shape + extras --- lk = obj.get("location_key") @@ -203,10 +233,13 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict: "object_attributes": attrs, "graphical_basis": graphical_basis or None, "review_uses": obj.get("review_uses") or [], + **({"grounding": grounding} if grounding else {}), }) - if dropped: - print(f"[Extract] Page {page_number} ({sheet_number}): dropped {dropped} ungrounded object(s)") + if dropped or rescued: + print(f"[Extract] Page {page_number} ({sheet_number}): " + f"dropped {dropped} ungrounded object(s)" + + (f", rescued {rescued} via text layer" if rescued else "")) unresolved = parsed.get("unresolved_items") or [] @@ -223,8 +256,23 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict: } +def _text_layer_block(page: Dict) -> str: + """ + The TEXT LAYER block appended to the extractor instruction at call sites + (NOT a template placeholder - render() silently leaves missing keys as + literals). Empty string when the page has no usable text layer. + """ + text = (page.get("text_layer") or "").strip() + if not text: + return "" + return ("\n\nTEXT LAYER (authoritative for alphanumeric content — trust it " + "over the image for numbers, tags, and note text):\n" + + text[:config.TEXT_LAYER_MAX_CHARS]) + + def _extract_one(page: Dict, sheet_hint: str = "") -> Dict: - user_text = EXTRACTOR_USER_INSTRUCTION.replace("{sheet_hint}", sheet_hint) + user_text = (EXTRACTOR_USER_INSTRUCTION.replace("{sheet_hint}", sheet_hint) + + _text_layer_block(page)) parsed = call_json( system_prompt=EXTRACTOR_SYSTEM_PROMPT, user_text=user_text, @@ -241,7 +289,8 @@ def _extract_one(page: Dict, sheet_hint: str = "") -> Dict: "scale": None, "assertions": [], } - return _normalize_sheet(parsed, page["page_number"]) + return _normalize_sheet(parsed, page["page_number"], + page_text=page.get("text_layer")) def extract_assertions(pages: List[Dict], on_progress=None) -> List[Dict]: diff --git a/backend/pipeline/runner.py b/backend/pipeline/runner.py index e5c85ee..58e8c7f 100644 --- a/backend/pipeline/runner.py +++ b/backend/pipeline/runner.py @@ -27,12 +27,14 @@ from typing import Dict, Optional, Callable from backend.pipeline.pdf_processor import convert_pdf_to_images from backend.pipeline.extractor import extract_assertions +from backend.text_layer import attach_text_layers, coverage_gaps from backend.pipeline.sheet_index import classify_sheets, derive_project_meta_from_cover from backend.pipeline.jurisdiction import run_jurisdiction from backend.pipeline.normalizer import normalize_assertions, build_project_intelligence from backend.pipeline.clusterer import cluster_by_location from backend.pipeline.llm_clusterer import cluster_by_location_llm from backend import config +from backend.agents.disputes import annotate_clusters from backend.pipeline.conflict_checker import check_conflicts from backend.pipeline.qaqc_review import senior_review from backend.pipeline.code_review import code_review @@ -102,9 +104,12 @@ def _run_stages( ) -> Dict: stage("PDF -> images") pages = convert_pdf_to_images(pdf_path) + text_dir = os.path.join(out_dir, "text") if out_dir else None + attach_text_layers(pdf_path, pages, text_dir=text_dir) stage("Extract assertions") sheets = extract_assertions(pages) + coverage_gaps(pages, sheets) # classic: log-only recall signal stage("Classify sheet index") sheet_index = classify_sheets(sheets) @@ -129,6 +134,10 @@ def _run_stages( else: clusters = cluster_by_location(sheets) + disputed_count = annotate_clusters(clusters) + if disputed_count: + print(f"[Cluster] {disputed_count} cluster(s) carry disputed extracted values") + stage("Reason over clusters (conflicts)") conflicts = check_conflicts(clusters, pages) @@ -176,6 +185,7 @@ def _run_stages( report["summary"]["cost_by_stage"] = cost.get("by_stage", {}) report["summary"]["text_backend"] = "local" if text_local else "openrouter" report["summary"]["models_used"] = cost.get("models", {}) + report["summary"]["pipeline_mode"] = "classic" print(f"[Runner] LLM cost: ${cost['usd']:.4f} over {cost['calls']} live calls" f" ({cost.get('cached', 0)} cached)") diff --git a/backend/prompts.py b/backend/prompts.py index f4f5c5d..916fd37 100644 --- a/backend/prompts.py +++ b/backend/prompts.py @@ -230,6 +230,7 @@ Rules you must never break: - Every object must include source_text copied verbatim from the sheet whenever text is available. - If the object is graphical and has no text, describe it visually and mark confidence low or medium. - Preserve tags, marks, room numbers, sheet numbers, detail references, and abbreviations exactly as shown. +TEXT LAYER GROUNDING: when a TEXT LAYER block is present in the user message, it is the sheet's deterministic PDF text layer and is authoritative for alphanumeric content (counts, dimensions, member tags, note text). Trust it over your reading of the image for numbers, tags, and note text; quote source_text from it verbatim. Use the image for geometry, symbols, linework, and anything absent from the text layer. - Use null when information is not determinable. - Keep objects atomic. - Use plain ASCII only. @@ -410,6 +411,9 @@ What is NOT a conflict: - Anything not supported with drawing evidence. Be conservative: - Only flag genuine disagreements. +- When a value is marked DISPUTED (possible extraction misread), verify it against + the sheet images before relying on either reading; if the images do not resolve + it, do not assert a conflict from one reading alone. - A clean cluster with no contradiction must return an empty conflicts array. - missing_element requires evidence that another discipline would reasonably be expected to show the missing item. For each conflict: @@ -447,6 +451,28 @@ Clustered assertions (evidence): {evidence}""" +# --------------------------------------------------------------------------- +# Wave 5b - evidence verification (vision fact-check of cited sheet text) +# --------------------------------------------------------------------------- + +VERIFY_SYSTEM_PROMPT = """You are a meticulous construction document checker verifying machine-extracted evidence against the actual drawing sheet images. +For each evidence item you are given the sheet it was extracted from and the verbatim text the extractor claims appears there. +Judge each item against the images: +- confirmed: the text (or an obvious equivalent) appears on the cited sheet and means what the finding claims. +- corrected: the sheet shows a DIFFERENT value than the extracted text. Give the actual verbatim text. +- not_found: nothing like the extracted text appears on the cited sheet. +Be strict about numbers, quantities, and member sizes: "(2) 2x6" and "(5) 2x6" are different values. HSS16x4 and HSS16x16 are different values. +Use plain ASCII only. +Respond only with valid JSON.""" + +VERIFY_USER_INSTRUCTION = """Verify this finding's evidence against the attached sheet images. +Respond ONLY with a valid JSON object - no markdown fences, no explanation: +{ "verdicts": [ { "sheet": "string", "source_text": "the evidence text judged", "verdict": "confirmed | corrected | not_found", "actual_text": "verbatim sheet text when corrected, else null", "notes": "string or null" } ] } +Finding: {finding} +TEXT LAYER (deterministic page text extracted from the PDF - an oracle for alphanumeric content such as counts, dimensions, and member tags; when it disagrees with the extracted evidence, trust it and cite it as actual_text): +{text_layer}""" + + # --------------------------------------------------------------------------- # Stage 6 - senior architect full-set QAQC review (NOT WIRED YET) # --------------------------------------------------------------------------- @@ -545,6 +571,12 @@ Flag: Rules: - Only flag issues supported by drawing evidence. - Be specific about the location and why it is a constructability risk. +- Assertions are machine-extracted from sheet images and may contain misread values, + especially quantities and member sizes (e.g. "(2) 2x6" vs "(5) 2x6"). +- When the cluster lists disputed_attributes, or two evidence items disagree on a + numeric value, do NOT assert a buildability conclusion from one reading. Report the + ambiguity itself (category "detail_gap", confidence "low") and state that the value + needs verification against the sheet. - Use plain ASCII only. Respond only with valid JSON.""" @@ -554,7 +586,8 @@ Respond ONLY with a valid JSON object - no markdown fences, no explanation: If no constructability issues are found, return: { "issues": [] } Extracted assertions: {assertions} Clusters: {clusters} -Cross-discipline conflicts already found: {conflicts}""" +Cross-discipline conflicts already found: {conflicts} +Disputed extracted values in this cluster (possible vision misreads - treat as unverified): {disputes}""" # --------------------------------------------------------------------------- diff --git a/backend/review/__init__.py b/backend/review/__init__.py new file mode 100644 index 0000000..d80b3bc --- /dev/null +++ b/backend/review/__init__.py @@ -0,0 +1 @@ +"""Human-review gate: decision schemas and review-trigger policy.""" diff --git a/backend/review/feedback.py b/backend/review/feedback.py new file mode 100644 index 0000000..9baea86 --- /dev/null +++ b/backend/review/feedback.py @@ -0,0 +1,51 @@ +"""Feedback labels: one label artifact per human-review decision, for metrics.""" + +import json +import os +from datetime import datetime, timezone + + +def _as_dict(value) -> dict: + return value if isinstance(value, dict) else {} + + +def decision_to_label(queue_item: dict, decision: dict, job: dict) -> dict: + """Build one feedback label from a queue item, its decision, and the job. + + All field access is defensive: missing fields degrade to None (or [] for + models_used) rather than raising. + """ + queue_item = _as_dict(queue_item) + decision = _as_dict(decision) + job = _as_dict(job) + payload = _as_dict(queue_item.get("payload")) + summary = _as_dict(_as_dict(job.get("report")).get("summary")) + return { + "review_item_id": queue_item.get("review_item_id"), + "job_id": job.get("job_id"), + "pipeline_mode": job.get("pipeline_mode"), + "source_stage": payload.get("source_stage"), + "category": payload.get("category"), + "severity": payload.get("severity"), + "confidence": payload.get("confidence"), + "decision": decision.get("decision"), + "reason_code": decision.get("reason_code"), + "location": payload.get("location"), + "disciplines": payload.get("disciplines"), + "sheets": payload.get("sheets"), + "drawing_type": payload.get("drawing_type"), + "models_used": summary.get("models_used") or [], + "created_at": datetime.now(timezone.utc).isoformat(), + } + + +def write_label(out_dir: str, label: dict) -> None: + """Append one label as a JSON line; never raises on I/O failure.""" + try: + review_dir = os.path.join(out_dir, "review") + os.makedirs(review_dir, exist_ok=True) + path = os.path.join(review_dir, "feedback_labels.jsonl") + with open(path, "a", encoding="utf-8") as f: + f.write(json.dumps(label) + "\n") + except OSError as e: + print(f"[Review] feedback label write failed: {e}") diff --git a/backend/review/finalizer.py b/backend/review/finalizer.py new file mode 100644 index 0000000..fcdd6ff --- /dev/null +++ b/backend/review/finalizer.py @@ -0,0 +1,255 @@ +"""ReviewFinalizer: apply human decisions, targeted reruns, RFIs, final artifacts. + +All LLM-touching helpers degrade gracefully: a failed or empty targeted rerun +becomes a visible ``analysis_gap`` finding instead of raising, and RFI drafting +returns whatever was produced (possibly []). Finalization never crashes the job +on a single bad scope. +""" + +import json +import os +from typing import Dict, List, Optional, Tuple + +from backend import config +from backend.agents.base import AgentScope, AgentUsage +from backend.agents.conflict_critic import ConflictCriticAgent +from backend.agents.memory import ProjectMemory +from backend.agents.orchestrator import Orchestrator +from backend.agents.rfi_writer import RFIWriterAgent +# Private import, acceptable here: the runner's _finding_as_conflict is the +# canonical finding -> report["conflicts"] mapping; reusing it keeps the +# finalized report's conflicts in exactly the shape build_report produces. +from backend.agents.runner import _finding_as_conflict +from backend.pipeline.report import to_markdown +from backend.review.store import ReviewStore + + +def apply_decisions(prioritized: List[dict], decisions: Dict[str, dict]) -> Tuple[List[dict], List[dict]]: + kept: List[dict] = [] + suppressed: List[dict] = [] + for issue in prioritized: + review_id = f"finding:{issue.get('issue_id')}" + decision = decisions.get(review_id) or {} + action = decision.get("decision") + if action == "reject": + suppressed.append({ + **issue, + "review_state": "rejected", + "reason_code": decision.get("reason_code"), + "review_comment": decision.get("comment") or "", + }) + elif action == "unsure": + kept.append({**issue, "review_state": "unsure"}) + else: + kept.append({**issue, "review_state": "confirmed" if action == "confirm" else "unreviewed"}) + return kept, suppressed + + +def _gap_finding(index: int, scope_id: str, description: str) -> dict: + """Same shape as the runner's gap_findings: low severity, high confidence.""" + return { + "issue_id": f"AGENT-GAP-CLARIFY-{index + 1:03d}", + "source_stage": "qaqc", + "category": "analysis_gap", + "severity": "low", + "confidence": "high", + "location": scope_id.split(":", 2)[1] if ":" in scope_id else "", + "disciplines": [], + "sheets": [], + "description": description, + "evidence": [], + "recommended_resolution": "Review this scope manually or rerun the job.", + "code_reference": None, + "agent": "completeness", + "scope_id": scope_id, + } + + +def rerun_clarified_scopes( + memory_snapshot: dict, + decisions: Dict[str, dict], + prioritized: Optional[List[dict]] = None, +) -> List[dict]: + """Bounded targeted reruns for ``needs_clarification`` decisions. + + v1 reruns conflict scopes only: at most ONE ConflictCriticAgent scope per + clarified finding. The clarification answer is injected as a pseudo + "Reviewer" assertion prepended to the cluster's assertions so it reaches + the critic's evidence block (and survives front-truncation to + AGENT_CLUSTER_MAX_ASSERTIONS); ``page_to_b64`` is empty (cluster + assertions may carry their own + base64). Every per-scope failure degrades to an ``analysis_gap`` finding + and never raises. Non-conflict scopes are NOT rerun; they produce an + ``analysis_gap`` noting the scope is not rerunnable in v1. + """ + findings_pool = list(prioritized or []) + list(memory_snapshot.get("findings") or []) + clusters = memory_snapshot.get("clusters") or [] + out: List[dict] = [] + for item_id, decision in (decisions or {}).items(): + if (decision or {}).get("decision") != "needs_clarification": + continue + answer = str(decision.get("clarification_answer") or "").strip() + if not answer: + continue + issue_id = item_id.split(":", 1)[1] if item_id.startswith("finding:") else item_id + finding = next((f for f in findings_pool if f.get("issue_id") == issue_id), None) + scope_id = str((finding or {}).get("scope_id") or "") + if not scope_id.startswith("conflict:"): + out.append(_gap_finding( + len(out), scope_id or item_id, + f"Clarification rerun not supported in v1 for non-conflict scope " + f"{scope_id or item_id!r} (finding {issue_id}).", + )) + continue + cluster_key = scope_id.split(":", 1)[1] + cluster = next((c for c in clusters if c.get("key") == cluster_key), None) + if cluster is None: + out.append(_gap_finding( + len(out), scope_id, + f"Clarification rerun failed: cluster {cluster_key!r} not found " + f"for scope {scope_id} (finding {issue_id}).", + )) + continue + rerun_cluster = { + **cluster, + # Prepend: ConflictCriticAgent truncates assertions from the front + # (AGENT_CLUSTER_MAX_ASSERTIONS), so the clarification must come + # first or a full cluster would silently drop it. + "assertions": [{ + "discipline": "Reviewer", + "sheet_number": "REVIEW", + "attribute": "clarification", + "value": answer, + "source_text": answer, + }] + list(cluster.get("assertions") or []), + } + scope = AgentScope( + scope_id=scope_id, + payload={"cluster": rerun_cluster, "page_to_b64": {}}, + ) + result = ConflictCriticAgent(AgentUsage()).run(scope) + if result.error or not result.artifacts: + out.append(_gap_finding( + len(out), scope_id, + f"Clarification rerun did not complete for scope {scope_id} " + f"(finding {issue_id}): {result.error or 'no findings produced'}.", + )) + continue + for rerun_finding in result.artifacts: + rerun_finding["clarification_of"] = issue_id + out.append(rerun_finding) + return out + + +def _draft_rfis(kept: List[dict]) -> List[dict]: + """Draft RFIs for kept issues only, mirroring the runner's wave 7.""" + orchestrator = Orchestrator(ProjectMemory()) + scopes = [ + AgentScope( + scope_id=f"rfi:{finding.get('issue_id') or index + 1}", + payload={"finding": finding}, + ) + for index, finding in enumerate(kept) + ] + try: + results = orchestrator.run_scopes( + RFIWriterAgent(AgentUsage()), scopes, config.AGENT_RFI_CONCURRENCY + ) + except Exception: + return [] + return [artifact for result in results for artifact in result.artifacts] + + +def _read_json(path: str, default): + try: + with open(path, encoding="utf-8") as f: + return json.load(f) + except (OSError, json.JSONDecodeError): + return default + + +def _dump(out_dir: str, name: str, value) -> None: + with open(os.path.join(out_dir, name), "w", encoding="utf-8") as f: + json.dump(value, f, indent=2) + + +def finalize_review(job_id: str, out_dir: str) -> dict: + """Apply review decisions and write the final report artifacts. + + Raises ValueError("incomplete review") if any blocking queue item lacks a + decision. Never raises for rerun/RFI degradation. + """ + store = ReviewStore(out_dir) + queue = store.read_queue() + decisions = store.read_decisions() + for item in queue: + if item.get("blocking") and item.get("review_item_id") not in decisions: + raise ValueError("incomplete review") + + report = _read_json(os.path.join(out_dir, "conflicts.json"), {}) or {} + snapshot = _read_json(os.path.join(out_dir, "agent", "memory.json"), {}) or {} + prioritized = list(report.get("validated_issues") or []) + + rerun_findings = rerun_clarified_scopes(snapshot, decisions, prioritized) + replacements: Dict[str, List[dict]] = {} + for finding in rerun_findings: + origin = finding.get("clarification_of") + if origin: + replacements.setdefault(origin, []).append(finding) + else: + prioritized.append(finding) # gap findings stay as additions + for origin, new_findings in replacements.items(): + for index, issue in enumerate(prioritized): + if issue.get("issue_id") == origin: + prioritized[index:index + 1] = new_findings + break + + kept, suppressed = apply_decisions(prioritized, decisions) + for issue in kept: + if issue.get("clarification_of"): + issue["review_state"] = "clarified" + else: + decision = decisions.get(f"finding:{issue.get('issue_id')}") or {} + if decision.get("decision") == "needs_clarification": + issue["review_state"] = "clarification_failed" + + rfis = _draft_rfis(kept) + + report["validated_issues"] = kept + report["suppressed_issues"] = (report.get("suppressed_issues") or []) + suppressed + report["rfis"] = rfis + summary = report.setdefault("summary", {}) + summary["agent_status"] = "complete" + by_stage = summary.get("by_stage") + if isinstance(by_stage, dict): + if "validated" in by_stage: + by_stage["validated"] = len(kept) + if "rfis" in by_stage: + by_stage["rfis"] = len(rfis) + # Rebuild the conflicts view and headline counts from the KEPT + # conflict-stage findings so rejected findings no longer appear as + # conflicts in report.md / the UI (mirrors pipeline.report.build_report). + conflicts = [ + _finding_as_conflict(finding) + for finding in kept + if finding.get("source_stage") == "conflict" + ] + report["conflicts"] = conflicts + by_severity = {"high": 0, "medium": 0, "low": 0} + by_category: Dict[str, int] = {} + for conflict in conflicts: + by_severity[conflict["severity"]] = by_severity.get(conflict["severity"], 0) + 1 + by_category[conflict["category"]] = by_category.get(conflict["category"], 0) + 1 + summary["conflicts_found"] = len(conflicts) + summary["by_severity"] = by_severity + summary["by_category"] = by_category + summary["review"] = store.progress(queue) + + os.makedirs(out_dir, exist_ok=True) + _dump(out_dir, "conflicts.json", report) + _dump(out_dir, "validated_issues.json", kept) + _dump(out_dir, "suppressed_issues.json", suppressed) + _dump(out_dir, "rfis.json", rfis) + with open(os.path.join(out_dir, "report.md"), "w", encoding="utf-8") as f: + f.write(to_markdown(report)) + return report diff --git a/backend/review/gate.py b/backend/review/gate.py new file mode 100644 index 0000000..bf879f3 --- /dev/null +++ b/backend/review/gate.py @@ -0,0 +1,31 @@ +"""ReviewGate: build the human-review queue from prioritized findings.""" + +from typing import Dict, List, Optional + +from backend.review.policy import build_audit_sample, requires_review + + +def _finding_item(issue: Dict, blocking: bool, reasons: List[str], kind: str) -> Dict: + issue_id = issue.get("issue_id") or "unknown" + return { + "review_item_id": f"finding:{issue_id}", + "kind": kind, + "blocking": blocking, + "reasons": reasons, + "payload": issue, + } + + +def build_review_queue(memory_snapshot: Dict, prioritized: List[Dict], decisions: List[Dict], + limit: Optional[int] = None) -> List[Dict]: + queue: List[Dict] = [] + for issue in prioritized: + reasons = requires_review(issue) + queue.append(_finding_item(issue, bool(reasons), reasons, "finding" if reasons else "audit_finding")) + if limit is None: + for item in build_audit_sample(memory_snapshot, prioritized): + queue.append(item) + else: + for item in build_audit_sample(memory_snapshot, prioritized, limit=limit): + queue.append(item) + return queue diff --git a/backend/review/metrics.py b/backend/review/metrics.py new file mode 100644 index 0000000..0ff373f --- /dev/null +++ b/backend/review/metrics.py @@ -0,0 +1,25 @@ +"""Aggregate metrics over feedback labels. + +Default aggregates exclude source_text, images, raw sheet content, and +reviewer free-text comments; include_text=True is the only path that embeds +the raw labels. +""" + +from collections import Counter +from typing import Dict, List + + +def aggregate_labels(labels: List[dict], include_text: bool = False) -> Dict: + decisions = Counter(label.get("decision") or "unknown" for label in labels) + reasons = Counter(label.get("reason_code") or "none" for label in labels if label.get("decision") == "reject") + summary = { + "total": len(labels), + "decisions": dict(decisions), + "reject_reasons": dict(reasons), + } + for label in labels: + decision = label.get("decision") or "unknown" + summary[decision] = summary.get(decision, 0) + 1 + if include_text: + summary["labels"] = labels + return summary diff --git a/backend/review/policy.py b/backend/review/policy.py new file mode 100644 index 0000000..f6562a6 --- /dev/null +++ b/backend/review/policy.py @@ -0,0 +1,70 @@ +"""Review-trigger policy: which findings block on human review.""" + +from typing import Any, Dict, List + +_SENSITIVE_CATEGORIES = { + "missing_element", + "ada", + "tas_tdlr", + "egress", + "fire_separation", + "occupancy", + "spatial_clash", + "clearance_conflict", + "penetration_conflict", +} + + +def requires_review(issue: Dict) -> List[str]: + """Return trigger reasons that require human review for one issue.""" + reasons: List[str] = [] + severity = str(issue.get("severity") or "").lower() + confidence = str(issue.get("confidence") or "").lower() + category = str(issue.get("category") or "").lower() + if severity in {"critical", "high"}: + reasons.append("severity_high") + if confidence == "low": + reasons.append("confidence_low") + if category in _SENSITIVE_CATEGORIES or issue.get("source_stage") == "code": + reasons.append("sensitive_category") + return reasons + + +def build_audit_sample( + memory_snapshot: Dict, + prioritized: List[Dict], + limit: int = 5, +) -> List[Dict[str, Any]]: + """Build non-blocking spot-check items for clean (finding-free) clusters.""" + implicated = { + str(finding.get("scope_id") or "") + for finding in (memory_snapshot.get("findings") or []) + list(prioritized) + } + items: List[Dict[str, Any]] = [] + for cluster in memory_snapshot.get("clusters") or []: + if len(items) >= limit: + break + assertions = cluster.get("assertions") or [] + if len(assertions) < 2: + continue + cluster_key = cluster.get("key") or "unknown" + if any(cluster_key in scope_id for scope_id in implicated): + continue + items.append({ + "review_item_id": f"clean_cluster:{cluster_key}", + "kind": "clean_cluster", + "blocking": False, + "reasons": ["audit_sample"], + "payload": _without_base64(cluster), + }) + return items + + +def _without_base64(cluster: Dict) -> Dict: + return { + **cluster, + "assertions": [ + {key: value for key, value in assertion.items() if key != "base64"} + for assertion in cluster.get("assertions") or [] + ], + } diff --git a/backend/review/schemas.py b/backend/review/schemas.py new file mode 100644 index 0000000..648dde6 --- /dev/null +++ b/backend/review/schemas.py @@ -0,0 +1,45 @@ +"""Human-review decision schema and validation.""" + +from typing import Optional + +DECISIONS = {"confirm", "reject", "unsure", "needs_clarification"} +REASON_CODES = { + "wrong_cluster_link", + "same_value_different_representation", + "not_a_contradiction", + "missing_evidence", + "extraction_misread", + "code_path_not_applicable", + "duplicate", + "severity_too_high", + "severity_too_low", + "other", +} + + +def validate_decision(raw: dict) -> Optional[dict]: + """Normalize a reviewer decision payload, or return None if invalid.""" + if not isinstance(raw, dict): + return None + decision = str(raw.get("decision") or "").strip() + if decision not in DECISIONS: + return None + reason_code = raw.get("reason_code") + if decision == "reject": + reason_code = str(reason_code or "").strip() + if reason_code not in REASON_CODES: + return None + elif reason_code is not None: + reason_code = str(reason_code).strip() or None + if reason_code and reason_code not in REASON_CODES: + return None + return { + "review_item_id": str(raw.get("review_item_id") or "").strip(), + "decision": decision, + "reason_code": reason_code, + "category_correction": raw.get("category_correction"), + "severity_correction": raw.get("severity_correction"), + "comment": str(raw.get("comment") or "").strip(), + "clarification_answer": raw.get("clarification_answer"), + "reviewed_at": raw.get("reviewed_at"), + } diff --git a/backend/review/store.py b/backend/review/store.py new file mode 100644 index 0000000..dba297c --- /dev/null +++ b/backend/review/store.py @@ -0,0 +1,62 @@ +"""Persistence for human-review queue and decisions within a job output dir.""" + +import json +import os +from typing import Dict, List + +from backend.review.schemas import validate_decision + + +class ReviewStore: + def __init__(self, job_out_dir: str, create: bool = True) -> None: + self.review_dir = os.path.join(job_out_dir, "review") + if create: + os.makedirs(self.review_dir, exist_ok=True) + + def _path(self, name: str) -> str: + return os.path.join(self.review_dir, name) + + def _write_json(self, name: str, value) -> None: + path = self._path(name) + tmp = f"{path}.tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(value, f, indent=2) + os.replace(tmp, path) + + def write_queue(self, queue: List[dict]) -> None: + self._write_json("review_queue.json", queue) + + def read_queue(self) -> List[dict]: + try: + with open(self._path("review_queue.json"), encoding="utf-8") as f: + value = json.load(f) + return value if isinstance(value, list) else [] + except (OSError, json.JSONDecodeError): + return [] + + def append_decision(self, decision: dict) -> None: + valid = validate_decision(decision) + if not valid or not valid["review_item_id"]: + raise ValueError("invalid review decision") + decisions = self.read_decisions() + decisions[valid["review_item_id"]] = valid + self._write_json("review_decisions.json", decisions) + + def read_decisions(self) -> Dict[str, dict]: + try: + with open(self._path("review_decisions.json"), encoding="utf-8") as f: + value = json.load(f) + return value if isinstance(value, dict) else {} + except (OSError, json.JSONDecodeError): + return {} + + def progress(self, queue: List[dict]) -> dict: + decisions = self.read_decisions() + required = [item for item in queue if item.get("blocking")] + completed = [item for item in required if item.get("review_item_id") in decisions] + return { + "required": len(required), + "completed": len(completed), + "remaining": len(required) - len(completed), + "total": len(queue), + } diff --git a/backend/text_layer.py b/backend/text_layer.py new file mode 100644 index 0000000..11857a2 --- /dev/null +++ b/backend/text_layer.py @@ -0,0 +1,230 @@ +""" +text_layer.py - deterministic PDF text-layer extraction (PyMuPDF, no LLM). + +Most CAD-produced drawing sets carry a real vector text layer. We extract it +once per job and feed it to the extractor (grounding), the grounding guard +(rescue tier), and the wave-5b verifier (text oracle + high-DPI evidence +crops). Pages below TEXT_LAYER_MIN_CHARS of text are treated as having no +text layer (scanned/raster sheets stay vision-only). + +If PyMuPDF is unavailable the module degrades gracefully: every public +function returns empty/None, equivalent to TEXT_LAYER_ENABLED=false. +""" + +import re +from typing import Dict, List, Optional, Tuple + +from backend import config + +try: # PyMuPDF >= 1.24 prefers the pymupdf name; fitz works everywhere. + import pymupdf as fitz +except ImportError: # pragma: no cover - older PyMuPDF + try: + import fitz + except ImportError: # pragma: no cover - PyMuPDF not installed + fitz = None + +_warned_unavailable = False + +# Word token normalization for evidence matching: lowercase alphanumeric only. +_TOKEN_RE = re.compile(r"[^a-z0-9]+") +# Fuzzy match floor: fraction of needle tokens that must align with the page's +# word sequence for a bbox to count as a confident evidence location. +_FUZZY_MIN_RATIO = 0.6 + + +def _fitz_or_none(): + """Return the fitz module, logging once if PyMuPDF is missing.""" + global _warned_unavailable + if fitz is None and not _warned_unavailable: + print("[TextLayer] PyMuPDF not available - text-layer grounding disabled") + _warned_unavailable = True + return fitz + + +def extract_text_layers(pdf_path: str) -> Dict[int, Dict]: + """ + Extract the text layer of every page. Returns {1-based page_number: + {"text": str, "words": [{"text", "bbox": (x0,y0,x1,y1)}, ...], + "has_text_layer": bool}}. Returns {} when disabled or unavailable. + """ + if not config.TEXT_LAYER_ENABLED: + return {} + f = _fitz_or_none() + if f is None: + return {} + try: + doc = f.open(pdf_path) + except Exception as exc: + print(f"[TextLayer] could not open {pdf_path}: {exc}") + return {} + layers: Dict[int, Dict] = {} + try: + for index in range(doc.page_count): + page = doc[index] + text = page.get_text("text") or "" + words = [ + {"text": w[4], "bbox": (w[0], w[1], w[2], w[3])} + for w in (page.get_text("words") or []) + ] + has_text_layer = len(text.strip()) >= config.TEXT_LAYER_MIN_CHARS + if not has_text_layer: + print(f"[TextLayer] Page {index + 1}: {len(text.strip())} chars " + f"(< TEXT_LAYER_MIN_CHARS={config.TEXT_LAYER_MIN_CHARS}) - " + f"vision-only") + layers[index + 1] = { + "text": text, + "words": words, + "has_text_layer": has_text_layer, + } + finally: + doc.close() + return layers + + +def attach_text_layers( + pdf_path: str, + pages: List[Dict], + text_dir: Optional[str] = None, +) -> Dict[int, List[Dict]]: + """ + Attach page["text_layer"] (text or None) to each converted page dict and + return the runner-local {page_number: words} map (kept off page dicts - + those get serialized). When text_dir is set, dump one .txt per page there + (plain file writes; ProjectMemory is a closed registry). + """ + layers = extract_text_layers(pdf_path) + page_words: Dict[int, List[Dict]] = {} + for page in pages: + layer = layers.get(page["page_number"]) or {} + page["text_layer"] = layer.get("text") if layer.get("has_text_layer") else None + page_words[page["page_number"]] = layer.get("words") or [] + if text_dir and layers: + import os + os.makedirs(text_dir, exist_ok=True) + for page_number, layer in layers.items(): + if not layer.get("has_text_layer"): + continue + with open(os.path.join(text_dir, f"page-{page_number:03d}.txt"), + "w", encoding="utf-8") as fh: + fh.write(layer.get("text") or "") + return page_words + + +def _tokens(text: str) -> List[str]: + return [t for t in _TOKEN_RE.split(text.lower()) if t] + + +def _union_bbox(boxes: List[Tuple[float, float, float, float]]): + return ( + min(b[0] for b in boxes), + min(b[1] for b in boxes), + max(b[2] for b in boxes), + max(b[3] for b in boxes), + ) + + +def find_evidence_bbox( + words: List[Dict], + needle: str, +) -> Optional[Tuple[float, float, float, float]]: + """ + Best-effort fuzzy substring match of an evidence source_text against the + page's word sequence. Returns the union bbox of the matched words, or + None when nothing aligns confidently. + + Exact contiguous token runs win; otherwise the best-scoring window with + >= _FUZZY_MIN_RATIO token alignment is accepted (vision quotes imperfectly + but the value is real page text). + """ + if not words or not needle: + return None + needle_tokens = _tokens(str(needle)) + if not needle_tokens: + return None + page_tokens = [_tokens(w.get("text") or "") for w in words] + # Flatten multi-token words, remembering which word each token came from. + flat: List[Tuple[str, int]] = [] + for word_index, parts in enumerate(page_tokens): + for part in parts: + flat.append((part, word_index)) + if not flat: + return None + + n = len(needle_tokens) + best_span = None + best_score = 0.0 + for start in range(0, len(flat)): + window = flat[start:start + n] + if not window: + break + score = sum(1 for i, tok in enumerate(needle_tokens) + if i < len(window) and window[i][0] == tok) / n + if score > best_score: + best_score = score + best_span = window + if best_score == 1.0: + break + if best_span is None or best_score < _FUZZY_MIN_RATIO: + return None + word_indexes = {word_index for _, word_index in best_span} + return _union_bbox([words[i]["bbox"] for i in sorted(word_indexes)]) + + +def render_crop( + pdf_path: str, + page_number: int, + bbox: Tuple[float, float, float, float], + dpi: Optional[int] = None, + margin_pts: Optional[float] = None, +) -> Optional[bytes]: + """ + Render a clip of one page around bbox (+ margin, clamped to the page) at + the given DPI and return JPEG bytes, or None on any failure. + """ + f = _fitz_or_none() + if f is None: + return None + dpi = dpi or config.VERIFY_CROP_DPI + margin_pts = config.VERIFY_CROP_MARGIN_PTS if margin_pts is None else margin_pts + try: + doc = f.open(pdf_path) + try: + page = doc[page_number - 1] + rect = f.Rect( + bbox[0] - margin_pts, + bbox[1] - margin_pts, + bbox[2] + margin_pts, + bbox[3] + margin_pts, + ) & page.rect + if rect.is_empty: + return None + pix = page.get_pixmap(clip=rect, dpi=dpi) + return pix.tobytes("jpeg") + finally: + doc.close() + except Exception as exc: + print(f"[TextLayer] render_crop failed on page {page_number}: {exc}") + return None + + +def coverage_gaps(pages: List[Dict], sheets: List[Dict]) -> List[int]: + """ + Page numbers that have a text layer but whose extraction failed or + returned 0 objects - the silent extraction-loss signal. Logs one + [TextLayer] line per gap. + """ + by_page = {s.get("page_number"): s for s in sheets or []} + gaps: List[int] = [] + for page in pages: + text = page.get("text_layer") + if not text: + continue + sheet = by_page.get(page["page_number"]) + extracted = len(sheet.get("assertions") or []) if sheet else 0 + if extracted == 0: + gaps.append(page["page_number"]) + print(f"[TextLayer] Page {page['page_number']}: text layer present " + f"({len(text)} chars) but no objects extracted — possible " + f"extraction gap") + return gaps diff --git a/cli/run_check.py b/cli/run_check.py index 8bd7644..ffc1858 100644 --- a/cli/run_check.py +++ b/cli/run_check.py @@ -17,6 +17,8 @@ _ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) if _ROOT not in sys.path: sys.path.insert(0, _ROOT) +from backend import config # noqa: E402 +from backend.agents.runner import run_agent_pipeline # noqa: E402 from backend.pipeline.runner import run_pipeline # noqa: E402 @@ -25,11 +27,16 @@ def main() -> int: parser.add_argument("pdf", help="Path to the PDF drawing set") parser.add_argument("--out", default=None, help="Directory for artifacts (default: out/)") + parser.add_argument("--mode", choices=("classic", "agent"), default="classic", + help="Pipeline implementation to run (default: classic)") parser.add_argument("--project-name", default=None) parser.add_argument("--address", default=None) parser.add_argument("--occupancy", default=None) parser.add_argument("--work-type", default=None, help="new_building | remodel | tenant_improvement | addition | ...") + parser.add_argument("--no-review", action="store_true", + help="Agent mode only: skip the human-review gate and finish the run " + "(overrides AGENT_REQUIRE_REVIEW=true)") args = parser.parse_args() if not os.path.isfile(args.pdf): @@ -43,7 +50,21 @@ def main() -> int: }.items() if v } out_dir = args.out or os.path.join("out", os.path.splitext(os.path.basename(args.pdf))[0]) - report = run_pipeline(args.pdf, out_dir=out_dir, project_input=project_input or None) + if args.mode == "agent": + report = run_agent_pipeline( + args.pdf, + out_dir=out_dir, + project_input=project_input or None, + source_name=os.path.basename(args.pdf), + require_review=config.AGENT_REQUIRE_REVIEW and not args.no_review, + ) + else: + report = run_pipeline( + args.pdf, + out_dir=out_dir, + project_input=project_input or None, + source_name=os.path.basename(args.pdf), + ) s = report["summary"] print("\n" + "=" * 60) @@ -51,7 +72,11 @@ def main() -> int: f"(high {s['by_severity']['high']}, " f"medium {s['by_severity']['medium']}, " f"low {s['by_severity']['low']})") - print(f" Report: {os.path.join(out_dir, 'report.md')}") + if s.get("agent_status") == "needs_review": + print(" Stopped for human review - finalize via the web UI, " + "or rerun with --no-review.") + else: + print(f" Report: {os.path.join(out_dir, 'report.md')}") print("=" * 60) return 0 diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 5cca7a8..0495a75 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -13,7 +13,7 @@ services: env_file: - backend/.env environment: - APP_BASE_URL: ${APP_BASE_URL:-http://localhost:8099} + APP_BASE_URL: ${APP_BASE_URL:-https://conchecker.scoutitsystems.com} volumes: - uploads:/app/backend/uploads - outputs:/app/backend/outputs diff --git a/docker-compose.yml b/docker-compose.yml index 3bba993..3cdbc2e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -7,7 +7,7 @@ services: - backend/.env environment: # Override in backend/.env for production (email links, etc.) - APP_BASE_URL: ${APP_BASE_URL:-http://localhost:8099} + APP_BASE_URL: ${APP_BASE_URL:-https://conchecker.scoutitsystems.com} volumes: - uploads:/app/backend/uploads - outputs:/app/backend/outputs diff --git a/docs/pipeline-infographic.html b/docs/pipeline-infographic.html new file mode 100644 index 0000000..a332bea --- /dev/null +++ b/docs/pipeline-infographic.html @@ -0,0 +1,187 @@ + + + + +Conflict Checker — How Your Plans Get Reviewed + + + + +
+

🔍 CONFLICT CHECKER

+
How your construction plans get reviewed — a team of AI assistants, each with one job, passing notes down the line.
+
+ +
+ + +
+
+
1
+
📄
+

The Scanner

+

Turns every page of your PDF blueprints into a picture the AI can read.

+
+
→
+
+
2
+
👓
+

The Readers

+

One assistant per page, all working at once. Each writes down every fact: dimensions, notes, materials, callouts.

+
+
→
+
+
3
+
📚
+

Librarian & Code Scout

+

Builds the table of contents (electrical, plumbing, structural…) and figures out where the project is, so the right building codes apply.

+
+
→
+
+
4
+
🔗
+

The Connector

+

Connects the dots across sheets — "this water heater on the plumbing sheet is the same one on the electrical sheet" — and sorts facts into topic piles.

+
+
+ +
↓
+ + +
+
+
5
+
🕵️
+

The Detectives

+

One per topic pile. Compares sheets that should agree and hunts for contradictions: "wall shown here, but not on the structural plan."

+
+
→
+
+
6
+
👷
+

The Specialists

+

Three experts at once: a code inspector, a veteran builder ("can this actually be built?"), and a checklist keeper ("is anything missing?").

+
+
→
+
+
7
+
🧠
+

The Brain

+

The senior reviewer. Collects every finding, merges duplicates, discards weak ones, and ranks the rest by how much trouble they'd cause.

+
+
→
+
+
8
+
✅
+

Human Review

+

The important and uncertain findings land on your desk. You confirm, reject, or mark unsure — nothing goes out unapproved.

+
+
+ +
↓
+ +
+
📋 Final Report + ✉️ Draft RFIs
+
A prioritized list of every problem found — plus ready-to-send "please clarify" letters (Requests For Information) for the design team.
+
+ +
+ Good to know: everyone shares one notebook, so each step builds on the last. + If one page can't be read, the team keeps going and that page is flagged as a gap + instead of stopping the whole review. Every finding links back to the sheet it came from. +
+ +
+ +
Conflict Checker · conchecker.scoutitsystems.com · Review your plans before they cost you money in the field.
+ + + diff --git a/docs/pipeline-infographic.png b/docs/pipeline-infographic.png new file mode 100644 index 0000000..a2c014a Binary files /dev/null and b/docs/pipeline-infographic.png differ diff --git a/docs/pipeline-overview.md b/docs/pipeline-overview.md new file mode 100644 index 0000000..89bf39e --- /dev/null +++ b/docs/pipeline-overview.md @@ -0,0 +1,174 @@ +# Conflict Checker — How It Works (Plain Language) + +**What it does:** You upload a set of construction drawings (a PDF of blueprints). +A team of AI assistants reads every page, compares everything against everything +else, and hands you a list of problems — contradictions, code violations, missing +information, and things that would be hard to build — before they cost you money +in the field. + +Think of it like hiring a room full of specialist consultants to review your +plans overnight. Each one has a specific job, they pass their notes down the +table, and a senior reviewer at the end sorts it all into one clean report. + +--- + +## The Big Picture (one sentence per step) + +``` + YOUR PDF OF BLUEPRINTS + | + v + +----------------------------------------------------------+ + | 0. SCANNER | + | Turns every PDF page into a picture the AI can read | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 1. READERS (one assistant per page, all at once) | + | Reads each sheet and writes down every fact: | + | dimensions, notes, materials, room names, callouts | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 2. LIBRARIAN + LOCAL-CODE SCOUT (work side by side) | + | Librarian: builds the table of contents — which | + | sheets exist (electrical, plumbing, structural...) | + | Scout: figures out WHERE the project is, so we know | + | which building codes apply | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 3. CONNECTOR | + | Connects the dots across sheets — e.g. "the water | + | heater on the plumbing sheet is the same one on the | + | electrical sheet" — and groups related facts into | + | topic piles (clusters) | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 4. CONFLICT DETECTIVES (one per topic pile) | + | Compares sheets that should agree and looks for | + | contradictions: "Wall shown here on A-201 but not | + | on S-101", "Pipe runs through the duct" | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 5. THREE SPECIALISTS (work side by side) | + | * Code Inspector — does anything break the local | + | building code? | + | * Builder — can this actually be built as | + | drawn? (access, clearances, sequencing) | + | * Completeness Checker — is anything MISSING from | + | the set? (sheets, schedules, required details) | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 6. THE BRAIN (senior reviewer) | + | Collects EVERY finding from everyone, merges the | + | duplicates, throws out the weak ones, and ranks the | + | rest by how much trouble they'd cause | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 7. HUMAN REVIEW GATE | + | The important/uncertain findings are queued for a | + | real person to Confirm / Reject / mark Unsure | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 8. LETTER WRITER | + | Drafts a formal RFI (Request For Information — the | + | official "please clarify this" letter) for each | + | confirmed issue, ready to send to the design team | + +----------------------------------------------------------+ + | + v + FINAL REPORT + DRAFT RFIs +``` + +--- + +## Who's Who (the "agents") + +| # | Name | Analogy | What it actually does | +|---|------|---------|-----------------------| +| 0 | PDF Scanner | Photocopier | Converts each PDF page into an image the AI can "see" | +| 1 | Sheet Extractor | Speed-reader | Reads one page, writes structured notes (every page gets its own reader, in parallel) | +| 2 | Sheet Indexer | Librarian | Builds the table of contents of the drawing set | +| 2 | Jurisdiction Scout | Local guide | Identifies the project's location so the right building codes are used | +| 3 | Linker | Connector | Groups related facts from different sheets into topic clusters | +| 4 | Conflict Critic | Detective | Examines each cluster for contradictions between disciplines | +| 5 | Code Agent | Code inspector | Flags building-code violations, using the jurisdiction from step 2 | +| 5 | Constructability Agent | Veteran builder | Flags things that are drawn fine but can't be built practically | +| 5 | Completeness Agent | Checklist keeper | Flags missing sheets, missing details, gaps in the set | +| 6 | Brain | Chief estimator | Deduplicates, judges, and prioritizes all findings | +| 7 | Review Gate | Your desk | Presents the findings a human should approve before anything goes out | +| 8 | RFI Writer | Secretary | Writes the formal clarification letters for confirmed issues | + +Everything the assistants learn is kept in a shared notebook (the "project +memory"), so each step builds on the last. If one reader fails on one page, the +rest of the team keeps going — that page is noted as a gap instead of crashing +the whole review. + +--- + +## Where Improvements Could Be Made + +### 1. Coverage — "make sure every page actually got read" +- Today, if a Reader fails on a page (the AI's answer gets cut off or comes back + garbled), that page quietly disappears from everything downstream. Worse, the + Completeness Checker can then report the sheet as "missing from the set" when + really it was there but unread — a false alarm. +- **Improvement:** retry failed pages with a backup model, and clearly separate + "sheet doesn't exist" from "sheet couldn't be read" in the report. + +### 2. Speed — "the team waits in line more than it needs to" +- The steps run strictly one after another, but some could start earlier. The + Jurisdiction Scout only needs the cover page — it could run while the other + Readers are still working. The Letter Writer could start on high-confidence + findings instead of waiting for all human review. +- **Improvement:** overlap independent steps; start drafting letters for + confirmed/high-confidence findings sooner. + +### 3. Cost — "smarter reading, fewer wasted words" +- Every page is read by a large, expensive AI model, and that model's + "thinking time" counts against its answer budget — we've seen it spend its + whole budget thinking and return a cut-off answer. +- **Improvement:** use cheaper models for simple pages (schedules, title + sheets), save the expensive model for dense drawings; keep tuning the + thinking budget knobs; reuse cached answers when the same plan set is + re-run. + +### 4. Smarter grouping — "better piles, better detective work" +- The Connector caps how many topic piles it keeps (a fixed limit), so on big + sets some connections may never be made. The Detectives only see one pile at + a time, so a contradiction spanning two piles can slip through. +- **Improvement:** revisit the pile limit, and let the Brain (or a second-pass + Detective) look for conflicts that span multiple piles. + +### 5. Human time — "review less, but review what matters" +- Today the review queue is built from rules about severity and confidence. +- **Improvement:** learn from your past Confirm/Reject decisions to sort the + queue better — the system already records your feedback, so it can get + smarter over time about what actually needs your eyes. + +### 6. Trust — "show the receipts" +- Findings carry evidence, but a non-technical reader can't easily see *where + on the drawing* the problem is. +- **Improvement:** attach a cropped image snippet of the exact spot on the + sheet to each finding, so anyone can verify it in seconds. + +--- + +*Technical reference for the curious: the pipeline lives in +`backend/agents/runner.py` (the waves above are the "Agent wave N" stages), the +team's shared notebook is `backend/agents/memory.py`, and the review queue is +`backend/review/gate.py` + `backend/review/finalizer.py`.* diff --git a/docs/superpowers/plans/2026-07-28-agent-human-review.md b/docs/superpowers/plans/2026-07-28-agent-human-review.md new file mode 100644 index 0000000..68ed077 --- /dev/null +++ b/docs/superpowers/plans/2026-07-28-agent-human-review.md @@ -0,0 +1,867 @@ +# Agent Human Review Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add required human review to the Agent pipeline so findings are confirmed, rejected, clarified, and measured before final RFIs/reports are issued. + +**Architecture:** Keep the existing Agent pipeline through Brain, then insert a ReviewGate that writes a persistent review queue and moves the job to `needs_review`. A ReviewFinalizer applies human decisions, performs bounded targeted reruns for clarification, drafts RFIs only for kept issues, and only then marks the job done and sends final email. + +**Tech Stack:** Python 3, FastAPI, pytest, vanilla JS frontend, JSON file artifacts under `backend/outputs//`. + +## Global Constraints + +- Do not change Classic pipeline behavior. +- Agent mode remains OpenRouter-only in v1. +- No final email before human review finalization. +- No raw `source_text`, sheet images, or drawing content in aggregate metrics by default. +- All new review logic must have non-LLM tests. +- Follow existing patterns: small modules, graceful degradation, JSON artifacts under job output dir. +- Review endpoints are state-changing and must be treated as sensitive in docs and deployment notes. + +--- + +### Task 1: Review schemas and policy + +**Files:** +- Create: `backend/review/__init__.py` +- Create: `backend/review/schemas.py` +- Create: `backend/review/policy.py` +- Test: `tests/review/test_policy.py` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `DECISIONS = {"confirm", "reject", "unsure", "needs_clarification"}` + - `REASON_CODES = {"wrong_cluster_link", "same_value_different_representation", "not_a_contradiction", "missing_evidence", "extraction_misread", "code_path_not_applicable", "duplicate", "severity_too_high", "severity_too_low", "other"}` + - `validate_decision(raw: dict) -> dict | None` + - `requires_review(issue: dict) -> list[str]` + - `build_audit_sample(memory_snapshot: dict, prioritized: list[dict], limit: int = 5) -> list[dict]` + +- [ ] **Step 1: Write failing policy tests** + +```python +from backend.review.policy import requires_review + + +def test_high_severity_requires_review(): + issue = {"severity": "high", "confidence": "high", "category": "note_or_spec_contradiction", "source_stage": "conflict"} + assert "severity_high" in requires_review(issue) + + +def test_low_confidence_requires_review(): + issue = {"severity": "low", "confidence": "low", "category": "note_or_spec_contradiction", "source_stage": "conflict"} + assert "confidence_low" in requires_review(issue) + + +def test_sensitive_code_category_requires_review(): + issue = {"severity": "medium", "confidence": "high", "category": "egress", "source_stage": "code"} + assert "sensitive_category" in requires_review(issue) + + +def test_medium_high_confidence_note_does_not_require_review(): + issue = {"severity": "medium", "confidence": "high", "category": "note_or_spec_contradiction", "source_stage": "conflict"} + assert requires_review(issue) == [] +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/review/test_policy.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'backend.review'` + +- [ ] **Step 3: Implement schemas and policy** + +```python +# backend/review/schemas.py +from typing import Optional + +DECISIONS = {"confirm", "reject", "unsure", "needs_clarification"} +REASON_CODES = { + "wrong_cluster_link", + "same_value_different_representation", + "not_a_contradiction", + "missing_evidence", + "extraction_misread", + "code_path_not_applicable", + "duplicate", + "severity_too_high", + "severity_too_low", + "other", +} + + +def validate_decision(raw: dict) -> Optional[dict]: + if not isinstance(raw, dict): + return None + decision = str(raw.get("decision") or "").strip() + if decision not in DECISIONS: + return None + reason_code = raw.get("reason_code") + if decision == "reject": + reason_code = str(reason_code or "").strip() + if reason_code not in REASON_CODES: + return None + elif reason_code is not None: + reason_code = str(reason_code).strip() or None + if reason_code and reason_code not in REASON_CODES: + return None + return { + "review_item_id": str(raw.get("review_item_id") or "").strip(), + "decision": decision, + "reason_code": reason_code, + "category_correction": raw.get("category_correction"), + "severity_correction": raw.get("severity_correction"), + "comment": str(raw.get("comment") or "").strip(), + "clarification_answer": raw.get("clarification_answer"), + "reviewed_at": raw.get("reviewed_at"), + } +``` + +```python +# backend/review/policy.py +from typing import Dict, List + +_SENSITIVE_CATEGORIES = { + "missing_element", + "ada", + "tas_tdlr", + "egress", + "fire_separation", + "occupancy", + "spatial_clash", + "clearance_conflict", + "penetration_conflict", +} + + +def requires_review(issue: Dict) -> List[str]: + reasons: List[str] = [] + severity = str(issue.get("severity") or "").lower() + confidence = str(issue.get("confidence") or "").lower() + category = str(issue.get("category") or "").lower() + if severity in {"critical", "high"}: + reasons.append("severity_high") + if confidence == "low": + reasons.append("confidence_low") + if category in _SENSITIVE_CATEGORIES or issue.get("source_stage") == "code": + reasons.append("sensitive_category") + return reasons +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pytest tests/review/test_policy.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/review tests/review/test_policy.py +git commit -m "Add review decision schema and trigger policy" +``` + +--- + +### Task 2: Review persistence + +**Files:** +- Create: `backend/review/store.py` +- Test: `tests/review/test_store.py` + +**Interfaces:** +- Consumes: `validate_decision` from Task 1. +- Produces: + - `ReviewStore(job_out_dir: str)` + - `.write_queue(queue: list[dict]) -> None` + - `.read_queue() -> list[dict]` + - `.append_decision(decision: dict) -> None` + - `.read_decisions() -> dict[str, dict]` + - `.progress(queue: list[dict]) -> dict` + +- [ ] **Step 1: Write failing persistence tests** + +```python +import json +from backend.review.store import ReviewStore + + +def test_queue_and_decisions_round_trip(tmp_path): + store = ReviewStore(str(tmp_path)) + queue = [{"review_item_id": "finding:1", "blocking": True}] + store.write_queue(queue) + assert store.read_queue() == queue + store.append_decision({"review_item_id": "finding:1", "decision": "confirm"}) + assert store.read_decisions()["finding:1"]["decision"] == "confirm" + + +def test_progress_counts_required_items(tmp_path): + store = ReviewStore(str(tmp_path)) + queue = [ + {"review_item_id": "a", "blocking": True}, + {"review_item_id": "b", "blocking": False}, + ] + store.write_queue(queue) + store.append_decision({"review_item_id": "a", "decision": "confirm"}) + progress = store.progress(queue) + assert progress["required"] == 1 + assert progress["completed"] == 1 +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/review/test_store.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'backend.review.store'` + +- [ ] **Step 3: Implement ReviewStore** + +```python +import json +import os +from typing import Dict, List + +from backend.review.schemas import validate_decision + + +class ReviewStore: + def __init__(self, job_out_dir: str) -> None: + self.review_dir = os.path.join(job_out_dir, "review") + os.makedirs(self.review_dir, exist_ok=True) + + def _path(self, name: str) -> str: + return os.path.join(self.review_dir, name) + + def _write_json(self, name: str, value) -> None: + path = self._path(name) + tmp = f"{path}.tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(value, f, indent=2) + os.replace(tmp, path) + + def write_queue(self, queue: List[dict]) -> None: + self._write_json("review_queue.json", queue) + + def read_queue(self) -> List[dict]: + try: + with open(self._path("review_queue.json"), encoding="utf-8") as f: + value = json.load(f) + return value if isinstance(value, list) else [] + except (OSError, json.JSONDecodeError): + return [] + + def append_decision(self, decision: dict) -> None: + valid = validate_decision(decision) + if not valid or not valid["review_item_id"]: + raise ValueError("invalid review decision") + decisions = self.read_decisions() + decisions[valid["review_item_id"]] = valid + self._write_json("review_decisions.json", decisions) + + def read_decisions(self) -> Dict[str, dict]: + try: + with open(self._path("review_decisions.json"), encoding="utf-8") as f: + value = json.load(f) + return value if isinstance(value, dict) else {} + except (OSError, json.JSONDecodeError): + return {} + + def progress(self, queue: List[dict]) -> dict: + decisions = self.read_decisions() + required = [item for item in queue if item.get("blocking")] + completed = [item for item in required if item.get("review_item_id") in decisions] + return { + "required": len(required), + "completed": len(completed), + "remaining": len(required) - len(completed), + "total": len(queue), + } +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pytest tests/review/test_store.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/review/store.py tests/review/test_store.py +git commit -m "Add persistent review store" +``` + +--- + +### Task 3: ReviewGate queue builder + +**Files:** +- Create: `backend/review/gate.py` +- Test: `tests/review/test_gate.py` + +**Interfaces:** +- Consumes: `requires_review`, `build_audit_sample` from Task 1. +- Produces: + - `build_review_queue(memory_snapshot: dict, prioritized: list[dict], decisions: list[dict]) -> list[dict]` + - queue item shape: `{ "review_item_id": str, "kind": "finding|audit_finding|clean_cluster", "blocking": bool, "reasons": list[str], "payload": dict }` + +- [ ] **Step 1: Write failing gate tests** + +```python +from backend.review.gate import build_review_queue + + +def test_gate_marks_blocking_and_audit_items(): + memory = {"clusters": [{"key": "room:101", "location": "Room 101", "assertions": [{"id": "a1"}, {"id": "a2"}]}], "findings": []} + prioritized = [ + {"issue_id": "AGENT-0001", "severity": "high", "confidence": "high", "category": "note_or_spec_contradiction", "source_stage": "conflict"}, + {"issue_id": "AGENT-0002", "severity": "low", "confidence": "high", "category": "note_or_spec_contradiction", "source_stage": "conflict"}, + ] + queue = build_review_queue(memory, prioritized, []) + by_id = {item["review_item_id"]: item for item in queue} + assert by_id["finding:AGENT-0001"]["blocking"] is True + assert by_id["finding:AGENT-0002"]["blocking"] is False + assert any(item["kind"] == "clean_cluster" for item in queue) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/review/test_gate.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'backend.review.gate'` + +- [ ] **Step 3: Implement ReviewGate** + +```python +from typing import Dict, List + +from backend.review.policy import build_audit_sample, requires_review + + +def _finding_item(issue: Dict, blocking: bool, reasons: List[str], kind: str) -> Dict: + issue_id = issue.get("issue_id") or "unknown" + return { + "review_item_id": f"finding:{issue_id}", + "kind": kind, + "blocking": blocking, + "reasons": reasons, + "payload": issue, + } + + +def build_review_queue(memory_snapshot: Dict, prioritized: List[Dict], decisions: List[Dict]) -> List[Dict]: + queue: List[Dict] = [] + for issue in prioritized: + reasons = requires_review(issue) + queue.append(_finding_item(issue, bool(reasons), reasons, "finding" if reasons else "audit_finding")) + for item in build_audit_sample(memory_snapshot, prioritized): + queue.append(item) + return queue +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pytest tests/review/test_gate.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/review/gate.py tests/review/test_gate.py +git commit -m "Add review gate queue builder" +``` + +--- + +### Task 4: Agent runner stops after Brain + +**Files:** +- Modify: `backend/agents/runner.py` +- Modify: `cli/run_check.py` +- Test: `tests/agents/test_runner_review_gate.py` + +**Interfaces:** +- Consumes: `build_review_queue`, `ReviewStore`. +- Produces: + - `run_agent_pipeline(..., require_review: bool = True) -> dict` + - candidate report contains `summary.agent_status = "needs_review"` and `summary.review = {"required": int, "completed": 0, "blocking": int}` when review is required. + +- [ ] **Step 1: Write failing runner gate test** + +```python +from backend.agents.runner import run_agent_pipeline + + +def test_agent_runner_can_enter_review_mode(monkeypatch, tmp_path): + monkeypatch.setattr("backend.agents.runner.convert_pdf_to_images", lambda path: [{"page_number": 1, "base64": "x"}]) + monkeypatch.setattr("backend.agents.runner.BrainAgent", lambda usage: type("B", (), {"run": lambda self, findings, sheet_index, jurisdiction: ([{"issue_id": "AGENT-0001", "severity": "high", "confidence": "high", "category": "note_or_spec_contradiction", "source_stage": "conflict"}], [])})()) + report = run_agent_pipeline("dummy.pdf", out_dir=str(tmp_path), require_review=True) + assert report["summary"]["agent_status"] == "needs_review" + assert report["summary"]["review"]["required"] == 1 +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `pytest tests/agents/test_runner_review_gate.py -v` +Expected: FAIL because `require_review` is not a supported argument. + +- [ ] **Step 3: Implement review-mode branch in runner** + +```python +from backend.review.gate import build_review_queue +from backend.review.store import ReviewStore + + +def run_agent_pipeline(..., require_review: bool = True) -> Dict: + # existing waves through Brain remain unchanged + if require_review: + memory_snapshot = memory.snapshot() + queue = build_review_queue(memory_snapshot, prioritized, decisions) + store = ReviewStore(out_dir) + store.write_queue(queue) + candidate_conflicts = [_finding_as_conflict(item) for item in conflict_findings] + report = build_report( + conflicts=candidate_conflicts, + sheets=sheets, + clusters=clusters, + source=source_name or os.path.basename(pdf_path), + ) + report.update({ + "project_input": merged_input, + "jurisdiction": jurisdiction, + "sheet_index": sheet_index, + "project_intelligence": object_graph, + "validated_issues": prioritized, + "rfis": [], + "suppressed_issues": [], + }) + progress = store.progress(queue) + report["summary"].update({ + "pipeline_mode": "agent", + "agent_status": "needs_review", + "review": progress, + }) + if out_dir: + _dump(out_dir, "conflicts.json", report) + _dump(out_dir, "validated_issues.json", prioritized) + return report + # existing RFI/report path remains for require_review=False +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `pytest tests/agents/test_runner_review_gate.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/agents/runner.py cli/run_check.py tests/agents/test_runner_review_gate.py +git commit -m "Gate agent runs behind required human review" +``` + +--- + +### Task 5: Job states and review API + +**Files:** +- Modify: `backend/jobs.py` +- Modify: `backend/main.py` +- Test: `tests/api/test_review_api.py` + +**Interfaces:** +- Consumes: `ReviewStore`, `validate_decision`. +- Produces: + - statuses: `needs_review`, `reviewing`, `finalizing`, `finalization_error` + - `GET /jobs/{job_id}/review -> {"queue": list[dict], "progress": dict}` + - `POST /jobs/{job_id}/review-decisions` + +- [ ] **Step 1: Write failing API tests** + +```python +from fastapi.testclient import TestClient +from backend.main import app + + +def test_review_queue_and_decision_save(monkeypatch, tmp_path): + client = TestClient(app) + monkeypatch.setattr("backend.main.get_job", lambda job_id: {"job_id": job_id, "status": "needs_review", "report": {"summary": {}}, "out_dir": str(tmp_path)}) + queue_response = client.get("/jobs/job1/review") + assert queue_response.status_code == 200 + decision_response = client.post("/jobs/job1/review-decisions", json={"decisions": [{"review_item_id": "finding:AGENT-0001", "decision": "confirm"}]}) + assert decision_response.status_code == 200 +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/api/test_review_api.py -v` +Expected: FAIL with 404 because review endpoints do not exist. + +- [ ] **Step 3: Implement job status and endpoints** + +```python +# backend/main.py +from backend.review.store import ReviewStore + + +@app.get("/jobs/{job_id}/review") +def review_queue(job_id: str): + job = get_job(job_id) + if not job: + raise HTTPException(status_code=404, detail="Job not found") + out_dir = job.get("out_dir") or os.path.join(config.OUTPUT_DIR, job_id) + store = ReviewStore(out_dir) + queue = store.read_queue() + return {"queue": queue, "progress": store.progress(queue)} + + +@app.post("/jobs/{job_id}/review-decisions") +def save_review_decisions(job_id: str, payload: dict): + job = get_job(job_id) + if not job: + raise HTTPException(status_code=404, detail="Job not found") + out_dir = job.get("out_dir") or os.path.join(config.OUTPUT_DIR, job_id) + store = ReviewStore(out_dir) + for decision in payload.get("decisions") or []: + store.append_decision(decision) + return {"progress": store.progress(store.read_queue())} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pytest tests/api/test_review_api.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/jobs.py backend/main.py tests/api/test_review_api.py +git commit -m "Add review job states and API endpoints" +``` + +--- + +### Task 6: Review finalizer and targeted rerun + +**Files:** +- Create: `backend/review/finalizer.py` +- Modify: `backend/agents/runner.py` +- Modify: `backend/main.py` +- Test: `tests/review/test_finalizer.py` + +**Interfaces:** +- Consumes: `ReviewStore`, queue items from Task 3, Agent runner helpers. +- Produces: + - `finalize_review(job_id: str, out_dir: str) -> dict` + - `apply_decisions(prioritized: list[dict], decisions: dict[str, dict]) -> tuple[list[dict], list[dict]]` + - `rerun_clarified_scopes(memory_snapshot: dict, decisions: dict[str, dict]) -> list[dict]` + - `POST /jobs/{job_id}/finalize-review` returns `409` until blocking decisions are complete + +- [ ] **Step 1: Write failing finalizer tests** + +```python +from backend.review.finalizer import apply_decisions + + +def test_reject_suppresses_with_reason(): + prioritized = [{"issue_id": "AGENT-0001", "severity": "high"}] + decisions = {"finding:AGENT-0001": {"decision": "reject", "reason_code": "duplicate"}} + kept, suppressed = apply_decisions(prioritized, decisions) + assert kept == [] + assert suppressed[0]["review_state"] == "rejected" + assert suppressed[0]["reason_code"] == "duplicate" + + +def test_unsure_is_kept_but_flagged(): + prioritized = [{"issue_id": "AGENT-0002", "severity": "medium"}] + decisions = {"finding:AGENT-0002": {"decision": "unsure"}} + kept, suppressed = apply_decisions(prioritized, decisions) + assert kept[0]["review_state"] == "unsure" + assert suppressed == [] +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/review/test_finalizer.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'backend.review.finalizer'` + +- [ ] **Step 3: Implement finalizer decision application** + +```python +from typing import Dict, List, Tuple + + +def apply_decisions(prioritized: List[dict], decisions: Dict[str, dict]) -> Tuple[List[dict], List[dict]]: + kept: List[dict] = [] + suppressed: List[dict] = [] + for issue in prioritized: + review_id = f"finding:{issue.get('issue_id')}" + decision = decisions.get(review_id) or {} + action = decision.get("decision") + if action == "reject": + suppressed.append({ + **issue, + "review_state": "rejected", + "reason_code": decision.get("reason_code"), + "review_comment": decision.get("comment") or "", + }) + elif action == "unsure": + kept.append({**issue, "review_state": "unsure"}) + else: + kept.append({**issue, "review_state": "confirmed" if action == "confirm" else "unreviewed"}) + return kept, suppressed +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pytest tests/review/test_finalizer.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/review/finalizer.py backend/agents/runner.py tests/review/test_finalizer.py +git commit -m "Finalize reviewed agent findings" +``` + +--- + +### Task 7: Feedback labels and metrics + +**Files:** +- Create: `backend/review/feedback.py` +- Create: `backend/review/metrics.py` +- Test: `tests/review/test_feedback.py` + +**Interfaces:** +- Consumes: queue items and validated decisions. +- Produces: + - `decision_to_label(queue_item: dict, decision: dict, job: dict) -> dict` + - `write_label(out_dir: str, label: dict) -> None` + - `aggregate_labels(labels: list[dict], include_text: bool = False) -> dict` + +- [ ] **Step 1: Write failing feedback tests** + +```python +from backend.review.metrics import aggregate_labels + + +def test_aggregate_redacts_text_by_default(): + labels = [{"decision": "reject", "reason_code": "missing_evidence", "comment": "secret", "payload": {"evidence": [{"source_text": "secret"}]}}] + summary = aggregate_labels(labels) + assert summary["reject"] == 1 + assert "secret" not in str(summary) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/review/test_feedback.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'backend.review.metrics'` + +- [ ] **Step 3: Implement label writing and aggregation** + +```python +from collections import Counter +from typing import Dict, List + + +def aggregate_labels(labels: List[dict], include_text: bool = False) -> Dict: + decisions = Counter(label.get("decision") or "unknown" for label in labels) + reasons = Counter(label.get("reason_code") or "none" for label in labels if label.get("decision") == "reject") + summary = { + "total": len(labels), + "decisions": dict(decisions), + "reject_reasons": dict(reasons), + } + for label in labels: + decision = label.get("decision") or "unknown" + summary[decision] = summary.get(decision, 0) + 1 + if include_text: + summary["labels"] = labels + return summary +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pytest tests/review/test_feedback.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/review/feedback.py backend/review/metrics.py tests/review/test_feedback.py +git commit -m "Add review feedback labels and aggregate metrics" +``` + +--- + +### Task 8: Two-phase email + +**Files:** +- Modify: `backend/email_sender.py` +- Modify: `backend/jobs.py` +- Test: `tests/api/test_review_email_flow.py` + +**Interfaces:** +- Consumes: existing `_smtp_ready` and `_send` helpers. +- Produces: + - `send_review_required(recipient_email: str, report: dict, review_url: str) -> bool` + +- [ ] **Step 1: Write failing email flow test** + +```python +from backend.email_sender import send_review_required + + +def test_review_required_email_skips_without_smtp(monkeypatch): + monkeypatch.setattr("backend.email_sender._smtp_ready", lambda: False) + assert send_review_required("user@example.com", {"source": "set.pdf", "summary": {}}, "http://localhost:8099/?job=abc") is False +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/api/test_review_email_flow.py -v` +Expected: FAIL with `ImportError: cannot import name 'send_review_required'` + +- [ ] **Step 3: Implement review-required email** + +```python +def send_review_required(recipient_email: str, report: dict, review_url: str) -> bool: + if not recipient_email or not _smtp_ready(): + return False + msg = EmailMessage() + msg["Subject"] = f"Conflict Checker - review required - {report.get('source', 'drawing set')}" + msg["From"] = config.SMTP_FROM or config.SMTP_USER + msg["To"] = recipient_email + review = report.get("summary", {}).get("review", {}) + msg.set_content( + "Agent analysis is complete and waiting for human review.\n\n" + f"Required review items: {review.get('required', 0)}\n" + f"Review URL: {review_url}\n" + ) + return _send(msg) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pytest tests/api/test_review_email_flow.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/email_sender.py backend/jobs.py tests/api/test_review_email_flow.py +git commit -m "Send review-required email before final report" +``` + +--- + +### Task 9: Frontend review queue + +**Files:** +- Modify: `frontend/index.html` +- Test: `tests/api/test_review_api.py` plus manual browser check + +**Interfaces:** +- Consumes: `GET /jobs/{id}`, `GET /jobs/{id}/review`, `POST /jobs/{id}/review-decisions`, `POST /jobs/{id}/finalize-review`. +- Produces: browser flow for `needs_review` jobs. + +- [ ] **Step 1: Add failing API expectation for review progress field** + +```python +def test_job_includes_review_progress(monkeypatch): + # Extend tests/api/test_review_api.py to assert get_job returns report.summary.review. + assert "review" in {"summary": {"review": {"required": 1, "completed": 0}}}["summary"] +``` + +- [ ] **Step 2: Run tests to verify current behavior** + +Run: `pytest tests/api/test_review_api.py -v` +Expected: PASS for API fields added in Task 5. + +- [ ] **Step 3: Implement minimal review UI** + +Add a `renderReview(job)` path in `frontend/index.html` that: +- fetches `/jobs/${jobId}/review`, +- renders blocking items first, +- shows `payload.description`, `payload.location`, `payload.category`, `payload.severity`, `payload.confidence`, and `payload.evidence`, +- requires a reason code when `reject` is selected, +- posts decisions to `/jobs/${jobId}/review-decisions`, +- calls `/jobs/${jobId}/finalize-review` only when `progress.remaining === 0`. + +- [ ] **Step 4: Manual browser check** + +Run: `uvicorn backend.main:app --reload --port 8099` +Expected: a synthetic `needs_review` job shows the queue, decisions persist across refresh, and finalize is blocked until required items are decided. + +- [ ] **Step 5: Commit** + +```bash +git add frontend/index.html tests/api/test_review_api.py +git commit -m "Add frontend human review queue" +``` + +--- + +### Task 10: Config, docs, and rollout + +**Files:** +- Modify: `backend/config.py` +- Modify: `backend/.env.example` +- Modify: `README.md` +- Test: `tests/review/test_policy.py`, `tests/review/test_store.py`, `tests/review/test_gate.py`, `tests/agents/test_runner_review_gate.py`, `tests/api/test_review_api.py`, `tests/review/test_finalizer.py`, `tests/review/test_feedback.py`, `tests/api/test_review_email_flow.py` + +**Interfaces:** +- Consumes: all previous tasks. +- Produces: + - `AGENT_REQUIRE_REVIEW = true` + - `AGENT_REVIEW_AUDIT_SAMPLE = 5` + - `REVIEW_AGGREGATE_INCLUDE_TEXT = false` + +- [ ] **Step 1: Add config assertions to existing policy test file** + +```python +from backend import config + + +def test_review_defaults(): + assert config.AGENT_REQUIRE_REVIEW is True + assert config.AGENT_REVIEW_AUDIT_SAMPLE == 5 + assert config.REVIEW_AGGREGATE_INCLUDE_TEXT is False +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `pytest tests/review/test_policy.py::test_review_defaults -v` +Expected: FAIL with `AttributeError` for missing config values. + +- [ ] **Step 3: Implement config and docs** + +Add to `backend/config.py`: + +```python +AGENT_REQUIRE_REVIEW = os.getenv("AGENT_REQUIRE_REVIEW", "true").strip().lower() in ("1", "true", "yes") +AGENT_REVIEW_AUDIT_SAMPLE = int(os.getenv("AGENT_REVIEW_AUDIT_SAMPLE", "5")) +REVIEW_AGGREGATE_INCLUDE_TEXT = os.getenv("REVIEW_AGGREGATE_INCLUDE_TEXT", "false").strip().lower() in ("1", "true", "yes") +``` + +Add the same keys to `backend/.env.example` and document the two-email flow and privacy boundary in `README.md`. + +- [ ] **Step 4: Run full test suite** + +Run: `pytest -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add backend/config.py backend/.env.example README.md tests +git commit -m "Configure required agent human review" +``` + +--- + +## Execution Handoff + +Plan complete and saved to `docs/superpowers/plans/2026-07-28-agent-human-review.md`. Two execution options: + +**1. Subagent-Driven (recommended)** - Dispatch a fresh subagent per task, review between tasks, fast iteration. + +**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints. + +Which approach? diff --git a/docs/superpowers/specs/2026-07-28-agent-human-review-design.md b/docs/superpowers/specs/2026-07-28-agent-human-review-design.md new file mode 100644 index 0000000..1f83e8b --- /dev/null +++ b/docs/superpowers/specs/2026-07-28-agent-human-review-design.md @@ -0,0 +1,245 @@ +# Required Human Review for Agent Pipeline Design + +**Date:** 2026-07-28 +**Status:** Approved +**Owner:** Conflict Checker Agent pipeline + +## Goal + +Make Agent mode produce higher-quality findings by requiring structured human review before final RFIs/reports are issued, and by turning review decisions into usable feedback for future prompt, rule, threshold, and evaluation improvements. + +## Background + +Agent mode is intended to replace Classic mode. Its advantage is the holistic project picture: sheet extraction, sheet index, jurisdiction, semantic linking, specialist findings, Brain consolidation, and RFI generation. The main quality risks are missed real conflicts, false positives, weak or unsupported findings, and silent stage/scope degradation. + +There are not enough known-good golden sets to rely only on golden-set regression. Human review becomes the feedback mechanism. The human is not expected to review every raw extraction; the human reviews a curated queue after Brain consolidation and before final report/RFI issuance. + +## Requirements + +### Functional requirements + +1. Agent web jobs must not reach `done` until required human review is complete. +2. The Agent pipeline runs through Brain, then enters `needs_review`. +3. RFI generation happens only after review finalization. +4. Required review items include: + - all critical/high severity findings, + - all low-confidence findings, + - sensitive categories: missing element, code/ADA/egress/fire separation, spatial clash/clearance, + - a small audit sample of medium/low findings and clean/no-finding clusters. +5. Review decisions support `confirm`, `reject`, `unsure`, and `needs_clarification`. +6. Rejections require a reason code. +7. Review progress persists to disk and survives server restart. +8. Rejected findings are suppressed, not deleted. +9. Clarifications are stored as first-class artifacts. +10. Where practical, clarification triggers targeted rerun of only the affected scope. +11. Aggregate feedback must not contain raw drawing text/images by default. +12. Classic mode remains unchanged. + +### Non-functional requirements + +- No automatic prompt mutation from human labels. +- No final email before review completion. +- Review endpoints must be treated as state-changing and sensitive. +- Review logic must be testable without LLM calls, PDFs, OpenRouter, or network access. +- Targeted reruns must degrade gracefully and must not crash finalization. + +## Architecture + +Add three small components. + +### ReviewGate + +Runs after Brain and before RFI/report finalization. + +Consumes: + +- `ProjectMemory` snapshot +- Brain prioritized issues +- Brain decisions +- review policy + +Produces: + +- `review/review_queue.json` +- candidate report with `summary.agent_status = "needs_review"` +- job transition to `needs_review` + +### ReviewStore + +Owns review persistence under the job output directory. + +Stores: + +- `review/review_queue.json` +- `review/review_decisions.json` +- `review/review_progress.json` + +Writes must be atomic using a temporary file plus `os.replace`, matching the existing LLM cache/report artifact style. + +### ReviewFinalizer + +Runs after required decisions are submitted. + +Responsibilities: + +- validate completeness, +- apply decisions, +- perform bounded targeted reruns for clarification where supported, +- re-run Brain only for affected findings, +- draft RFIs only for kept/confirmed issues, +- write final artifacts, +- transition to `done`, +- send final email. + +## Job lifecycle + +Current lifecycle: + +`queued -> running -> done -> email` + +New Agent lifecycle: + +`queued -> running -> needs_review -> reviewing -> finalizing -> done -> email` + +Additional failure state: + +- `finalization_error` + +If the server restarts while a job is in `needs_review` or `reviewing`, the backend rebuilds state from `outputs//conflicts.json`, `outputs//review/review_queue.json`, and `outputs//review/review_decisions.json`. + +## Email behavior + +If email is enabled, Agent mode sends two emails: + +1. **Review required** when the job enters `needs_review`. +2. **Final report** only after review finalization. + +If SMTP is not configured, the UI still shows `needs_review` and no email failure crashes the job. + +## Review queue policy + +Blocking review items are findings that meet any of these rules: + +- severity is `critical` or `high`, +- confidence is `low`, +- category is `missing_element`, +- source stage is `code`, +- category is in `ada`, `tas_tdlr`, `egress`, `fire_separation`, `occupancy`, `spatial_clash`, `clearance_conflict`, or `penetration_conflict`. + +Audit sample items are selected deterministically from: + +- medium/low findings not already blocking, +- clean clusters with no findings, +- no-finding scopes when available. + +Default audit sample size is 5 items. + +## Review decision schema + +```json +{ + "review_item_id": "finding:AGENT-0007", + "decision": "reject", + "reason_code": "same_value_different_representation", + "category_correction": null, + "severity_correction": null, + "comment": "9'-0\" AFF and 108 inches are the same value here.", + "clarification_answer": null, + "reviewed_at": "2026-07-28T12:00:00Z" +} +``` + +Allowed reason codes: + +- `wrong_cluster_link` +- `same_value_different_representation` +- `not_a_contradiction` +- `missing_evidence` +- `extraction_misread` +- `code_path_not_applicable` +- `duplicate` +- `severity_too_high` +- `severity_too_low` +- `other` + +## Finalization rules + +- All blocking review items must have a valid decision before finalization. +- Confirmed findings become final `validated_issues`. +- Unsure findings remain included but are flagged as `review_state = "unsure"`. +- Rejected findings become `suppressed_issues` with reason code and comment. +- Clarification answers are stored and, when the affected scope is rerunnable, trigger a targeted rerun. +- Targeted rerun failure creates an `analysis_gap` finding and does not block finalization unless the reviewer chooses to reject the affected item. +- RFIs are drafted only for final kept issues. + +## Feedback labels + +Every decision emits a label artifact for metrics: + +```json +{ + "review_item_id": "finding:AGENT-0007", + "job_id": "abc123", + "pipeline_mode": "agent", + "source_stage": "conflict", + "category": "elevation_disagreement", + "severity": "high", + "confidence": "medium", + "decision": "reject", + "reason_code": "same_value_different_representation", + "location": "Room 204 / Level 2", + "disciplines": ["Architectural", "Mechanical"], + "sheets": ["A2.1", "M2.1"], + "drawing_type": "floor_plan", + "models_used": ["google/gemini-2.5-pro"], + "created_at": "2026-07-28T12:00:00Z" +} +``` + +Default aggregate metrics exclude `source_text`, images, raw sheet content, and reviewer free-text comments. + +## API shape + +- `GET /jobs/{job_id}` includes `needs_review`, `reviewing`, `finalizing`, `done`, `error`, or `finalization_error` plus review progress. +- `GET /jobs/{job_id}/review` returns `{ "queue": [...], "progress": {...} }`. +- `POST /jobs/{job_id}/review-decisions` saves one or more decisions. +- `POST /jobs/{job_id}/finalize-review` validates completeness and finalizes the job. + +## Security and privacy + +Review endpoints are more sensitive than read-only report endpoints because they mutate job state and expose evidence. Before required review is enabled beyond a trusted LAN, the app should have reverse-proxy auth, a shared access token, or explicit deployment documentation stating that the UI/API must not be exposed publicly. + +Review artifacts stay job-local by default. Cross-job aggregate metrics use metadata and reason codes only unless richer retention is explicitly enabled later. + +## Testing strategy + +Tests must not require PDFs, LLMs, OpenRouter, or network access. + +Cover: + +- required-review trigger policy, +- review queue construction, +- decision validation and reason codes, +- finalization behavior for confirm/reject/unsure/clarification, +- restart recovery from review artifacts, +- targeted rerun failure degradation, +- metrics redaction, +- API state transitions, +- email flow blocking until finalization. + +## Rollout + +- Classic mode is unchanged. +- Agent web jobs default to required human review. +- CLI supports an explicit bypass flag, `--no-review`, for tuning/debug runs. +- Review state and decisions are always written to job artifacts. +- Aggregate feedback is metadata-only by default. + +## Acceptance criteria + +- An Agent web job cannot reach `done` or send the final email while required review items are undecided. +- Rejected findings are suppressed with reason codes and remain auditable. +- Review progress survives server restart. +- Clarification failures degrade to visible `analysis_gap`, not job failure. +- Aggregate feedback contains no raw drawing text/images by default. +- New tests cover the review gate without requiring LLM calls. diff --git a/docs/superpowers/specs/2026-08-12-text-layer-grounding-design.md b/docs/superpowers/specs/2026-08-12-text-layer-grounding-design.md new file mode 100644 index 0000000..e7be7f7 --- /dev/null +++ b/docs/superpowers/specs/2026-08-12-text-layer-grounding-design.md @@ -0,0 +1,166 @@ +# Text-Layer Grounding — Design Spec + +**Date:** 2026-08-12 · **Branch:** `agent-mode` · **Status:** approved by user (2026-08-12) + +## Problem + +The pipeline is vision-only for extraction, but most CAD-produced drawing sets +carry a real vector text layer. Two worst documented failure modes are text +problems being solved with pixels: + +1. **Wave-1 text misreads propagate immutably** — e.g. job `959e16407573`: + vision read "(2) 2x6 STUD PACK" where the sheet says "(5)"; text-only + downstream specialists treated the misread as ground truth → confident + false-positive findings. +2. **Silent extraction loss** — failed/under-extracted pages are invisible + (job `475a6f184dd1`: 42% extraction loss), producing false + `missing_expected_sheets` warnings and missed conflicts. + +Priority (user, 2026-08-12): reduce false positives **and** missed items; +more accurate conflicts. + +## Approach + +Extract the PDF text layer deterministically (PyMuPDF) once per job, and make +it a first-class citizen at three points: extractor grounding, the grounding +guard, and the wave-5b verifier (as text oracle + high-DPI evidence crops). + +Inspired by `hamzaabduljabbar/construction-drawing-analyzer` (patterns only — +its license is source-available/no-resale; all code here is original). + +## Components + +### 1. New module `backend/text_layer.py` (deterministic, no LLM) + +- `extract_text_layers(pdf_path) -> Dict[int, dict]` — per 1-based page: + `{"text": str, "words": [{"text", "bbox": (x0,y0,x1,y1)}, ...], + "has_text_layer": bool}`. Pages with < `TEXT_LAYER_MIN_CHARS` of text are + `has_text_layer=False` (scanned/raster sheets stay vision-only; logged). +- `find_evidence_bbox(words, needle) -> bbox | None` — best-effort fuzzy + substring match of an evidence `source_text` against word sequence; returns + union rect of matched words. +- `render_crop(pdf_path, page_number, bbox, dpi, margin_pts) -> bytes` — + PyMuPDF `page.get_pixmap(clip=rect, dpi=dpi)` → JPEG bytes. + +Both runners call `extract_text_layers` right after `convert_pdf_to_images` +and attach `page["text_layer"] = ` to each page dict. Word +positions stay in a separate `page_words: Dict[int, list]` runner-local map +(not attached to page dicts — they get serialized). + +### 2. Extractor grounding (both pipelines) + +- Static paragraph added to `_EXTRACTOR_SYSTEM_TEMPLATE` in + `backend/prompts.py` (no new placeholder): when a TEXT LAYER block is + present in the user message it is **authoritative for alphanumeric content** + (counts, dimensions, member tags, notes); the image is for geometry, + symbols, linework, and anything absent from the text layer. +- Text-layer content is **appended programmatically** at each extractor call + site (classic `extractor.py::_extract_one`, agent + `extractors.py::SheetExtractorAgent.run`) — NOT a new `{placeholder}` in the + shared template (two-render-path trap: `render()` silently leaves missing + keys as literals). Block capped at `TEXT_LAYER_MAX_CHARS`. + Format: `\n\nTEXT LAYER (authoritative for alphanumeric content — trust it + over the image for numbers, tags, and note text):\n` + +### 3. Grounding guard rescue tier (`pipeline/extractor.py::_normalize_sheet`) + +Current guard drops an object when its primary value's digit-runs aren't in +its own `source_text`. New tier, only when a text layer exists for the page: + +- digits ⊆ source_text → keep (unchanged) +- digits ⊆ page text layer but ⊄ source_text → keep, stamp + `grounding: "text_layer"` on the assertion (recall rescue — vision quoted + imperfectly but the value is real page text) +- otherwise → drop (unchanged) + +`_is_grounded` gains an optional `page_text` param; existing callers/tests +unaffected. Dropped/ rescued counts logged per page. + +### 4. Verifier: text oracle + high-DPI crops (wave 5b) + +Wherever verify scopes are built (agent runner confirmed; classic runner to be +checked — integrate at both if present): + +- Scope payload gains `text_layer_excerpt`: concatenated text of the finding's + cited sheets, capped at `VERIFY_TEXT_MAX_CHARS`. `VERIFY_USER_INSTRUCTION` + gains a `{text_layer}` placeholder with instructions to treat it as + deterministic page text (verdicts may cite it as `actual_text`). **Both + render sites** (agent verifier + any classic-path render) must substitute it + — grep the template name across `backend/agents/` and `backend/pipeline/`. +- When `VERIFY_HI_DPI_CROPS` and the page has words: for each evidence item, + `find_evidence_bbox` on the cited page's words; on hit, `render_crop` at + `VERIFY_CROP_DPI` with margin → crop images replace full-page images (up to + `AGENT_CONFLICT_MAX_IMAGES`). On any miss/failure → fall back to the current + full-page image. Zero-resolved-images ⇒ scope skipped (I2 guard preserved). + +### 5. Coverage signal (recall) + +After extraction in both runners: for each page with `has_text_layer=True` +whose extraction failed or returned 0 objects, log +`[TextLayer] Page N: text layer present (M chars) but no objects extracted — +possible extraction gap` and add the page to the existing gap-finding path +(agent: `orchestrator.stats.failed_scopes`-style finding; classic: log only). + +## Config knobs (`backend/config.py`, env-overridable, documented in `.env.example`) + +| Key | Default | Effect | +|-----|---------|--------| +| `TEXT_LAYER_ENABLED` | `true` | Master switch | +| `TEXT_LAYER_MIN_CHARS` | `20` | Below this per page → `has_text_layer=False` | +| `TEXT_LAYER_MAX_CHARS` | `12000` | Cap per sheet injected into extractor prompt | +| `VERIFY_TEXT_MAX_CHARS` | `8000` | Cap of text-layer excerpt in verify scope | +| `VERIFY_HI_DPI_CROPS` | `true` | Evidence-located crops in verifier | +| `VERIFY_CROP_DPI` | `300` | Crop render DPI | +| `VERIFY_CROP_MARGIN_PTS` | `36` | Padding around evidence bbox (PDF points) | + +## Known traps (from project history — designed around) + +- **Two render paths:** no new `{placeholder}` in extractor templates; the one + new placeholder (`{text_layer}` in VERIFY_USER_INSTRUCTION) substituted at + every render site; a render test asserts no `{...}` literals remain. +- **ProjectMemory closed registry:** no new memory keys. Text artifacts dump + via plain file writes under `outputs//text/` (agent: under `agent/`). +- **`slim_clusters`:** no new cluster fields — unchanged. +- **I2 zero-image path:** crops replace full-page images only on confident + bbox match; never reduce image count to zero. +- **Base64 hygiene:** page dicts already carry base64; `text_layer` strings + must not leak into `clusters.json` dumps — reuse `_without_base64` pattern + if assertions ever carry page refs (they don't today). + +## Dependencies + +`PyMuPDF>=1.23` added to `requirements.txt` (Docker image rebuild picks it up; +pdf2image/poppler unchanged). + +## Testing + +- `tests/test_text_layer.py` — build tiny PDFs with PyMuPDF in-test: + extraction, `has_text_layer` thresholds, `find_evidence_bbox` hit/miss, + `render_crop` dimensions. +- Extractor guard: rescue-tier unit tests (keep-with-flag, still-drop, + unchanged behavior without text layer). +- Prompt render test: extractor + verify instructions fully substituted at + every site (both pipelines). +- Runner-level (pattern from `tests/agents/test_wave5b_suppression.py`): + stubbed waves, assert text layer reaches extract scopes and verify scopes + (excerpt present, crop fallback on no-match), full `run_agent_pipeline`. +- Full `pytest tests/` green before push. + +## Validation (post-deploy) + +Re-run the Cypress set (source PDF persists at +`/app/backend/outputs/959e16407573/source.pdf` on sits-docker) per the +documented re-run workflow. Success criteria: + +1. The "(2) vs (5)"-class findings are not generated, or are verifier-refuted + with text-layer evidence cited. +2. Coverage-gap log lines appear for any page with text but no objects. +3. No new `finish_reason=length` in waves 1/4; cost delta reported vs + baseline job. + +## Out of scope (future PRs) + +- Legend/symbol-library wave injected into extractor + critic prompts. +- Deterministic schedule-row recall pass (text-layer tables → assertions). +- pdf-markup export for the review UI. +- Takeoff/polygon geometry (belongs to AI_Takeoffs, not this product). diff --git a/frontend/index.html b/frontend/index.html index e13494b..b98ee25 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -3,6 +3,9 @@ + + + Conflict Checker