Compare commits
18
Commits
0ea0b0e897
...
agent-mode
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bae608a505 | ||
|
|
fe09e4a66b | ||
|
|
48fefa4007 | ||
|
|
23f6d7fe89 | ||
|
|
06e108142e | ||
|
|
0d109fb5cd | ||
|
|
570300324f | ||
|
|
349b357e5c | ||
|
|
b174b531cd | ||
|
|
3df359500c | ||
|
|
82952df307 | ||
|
|
c8af430143 | ||
|
|
f21eb5d912 | ||
|
|
4a3f33a245 | ||
|
|
15297038a2 | ||
|
|
d431a026ce | ||
|
|
8631a26006 | ||
|
|
df4d15fd0c |
@@ -0,0 +1,936 @@
|
|||||||
|
# Evidence Verification + Cross-Sheet Correlation Implementation Plan
|
||||||
|
|
||||||
|
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||||||
|
|
||||||
|
**Goal:** Stop vision-extraction misreads (e.g. "(2) 2x6 STUD PACK" vs the actual "(5) 2x6") from becoming confident downstream findings, and correlate the same physical element across sheets (S101/S205/S401) so no stage reasons from one sheet's text in isolation.
|
||||||
|
|
||||||
|
**Architecture:** Three independently shippable phases on the agent pipeline (`backend/agents/runner.py`):
|
||||||
|
1. **Disputed-value detection** — deterministic post-link pass that flags contradictory extracted values inside a cluster and surfaces them to the critic/specialist prompts.
|
||||||
|
2. **Cross-sheet xref linking** — linker gains detail-reference/tag buckets that join assertions across levels (today `(level, family)` bucketing splits S101/S205/S401 apart).
|
||||||
|
3. **Evidence verification wave (5b)** — a bounded vision fact-check agent re-reads the cited sheet images for high-severity / disputed findings before the Brain merge, annotates or suppresses findings built on phantom text.
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.14, pytest (`tests/`), existing `call_json` LLM wrapper (supports `images_b64`, `reasoning_effort`, `reasoning_max_tokens`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current context / root cause (from job 959e16407573)
|
||||||
|
|
||||||
|
Finding `validated_issues[3]` ("FRONT PERSPECTIVE detail on Sheet S401", HSS16x4 on
|
||||||
|
"(2) 2x6 STUD PACK", severity critical) is a **false positive built on a wave-1 vision
|
||||||
|
misread**. The sheet actually shows a (5) 2x6 stud pack (matching S205/S101). Chain of failure:
|
||||||
|
|
||||||
|
1. Wave 1 (`SheetExtractorAgent`) froze the misread into text. From then on it is "ground truth".
|
||||||
|
2. Wave 3 linker (`backend/agents/linker.py:33` `build_link_scopes`) buckets by
|
||||||
|
`(level, family)`. S101 (foundation), S205 (details), S401 (sections) get different
|
||||||
|
`level` values, so assertions about the same front-wall header never share a link scope
|
||||||
|
or cluster. No cross-sheet corroboration happened.
|
||||||
|
3. Wave 5 `ConstructabilityAgent` (`backend/agents/construct_agent.py:53`) calls
|
||||||
|
`call_json` **with no images** — in this job 120/120 constructability calls were `+0img`.
|
||||||
|
It reasoned arithmetically from the misread text ("2 x 1.5in = 3in < 4in -> unbuildable").
|
||||||
|
It even held the "(5) 2x6 STUD PACK" assertion in the same scope but labeled it
|
||||||
|
"Ambiguous column size specification" instead of arbitrating.
|
||||||
|
4. Nothing between wave 5 and the report ever looks at a sheet image again. Only the wave-4
|
||||||
|
conflict critic receives images, and only for its own cluster's pages.
|
||||||
|
|
||||||
|
Also confirmed in this log (separate known bug, fixed in Task 7 while we're here): wave-4
|
||||||
|
conflict critic truncates on Gemini thinking tokens because `conflict_critic.py:59` passes
|
||||||
|
`max_tokens=config.REASON_MAX_TOKENS` (4096) with no reasoning budget — 13/121 calls hit
|
||||||
|
`finish_reason=length`.
|
||||||
|
|
||||||
|
## Assumptions
|
||||||
|
|
||||||
|
- Assertions carry `id`, `attribute`, `value`, `source_text`, `location_key`
|
||||||
|
(`room`/`grid`/`detail_reference`/`tag`/`level`) — see `linker._payload` and
|
||||||
|
`_serialize.slim_assertion`.
|
||||||
|
- Extractor assertions already carry a `confidence` field (per test fixtures).
|
||||||
|
- `validate_issue` in `backend/pipeline/_stage.py` guarantees each finding an `issue_id`.
|
||||||
|
- Test convention: `unittest.mock.patch("backend.agents.<module>.call_json", ...)` —
|
||||||
|
see `tests/agents/test_sheet_extractor_fallback.py`. Run tests with
|
||||||
|
`.venv/bin/python -m pytest tests/ -x -q`.
|
||||||
|
- `AgentResult.error` defaults to `""` (not None) in assertions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — Disputed-value detection + prompt hardening
|
||||||
|
|
||||||
|
### Task 1: `find_disputes` pure function (TDD)
|
||||||
|
|
||||||
|
**Objective:** Detect "same attribute, different values" inside one cluster's assertions.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `backend/agents/disputes.py`
|
||||||
|
- Test: `tests/agents/test_disputes.py`
|
||||||
|
|
||||||
|
**Step 1: Write failing test**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/agents/test_disputes.py
|
||||||
|
from backend.agents.disputes import annotate_clusters, find_disputes
|
||||||
|
|
||||||
|
|
||||||
|
def _a(id_, attribute, value):
|
||||||
|
return {"id": id_, "attribute": attribute, "value": value,
|
||||||
|
"source_text": value}
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_disputes_flags_same_attribute_different_values():
|
||||||
|
assertions = [
|
||||||
|
_a("a1", "stud_pack_size", "(2) 2x6 STUD PACK"),
|
||||||
|
_a("a2", "stud_pack_size", "(5) 2x6 STUD PACK"),
|
||||||
|
_a("a3", "beam_size", "HSS16X4X5/8"),
|
||||||
|
]
|
||||||
|
disputes = find_disputes(assertions)
|
||||||
|
assert len(disputes) == 1
|
||||||
|
assert disputes[0]["attribute"] == "stud_pack_size"
|
||||||
|
assert disputes[0]["values"] == ["(2) 2x6 STUD PACK", "(5) 2x6 STUD PACK"]
|
||||||
|
assert disputes[0]["assertion_ids"] == ["a1", "a2"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_disputes_ignores_agreeing_values_and_blanks():
|
||||||
|
assertions = [
|
||||||
|
_a("a1", "beam_size", "HSS16X4X5/8"),
|
||||||
|
_a("a2", "beam_size", " hss16x4x5/8 "), # same after normalize
|
||||||
|
_a("a3", "", "orphan"), # no attribute -> skipped
|
||||||
|
_a("a4", "beam_size", ""), # no value -> skipped
|
||||||
|
]
|
||||||
|
assert find_disputes(assertions) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_annotate_clusters_writes_disputed_attributes():
|
||||||
|
clusters = [
|
||||||
|
{"key": "c1", "assertions": [
|
||||||
|
_a("a1", "stud_pack_size", "(2) 2x6"),
|
||||||
|
_a("a2", "stud_pack_size", "(5) 2x6"),
|
||||||
|
]},
|
||||||
|
{"key": "c2", "assertions": [_a("a3", "x", "1"), _a("a4", "x", "1")]},
|
||||||
|
]
|
||||||
|
assert annotate_clusters(clusters) == 1
|
||||||
|
assert clusters[0]["disputed_attributes"][0]["attribute"] == "stud_pack_size"
|
||||||
|
assert "disputed_attributes" not in clusters[1]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Run test to verify failure**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py -v`
|
||||||
|
Expected: FAIL — `ModuleNotFoundError: backend.agents.disputes`
|
||||||
|
|
||||||
|
**Step 3: Implement**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# backend/agents/disputes.py
|
||||||
|
"""Deterministic detection of contradictory extracted values within a cluster.
|
||||||
|
|
||||||
|
Extraction is a vision pass: quantities and sizes can be misread ("(2) 2x6" vs
|
||||||
|
"(5) 2x6"). Cluster members are supposed to describe the same real-world
|
||||||
|
element, so two members asserting different values for the same attribute are
|
||||||
|
a probable misread. Flag these so downstream text-only stages treat the value
|
||||||
|
as unverified instead of reasoning from one reading.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict, List
|
||||||
|
|
||||||
|
|
||||||
|
def _norm(value) -> str:
|
||||||
|
return re.sub(r"\s+", " ", str(value or "").strip().lower())
|
||||||
|
|
||||||
|
|
||||||
|
def find_disputes(assertions: List[Dict]) -> List[Dict]:
|
||||||
|
"""Same attribute with >= 2 distinct normalized values = disputed."""
|
||||||
|
groups: Dict[str, Dict[str, set]] = {}
|
||||||
|
for assertion in assertions:
|
||||||
|
attribute = _norm(assertion.get("attribute"))
|
||||||
|
value = _norm(assertion.get("value"))
|
||||||
|
if not attribute or not value:
|
||||||
|
continue
|
||||||
|
groups.setdefault(attribute, {}).setdefault(value, set()).add(
|
||||||
|
assertion.get("id")
|
||||||
|
)
|
||||||
|
disputes = []
|
||||||
|
for attribute, values in sorted(groups.items()):
|
||||||
|
if len(values) < 2:
|
||||||
|
continue
|
||||||
|
disputes.append({
|
||||||
|
"attribute": attribute,
|
||||||
|
"values": sorted(values),
|
||||||
|
"assertion_ids": sorted(
|
||||||
|
aid for ids in values.values() for aid in ids if aid
|
||||||
|
),
|
||||||
|
})
|
||||||
|
return disputes
|
||||||
|
|
||||||
|
|
||||||
|
def annotate_clusters(clusters: List[Dict]) -> int:
|
||||||
|
"""Attach disputed_attributes to each cluster that has any. Returns count."""
|
||||||
|
annotated = 0
|
||||||
|
for cluster in clusters:
|
||||||
|
disputes = find_disputes(cluster.get("assertions") or [])
|
||||||
|
if disputes:
|
||||||
|
cluster["disputed_attributes"] = disputes
|
||||||
|
annotated += 1
|
||||||
|
return annotated
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4: Run test to verify pass**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py -v`
|
||||||
|
Expected: 3 passed
|
||||||
|
|
||||||
|
**Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add backend/agents/disputes.py tests/agents/test_disputes.py
|
||||||
|
git commit -m "feat: deterministic disputed-value detection for cluster assertions"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Wire `annotate_clusters` into the runner + serialization
|
||||||
|
|
||||||
|
**Objective:** Disputes must be visible to the wave-4 critic and wave-5 constructability prompts.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `backend/agents/runner.py` (after `memory.replace("clusters", clusters)`, ~line 116)
|
||||||
|
- Modify: `backend/pipeline/_serialize.py` (`slim_clusters`, line 46)
|
||||||
|
- Test: `tests/agents/test_disputes.py` (append)
|
||||||
|
|
||||||
|
**Step 1: Write failing test**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_slim_clusters_preserves_disputed_attributes():
|
||||||
|
from backend.pipeline._serialize import slim_clusters
|
||||||
|
cluster = {"key": "c1", "assertions": [],
|
||||||
|
"disputed_attributes": [{"attribute": "a", "values": ["1", "2"],
|
||||||
|
"assertion_ids": ["x", "y"]}]}
|
||||||
|
slim = slim_clusters([cluster])[0]
|
||||||
|
assert slim["disputed_attributes"][0]["values"] == ["1", "2"]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Run test to verify failure**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/agents/test_disputes.py::test_slim_clusters_preserves_disputed_attributes -v`
|
||||||
|
Expected: FAIL — `KeyError: 'disputed_attributes'`
|
||||||
|
|
||||||
|
**Step 3: Implement**
|
||||||
|
|
||||||
|
In `backend/pipeline/_serialize.py` `slim_clusters`, add the key:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def slim_clusters(clusters: List[Dict]) -> List[Dict]:
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"key": c.get("key"),
|
||||||
|
"location": c.get("location"),
|
||||||
|
"disciplines": c.get("disciplines"),
|
||||||
|
"kind": c.get("kind"),
|
||||||
|
**({"disputed_attributes": c["disputed_attributes"]}
|
||||||
|
if c.get("disputed_attributes") else {}),
|
||||||
|
"assertions": [slim_assertion(a) for a in c.get("assertions", [])],
|
||||||
|
}
|
||||||
|
for c in clusters
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
In `backend/agents/runner.py`, right after `clusters = [...]` / `object_graph = build_object_graph(clusters)` (before `memory.replace("clusters", clusters)`):
|
||||||
|
|
||||||
|
```python
|
||||||
|
from backend.agents.disputes import annotate_clusters
|
||||||
|
...
|
||||||
|
object_graph = build_object_graph(clusters)
|
||||||
|
disputed_count = annotate_clusters(clusters)
|
||||||
|
if disputed_count:
|
||||||
|
orchestrator.log(
|
||||||
|
f"[Link] {disputed_count} clusters carry disputed extracted values"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
(Check `Orchestrator` for the actual log method name — `orchestrator.stage(...)` exists;
|
||||||
|
if no `.log`, use the module's existing logging/print convention. Adjust to match.)
|
||||||
|
|
||||||
|
**Step 4: Run tests**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/agents/ -v`
|
||||||
|
Expected: all pass (including existing `test_runner_review_gate.py`)
|
||||||
|
|
||||||
|
**Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add backend/agents/runner.py backend/pipeline/_serialize.py tests/agents/test_disputes.py
|
||||||
|
git commit -m "feat: surface disputed extracted values to critic and specialist prompts"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Prompt hardening — extracted text is fallible
|
||||||
|
|
||||||
|
**Objective:** Tell text-only specialists how to handle disputed/unverified values so they stop asserting buildability conclusions from a single (possibly misread) number.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `backend/prompts.py` `CONSTRUCTABILITY_SYSTEM_PROMPT` (line 533) and `CONSTRUCTABILITY_USER_INSTRUCTION` (line 551)
|
||||||
|
|
||||||
|
**Step 1: Edit prompts**
|
||||||
|
|
||||||
|
Append to `CONSTRUCTABILITY_SYSTEM_PROMPT` Rules list (after line 547, before "Use plain ASCII"):
|
||||||
|
|
||||||
|
```
|
||||||
|
- Assertions are machine-extracted from sheet images and may contain misread values,
|
||||||
|
especially quantities and member sizes (e.g. "(2) 2x6" vs "(5) 2x6").
|
||||||
|
- When the cluster lists disputed_attributes, or two evidence items disagree on a
|
||||||
|
numeric value, do NOT assert a buildability conclusion from one reading. Report the
|
||||||
|
ambiguity itself (category "detail_gap", confidence "low") and state that the value
|
||||||
|
needs verification against the sheet.
|
||||||
|
```
|
||||||
|
|
||||||
|
Append to `CONSTRUCTABILITY_USER_INSTRUCTION` after the `Cross-discipline conflicts already found: {conflicts}` line:
|
||||||
|
|
||||||
|
```
|
||||||
|
Disputed extracted values in this cluster (possible vision misreads - treat as unverified): {disputes}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Wire the `{disputes}` placeholder in `construct_agent.py`**
|
||||||
|
|
||||||
|
In `backend/agents/construct_agent.py` `run()`, extend the `substitutions` dict:
|
||||||
|
|
||||||
|
```python
|
||||||
|
substitutions = {
|
||||||
|
"assertions": dumps(cluster["assertions"]),
|
||||||
|
"clusters": dumps(slim_clusters([cluster])),
|
||||||
|
"conflicts": dumps(scope.payload.get("conflicts") or []),
|
||||||
|
"disputes": dumps(cluster.get("disputed_attributes") or []),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Run full test suite (prompt edits can break runner tests that snapshot prompts)**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/ -q`
|
||||||
|
Expected: all pass
|
||||||
|
|
||||||
|
**Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add backend/prompts.py backend/agents/construct_agent.py
|
||||||
|
git commit -m "feat: constructability prompt treats disputed extracted values as unverified"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — Cross-sheet xref linking
|
||||||
|
|
||||||
|
### Task 4: detail-reference / tag xref buckets in the linker (TDD)
|
||||||
|
|
||||||
|
**Objective:** Assertions sharing a `detail_reference` or a member `tag` get linked across levels, so S101/S205/S401 details of the same physical element land in one scope.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `backend/agents/linker.py` (`build_link_scopes`, line 33)
|
||||||
|
- Test: `tests/agents/test_linker_xref.py`
|
||||||
|
|
||||||
|
**Step 1: Write failing test**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/agents/test_linker_xref.py
|
||||||
|
from backend.agents.base import AgentScope
|
||||||
|
from backend.agents.linker import build_link_scopes
|
||||||
|
|
||||||
|
|
||||||
|
def _sheet(number, page, level, assertions):
|
||||||
|
return {"sheet_number": number, "page_number": page,
|
||||||
|
"discipline": "Structural", "level": level,
|
||||||
|
"assertions": assertions}
|
||||||
|
|
||||||
|
|
||||||
|
def _assertion(id_, ref=None, tag=None, level=None):
|
||||||
|
return {"id": id_, "attribute": "stud_pack_size", "value": "(5) 2x6",
|
||||||
|
"source_text": "(5) 2x6 STUD PACK",
|
||||||
|
"location_key": {"detail_reference": ref, "tag": tag,
|
||||||
|
"level": level}}
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_joins_same_detail_reference_across_levels():
|
||||||
|
sheets = [
|
||||||
|
_sheet("S101", 10, "foundation", [_assertion("a1", ref="A/S205")]),
|
||||||
|
_sheet("S205", 20, "roof", [_assertion("a2", ref="A/S205")]),
|
||||||
|
_sheet("S401", 30, "roof", [_assertion("a3", ref="A/S205")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
xref = [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
assert xref, "expected a cross-level detail-reference scope"
|
||||||
|
ids = {a["id"] for s in xref for a in s.payload["assertions"]}
|
||||||
|
assert ids == {"a1", "a2", "a3"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_requires_two_distinct_sheets():
|
||||||
|
sheets = [
|
||||||
|
_sheet("S401", 30, "roof", [_assertion("a1", ref="A/S205"),
|
||||||
|
_assertion("a2", ref="A/S205")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
assert not [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_joins_shared_member_tag():
|
||||||
|
sheets = [
|
||||||
|
_sheet("S102", 5, "roof", [_assertion("a1", tag="HSS16X4X5/8")]),
|
||||||
|
_sheet("S401", 30, "unknown", [_assertion("a2", tag="HSS16X4X5/8")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
xref = [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
assert xref
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Run test to verify failure**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/agents/test_linker_xref.py -v`
|
||||||
|
Expected: FAIL — no `xref:` scopes produced
|
||||||
|
|
||||||
|
**Step 3: Implement**
|
||||||
|
|
||||||
|
Rewrite `build_link_scopes` in `backend/agents/linker.py` (keep the existing
|
||||||
|
`(level, family)` bucketing, add the xref pass):
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _xref_keys(assertion: Dict) -> List[str]:
|
||||||
|
"""Cross-level join keys: detail references and member tags."""
|
||||||
|
location = assertion.get("location_key") or {}
|
||||||
|
keys = []
|
||||||
|
ref = re.sub(r"\s+", "", str(location.get("detail_reference") or "")).upper()
|
||||||
|
if ref:
|
||||||
|
keys.append(f"detail:{ref}")
|
||||||
|
tag = re.sub(r"\s+", "", str(location.get("tag") or "")).upper()
|
||||||
|
if re.match(r"^[A-Z]{2,}\d", tag): # member marks: HSS16X4X5/8, W12X26, ...
|
||||||
|
keys.append(f"tag:{tag}")
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
|
def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]:
|
||||||
|
"""Partition facts by level and object/tag family, then enforce a hard cap.
|
||||||
|
|
||||||
|
A second pass joins assertions that share a detail_reference or member tag
|
||||||
|
ACROSS levels, so plan/detail/section sheets describing the same physical
|
||||||
|
element are linked together even though their levels differ.
|
||||||
|
"""
|
||||||
|
buckets: Dict[Tuple[str, str], List[Dict]] = defaultdict(list)
|
||||||
|
xref: Dict[str, List[Dict]] = defaultdict(list)
|
||||||
|
for sheet in sheets:
|
||||||
|
for assertion in sheet.get("assertions", []):
|
||||||
|
enriched = {
|
||||||
|
**assertion,
|
||||||
|
"discipline": sheet.get("discipline") or "Unknown",
|
||||||
|
"sheet_number": sheet.get("sheet_number"),
|
||||||
|
"page_number": sheet.get("page_number"),
|
||||||
|
}
|
||||||
|
level = str((assertion.get("location_key") or {}).get("level")
|
||||||
|
or sheet.get("level") or "unknown").lower()
|
||||||
|
buckets[(level, _family(assertion))].append(enriched)
|
||||||
|
for key in _xref_keys(assertion):
|
||||||
|
xref[key].append(enriched)
|
||||||
|
|
||||||
|
scopes: List[AgentScope] = []
|
||||||
|
cap = max(2, config.AGENT_LINK_MAX_ASSERTIONS)
|
||||||
|
for (level, family), assertions in sorted(buckets.items()):
|
||||||
|
for offset in range(0, len(assertions), cap):
|
||||||
|
chunk = assertions[offset:offset + cap]
|
||||||
|
if len(chunk) < 2:
|
||||||
|
continue
|
||||||
|
scopes.append(AgentScope(
|
||||||
|
scope_id=f"{level}:{family}:{offset // cap + 1}",
|
||||||
|
payload={"assertions": chunk, "level": level, "family": family},
|
||||||
|
))
|
||||||
|
for key, assertions in sorted(xref.items()):
|
||||||
|
sheets_present = {a.get("sheet_number") for a in assertions}
|
||||||
|
if len(assertions) < 2 or len(sheets_present) < 2:
|
||||||
|
continue
|
||||||
|
scopes.append(AgentScope(
|
||||||
|
scope_id=f"xref:{key}",
|
||||||
|
payload={"assertions": assertions[:cap],
|
||||||
|
"level": "xref", "family": key},
|
||||||
|
))
|
||||||
|
return scopes
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4: Run tests**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/agents/test_linker_xref.py tests/agents/ -v`
|
||||||
|
Expected: all pass (watch existing runner tests for scope-count coupling)
|
||||||
|
|
||||||
|
**Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add backend/agents/linker.py tests/agents/test_linker_xref.py
|
||||||
|
git commit -m "feat: cross-level xref link scopes via detail_reference and member tag"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3 — Evidence verification wave (5b)
|
||||||
|
|
||||||
|
### Task 5: Config knobs + verify prompts
|
||||||
|
|
||||||
|
**Objective:** Add the tuning surface and prompts for the vision fact-check agent.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `backend/config.py` (near line 44, with the other AGENT_* knobs)
|
||||||
|
- Modify: `backend/prompts.py` (append near the CONFLICT prompts, ~line 460)
|
||||||
|
- Modify: `backend/.env.example`
|
||||||
|
|
||||||
|
**Step 1: Add config knobs to `backend/config.py`**
|
||||||
|
|
||||||
|
```python
|
||||||
|
AGENT_VERIFY_MODEL = os.getenv("AGENT_VERIFY_MODEL", "") or MODEL
|
||||||
|
AGENT_VERIFY_CONCURRENCY = int(os.getenv("AGENT_VERIFY_CONCURRENCY", "4"))
|
||||||
|
AGENT_VERIFY_MAX_CHECKS = int(os.getenv("AGENT_VERIFY_MAX_CHECKS", "20"))
|
||||||
|
AGENT_VERIFY_SEVERITIES = {
|
||||||
|
s.strip().lower()
|
||||||
|
for s in os.getenv("AGENT_VERIFY_SEVERITIES", "critical,high").split(",")
|
||||||
|
if s.strip()
|
||||||
|
}
|
||||||
|
AGENT_VERIFY_REASONING_EFFORT = os.getenv("AGENT_VERIFY_REASONING_EFFORT", "low").strip()
|
||||||
|
VERIFY_MAX_TOKENS = int(os.getenv("VERIFY_MAX_TOKENS", "8192"))
|
||||||
|
```
|
||||||
|
|
||||||
|
Append to `backend/.env.example`:
|
||||||
|
|
||||||
|
```
|
||||||
|
# Wave 5b evidence verification (vision fact-check of cited sheet text)
|
||||||
|
AGENT_VERIFY_MAX_CHECKS=20
|
||||||
|
AGENT_VERIFY_SEVERITIES=critical,high
|
||||||
|
AGENT_VERIFY_REASONING_EFFORT=low
|
||||||
|
VERIFY_MAX_TOKENS=8192
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Add prompts to `backend/prompts.py`**
|
||||||
|
|
||||||
|
```python
|
||||||
|
VERIFY_SYSTEM_PROMPT = """You are a meticulous construction document checker verifying machine-extracted evidence against the actual drawing sheet images.
|
||||||
|
For each evidence item you are given the sheet it was extracted from and the verbatim text the extractor claims appears there.
|
||||||
|
Judge each item against the images:
|
||||||
|
- confirmed: the text (or an obvious equivalent) appears on the cited sheet and means what the finding claims.
|
||||||
|
- corrected: the sheet shows a DIFFERENT value than the extracted text. Give the actual verbatim text.
|
||||||
|
- not_found: nothing like the extracted text appears on the cited sheet.
|
||||||
|
Be strict about numbers, quantities, and member sizes: "(2) 2x6" and "(5) 2x6" are different values. HSS16x4 and HSS16x16 are different values.
|
||||||
|
Use plain ASCII only.
|
||||||
|
Respond only with valid JSON."""
|
||||||
|
|
||||||
|
VERIFY_USER_INSTRUCTION = """Verify this finding's evidence against the attached sheet images.
|
||||||
|
Respond ONLY with a valid JSON object - no markdown fences, no explanation:
|
||||||
|
{ "verdicts": [ { "sheet": "string", "source_text": "the evidence text judged", "verdict": "confirmed | corrected | not_found", "actual_text": "verbatim sheet text when corrected, else null", "notes": "string or null" } ] }
|
||||||
|
Finding: {finding}"""
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Sanity check**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -c "from backend import config, prompts; print(config.AGENT_VERIFY_MAX_CHECKS, config.AGENT_VERIFY_SEVERITIES); print(prompts.VERIFY_SYSTEM_PROMPT[:40])"`
|
||||||
|
Expected: `20 {'critical', 'high'}` and prompt text
|
||||||
|
|
||||||
|
**Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add backend/config.py backend/prompts.py backend/.env.example
|
||||||
|
git commit -m "feat: config knobs and prompts for evidence verification wave"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 6: `EvidenceVerifierAgent` + runner wave 5b (TDD)
|
||||||
|
|
||||||
|
**Objective:** Re-read cited sheet images for selected findings; annotate verified findings, suppress refuted ones before the Brain merge.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `backend/agents/verifier.py`
|
||||||
|
- Modify: `backend/agents/runner.py` (new wave between wave 5 and wave 6, ~line 167)
|
||||||
|
- Modify: `backend/agents/construct_agent.py` line 67 (stamp `cluster_key` for dispute-based selection)
|
||||||
|
- Test: `tests/agents/test_verifier.py`
|
||||||
|
|
||||||
|
**Step 1: Write failing test**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/agents/test_verifier.py
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from backend.agents.base import AgentScope, AgentUsage
|
||||||
|
from backend.agents.verifier import (
|
||||||
|
EvidenceVerifierAgent, apply_verdicts, select_findings,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _finding(sev="critical", issue_id="i1", sheets=("S401",), cluster_key=None):
|
||||||
|
f = {"issue_id": issue_id, "severity": sev, "confidence": "high",
|
||||||
|
"source_stage": "constructability", "sheets": list(sheets),
|
||||||
|
"description": "HSS16x4 on (2) 2x6 STUD PACK is unbuildable",
|
||||||
|
"evidence": [{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
|
||||||
|
"asserted_value": "3-inch width"}]}
|
||||||
|
if cluster_key:
|
||||||
|
f["cluster_key"] = cluster_key
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_findings_by_severity_and_dispute():
|
||||||
|
findings = [_finding("critical"), _finding("low", "i2"),
|
||||||
|
_finding("medium", "i3", cluster_key="c9")]
|
||||||
|
clusters = [{"key": "c9", "disputed_attributes": [{"attribute": "a"}]}]
|
||||||
|
selected = select_findings(findings, clusters, max_checks=20,
|
||||||
|
severities={"critical", "high"})
|
||||||
|
assert [f["issue_id"] for f in selected] == ["i1", "i3"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_findings_respects_cap():
|
||||||
|
findings = [_finding("critical", f"i{n}") for n in range(30)]
|
||||||
|
selected = select_findings(findings, [], max_checks=5,
|
||||||
|
severities={"critical"})
|
||||||
|
assert len(selected) == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_attaches_verdicts_and_marks_refuted():
|
||||||
|
agent = EvidenceVerifierAgent(usage=AgentUsage())
|
||||||
|
scope = AgentScope(scope_id="verify:0", payload={
|
||||||
|
"finding_index": 0,
|
||||||
|
"finding": _finding(),
|
||||||
|
"images_b64": ["QUJD"],
|
||||||
|
})
|
||||||
|
verdicts = {"verdicts": [
|
||||||
|
{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
|
||||||
|
"verdict": "corrected", "actual_text": "(5) 2x6 STUD PACK",
|
||||||
|
"notes": "callout reads (5)"},
|
||||||
|
]}
|
||||||
|
with patch("backend.agents.verifier.call_json", return_value=verdicts):
|
||||||
|
result = agent.run(scope)
|
||||||
|
assert not result.error
|
||||||
|
artifact = result.artifacts[0]
|
||||||
|
assert artifact["finding_index"] == 0
|
||||||
|
assert artifact["status"] == "refuted" # no evidence confirmed
|
||||||
|
assert artifact["verdicts"][0]["actual_text"] == "(5) 2x6 STUD PACK"
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_verdicts_annotates_and_suppresses():
|
||||||
|
findings = [_finding("critical", "i1"), _finding("high", "i2")]
|
||||||
|
from backend.agents.base import AgentResult
|
||||||
|
results = [AgentResult(scope_id="verify:0", artifacts=[
|
||||||
|
{"finding_index": 0, "status": "refuted", "verdicts": []},
|
||||||
|
{"finding_index": 1, "status": "confirmed", "verdicts": []},
|
||||||
|
])]
|
||||||
|
suppressed = apply_verdicts(findings, results)
|
||||||
|
assert suppressed == [findings[0]]
|
||||||
|
assert findings[0]["verification"]["status"] == "refuted"
|
||||||
|
assert findings[1]["verification"]["status"] == "confirmed"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Run test to verify failure**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/agents/test_verifier.py -v`
|
||||||
|
Expected: FAIL — `ModuleNotFoundError: backend.agents.verifier`
|
||||||
|
|
||||||
|
**Step 3: Implement `backend/agents/verifier.py`**
|
||||||
|
|
||||||
|
```python
|
||||||
|
"""Wave 5b: vision fact-check of extracted evidence against cited sheet images.
|
||||||
|
|
||||||
|
Downstream specialists are text-only; a wave-1 vision misread ("(2) 2x6" vs
|
||||||
|
"(5) 2x6") otherwise becomes immutable ground truth. For high-severity or
|
||||||
|
dispute-linked findings, re-read the cited sheets and adjudicate each evidence
|
||||||
|
item: confirmed / corrected / not_found. Findings whose evidence is entirely
|
||||||
|
unconfirmed are suppressed before the Brain merge.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Dict, List, Optional, Set
|
||||||
|
|
||||||
|
from backend import config
|
||||||
|
from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure
|
||||||
|
from backend.llm import call_json
|
||||||
|
from backend.pipeline._serialize import dumps
|
||||||
|
from backend.pipeline._stage import collect_list, render
|
||||||
|
from backend.prompts import VERIFY_SYSTEM_PROMPT, VERIFY_USER_INSTRUCTION
|
||||||
|
|
||||||
|
_SEVERITY_RANK = {"critical": 0, "high": 1, "medium": 2, "low": 3}
|
||||||
|
_VERDICTS = ("confirmed", "corrected", "not_found")
|
||||||
|
|
||||||
|
|
||||||
|
def select_findings(
|
||||||
|
findings: List[Dict],
|
||||||
|
clusters: List[Dict],
|
||||||
|
max_checks: int,
|
||||||
|
severities: Set[str],
|
||||||
|
) -> List[Dict]:
|
||||||
|
"""Severity-gated selection plus any finding tied to a disputed cluster."""
|
||||||
|
disputed_keys = {
|
||||||
|
cluster.get("key") for cluster in clusters
|
||||||
|
if cluster.get("disputed_attributes")
|
||||||
|
}
|
||||||
|
selected = [
|
||||||
|
finding for finding in findings
|
||||||
|
if str(finding.get("severity") or "").lower() in severities
|
||||||
|
or finding.get("cluster_key") in disputed_keys
|
||||||
|
]
|
||||||
|
selected.sort(key=lambda f: _SEVERITY_RANK.get(
|
||||||
|
str(f.get("severity") or "").lower(), 9))
|
||||||
|
return selected[:max_checks]
|
||||||
|
|
||||||
|
|
||||||
|
def _valid_verdict(item: Dict) -> Optional[Dict]:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
return None
|
||||||
|
verdict = str(item.get("verdict") or "").lower()
|
||||||
|
if verdict not in _VERDICTS:
|
||||||
|
return None
|
||||||
|
return {
|
||||||
|
"sheet": item.get("sheet") or "",
|
||||||
|
"source_text": item.get("source_text") or "",
|
||||||
|
"verdict": verdict,
|
||||||
|
"actual_text": item.get("actual_text"),
|
||||||
|
"notes": item.get("notes"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _status(verdicts: List[Dict]) -> str:
|
||||||
|
if not verdicts:
|
||||||
|
return "unverified"
|
||||||
|
confirmed = sum(1 for v in verdicts if v["verdict"] == "confirmed")
|
||||||
|
if confirmed == len(verdicts):
|
||||||
|
return "confirmed"
|
||||||
|
if confirmed == 0:
|
||||||
|
return "refuted"
|
||||||
|
return "mixed"
|
||||||
|
|
||||||
|
|
||||||
|
class EvidenceVerifierAgent:
|
||||||
|
name = "verify"
|
||||||
|
|
||||||
|
def __init__(self, usage: AgentUsage) -> None:
|
||||||
|
self.usage = usage
|
||||||
|
|
||||||
|
def run(self, scope: AgentScope) -> AgentResult:
|
||||||
|
try:
|
||||||
|
finding = scope.payload["finding"]
|
||||||
|
instruction = render(
|
||||||
|
VERIFY_USER_INSTRUCTION, {"finding": dumps(finding)}
|
||||||
|
)
|
||||||
|
parsed = call_json(
|
||||||
|
system_prompt=VERIFY_SYSTEM_PROMPT,
|
||||||
|
user_text=instruction,
|
||||||
|
images_b64=scope.payload.get("images_b64") or [],
|
||||||
|
max_tokens=config.VERIFY_MAX_TOKENS,
|
||||||
|
model=config.AGENT_VERIFY_MODEL,
|
||||||
|
reasoning_effort=config.AGENT_VERIFY_REASONING_EFFORT or None,
|
||||||
|
usage_tracker=self.usage,
|
||||||
|
usage_stage="agent.verify",
|
||||||
|
)
|
||||||
|
verdicts = collect_list(parsed, "verdicts", _valid_verdict)
|
||||||
|
return AgentResult(scope_id=scope.scope_id, artifacts=[{
|
||||||
|
"finding_index": scope.payload["finding_index"],
|
||||||
|
"status": _status(verdicts),
|
||||||
|
"verdicts": verdicts,
|
||||||
|
}])
|
||||||
|
except Exception as exc:
|
||||||
|
return failure(scope, exc)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_verdicts(
|
||||||
|
findings: List[Dict], verify_results: List[AgentResult]
|
||||||
|
) -> List[Dict]:
|
||||||
|
"""Annotate findings with verification; return refuted ones to suppress."""
|
||||||
|
by_index: Dict[int, Dict] = {}
|
||||||
|
for result in verify_results:
|
||||||
|
for artifact in result.artifacts:
|
||||||
|
by_index[artifact["finding_index"]] = artifact
|
||||||
|
suppressed = []
|
||||||
|
for index, finding in enumerate(findings):
|
||||||
|
artifact = by_index.get(index)
|
||||||
|
if not artifact:
|
||||||
|
continue
|
||||||
|
finding["verification"] = {
|
||||||
|
"status": artifact["status"],
|
||||||
|
"verdicts": artifact["verdicts"],
|
||||||
|
}
|
||||||
|
if artifact["status"] == "refuted":
|
||||||
|
finding["confidence"] = "low"
|
||||||
|
suppressed.append(finding)
|
||||||
|
return suppressed
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4: Stamp `cluster_key` on constructability findings**
|
||||||
|
|
||||||
|
In `backend/agents/construct_agent.py` line 66-67, change:
|
||||||
|
|
||||||
|
```python
|
||||||
|
for finding in findings:
|
||||||
|
finding.update(agent=self.name, scope_id=scope.scope_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
```python
|
||||||
|
for finding in findings:
|
||||||
|
finding.update(agent=self.name, scope_id=scope.scope_id,
|
||||||
|
cluster_key=cluster.get("key"))
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 5: Wire wave 5b into `backend/agents/runner.py`**
|
||||||
|
|
||||||
|
After `memory.extend("findings", specialist_findings)` (line 167) and before
|
||||||
|
`gap_findings` / wave 6:
|
||||||
|
|
||||||
|
```python
|
||||||
|
orchestrator.stage("Agent wave 5b: evidence verification")
|
||||||
|
sheet_to_page = {
|
||||||
|
sheet.get("sheet_number"): sheet.get("page_number") for sheet in sheets
|
||||||
|
}
|
||||||
|
verify_targets = select_findings(
|
||||||
|
specialist_findings, clusters,
|
||||||
|
max_checks=config.AGENT_VERIFY_MAX_CHECKS,
|
||||||
|
severities=config.AGENT_VERIFY_SEVERITIES,
|
||||||
|
)
|
||||||
|
target_indexes = {id(f): i for i, f in enumerate(specialist_findings)}
|
||||||
|
verify_scopes = [
|
||||||
|
AgentScope(
|
||||||
|
scope_id=f"verify:{target_indexes[id(finding)]}",
|
||||||
|
payload={
|
||||||
|
"finding_index": target_indexes[id(finding)],
|
||||||
|
"finding": finding,
|
||||||
|
"images_b64": [
|
||||||
|
page_to_b64[sheet_to_page[name]]
|
||||||
|
for name in (finding.get("sheets") or [])
|
||||||
|
[:config.AGENT_CONFLICT_MAX_IMAGES]
|
||||||
|
if sheet_to_page.get(name) in page_to_b64
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for finding in verify_targets
|
||||||
|
]
|
||||||
|
verify_results = orchestrator.run_scopes(
|
||||||
|
EvidenceVerifierAgent(usage), verify_scopes,
|
||||||
|
config.AGENT_VERIFY_CONCURRENCY,
|
||||||
|
)
|
||||||
|
suppressed = apply_verdicts(specialist_findings, verify_results)
|
||||||
|
if suppressed:
|
||||||
|
suppressed_ids = {id(f) for f in suppressed}
|
||||||
|
specialist_findings = [
|
||||||
|
f for f in specialist_findings if id(f) not in suppressed_ids
|
||||||
|
]
|
||||||
|
memory.replace("suppressed", suppressed)
|
||||||
|
memory.extend("findings", specialist_findings) # see note below
|
||||||
|
```
|
||||||
|
|
||||||
|
NOTE for implementer: `memory.extend("findings", ...)` already ran with the
|
||||||
|
un-suppressed list. Adjust ordering so verification happens BEFORE
|
||||||
|
`memory.extend("findings", specialist_findings)` — i.e. move the extend to after
|
||||||
|
wave 5b — so the Brain never sees refuted findings. Keep `gap_findings` logic
|
||||||
|
unchanged. Also add imports at top of runner.py:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from backend.agents.verifier import (
|
||||||
|
EvidenceVerifierAgent, apply_verdicts, select_findings,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
And in the report dicts (both the `require_review` branch ~line 217-225 and the
|
||||||
|
wave-7 branch ~line 293-300), populate suppressed issues:
|
||||||
|
|
||||||
|
```python
|
||||||
|
"suppressed_issues": memory.snapshot().get("suppressed") or [],
|
||||||
|
```
|
||||||
|
|
||||||
|
Finally, `verify` results cost shows up as `agent.verify` in
|
||||||
|
`summary.cost_by_stage` automatically via `usage_stage="agent.verify"`.
|
||||||
|
|
||||||
|
**Step 6: Update existing runner tests**
|
||||||
|
|
||||||
|
`tests/agents/test_runner_review_gate.py` monkeypatches `BrainAgent` and
|
||||||
|
`convert_pdf_to_images` but lets waves 1-5 run against... check how LLM calls
|
||||||
|
are stubbed there (likely `call_json` returns None -> empty artifacts, which is
|
||||||
|
fine). The new wave must no-op cleanly when `select_findings` returns `[]`
|
||||||
|
(zero scopes -> `run_scopes` returns `[]` per orchestrator.py:60). Verify by
|
||||||
|
running the suite; if a runner test now fails because verification selects a
|
||||||
|
stubbed finding, monkeypatch `select_findings` to `lambda *a, **k: []` in that
|
||||||
|
test file's `_patch_brain` helper.
|
||||||
|
|
||||||
|
**Step 7: Run full suite**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/ -q`
|
||||||
|
Expected: all pass
|
||||||
|
|
||||||
|
**Step 8: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add backend/agents/verifier.py backend/agents/runner.py backend/agents/construct_agent.py tests/agents/test_verifier.py tests/agents/test_runner_review_gate.py
|
||||||
|
git commit -m "feat: wave 5b evidence verification - vision fact-check before Brain merge"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 7 (small, related): reasoning budget for the conflict critic
|
||||||
|
|
||||||
|
**Objective:** Fix the wave-4 truncation found in this same job (13/121 calls hit `finish_reason=length` at the 4096 cap with ~3.7k thinking tokens).
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `backend/agents/conflict_critic.py:55-63`
|
||||||
|
- Modify: `backend/pipeline/conflict_checker.py:85` (same pattern, classic path)
|
||||||
|
|
||||||
|
**Step 1: Apply the extractor's reasoning-knob pattern**
|
||||||
|
|
||||||
|
```python
|
||||||
|
parsed = call_json(
|
||||||
|
system_prompt=CONFLICT_SYSTEM_PROMPT,
|
||||||
|
user_text=instruction,
|
||||||
|
images_b64=images,
|
||||||
|
max_tokens=config.REASON_MAX_TOKENS,
|
||||||
|
model=config.AGENT_CONFLICT_MODEL,
|
||||||
|
reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
|
||||||
|
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
|
||||||
|
usage_tracker=self.usage,
|
||||||
|
usage_stage="agent.conflict",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
(Match the exact kwarg names `SheetExtractorAgent` uses — check
|
||||||
|
`backend/agents/extractors.py` for whether it passes `None` when the budget is
|
||||||
|
0, and mirror that guard.)
|
||||||
|
|
||||||
|
**Step 2: Run tests**
|
||||||
|
|
||||||
|
Run: `.venv/bin/python -m pytest tests/ -q`
|
||||||
|
Expected: all pass
|
||||||
|
|
||||||
|
**Step 3: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add backend/agents/conflict_critic.py backend/pipeline/conflict_checker.py
|
||||||
|
git commit -m "fix: reasoning budget for conflict critic (wave-4 max_tokens truncation)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Tests / validation
|
||||||
|
|
||||||
|
1. `.venv/bin/python -m pytest tests/ -q` — full suite green.
|
||||||
|
2. **Targeted repro of the original failure:** pull the S401 page image from job
|
||||||
|
959e16407573's output dir on sits-docker (or re-render the PDF page), then run
|
||||||
|
one `EvidenceVerifierAgent` scope locally against the finding JSON from
|
||||||
|
`validated_issues[3]`. Expected: verdict `corrected`,
|
||||||
|
`actual_text: "(5) 2x6 STUD PACK"`, status `refuted`.
|
||||||
|
3. **End-to-end:** rerun the same Cypress TX PDF through the pipeline (local with
|
||||||
|
`LLM_CACHE`/`LLM_RAW_DUMP` per the conflict-checker skill). Expected:
|
||||||
|
- log shows `Agent wave 5b: evidence verification` with a bounded number of calls;
|
||||||
|
- the S401 stud-pack finding is either absent from `validated_issues` and present
|
||||||
|
in `suppressed_issues` with verification verdicts, or downgraded to low confidence;
|
||||||
|
- `agent.verify` appears in `summary.cost_by_stage`;
|
||||||
|
- zero `finish_reason=length` lines in wave 4 (Task 7).
|
||||||
|
4. Cost check: wave 5b adds at most `AGENT_VERIFY_MAX_CHECKS` (20) vision calls —
|
||||||
|
for this job's profile that is well under $1.
|
||||||
|
|
||||||
|
## Risks, tradeoffs, open questions
|
||||||
|
|
||||||
|
- **Dispute false positives:** cluster members with legitimately different values
|
||||||
|
(e.g. two doors in one door cluster) will produce `disputed_attributes`. Mitigation:
|
||||||
|
prompts treat disputes as "unverified", not "wrong"; only severity-gated findings
|
||||||
|
burn verification calls. Tune later by restricting `find_disputes` to numeric-ish
|
||||||
|
values if noise is high.
|
||||||
|
- **Verifier can also misread.** It is one model checking another with the same eyes.
|
||||||
|
Mitigation: verdict requires `actual_text` verbatim evidence for `corrected`, and
|
||||||
|
only fully-unconfirmed findings are suppressed (mixed keeps the finding with a note).
|
||||||
|
- **xref cost:** extra link scopes. Bounded by the >= 2 distinct sheets gate and the
|
||||||
|
existing assertion cap; expect a handful of extra scopes per set.
|
||||||
|
- **Suppression in review mode:** refuted findings land in `suppressed_issues` — the
|
||||||
|
review UI/finalizer must tolerate that list being non-empty (it is currently always
|
||||||
|
`[]` in agent mode). Open question: surface suppressed items in the human review
|
||||||
|
queue as informational, or keep them report-only?
|
||||||
|
- **Open question:** should wave-4 conflict findings (which already saw images) also be
|
||||||
|
verification-eligible? Plan says no (they had the pixels); revisit if critics show
|
||||||
|
the same misread pattern.
|
||||||
@@ -72,3 +72,24 @@ SMTP_PASSWORD=
|
|||||||
SMTP_FROM=
|
SMTP_FROM=
|
||||||
SMTP_USE_TLS=true
|
SMTP_USE_TLS=true
|
||||||
SMTP_USE_SSL=false
|
SMTP_USE_SSL=false
|
||||||
|
|
||||||
|
# Wave 5b evidence verification (vision fact-check of cited sheet text)
|
||||||
|
AGENT_VERIFY_MAX_CHECKS=20
|
||||||
|
AGENT_VERIFY_SEVERITIES=critical,high
|
||||||
|
AGENT_VERIFY_REASONING_EFFORT=low
|
||||||
|
VERIFY_MAX_TOKENS=8192
|
||||||
|
|
||||||
|
# Text-layer grounding (deterministic PDF text layer via PyMuPDF)
|
||||||
|
# TEXT_LAYER_ENABLED: master switch for text-layer extraction/grounding
|
||||||
|
# TEXT_LAYER_MIN_CHARS: below this per page the sheet stays vision-only
|
||||||
|
# TEXT_LAYER_MAX_CHARS: cap of text layer injected into the extractor prompt
|
||||||
|
# VERIFY_TEXT_MAX_CHARS: cap of the text-layer excerpt in verify scopes
|
||||||
|
# VERIFY_HI_DPI_CROPS: evidence-located high-DPI crops in the verifier
|
||||||
|
# VERIFY_CROP_DPI / VERIFY_CROP_MARGIN_PTS: crop render DPI / padding (PDF points)
|
||||||
|
TEXT_LAYER_ENABLED=true
|
||||||
|
TEXT_LAYER_MIN_CHARS=20
|
||||||
|
TEXT_LAYER_MAX_CHARS=12000
|
||||||
|
VERIFY_TEXT_MAX_CHARS=8000
|
||||||
|
VERIFY_HI_DPI_CROPS=true
|
||||||
|
VERIFY_CROP_DPI=300
|
||||||
|
VERIFY_CROP_MARGIN_PTS=36
|
||||||
|
|||||||
@@ -60,6 +60,8 @@ class ConflictCriticAgent:
|
|||||||
model=config.AGENT_CONFLICT_MODEL,
|
model=config.AGENT_CONFLICT_MODEL,
|
||||||
usage_tracker=self.usage,
|
usage_tracker=self.usage,
|
||||||
usage_stage="agent.conflict",
|
usage_stage="agent.conflict",
|
||||||
|
reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
|
||||||
|
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
|
||||||
)
|
)
|
||||||
candidates = parsed if isinstance(parsed, list) else (
|
candidates = parsed if isinstance(parsed, list) else (
|
||||||
parsed.get("conflicts") if isinstance(parsed, dict) else []
|
parsed.get("conflicts") if isinstance(parsed, dict) else []
|
||||||
|
|||||||
@@ -47,6 +47,7 @@ class ConstructabilityAgent:
|
|||||||
"assertions": dumps(cluster["assertions"]),
|
"assertions": dumps(cluster["assertions"]),
|
||||||
"clusters": dumps(slim_clusters([cluster])),
|
"clusters": dumps(slim_clusters([cluster])),
|
||||||
"conflicts": dumps(scope.payload.get("conflicts") or []),
|
"conflicts": dumps(scope.payload.get("conflicts") or []),
|
||||||
|
"disputes": dumps(cluster.get("disputed_attributes") or []),
|
||||||
}
|
}
|
||||||
for key, value in substitutions.items():
|
for key, value in substitutions.items():
|
||||||
instruction = instruction.replace("{" + key + "}", value)
|
instruction = instruction.replace("{" + key + "}", value)
|
||||||
@@ -64,7 +65,8 @@ class ConstructabilityAgent:
|
|||||||
lambda item: validate_issue(item, "constructability"),
|
lambda item: validate_issue(item, "constructability"),
|
||||||
)
|
)
|
||||||
for finding in findings:
|
for finding in findings:
|
||||||
finding.update(agent=self.name, scope_id=scope.scope_id)
|
finding.update(agent=self.name, scope_id=scope.scope_id,
|
||||||
|
cluster_key=cluster.get("key"))
|
||||||
return AgentResult(scope_id=scope.scope_id, artifacts=findings)
|
return AgentResult(scope_id=scope.scope_id, artifacts=findings)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
return failure(scope, exc)
|
return failure(scope, exc)
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
"""Deterministic detection of contradictory extracted values within a cluster.
|
||||||
|
|
||||||
|
Extraction is a vision pass: quantities and sizes can be misread ("(2) 2x6" vs
|
||||||
|
"(5) 2x6"). Cluster members are supposed to describe the same real-world
|
||||||
|
element, so two members asserting different values for the same attribute are
|
||||||
|
a probable misread. Flag these so downstream text-only stages treat the value
|
||||||
|
as unverified instead of reasoning from one reading.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict, List
|
||||||
|
|
||||||
|
|
||||||
|
def _norm(value) -> str:
|
||||||
|
return re.sub(r"\s+", " ", ("" if value is None else str(value)).strip().lower())
|
||||||
|
|
||||||
|
|
||||||
|
def find_disputes(assertions: List[Dict]) -> List[Dict]:
|
||||||
|
"""Same attribute with >= 2 distinct normalized values = disputed."""
|
||||||
|
groups: Dict[str, Dict[str, Dict]] = {}
|
||||||
|
for assertion in assertions:
|
||||||
|
attribute = _norm(assertion.get("attribute"))
|
||||||
|
value = _norm(assertion.get("value"))
|
||||||
|
if not attribute or not value:
|
||||||
|
continue
|
||||||
|
# Group on the normalized value, but keep the original (whitespace-
|
||||||
|
# collapsed) text so disputes read like the sheet, not a lowercase munge.
|
||||||
|
original = re.sub(r"\s+", " ", str(assertion.get("value")).strip())
|
||||||
|
bucket = groups.setdefault(attribute, {}).setdefault(
|
||||||
|
value, {"original": original, "ids": set()}
|
||||||
|
)
|
||||||
|
bucket["ids"].add(assertion.get("id"))
|
||||||
|
disputes = []
|
||||||
|
for attribute, values in sorted(groups.items()):
|
||||||
|
if len(values) < 2:
|
||||||
|
continue
|
||||||
|
disputes.append({
|
||||||
|
"attribute": attribute,
|
||||||
|
"values": sorted(v["original"] for v in values.values()),
|
||||||
|
"assertion_ids": sorted(
|
||||||
|
aid for v in values.values() for aid in v["ids"] if aid
|
||||||
|
),
|
||||||
|
})
|
||||||
|
return disputes
|
||||||
|
|
||||||
|
|
||||||
|
def annotate_clusters(clusters: List[Dict]) -> int:
|
||||||
|
"""Attach disputed_attributes to each cluster that has any. Returns count."""
|
||||||
|
annotated = 0
|
||||||
|
for cluster in clusters:
|
||||||
|
disputes = find_disputes(cluster.get("assertions") or [])
|
||||||
|
if disputes:
|
||||||
|
cluster["disputed_attributes"] = disputes
|
||||||
|
annotated += 1
|
||||||
|
return annotated
|
||||||
@@ -6,7 +6,11 @@ from typing import Dict
|
|||||||
from backend import config
|
from backend import config
|
||||||
from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure
|
from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure
|
||||||
from backend.llm import call_json
|
from backend.llm import call_json
|
||||||
from backend.pipeline.extractor import _normalize_sheet
|
from backend.pipeline.extractor import (
|
||||||
|
_normalize_sheet,
|
||||||
|
_text_layer_block,
|
||||||
|
discipline_from_sheet_number,
|
||||||
|
)
|
||||||
from backend.pipeline.sheet_index import _index_input
|
from backend.pipeline.sheet_index import _index_input
|
||||||
from backend.prompts import (
|
from backend.prompts import (
|
||||||
EXTRACTOR_SYSTEM_PROMPT,
|
EXTRACTOR_SYSTEM_PROMPT,
|
||||||
@@ -60,12 +64,39 @@ class SheetExtractorAgent:
|
|||||||
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
|
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def _text_structuring_call(self, instruction_page: Dict, sheet_hint: str):
|
||||||
|
"""Rung 2: text-only structuring pass over the page's text layer
|
||||||
|
(no image). Recovers text content the vision pass missed."""
|
||||||
|
from backend.prompts import (TEXT_STRUCTURING_SYSTEM_PROMPT,
|
||||||
|
TEXT_STRUCTURING_USER_INSTRUCTION)
|
||||||
|
instruction = (TEXT_STRUCTURING_USER_INSTRUCTION
|
||||||
|
.replace("{sheet_hint}", str(sheet_hint or ""))
|
||||||
|
.replace("{text_layer}",
|
||||||
|
(instruction_page.get("text_layer") or "")
|
||||||
|
[:config.TEXT_LAYER_MAX_CHARS]))
|
||||||
|
return call_json(
|
||||||
|
system_prompt=TEXT_STRUCTURING_SYSTEM_PROMPT,
|
||||||
|
user_text=instruction,
|
||||||
|
images_b64=None,
|
||||||
|
max_tokens=config.EXTRACT_MAX_TOKENS,
|
||||||
|
model=config.AGENT_EXTRACT_MODEL,
|
||||||
|
usage_tracker=self.usage,
|
||||||
|
usage_stage="agent.extract_text",
|
||||||
|
reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
|
||||||
|
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
|
||||||
|
)
|
||||||
|
|
||||||
def run(self, scope: AgentScope) -> AgentResult:
|
def run(self, scope: AgentScope) -> AgentResult:
|
||||||
|
from backend.text_coverage import (fallback_objects, merge_objects,
|
||||||
|
recover_sheet_number, text_coverage)
|
||||||
try:
|
try:
|
||||||
page = scope.payload["page"]
|
page = scope.payload["page"]
|
||||||
|
hint = scope.payload.get("sheet_hint") or ""
|
||||||
|
page_text = page.get("text_layer")
|
||||||
instruction = EXTRACTOR_USER_INSTRUCTION.replace(
|
instruction = EXTRACTOR_USER_INSTRUCTION.replace(
|
||||||
"{sheet_hint}", str(scope.payload.get("sheet_hint") or "")
|
"{sheet_hint}", str(hint)) + _text_layer_block(page)
|
||||||
)
|
|
||||||
|
# Rung 1: vision pass (unchanged behaviour, incl. compact retry)
|
||||||
parsed = _wrap_bare_list(self._call(instruction, page),
|
parsed = _wrap_bare_list(self._call(instruction, page),
|
||||||
page["page_number"])
|
page["page_number"])
|
||||||
if not isinstance(parsed, dict):
|
if not isinstance(parsed, dict):
|
||||||
@@ -78,8 +109,74 @@ class SheetExtractorAgent:
|
|||||||
page["page_number"],
|
page["page_number"],
|
||||||
)
|
)
|
||||||
if not isinstance(parsed, dict):
|
if not isinstance(parsed, dict):
|
||||||
raise ValueError("no structured extraction returned")
|
# Don't give up on the page - the ladder below can still
|
||||||
sheet = _normalize_sheet(parsed, page["page_number"])
|
# rescue it from the text layer.
|
||||||
|
parsed = {"sheet": {}, "objects": []}
|
||||||
|
|
||||||
|
sheet = _normalize_sheet(parsed, page["page_number"],
|
||||||
|
page_text=page_text)
|
||||||
|
cov = text_coverage(page_text or "", sheet["assertions"])
|
||||||
|
sheet["coverage"] = cov
|
||||||
|
|
||||||
|
# Rung 2: text-only structuring when coverage is below floor.
|
||||||
|
# MERGE, never replace: vision keeps every object it found
|
||||||
|
# (graphical_basis content exists only in the image); the text
|
||||||
|
# pass fills in the text content the vision pass missed.
|
||||||
|
if (page_text and config.EXTRACT_TEXT_RETRY_ENABLED
|
||||||
|
and cov["ratio"] < config.EXTRACT_COVERAGE_FLOOR):
|
||||||
|
print(f"[Extract] Page {page['page_number']}: coverage "
|
||||||
|
f"{cov['ratio']:.0%} < floor - text-only structuring pass")
|
||||||
|
parsed2 = _wrap_bare_list(
|
||||||
|
self._text_structuring_call(page, hint), page["page_number"])
|
||||||
|
if isinstance(parsed2, dict):
|
||||||
|
sheet2 = _normalize_sheet(parsed2, page["page_number"],
|
||||||
|
page_text=page_text)
|
||||||
|
before = len(sheet["assertions"])
|
||||||
|
sheet["assertions"] = merge_objects(sheet["assertions"],
|
||||||
|
sheet2["assertions"])
|
||||||
|
# Fill header gaps the vision pass left null
|
||||||
|
for key in ("sheet_number", "sheet_title", "discipline",
|
||||||
|
"level", "scale", "drawing_type"):
|
||||||
|
if not sheet.get(key) and sheet2.get(key):
|
||||||
|
sheet[key] = sheet2[key]
|
||||||
|
cov = text_coverage(page_text, sheet["assertions"])
|
||||||
|
sheet["coverage"] = cov
|
||||||
|
print(f"[Extract] Page {page['page_number']}: merged "
|
||||||
|
f"{len(sheet['assertions']) - before} text-structured "
|
||||||
|
f"object(s), coverage now {cov['ratio']:.0%}")
|
||||||
|
|
||||||
|
# Rung 3: deterministic fallback - dark sheets are impossible.
|
||||||
|
# Also merged (deduped) so stub notes never double up with
|
||||||
|
# objects the earlier rungs already captured.
|
||||||
|
if (page_text and config.EXTRACT_FALLBACK_ENABLED
|
||||||
|
and cov["ratio"] < config.EXTRACT_COVERAGE_FLOOR):
|
||||||
|
stubs = fallback_objects(page_text, page["page_number"],
|
||||||
|
config.EXTRACT_FALLBACK_MAX_OBJECTS)
|
||||||
|
stubs = _normalize_sheet({"sheet": {}, "objects": stubs},
|
||||||
|
page["page_number"],
|
||||||
|
page_text=page_text)["assertions"]
|
||||||
|
# _normalize_sheet only stamps its own "text_layer" rescue
|
||||||
|
# grounding; restore the explicit fallback provenance.
|
||||||
|
for stub in stubs:
|
||||||
|
stub["grounding"] = "text_layer_fallback"
|
||||||
|
before = len(sheet["assertions"])
|
||||||
|
sheet["assertions"] = merge_objects(sheet["assertions"], stubs)
|
||||||
|
print(f"[Extract] Page {page['page_number']}: fallback merged "
|
||||||
|
f"{len(sheet['assertions']) - before} text-layer stub(s)")
|
||||||
|
sheet["coverage"] = text_coverage(page_text,
|
||||||
|
sheet["assertions"])
|
||||||
|
|
||||||
|
# Identity recovery: never leave a text-bearing page sheet-less
|
||||||
|
if not sheet.get("sheet_number") and page_text:
|
||||||
|
recovered = recover_sheet_number(page_text)
|
||||||
|
if recovered:
|
||||||
|
sheet["sheet_number"] = recovered
|
||||||
|
sheet["discipline"] = (
|
||||||
|
discipline_from_sheet_number(recovered)
|
||||||
|
or sheet.get("discipline") or "Unknown")
|
||||||
|
print(f"[Extract] Page {page['page_number']}: sheet number "
|
||||||
|
f"recovered from text layer -> {recovered}")
|
||||||
|
|
||||||
return AgentResult(scope_id=scope.scope_id, artifacts=[sheet])
|
return AgentResult(scope_id=scope.scope_id, artifacts=[sheet])
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
return failure(scope, exc)
|
return failure(scope, exc)
|
||||||
|
|||||||
@@ -30,9 +30,23 @@ def _family(assertion: Dict) -> str:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _xref_keys(assertion: Dict) -> List[str]:
|
||||||
|
"""Cross-level join keys: detail references and member tags."""
|
||||||
|
location = assertion.get("location_key") or {}
|
||||||
|
keys = []
|
||||||
|
ref = re.sub(r"\s+", "", str(location.get("detail_reference") or "")).upper()
|
||||||
|
if ref:
|
||||||
|
keys.append(f"detail:{ref}")
|
||||||
|
tag = re.sub(r"\s+", "", str(location.get("tag") or "")).upper()
|
||||||
|
if re.match(r"^[A-Z]+\d", tag): # member marks: W12X26, HSS16X4X5/8, ...
|
||||||
|
keys.append(f"tag:{tag}")
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]:
|
def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]:
|
||||||
"""Partition facts by level and object/tag family, then enforce a hard cap."""
|
"""Partition facts by level and object/tag family, then enforce a hard cap."""
|
||||||
buckets: Dict[Tuple[str, str], List[Dict]] = defaultdict(list)
|
buckets: Dict[Tuple[str, str], List[Dict]] = defaultdict(list)
|
||||||
|
xref: Dict[str, List[Dict]] = defaultdict(list)
|
||||||
for sheet in sheets:
|
for sheet in sheets:
|
||||||
for assertion in sheet.get("assertions", []):
|
for assertion in sheet.get("assertions", []):
|
||||||
enriched = {
|
enriched = {
|
||||||
@@ -44,6 +58,8 @@ def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]:
|
|||||||
level = str((assertion.get("location_key") or {}).get("level")
|
level = str((assertion.get("location_key") or {}).get("level")
|
||||||
or sheet.get("level") or "unknown").lower()
|
or sheet.get("level") or "unknown").lower()
|
||||||
buckets[(level, _family(assertion))].append(enriched)
|
buckets[(level, _family(assertion))].append(enriched)
|
||||||
|
for key in _xref_keys(assertion):
|
||||||
|
xref[key].append(enriched)
|
||||||
|
|
||||||
scopes: List[AgentScope] = []
|
scopes: List[AgentScope] = []
|
||||||
cap = max(2, config.AGENT_LINK_MAX_ASSERTIONS)
|
cap = max(2, config.AGENT_LINK_MAX_ASSERTIONS)
|
||||||
@@ -56,6 +72,18 @@ def build_link_scopes(sheets: List[Dict]) -> List[AgentScope]:
|
|||||||
scope_id=f"{level}:{family}:{offset // cap + 1}",
|
scope_id=f"{level}:{family}:{offset // cap + 1}",
|
||||||
payload={"assertions": chunk, "level": level, "family": family},
|
payload={"assertions": chunk, "level": level, "family": family},
|
||||||
))
|
))
|
||||||
|
for key, assertions in sorted(xref.items()):
|
||||||
|
sheets_present = {a.get("sheet_number") for a in assertions}
|
||||||
|
if len(assertions) < 2 or len(sheets_present) < 2:
|
||||||
|
continue
|
||||||
|
chunk = assertions[:cap]
|
||||||
|
if len({a.get("sheet_number") for a in chunk}) < 2:
|
||||||
|
continue # cap landed on a single sheet — xref adds nothing
|
||||||
|
scopes.append(AgentScope(
|
||||||
|
scope_id=f"xref:{key}",
|
||||||
|
payload={"assertions": chunk,
|
||||||
|
"level": "xref", "family": key},
|
||||||
|
))
|
||||||
return scopes
|
return scopes
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,8 @@ import threading
|
|||||||
from typing import Any, Dict, Iterable, Optional
|
from typing import Any, Dict, Iterable, Optional
|
||||||
|
|
||||||
|
|
||||||
_COLLECTION_KEYS = {"sheets", "clusters", "findings", "decisions", "rfis"}
|
_COLLECTION_KEYS = {"sheets", "clusters", "findings", "decisions", "rfis",
|
||||||
|
"suppressed"}
|
||||||
_MAPPING_KEYS = {"sheet_index", "jurisdiction", "object_graph"}
|
_MAPPING_KEYS = {"sheet_index", "jurisdiction", "object_graph"}
|
||||||
_MEMORY_KEYS = _COLLECTION_KEYS | _MAPPING_KEYS
|
_MEMORY_KEYS = _COLLECTION_KEYS | _MAPPING_KEYS
|
||||||
|
|
||||||
|
|||||||
+130
-1
@@ -1,5 +1,6 @@
|
|||||||
"""Public entry point for the scoped Agent-mode pipeline."""
|
"""Public entry point for the scoped Agent-mode pipeline."""
|
||||||
|
|
||||||
|
import base64
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
from typing import Callable, Dict, Optional
|
from typing import Callable, Dict, Optional
|
||||||
@@ -11,6 +12,7 @@ from backend.agents.code_agent import CodeAgent, build_code_scopes
|
|||||||
from backend.agents.completeness import CompletenessAgent, build_sheet_summaries
|
from backend.agents.completeness import CompletenessAgent, build_sheet_summaries
|
||||||
from backend.agents.conflict_critic import ConflictCriticAgent
|
from backend.agents.conflict_critic import ConflictCriticAgent
|
||||||
from backend.agents.construct_agent import ConstructabilityAgent, build_construct_scopes
|
from backend.agents.construct_agent import ConstructabilityAgent, build_construct_scopes
|
||||||
|
from backend.agents.disputes import annotate_clusters
|
||||||
from backend.agents.extractors import (
|
from backend.agents.extractors import (
|
||||||
JurisdictionAgent,
|
JurisdictionAgent,
|
||||||
SheetExtractorAgent,
|
SheetExtractorAgent,
|
||||||
@@ -20,12 +22,19 @@ from backend.agents.linker import LinkerAgent, build_link_scopes, build_object_g
|
|||||||
from backend.agents.memory import ProjectMemory
|
from backend.agents.memory import ProjectMemory
|
||||||
from backend.agents.orchestrator import Orchestrator
|
from backend.agents.orchestrator import Orchestrator
|
||||||
from backend.agents.rfi_writer import RFIWriterAgent
|
from backend.agents.rfi_writer import RFIWriterAgent
|
||||||
|
from backend.agents.verifier import (
|
||||||
|
EvidenceVerifierAgent, apply_verdicts, select_findings,
|
||||||
|
)
|
||||||
from backend.llm import reset_cost
|
from backend.llm import reset_cost
|
||||||
from backend.pipeline.pdf_processor import convert_pdf_to_images
|
from backend.pipeline.pdf_processor import convert_pdf_to_images
|
||||||
from backend.pipeline.report import build_report, to_markdown
|
from backend.pipeline.report import build_report, to_markdown
|
||||||
from backend.pipeline.sheet_index import derive_project_meta_from_cover
|
from backend.pipeline.sheet_index import derive_project_meta_from_cover
|
||||||
from backend.review.gate import build_review_queue
|
from backend.review.gate import build_review_queue
|
||||||
from backend.review.store import ReviewStore
|
from backend.review.store import ReviewStore
|
||||||
|
from backend.sheet_reconcile import declared_sheet_list, reconcile_sheets
|
||||||
|
from backend.text_layer import (
|
||||||
|
attach_text_layers, coverage_gaps, find_evidence_bbox, render_crop,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def run_agent_pipeline(
|
def run_agent_pipeline(
|
||||||
@@ -53,6 +62,9 @@ def run_agent_pipeline(
|
|||||||
orchestrator.stage("Agent ingest: PDF -> images")
|
orchestrator.stage("Agent ingest: PDF -> images")
|
||||||
pages = convert_pdf_to_images(pdf_path)
|
pages = convert_pdf_to_images(pdf_path)
|
||||||
page_to_b64 = {page["page_number"]: page["base64"] for page in pages}
|
page_to_b64 = {page["page_number"]: page["base64"] for page in pages}
|
||||||
|
text_dir = os.path.join(agent_dir, "text") if agent_dir else None
|
||||||
|
page_words = attach_text_layers(pdf_path, pages, text_dir=text_dir)
|
||||||
|
page_to_text = {page["page_number"]: page.get("text_layer") for page in pages}
|
||||||
|
|
||||||
orchestrator.stage("Agent wave 1: extract sheets")
|
orchestrator.stage("Agent wave 1: extract sheets")
|
||||||
extract_scopes = [
|
extract_scopes = [
|
||||||
@@ -74,6 +86,27 @@ def run_agent_pipeline(
|
|||||||
memory.replace("sheets", sheets)
|
memory.replace("sheets", sheets)
|
||||||
memory.dump("01-extract.json")
|
memory.dump("01-extract.json")
|
||||||
|
|
||||||
|
# Deterministic reconciliation: the cover sheet's own sheet index
|
||||||
|
# declares what the set should contain; compare against what wave 1
|
||||||
|
# identified (catches missed sheets AND phantom/misread sheet numbers).
|
||||||
|
sheet_recon = reconcile_sheets(sheets, declared_sheet_list(page_to_text))
|
||||||
|
if sheet_recon["declared_total"]:
|
||||||
|
print(f"[SheetIndex] cover declares {sheet_recon['declared_total']} "
|
||||||
|
f"sheets; {sheet_recon['found_total']} identified in set")
|
||||||
|
if sheet_recon["declared_not_in_set"]:
|
||||||
|
print(f"[SheetIndex] declared but not in set: "
|
||||||
|
f"{', '.join(sheet_recon['declared_not_in_set'][:20])}")
|
||||||
|
if sheet_recon["in_set_not_declared"]:
|
||||||
|
print(f"[SheetIndex] in set but not declared: "
|
||||||
|
f"{', '.join(sheet_recon['in_set_not_declared'][:20])}")
|
||||||
|
# Coverage signal: text layer present but extraction failed/empty reuses
|
||||||
|
# the failed-scopes gap-finding path (finding built below wave 6).
|
||||||
|
for gap_page in coverage_gaps(pages, sheets):
|
||||||
|
orchestrator.stats.failed_scopes.append(
|
||||||
|
f"sheet_extractor:sheet:{gap_page}: extraction gap "
|
||||||
|
f"(text layer present, no objects extracted)"
|
||||||
|
)
|
||||||
|
|
||||||
cover_meta = derive_project_meta_from_cover(
|
cover_meta = derive_project_meta_from_cover(
|
||||||
sheets, source_name or os.path.basename(pdf_path)
|
sheets, source_name or os.path.basename(pdf_path)
|
||||||
)
|
)
|
||||||
@@ -113,6 +146,9 @@ def run_agent_pipeline(
|
|||||||
for artifact in result.artifacts
|
for artifact in result.artifacts
|
||||||
][:config.CLUSTER_MAX]
|
][:config.CLUSTER_MAX]
|
||||||
object_graph = build_object_graph(clusters)
|
object_graph = build_object_graph(clusters)
|
||||||
|
disputed_count = annotate_clusters(clusters)
|
||||||
|
if disputed_count:
|
||||||
|
orchestrator.stage(f"[Link] {disputed_count} clusters carry disputed extracted values")
|
||||||
memory.replace("clusters", clusters)
|
memory.replace("clusters", clusters)
|
||||||
memory.replace("object_graph", object_graph)
|
memory.replace("object_graph", object_graph)
|
||||||
memory.dump("03-link.json")
|
memory.dump("03-link.json")
|
||||||
@@ -164,6 +200,56 @@ def run_agent_pipeline(
|
|||||||
for result in code_results + construct_results + completeness_results
|
for result in code_results + construct_results + completeness_results
|
||||||
for artifact in result.artifacts
|
for artifact in result.artifacts
|
||||||
]
|
]
|
||||||
|
|
||||||
|
orchestrator.stage("Agent wave 5b: evidence verification")
|
||||||
|
sheet_to_page = {str(s.get("sheet_number")): s.get("page_number")
|
||||||
|
for s in sheets}
|
||||||
|
verify_targets = select_findings(
|
||||||
|
specialist_findings, clusters,
|
||||||
|
max_checks=config.AGENT_VERIFY_MAX_CHECKS,
|
||||||
|
severities=config.AGENT_VERIFY_SEVERITIES,
|
||||||
|
)
|
||||||
|
target_indexes = {id(f): i for i, f in enumerate(specialist_findings)}
|
||||||
|
verify_scopes = []
|
||||||
|
for finding in verify_targets:
|
||||||
|
cited_pages = [
|
||||||
|
sheet_to_page[str(name)]
|
||||||
|
for name in (finding.get("sheets") or [])
|
||||||
|
if sheet_to_page.get(str(name)) in page_to_b64
|
||||||
|
]
|
||||||
|
images = [
|
||||||
|
page_to_b64[p]
|
||||||
|
for p in cited_pages[:config.AGENT_CONFLICT_MAX_IMAGES]
|
||||||
|
]
|
||||||
|
if not images:
|
||||||
|
continue # never judge evidence against images we could not load
|
||||||
|
# Text oracle: concatenated text layer of the cited sheets, capped.
|
||||||
|
excerpt = "\n\n".join(
|
||||||
|
f"--- Page {p} ---\n{page_to_text[p]}"
|
||||||
|
for p in cited_pages
|
||||||
|
if page_to_text.get(p)
|
||||||
|
)[:config.VERIFY_TEXT_MAX_CHARS]
|
||||||
|
if config.VERIFY_HI_DPI_CROPS:
|
||||||
|
images = _evidence_crops(finding, cited_pages, sheet_to_page,
|
||||||
|
page_words, page_to_b64, pdf_path,
|
||||||
|
fallback=images)
|
||||||
|
verify_scopes.append(AgentScope(
|
||||||
|
scope_id=f"verify:{target_indexes[id(finding)]}",
|
||||||
|
payload={
|
||||||
|
"finding_index": target_indexes[id(finding)],
|
||||||
|
"finding": finding,
|
||||||
|
"images_b64": images,
|
||||||
|
"text_layer_excerpt": excerpt,
|
||||||
|
},
|
||||||
|
))
|
||||||
|
verify_results = orchestrator.run_scopes(
|
||||||
|
EvidenceVerifierAgent(usage), verify_scopes, config.AGENT_VERIFY_CONCURRENCY)
|
||||||
|
suppressed = apply_verdicts(specialist_findings, verify_results)
|
||||||
|
if suppressed:
|
||||||
|
suppressed_ids = {id(f) for f in suppressed}
|
||||||
|
specialist_findings = [f for f in specialist_findings if id(f) not in suppressed_ids]
|
||||||
|
memory.replace("suppressed", suppressed)
|
||||||
|
|
||||||
memory.extend("findings", specialist_findings)
|
memory.extend("findings", specialist_findings)
|
||||||
gap_findings = [
|
gap_findings = [
|
||||||
{
|
{
|
||||||
@@ -218,10 +304,11 @@ def run_agent_pipeline(
|
|||||||
"project_input": merged_input,
|
"project_input": merged_input,
|
||||||
"jurisdiction": jurisdiction,
|
"jurisdiction": jurisdiction,
|
||||||
"sheet_index": sheet_index,
|
"sheet_index": sheet_index,
|
||||||
|
"sheet_reconciliation": sheet_recon,
|
||||||
"project_intelligence": object_graph,
|
"project_intelligence": object_graph,
|
||||||
"validated_issues": prioritized,
|
"validated_issues": prioritized,
|
||||||
"rfis": [],
|
"rfis": [],
|
||||||
"suppressed_issues": [],
|
"suppressed_issues": memory.snapshot().get("suppressed") or [],
|
||||||
})
|
})
|
||||||
progress = store.progress(queue)
|
progress = store.progress(queue)
|
||||||
# Same usage/stats summary block as the wave-7 path (rfis: 0 — they
|
# Same usage/stats summary block as the wave-7 path (rfis: 0 — they
|
||||||
@@ -294,9 +381,11 @@ def run_agent_pipeline(
|
|||||||
"project_input": merged_input,
|
"project_input": merged_input,
|
||||||
"jurisdiction": jurisdiction,
|
"jurisdiction": jurisdiction,
|
||||||
"sheet_index": sheet_index,
|
"sheet_index": sheet_index,
|
||||||
|
"sheet_reconciliation": sheet_recon,
|
||||||
"project_intelligence": object_graph,
|
"project_intelligence": object_graph,
|
||||||
"validated_issues": prioritized,
|
"validated_issues": prioritized,
|
||||||
"rfis": rfis,
|
"rfis": rfis,
|
||||||
|
"suppressed_issues": memory.snapshot().get("suppressed") or [],
|
||||||
})
|
})
|
||||||
cost = usage.snapshot()
|
cost = usage.snapshot()
|
||||||
orchestrator.stats.calls = cost["calls"]
|
orchestrator.stats.calls = cost["calls"]
|
||||||
@@ -351,6 +440,46 @@ def _dump(out_dir: str, name: str, value) -> None:
|
|||||||
json.dump(value, f, indent=2)
|
json.dump(value, f, indent=2)
|
||||||
|
|
||||||
|
|
||||||
|
def _evidence_crops(
|
||||||
|
finding: Dict,
|
||||||
|
cited_pages: list,
|
||||||
|
sheet_to_page: Dict,
|
||||||
|
page_words: Dict,
|
||||||
|
page_to_b64: Dict,
|
||||||
|
pdf_path: str,
|
||||||
|
fallback: list,
|
||||||
|
) -> list:
|
||||||
|
"""High-DPI crops around each evidence item's source_text, located via the
|
||||||
|
page text layer. Crops REPLACE full-page images when at least one evidence
|
||||||
|
location resolves confidently; otherwise the full-page fallback is kept.
|
||||||
|
Never returns an empty list when fallback is non-empty (I2 guard)."""
|
||||||
|
crops: list = []
|
||||||
|
for item in finding.get("evidence") or []:
|
||||||
|
if len(crops) >= config.AGENT_CONFLICT_MAX_IMAGES:
|
||||||
|
break
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
source_text = item.get("source_text") or ""
|
||||||
|
if not source_text:
|
||||||
|
continue
|
||||||
|
# Prefer the page named on the evidence item, then any cited page.
|
||||||
|
candidates = []
|
||||||
|
named_page = sheet_to_page.get(str(item.get("sheet") or ""))
|
||||||
|
if named_page in cited_pages:
|
||||||
|
candidates.append(named_page)
|
||||||
|
candidates.extend(p for p in cited_pages if p not in candidates)
|
||||||
|
for page in candidates:
|
||||||
|
bbox = find_evidence_bbox(page_words.get(page) or [], source_text)
|
||||||
|
if bbox is None:
|
||||||
|
continue
|
||||||
|
crop = render_crop(pdf_path, page, bbox)
|
||||||
|
if not crop:
|
||||||
|
continue
|
||||||
|
crops.append(base64.b64encode(crop).decode("utf-8"))
|
||||||
|
break
|
||||||
|
return crops or fallback
|
||||||
|
|
||||||
|
|
||||||
def _counts(items, key: str) -> Dict[str, int]:
|
def _counts(items, key: str) -> Dict[str, int]:
|
||||||
counts: Dict[str, int] = {}
|
counts: Dict[str, int] = {}
|
||||||
for item in items:
|
for item in items:
|
||||||
|
|||||||
@@ -0,0 +1,99 @@
|
|||||||
|
"""Wave 5b: vision fact-check of extracted evidence against cited sheet images."""
|
||||||
|
|
||||||
|
from backend import config
|
||||||
|
from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure
|
||||||
|
from backend.llm import call_json
|
||||||
|
from backend.pipeline._serialize import dumps
|
||||||
|
from backend.pipeline._stage import collect_list, render
|
||||||
|
from backend.prompts import VERIFY_SYSTEM_PROMPT, VERIFY_USER_INSTRUCTION
|
||||||
|
|
||||||
|
_SEVERITY_RANK = {"critical": 0, "high": 1, "medium": 2, "low": 3}
|
||||||
|
_VERDICTS = ("confirmed", "corrected", "not_found")
|
||||||
|
|
||||||
|
|
||||||
|
def select_findings(findings, clusters, max_checks, severities):
|
||||||
|
"""Severity-gated selection plus any finding tied to a disputed cluster."""
|
||||||
|
disputed_keys = {c.get("key") for c in clusters if c.get("disputed_attributes")}
|
||||||
|
selected = [f for f in findings
|
||||||
|
if str(f.get("severity") or "").lower() in severities
|
||||||
|
or f.get("cluster_key") in disputed_keys]
|
||||||
|
selected.sort(key=lambda f: _SEVERITY_RANK.get(
|
||||||
|
str(f.get("severity") or "").lower(), 9))
|
||||||
|
return selected[:max_checks]
|
||||||
|
|
||||||
|
|
||||||
|
def _valid_verdict(item):
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
return None
|
||||||
|
verdict = str(item.get("verdict") or "").lower()
|
||||||
|
if verdict not in _VERDICTS:
|
||||||
|
return None
|
||||||
|
return {"sheet": item.get("sheet") or "",
|
||||||
|
"source_text": item.get("source_text") or "",
|
||||||
|
"verdict": verdict,
|
||||||
|
"actual_text": item.get("actual_text"),
|
||||||
|
"notes": item.get("notes")}
|
||||||
|
|
||||||
|
|
||||||
|
def _status(verdicts):
|
||||||
|
if not verdicts:
|
||||||
|
return "unverified"
|
||||||
|
confirmed = sum(1 for v in verdicts if v["verdict"] == "confirmed")
|
||||||
|
if confirmed == len(verdicts):
|
||||||
|
return "confirmed"
|
||||||
|
if confirmed == 0:
|
||||||
|
return "refuted"
|
||||||
|
return "mixed"
|
||||||
|
|
||||||
|
|
||||||
|
class EvidenceVerifierAgent:
|
||||||
|
name = "verify"
|
||||||
|
|
||||||
|
def __init__(self, usage: AgentUsage) -> None:
|
||||||
|
self.usage = usage
|
||||||
|
|
||||||
|
def run(self, scope: AgentScope) -> AgentResult:
|
||||||
|
try:
|
||||||
|
finding = scope.payload["finding"]
|
||||||
|
instruction = render(VERIFY_USER_INSTRUCTION, {
|
||||||
|
"finding": dumps(finding),
|
||||||
|
"text_layer": scope.payload.get("text_layer_excerpt")
|
||||||
|
or "(no text layer available for the cited sheets)",
|
||||||
|
})
|
||||||
|
parsed = call_json(
|
||||||
|
system_prompt=VERIFY_SYSTEM_PROMPT,
|
||||||
|
user_text=instruction,
|
||||||
|
images_b64=scope.payload.get("images_b64") or [],
|
||||||
|
max_tokens=config.VERIFY_MAX_TOKENS,
|
||||||
|
model=config.AGENT_VERIFY_MODEL,
|
||||||
|
reasoning_effort=config.AGENT_VERIFY_REASONING_EFFORT or None,
|
||||||
|
usage_tracker=self.usage,
|
||||||
|
usage_stage="agent.verify",
|
||||||
|
)
|
||||||
|
verdicts = collect_list(parsed, "verdicts", _valid_verdict)
|
||||||
|
return AgentResult(scope_id=scope.scope_id, artifacts=[{
|
||||||
|
"finding_index": scope.payload["finding_index"],
|
||||||
|
"status": _status(verdicts),
|
||||||
|
"verdicts": verdicts,
|
||||||
|
}])
|
||||||
|
except Exception as exc:
|
||||||
|
return failure(scope, exc)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_verdicts(findings, verify_results):
|
||||||
|
"""Annotate findings with verification; return refuted ones to suppress."""
|
||||||
|
by_index = {}
|
||||||
|
for result in verify_results:
|
||||||
|
for artifact in result.artifacts:
|
||||||
|
by_index[artifact["finding_index"]] = artifact
|
||||||
|
suppressed = []
|
||||||
|
for index, finding in enumerate(findings):
|
||||||
|
artifact = by_index.get(index)
|
||||||
|
if not artifact:
|
||||||
|
continue
|
||||||
|
finding["verification"] = {"status": artifact["status"],
|
||||||
|
"verdicts": artifact["verdicts"]}
|
||||||
|
if artifact["status"] == "refuted":
|
||||||
|
finding["confidence"] = "low"
|
||||||
|
suppressed.append(finding)
|
||||||
|
return suppressed
|
||||||
@@ -47,6 +47,30 @@ AGENT_CONFLICT_CONCURRENCY = int(os.getenv("AGENT_CONFLICT_CONCURRENCY", "4"))
|
|||||||
AGENT_SPECIALIST_CONCURRENCY = int(os.getenv("AGENT_SPECIALIST_CONCURRENCY", "4"))
|
AGENT_SPECIALIST_CONCURRENCY = int(os.getenv("AGENT_SPECIALIST_CONCURRENCY", "4"))
|
||||||
AGENT_RFI_CONCURRENCY = int(os.getenv("AGENT_RFI_CONCURRENCY", "4"))
|
AGENT_RFI_CONCURRENCY = int(os.getenv("AGENT_RFI_CONCURRENCY", "4"))
|
||||||
|
|
||||||
|
# Wave 5b evidence verification (vision fact-check of cited sheet text)
|
||||||
|
AGENT_VERIFY_MODEL = os.getenv("AGENT_VERIFY_MODEL", "") or MODEL
|
||||||
|
AGENT_VERIFY_CONCURRENCY = int(os.getenv("AGENT_VERIFY_CONCURRENCY", "4"))
|
||||||
|
AGENT_VERIFY_MAX_CHECKS = int(os.getenv("AGENT_VERIFY_MAX_CHECKS", "20"))
|
||||||
|
AGENT_VERIFY_SEVERITIES = {
|
||||||
|
s.strip().lower()
|
||||||
|
for s in os.getenv("AGENT_VERIFY_SEVERITIES", "critical,high").split(",")
|
||||||
|
if s.strip()
|
||||||
|
}
|
||||||
|
AGENT_VERIFY_REASONING_EFFORT = os.getenv("AGENT_VERIFY_REASONING_EFFORT", "low").strip()
|
||||||
|
VERIFY_MAX_TOKENS = int(os.getenv("VERIFY_MAX_TOKENS", "8192"))
|
||||||
|
|
||||||
|
# -- Text-layer grounding (deterministic PDF text layer via PyMuPDF) ----
|
||||||
|
# The vector text layer is extracted once per job and grounds the extractor,
|
||||||
|
# rescues misquoted-but-real values in the grounding guard, and serves the
|
||||||
|
# wave-5b verifier as a text oracle plus high-DPI evidence crops.
|
||||||
|
TEXT_LAYER_ENABLED = os.getenv("TEXT_LAYER_ENABLED", "true").strip().lower() in ("1", "true", "yes")
|
||||||
|
TEXT_LAYER_MIN_CHARS = int(os.getenv("TEXT_LAYER_MIN_CHARS", "20")) # below this per page -> no text layer
|
||||||
|
TEXT_LAYER_MAX_CHARS = int(os.getenv("TEXT_LAYER_MAX_CHARS", "12000")) # cap per sheet in extractor prompt
|
||||||
|
VERIFY_TEXT_MAX_CHARS = int(os.getenv("VERIFY_TEXT_MAX_CHARS", "8000"))# cap of excerpt in verify scope
|
||||||
|
VERIFY_HI_DPI_CROPS = os.getenv("VERIFY_HI_DPI_CROPS", "true").strip().lower() in ("1", "true", "yes")
|
||||||
|
VERIFY_CROP_DPI = int(os.getenv("VERIFY_CROP_DPI", "300"))
|
||||||
|
VERIFY_CROP_MARGIN_PTS = int(os.getenv("VERIFY_CROP_MARGIN_PTS", "36"))# padding around evidence bbox (PDF points)
|
||||||
|
|
||||||
# Agent-mode human-review gate. When on (default), Agent runs stop after the
|
# Agent-mode human-review gate. When on (default), Agent runs stop after the
|
||||||
# Brain merge and wait for human decisions before RFIs/final report/email go
|
# Brain merge and wait for human decisions before RFIs/final report/email go
|
||||||
# out. AGENT_REVIEW_AUDIT_SAMPLE caps how many clean clusters get added to the
|
# out. AGENT_REVIEW_AUDIT_SAMPLE caps how many clean clusters get added to the
|
||||||
@@ -88,6 +112,15 @@ EXTRACT_REASONING_EFFORT = os.getenv("EXTRACT_REASONING_EFFORT", "low").strip()
|
|||||||
# the budget into visible output. 0 disables -> falls back to the effort knob.
|
# the budget into visible output. 0 disables -> falls back to the effort knob.
|
||||||
# Mutually exclusive with effort when set (OpenRouter rejects both together).
|
# Mutually exclusive with effort when set (OpenRouter rejects both together).
|
||||||
EXTRACT_REASONING_MAX_TOKENS = int(os.getenv("EXTRACT_REASONING_MAX_TOKENS", "2048"))
|
EXTRACT_REASONING_MAX_TOKENS = int(os.getenv("EXTRACT_REASONING_MAX_TOKENS", "2048"))
|
||||||
|
# Coverage-driven extraction retry ladder. After the vision pass, the fraction
|
||||||
|
# of meaningful text-layer lines represented in extracted objects is measured;
|
||||||
|
# below EXTRACT_COVERAGE_FLOOR the page climbs the ladder: text-only
|
||||||
|
# structuring pass (rung 2), then deterministic text-layer fallback stubs
|
||||||
|
# (rung 3) so no text-bearing page goes dark.
|
||||||
|
EXTRACT_COVERAGE_FLOOR = float(os.getenv("EXTRACT_COVERAGE_FLOOR", "0.6"))
|
||||||
|
EXTRACT_TEXT_RETRY_ENABLED = os.getenv("EXTRACT_TEXT_RETRY_ENABLED", "true").lower() == "true"
|
||||||
|
EXTRACT_FALLBACK_ENABLED = os.getenv("EXTRACT_FALLBACK_ENABLED", "true").lower() == "true"
|
||||||
|
EXTRACT_FALLBACK_MAX_OBJECTS = int(os.getenv("EXTRACT_FALLBACK_MAX_OBJECTS", "200"))
|
||||||
REASON_MAX_TOKENS = int(os.getenv("REASON_MAX_TOKENS", "4096"))
|
REASON_MAX_TOKENS = int(os.getenv("REASON_MAX_TOKENS", "4096"))
|
||||||
|
|
||||||
# -- QAQC stage knobs (Stages 0-1, 3, 6-11) -------------------------
|
# -- QAQC stage knobs (Stages 0-1, 3, 6-11) -------------------------
|
||||||
|
|||||||
@@ -54,6 +54,8 @@ def slim_clusters(clusters: List[Dict]) -> List[Dict]:
|
|||||||
"location": c.get("location"),
|
"location": c.get("location"),
|
||||||
"disciplines": c.get("disciplines"),
|
"disciplines": c.get("disciplines"),
|
||||||
"kind": c.get("kind"),
|
"kind": c.get("kind"),
|
||||||
|
**({"disputed_attributes": c["disputed_attributes"]}
|
||||||
|
if c.get("disputed_attributes") else {}),
|
||||||
"assertions": [slim_assertion(a) for a in c.get("assertions", [])],
|
"assertions": [slim_assertion(a) for a in c.get("assertions", [])],
|
||||||
}
|
}
|
||||||
for c in clusters
|
for c in clusters
|
||||||
|
|||||||
@@ -32,6 +32,16 @@ def _evidence_block(cluster: Dict) -> str:
|
|||||||
f"{a.get('attribute','')} = {a.get('value','')} | "
|
f"{a.get('attribute','')} = {a.get('value','')} | "
|
||||||
f"\"{a.get('source_text','')}\""
|
f"\"{a.get('source_text','')}\""
|
||||||
)
|
)
|
||||||
|
disputes = cluster.get("disputed_attributes") or []
|
||||||
|
if disputes:
|
||||||
|
lines.append("")
|
||||||
|
for d in disputes:
|
||||||
|
lines.append(
|
||||||
|
"DISPUTED VALUE (possible extraction misread): "
|
||||||
|
f"attribute={d.get('attribute','')} "
|
||||||
|
f"values={' | '.join(d.get('values') or [])} "
|
||||||
|
f"(assertions {', '.join(d.get('assertion_ids') or [])})"
|
||||||
|
)
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
@@ -83,6 +93,8 @@ def _check_one(cluster: Dict, page_to_b64: Dict[int, str]) -> List[Dict]:
|
|||||||
user_text=user_text,
|
user_text=user_text,
|
||||||
images_b64=_images_for(cluster, page_to_b64),
|
images_b64=_images_for(cluster, page_to_b64),
|
||||||
max_tokens=config.REASON_MAX_TOKENS,
|
max_tokens=config.REASON_MAX_TOKENS,
|
||||||
|
reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
|
||||||
|
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
|
||||||
)
|
)
|
||||||
if isinstance(parsed, list):
|
if isinstance(parsed, list):
|
||||||
candidates = parsed
|
candidates = parsed
|
||||||
|
|||||||
@@ -27,6 +27,10 @@ def constructability_review(sheets: List[Dict], clusters: List[Dict],
|
|||||||
"assertions": dumps(slim_sheets(sheets)),
|
"assertions": dumps(slim_sheets(sheets)),
|
||||||
"clusters": dumps(slim_clusters(clusters)),
|
"clusters": dumps(slim_clusters(clusters)),
|
||||||
"conflicts": dumps(conflicts),
|
"conflicts": dumps(conflicts),
|
||||||
|
"disputes": dumps([
|
||||||
|
d for cluster in clusters
|
||||||
|
for d in (cluster.get("disputed_attributes") or [])
|
||||||
|
]),
|
||||||
},
|
},
|
||||||
max_tokens=config.CONSTRUCT_MAX_TOKENS,
|
max_tokens=config.CONSTRUCT_MAX_TOKENS,
|
||||||
)
|
)
|
||||||
|
|||||||
+163
-17
@@ -21,9 +21,18 @@ from backend.llm import call_json
|
|||||||
from backend.prompts import (
|
from backend.prompts import (
|
||||||
EXTRACTOR_SYSTEM_PROMPT,
|
EXTRACTOR_SYSTEM_PROMPT,
|
||||||
EXTRACTOR_USER_INSTRUCTION,
|
EXTRACTOR_USER_INSTRUCTION,
|
||||||
|
TEXT_STRUCTURING_SYSTEM_PROMPT,
|
||||||
|
TEXT_STRUCTURING_USER_INSTRUCTION,
|
||||||
DISCIPLINE_PREFIXES,
|
DISCIPLINE_PREFIXES,
|
||||||
ATTRIBUTE_VOCAB,
|
ATTRIBUTE_VOCAB,
|
||||||
)
|
)
|
||||||
|
from backend.text_coverage import (
|
||||||
|
_norm,
|
||||||
|
fallback_objects,
|
||||||
|
merge_objects,
|
||||||
|
recover_sheet_number,
|
||||||
|
text_coverage,
|
||||||
|
)
|
||||||
|
|
||||||
# prefix (upper) -> discipline, longest-prefix-first for greedy matching
|
# prefix (upper) -> discipline, longest-prefix-first for greedy matching
|
||||||
_PREFIX_TO_DISCIPLINE = sorted(
|
_PREFIX_TO_DISCIPLINE = sorted(
|
||||||
@@ -64,7 +73,8 @@ def discipline_from_sheet_number(sheet_number: Optional[str]) -> Optional[str]:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _is_grounded(value: str, source_text: str, graphical_basis: str = "") -> bool:
|
def _is_grounded(value: str, source_text: str, graphical_basis: str = "",
|
||||||
|
page_text: Optional[str] = None) -> bool:
|
||||||
"""
|
"""
|
||||||
Keep an object only if its primary value is supported by its source_text,
|
Keep an object only if its primary value is supported by its source_text,
|
||||||
OR it is a graphical object (has graphical_basis with no text to quote).
|
OR it is a graphical object (has graphical_basis with no text to quote).
|
||||||
@@ -72,6 +82,9 @@ def _is_grounded(value: str, source_text: str, graphical_basis: str = "") -> boo
|
|||||||
- If graphical_basis is set and source_text is absent, the object is valid.
|
- If graphical_basis is set and source_text is absent, the object is valid.
|
||||||
- If the value contains digits, every distinct digit-run must appear in
|
- If the value contains digits, every distinct digit-run must appear in
|
||||||
source_text (catches invented dimensions/counts/elevations).
|
source_text (catches invented dimensions/counts/elevations).
|
||||||
|
- Rescue tier: when page_text (the deterministic text layer) is given,
|
||||||
|
digit-runs absent from source_text but present in the page text are
|
||||||
|
still grounded - vision quoted imperfectly but the value is real.
|
||||||
- If the value has no digits, require some alphabetic-token overlap.
|
- If the value has no digits, require some alphabetic-token overlap.
|
||||||
"""
|
"""
|
||||||
# Graphical objects (no readable text on sheet) are always allowed through.
|
# Graphical objects (no readable text on sheet) are always allowed through.
|
||||||
@@ -85,7 +98,11 @@ def _is_grounded(value: str, source_text: str, graphical_basis: str = "") -> boo
|
|||||||
val_digits = set(_DIGITS_RE.findall(value))
|
val_digits = set(_DIGITS_RE.findall(value))
|
||||||
if val_digits:
|
if val_digits:
|
||||||
src_digits = set(_DIGITS_RE.findall(source_text))
|
src_digits = set(_DIGITS_RE.findall(source_text))
|
||||||
return val_digits.issubset(src_digits)
|
if val_digits.issubset(src_digits):
|
||||||
|
return True
|
||||||
|
if page_text:
|
||||||
|
return val_digits.issubset(set(_DIGITS_RE.findall(page_text)))
|
||||||
|
return False
|
||||||
|
|
||||||
# No digits: text-based grounding.
|
# No digits: text-based grounding.
|
||||||
val_norm = re.sub(r"[^a-z0-9]+", " ", value.lower()).strip()
|
val_norm = re.sub(r"[^a-z0-9]+", " ", value.lower()).strip()
|
||||||
@@ -109,7 +126,24 @@ def _primary_value(obj: Dict) -> str:
|
|||||||
or obj.get("name") or obj.get("tag") or "")
|
or obj.get("name") or obj.get("tag") or "")
|
||||||
|
|
||||||
|
|
||||||
def _normalize_sheet(parsed: Dict, page_number: int) -> Dict:
|
def _grounding_stamp(value: str, source_text: str,
|
||||||
|
page_text: Optional[str]) -> Optional[str]:
|
||||||
|
"""\"text_layer\" when the object survived only via the text-layer rescue
|
||||||
|
tier (digits absent from source_text but present in the page text)."""
|
||||||
|
if not page_text:
|
||||||
|
return None
|
||||||
|
val_digits = set(_DIGITS_RE.findall(str(value)))
|
||||||
|
if not val_digits:
|
||||||
|
return None
|
||||||
|
if val_digits.issubset(set(_DIGITS_RE.findall(source_text))):
|
||||||
|
return None
|
||||||
|
if val_digits.issubset(set(_DIGITS_RE.findall(page_text))):
|
||||||
|
return "text_layer"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_sheet(parsed: Dict, page_number: int,
|
||||||
|
page_text: Optional[str] = None) -> Dict:
|
||||||
"""
|
"""
|
||||||
Validate + clean one parsed sheet result, attaching page_number and ids.
|
Validate + clean one parsed sheet result, attaching page_number and ids.
|
||||||
|
|
||||||
@@ -138,6 +172,9 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict:
|
|||||||
raw_objects = parsed.get("objects") or parsed.get("assertions") or []
|
raw_objects = parsed.get("objects") or parsed.get("assertions") or []
|
||||||
clean: List[Dict] = []
|
clean: List[Dict] = []
|
||||||
dropped = 0
|
dropped = 0
|
||||||
|
rescued = 0
|
||||||
|
unverified = 0
|
||||||
|
page_norm = _norm(page_text) if page_text else ""
|
||||||
|
|
||||||
for idx, obj in enumerate(raw_objects):
|
for idx, obj in enumerate(raw_objects):
|
||||||
if not isinstance(obj, dict):
|
if not isinstance(obj, dict):
|
||||||
@@ -149,9 +186,23 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict:
|
|||||||
# Derive a primary value for the grounding check
|
# Derive a primary value for the grounding check
|
||||||
primary_val = _primary_value(obj)
|
primary_val = _primary_value(obj)
|
||||||
|
|
||||||
if not _is_grounded(primary_val, source_text, graphical_basis):
|
if not _is_grounded(primary_val, source_text, graphical_basis,
|
||||||
|
page_text=page_text):
|
||||||
dropped += 1
|
dropped += 1
|
||||||
continue
|
continue
|
||||||
|
# Pre-set stamps (fallback/merge rungs) win; otherwise compute the
|
||||||
|
# text-layer rescue stamp.
|
||||||
|
grounding = obj.get("grounding") or _grounding_stamp(
|
||||||
|
primary_val, source_text, page_text)
|
||||||
|
if grounding == "text_layer":
|
||||||
|
rescued += 1
|
||||||
|
if not grounding and page_text and source_text:
|
||||||
|
# Vision-unverified: survived the digit guard, but the quoted
|
||||||
|
# source_text is not present in the deterministic text layer.
|
||||||
|
# Kept and stamped - the wave-5b verifier prioritizes these.
|
||||||
|
if _norm(str(source_text)) not in page_norm:
|
||||||
|
grounding = "vision_unverified"
|
||||||
|
unverified += 1
|
||||||
|
|
||||||
# --- location_key: new schema is richer; map to legacy shape + extras ---
|
# --- location_key: new schema is richer; map to legacy shape + extras ---
|
||||||
lk = obj.get("location_key")
|
lk = obj.get("location_key")
|
||||||
@@ -203,10 +254,14 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict:
|
|||||||
"object_attributes": attrs,
|
"object_attributes": attrs,
|
||||||
"graphical_basis": graphical_basis or None,
|
"graphical_basis": graphical_basis or None,
|
||||||
"review_uses": obj.get("review_uses") or [],
|
"review_uses": obj.get("review_uses") or [],
|
||||||
|
**({"grounding": grounding} if grounding else {}),
|
||||||
})
|
})
|
||||||
|
|
||||||
if dropped:
|
if dropped or rescued or unverified:
|
||||||
print(f"[Extract] Page {page_number} ({sheet_number}): dropped {dropped} ungrounded object(s)")
|
print(f"[Extract] Page {page_number} ({sheet_number}): "
|
||||||
|
f"dropped {dropped} ungrounded object(s)"
|
||||||
|
+ (f", rescued {rescued} via text layer" if rescued else "")
|
||||||
|
+ (f", {unverified} vision-unverified" if unverified else ""))
|
||||||
|
|
||||||
unresolved = parsed.get("unresolved_items") or []
|
unresolved = parsed.get("unresolved_items") or []
|
||||||
|
|
||||||
@@ -223,8 +278,41 @@ def _normalize_sheet(parsed: Dict, page_number: int) -> Dict:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _text_layer_block(page: Dict) -> str:
|
||||||
|
"""
|
||||||
|
The TEXT LAYER block appended to the extractor instruction at call sites
|
||||||
|
(NOT a template placeholder - render() silently leaves missing keys as
|
||||||
|
literals). Empty string when the page has no usable text layer.
|
||||||
|
"""
|
||||||
|
text = (page.get("text_layer") or "").strip()
|
||||||
|
if not text:
|
||||||
|
return ""
|
||||||
|
return ("\n\nTEXT LAYER (authoritative for alphanumeric content — trust it "
|
||||||
|
"over the image for numbers, tags, and note text):\n"
|
||||||
|
+ text[:config.TEXT_LAYER_MAX_CHARS])
|
||||||
|
|
||||||
|
|
||||||
|
def _text_structuring_extract(page: Dict, sheet_hint: str = ""):
|
||||||
|
"""Rung 2 of the extraction ladder: text-only structuring call (no
|
||||||
|
image). The text layer is authoritative for alphanumeric content - the
|
||||||
|
model segments it instead of transcribing pixels, so vision misreads
|
||||||
|
are impossible on this rung."""
|
||||||
|
instruction = (TEXT_STRUCTURING_USER_INSTRUCTION
|
||||||
|
.replace("{sheet_hint}", sheet_hint or "")
|
||||||
|
.replace("{text_layer}",
|
||||||
|
(page.get("text_layer") or "")
|
||||||
|
[:config.TEXT_LAYER_MAX_CHARS]))
|
||||||
|
return call_json(
|
||||||
|
system_prompt=TEXT_STRUCTURING_SYSTEM_PROMPT,
|
||||||
|
user_text=instruction,
|
||||||
|
max_tokens=config.EXTRACT_MAX_TOKENS,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _extract_one(page: Dict, sheet_hint: str = "") -> Dict:
|
def _extract_one(page: Dict, sheet_hint: str = "") -> Dict:
|
||||||
user_text = EXTRACTOR_USER_INSTRUCTION.replace("{sheet_hint}", sheet_hint)
|
page_text = page.get("text_layer")
|
||||||
|
user_text = (EXTRACTOR_USER_INSTRUCTION.replace("{sheet_hint}", sheet_hint)
|
||||||
|
+ _text_layer_block(page))
|
||||||
parsed = call_json(
|
parsed = call_json(
|
||||||
system_prompt=EXTRACTOR_SYSTEM_PROMPT,
|
system_prompt=EXTRACTOR_SYSTEM_PROMPT,
|
||||||
user_text=user_text,
|
user_text=user_text,
|
||||||
@@ -232,16 +320,74 @@ def _extract_one(page: Dict, sheet_hint: str = "") -> Dict:
|
|||||||
max_tokens=config.EXTRACT_MAX_TOKENS,
|
max_tokens=config.EXTRACT_MAX_TOKENS,
|
||||||
)
|
)
|
||||||
if not isinstance(parsed, dict):
|
if not isinstance(parsed, dict):
|
||||||
return {
|
if not page_text:
|
||||||
"page_number": page["page_number"],
|
# Scanned/raster page: vision-only, keep the legacy failure shape.
|
||||||
"sheet_number": None,
|
return {
|
||||||
"discipline": "Unknown",
|
"page_number": page["page_number"],
|
||||||
"sheet_title": f"Page {page['page_number']} (extraction failed)",
|
"sheet_number": None,
|
||||||
"level": None,
|
"discipline": "Unknown",
|
||||||
"scale": None,
|
"sheet_title": f"Page {page['page_number']} (extraction failed)",
|
||||||
"assertions": [],
|
"level": None,
|
||||||
}
|
"scale": None,
|
||||||
return _normalize_sheet(parsed, page["page_number"])
|
"assertions": [],
|
||||||
|
}
|
||||||
|
# Text-bearing page: climb the ladder instead of going dark.
|
||||||
|
parsed = {"sheet": {}, "objects": []}
|
||||||
|
|
||||||
|
sheet = _normalize_sheet(parsed, page["page_number"], page_text=page_text)
|
||||||
|
cov = text_coverage(page_text or "", sheet["assertions"])
|
||||||
|
sheet["coverage"] = cov
|
||||||
|
|
||||||
|
# Rung 2: text-only structuring when coverage is below floor. MERGE,
|
||||||
|
# never replace - vision objects (graphical_basis content exists only
|
||||||
|
# in the image) are kept; the text pass fills what vision missed.
|
||||||
|
if (page_text and config.EXTRACT_TEXT_RETRY_ENABLED
|
||||||
|
and cov["ratio"] < config.EXTRACT_COVERAGE_FLOOR):
|
||||||
|
print(f"[Extract] Page {page['page_number']}: coverage "
|
||||||
|
f"{cov['ratio']:.0%} < floor - text-only structuring pass")
|
||||||
|
parsed2 = _text_structuring_extract(page, sheet_hint)
|
||||||
|
if isinstance(parsed2, dict):
|
||||||
|
sheet2 = _normalize_sheet(parsed2, page["page_number"],
|
||||||
|
page_text=page_text)
|
||||||
|
before = len(sheet["assertions"])
|
||||||
|
sheet["assertions"] = merge_objects(sheet["assertions"],
|
||||||
|
sheet2["assertions"])
|
||||||
|
for key in ("sheet_number", "sheet_title", "discipline",
|
||||||
|
"level", "scale", "drawing_type"):
|
||||||
|
if not sheet.get(key) and sheet2.get(key):
|
||||||
|
sheet[key] = sheet2[key]
|
||||||
|
cov = text_coverage(page_text, sheet["assertions"])
|
||||||
|
sheet["coverage"] = cov
|
||||||
|
print(f"[Extract] Page {page['page_number']}: merged "
|
||||||
|
f"{len(sheet['assertions']) - before} text-structured "
|
||||||
|
f"object(s), coverage now {cov['ratio']:.0%}")
|
||||||
|
|
||||||
|
# Rung 3: deterministic fallback - a dark text-bearing sheet is
|
||||||
|
# impossible. Stubs are deduped against earlier rungs.
|
||||||
|
if (page_text and config.EXTRACT_FALLBACK_ENABLED
|
||||||
|
and cov["ratio"] < config.EXTRACT_COVERAGE_FLOOR):
|
||||||
|
stubs = fallback_objects(page_text, page["page_number"],
|
||||||
|
config.EXTRACT_FALLBACK_MAX_OBJECTS)
|
||||||
|
stubs = _normalize_sheet({"sheet": {}, "objects": stubs},
|
||||||
|
page["page_number"],
|
||||||
|
page_text=page_text)["assertions"]
|
||||||
|
before = len(sheet["assertions"])
|
||||||
|
sheet["assertions"] = merge_objects(sheet["assertions"], stubs)
|
||||||
|
print(f"[Extract] Page {page['page_number']}: fallback merged "
|
||||||
|
f"{len(sheet['assertions']) - before} text-layer stub(s)")
|
||||||
|
sheet["coverage"] = text_coverage(page_text, sheet["assertions"])
|
||||||
|
|
||||||
|
# Identity recovery: never leave a text-bearing page sheet-less.
|
||||||
|
if not sheet.get("sheet_number") and page_text:
|
||||||
|
recovered = recover_sheet_number(page_text)
|
||||||
|
if recovered:
|
||||||
|
sheet["sheet_number"] = recovered
|
||||||
|
sheet["discipline"] = (discipline_from_sheet_number(recovered)
|
||||||
|
or sheet.get("discipline") or "Unknown")
|
||||||
|
print(f"[Extract] Page {page['page_number']}: sheet number "
|
||||||
|
f"recovered from text layer -> {recovered}")
|
||||||
|
|
||||||
|
return sheet
|
||||||
|
|
||||||
|
|
||||||
def extract_assertions(pages: List[Dict], on_progress=None) -> List[Dict]:
|
def extract_assertions(pages: List[Dict], on_progress=None) -> List[Dict]:
|
||||||
|
|||||||
+45
-11
@@ -1,5 +1,4 @@
|
|||||||
"""
|
"""report.py - Stage 4: assemble the final report.
|
||||||
report.py - Stage 4: assemble the final report.
|
|
||||||
|
|
||||||
Produces a single JSON object (also the web API payload) and a human-readable
|
Produces a single JSON object (also the web API payload) and a human-readable
|
||||||
Markdown summary grouped by severity.
|
Markdown summary grouped by severity.
|
||||||
@@ -8,6 +7,37 @@ Markdown summary grouped by severity.
|
|||||||
from typing import List, Dict
|
from typing import List, Dict
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from backend import config
|
||||||
|
|
||||||
|
|
||||||
|
def _extraction_coverage(sheets: List[Dict]) -> Dict | None:
|
||||||
|
"""Summarize per-sheet extraction coverage for the report summary.
|
||||||
|
|
||||||
|
Sheets may carry a ``coverage`` dict (``total_lines``/``covered_lines``/
|
||||||
|
``ratio``) attached during wave-1 extraction. Older paths and scanned
|
||||||
|
pages have none; when no sheet is measured, return None so callers can
|
||||||
|
omit the key entirely.
|
||||||
|
"""
|
||||||
|
measured = [s for s in sheets if isinstance(s.get("coverage"), dict)]
|
||||||
|
if not measured:
|
||||||
|
return None
|
||||||
|
floor = getattr(config, "EXTRACT_COVERAGE_FLOOR", 0.6)
|
||||||
|
return {
|
||||||
|
"pages_measured": len(measured),
|
||||||
|
"pages_below_floor": [
|
||||||
|
s.get("page_number") for s in measured
|
||||||
|
if s["coverage"].get("ratio", 0.0) < floor
|
||||||
|
],
|
||||||
|
"fallback_pages": [
|
||||||
|
s.get("page_number") for s in measured
|
||||||
|
if any(a.get("grounding") == "text_layer_fallback"
|
||||||
|
for a in s.get("assertions", []))
|
||||||
|
],
|
||||||
|
"mean_ratio": round(
|
||||||
|
sum(s["coverage"].get("ratio", 0.0) for s in measured)
|
||||||
|
/ len(measured), 3),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def build_report(conflicts: List[Dict], sheets: List[Dict], clusters: List[Dict],
|
def build_report(conflicts: List[Dict], sheets: List[Dict], clusters: List[Dict],
|
||||||
source: str = "") -> Dict:
|
source: str = "") -> Dict:
|
||||||
@@ -19,18 +49,22 @@ def build_report(conflicts: List[Dict], sheets: List[Dict], clusters: List[Dict]
|
|||||||
by_cat[c["category"]] = by_cat.get(c["category"], 0) + 1
|
by_cat[c["category"]] = by_cat.get(c["category"], 0) + 1
|
||||||
|
|
||||||
disciplines = sorted({s["discipline"] for s in sheets if s.get("discipline")})
|
disciplines = sorted({s["discipline"] for s in sheets if s.get("discipline")})
|
||||||
|
summary = {
|
||||||
|
"sheets_analyzed": len(sheets),
|
||||||
|
"disciplines": disciplines,
|
||||||
|
"assertions_extracted": sum(len(s.get("assertions", [])) for s in sheets),
|
||||||
|
"clusters_checked": len(clusters),
|
||||||
|
"conflicts_found": len(conflicts),
|
||||||
|
"by_severity": by_sev,
|
||||||
|
"by_category": by_cat,
|
||||||
|
}
|
||||||
|
coverage = _extraction_coverage(sheets)
|
||||||
|
if coverage is not None:
|
||||||
|
summary["extraction_coverage"] = coverage
|
||||||
return {
|
return {
|
||||||
"source": source,
|
"source": source,
|
||||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||||
"summary": {
|
"summary": summary,
|
||||||
"sheets_analyzed": len(sheets),
|
|
||||||
"disciplines": disciplines,
|
|
||||||
"assertions_extracted": sum(len(s.get("assertions", [])) for s in sheets),
|
|
||||||
"clusters_checked": len(clusters),
|
|
||||||
"conflicts_found": len(conflicts),
|
|
||||||
"by_severity": by_sev,
|
|
||||||
"by_category": by_cat,
|
|
||||||
},
|
|
||||||
"conflicts": conflicts,
|
"conflicts": conflicts,
|
||||||
"sheets": [
|
"sheets": [
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -27,12 +27,15 @@ from typing import Dict, Optional, Callable
|
|||||||
|
|
||||||
from backend.pipeline.pdf_processor import convert_pdf_to_images
|
from backend.pipeline.pdf_processor import convert_pdf_to_images
|
||||||
from backend.pipeline.extractor import extract_assertions
|
from backend.pipeline.extractor import extract_assertions
|
||||||
|
from backend.sheet_reconcile import declared_sheet_list, reconcile_sheets
|
||||||
|
from backend.text_layer import attach_text_layers, coverage_gaps
|
||||||
from backend.pipeline.sheet_index import classify_sheets, derive_project_meta_from_cover
|
from backend.pipeline.sheet_index import classify_sheets, derive_project_meta_from_cover
|
||||||
from backend.pipeline.jurisdiction import run_jurisdiction
|
from backend.pipeline.jurisdiction import run_jurisdiction
|
||||||
from backend.pipeline.normalizer import normalize_assertions, build_project_intelligence
|
from backend.pipeline.normalizer import normalize_assertions, build_project_intelligence
|
||||||
from backend.pipeline.clusterer import cluster_by_location
|
from backend.pipeline.clusterer import cluster_by_location
|
||||||
from backend.pipeline.llm_clusterer import cluster_by_location_llm
|
from backend.pipeline.llm_clusterer import cluster_by_location_llm
|
||||||
from backend import config
|
from backend import config
|
||||||
|
from backend.agents.disputes import annotate_clusters
|
||||||
from backend.pipeline.conflict_checker import check_conflicts
|
from backend.pipeline.conflict_checker import check_conflicts
|
||||||
from backend.pipeline.qaqc_review import senior_review
|
from backend.pipeline.qaqc_review import senior_review
|
||||||
from backend.pipeline.code_review import code_review
|
from backend.pipeline.code_review import code_review
|
||||||
@@ -102,9 +105,25 @@ def _run_stages(
|
|||||||
) -> Dict:
|
) -> Dict:
|
||||||
stage("PDF -> images")
|
stage("PDF -> images")
|
||||||
pages = convert_pdf_to_images(pdf_path)
|
pages = convert_pdf_to_images(pdf_path)
|
||||||
|
text_dir = os.path.join(out_dir, "text") if out_dir else None
|
||||||
|
attach_text_layers(pdf_path, pages, text_dir=text_dir)
|
||||||
|
|
||||||
stage("Extract assertions")
|
stage("Extract assertions")
|
||||||
sheets = extract_assertions(pages)
|
sheets = extract_assertions(pages)
|
||||||
|
coverage_gaps(pages, sheets) # classic: log-only recall signal
|
||||||
|
|
||||||
|
# Deterministic reconciliation: cover-sheet index vs identified sheets.
|
||||||
|
page_to_text = {p["page_number"]: p.get("text_layer") for p in pages}
|
||||||
|
sheet_recon = reconcile_sheets(sheets, declared_sheet_list(page_to_text))
|
||||||
|
if sheet_recon["declared_total"]:
|
||||||
|
print(f"[SheetIndex] cover declares {sheet_recon['declared_total']} "
|
||||||
|
f"sheets; {sheet_recon['found_total']} identified in set")
|
||||||
|
if sheet_recon["declared_not_in_set"]:
|
||||||
|
print(f"[SheetIndex] declared but not in set: "
|
||||||
|
f"{', '.join(sheet_recon['declared_not_in_set'][:20])}")
|
||||||
|
if sheet_recon["in_set_not_declared"]:
|
||||||
|
print(f"[SheetIndex] in set but not declared: "
|
||||||
|
f"{', '.join(sheet_recon['in_set_not_declared'][:20])}")
|
||||||
|
|
||||||
stage("Classify sheet index")
|
stage("Classify sheet index")
|
||||||
sheet_index = classify_sheets(sheets)
|
sheet_index = classify_sheets(sheets)
|
||||||
@@ -129,6 +148,10 @@ def _run_stages(
|
|||||||
else:
|
else:
|
||||||
clusters = cluster_by_location(sheets)
|
clusters = cluster_by_location(sheets)
|
||||||
|
|
||||||
|
disputed_count = annotate_clusters(clusters)
|
||||||
|
if disputed_count:
|
||||||
|
print(f"[Cluster] {disputed_count} cluster(s) carry disputed extracted values")
|
||||||
|
|
||||||
stage("Reason over clusters (conflicts)")
|
stage("Reason over clusters (conflicts)")
|
||||||
conflicts = check_conflicts(clusters, pages)
|
conflicts = check_conflicts(clusters, pages)
|
||||||
|
|
||||||
@@ -158,6 +181,7 @@ def _run_stages(
|
|||||||
report["project_input"] = merged_input
|
report["project_input"] = merged_input
|
||||||
report["jurisdiction"] = jurisdiction
|
report["jurisdiction"] = jurisdiction
|
||||||
report["sheet_index"] = sheet_index
|
report["sheet_index"] = sheet_index
|
||||||
|
report["sheet_reconciliation"] = sheet_recon
|
||||||
report["project_intelligence"] = project_intel
|
report["project_intelligence"] = project_intel
|
||||||
report["validated_issues"] = prioritized
|
report["validated_issues"] = prioritized
|
||||||
report["rfis"] = rfis
|
report["rfis"] = rfis
|
||||||
|
|||||||
+58
-1
@@ -230,6 +230,7 @@ Rules you must never break:
|
|||||||
- Every object must include source_text copied verbatim from the sheet whenever text is available.
|
- Every object must include source_text copied verbatim from the sheet whenever text is available.
|
||||||
- If the object is graphical and has no text, describe it visually and mark confidence low or medium.
|
- If the object is graphical and has no text, describe it visually and mark confidence low or medium.
|
||||||
- Preserve tags, marks, room numbers, sheet numbers, detail references, and abbreviations exactly as shown.
|
- Preserve tags, marks, room numbers, sheet numbers, detail references, and abbreviations exactly as shown.
|
||||||
|
TEXT LAYER GROUNDING: when a TEXT LAYER block is present in the user message, it is the sheet's deterministic PDF text layer and is authoritative for alphanumeric content (counts, dimensions, member tags, note text). Trust it over your reading of the image for numbers, tags, and note text; quote source_text from it verbatim. Use the image for geometry, symbols, linework, and anything absent from the text layer.
|
||||||
- Use null when information is not determinable.
|
- Use null when information is not determinable.
|
||||||
- Keep objects atomic.
|
- Keep objects atomic.
|
||||||
- Use plain ASCII only.
|
- Use plain ASCII only.
|
||||||
@@ -280,6 +281,30 @@ If the sheet has no extractable objects, return an empty objects array.
|
|||||||
Optional sheet hint: {sheet_hint}"""
|
Optional sheet hint: {sheet_hint}"""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Stage 2b - text-only structuring (extraction retry ladder, rung 2)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
TEXT_STRUCTURING_SYSTEM_PROMPT = """You are a construction document structuring engine.
|
||||||
|
You receive the deterministic text layer extracted from one drawing sheet. It is complete and authoritative.
|
||||||
|
Your ONLY job is to segment it into structured objects. You are NOT reading an image. You must NOT invent, complete, or correct any text.
|
||||||
|
Rules:
|
||||||
|
- Every numbered note, schedule row, callout, tag, legend entry, and title-block field becomes its own object.
|
||||||
|
- source_text must be copied VERBATIM from the input, character-for-character. Never paraphrase.
|
||||||
|
- Cover the ENTIRE input. Omitting a note is a failure. When unsure of an object's type, use general_note with confidence low.
|
||||||
|
- Numbers, model numbers, dimensions, and tags must appear in source_text exactly as in the input.
|
||||||
|
Respond only with valid JSON."""
|
||||||
|
|
||||||
|
TEXT_STRUCTURING_USER_INSTRUCTION = """Segment this sheet's text layer into structured construction objects.
|
||||||
|
Every note, schedule row, callout, tag, and title-block field in the text layer must become an object - omit nothing.
|
||||||
|
Respond ONLY with a valid JSON object - no markdown fences:
|
||||||
|
{ "sheet": { "sheet_number": "string or null", "sheet_title": "string or null", "discipline": "string or null", "drawing_type": "string or null", "level": "string or null", "scale": "string or null" }, "objects": [ { "object_id": "string", "object_type": "room | door | window | wall | finish | ceiling | dimension | grid | callout | keynote | general_note | equipment | plumbing_fixture | mechanical_equipment | electrical_device | lighting_fixture | structural_element | schedule_reference | symbol | abbreviation", "category": "architectural | structural | mechanical | electrical | plumbing | code | general", "tag": "string or null", "name": "string or null", "description": "string or null", "attributes": { "attribute_name": "attribute_value" }, "location_key": { "room_number": "string or null", "grid": "string or null", "detail_reference": "string or null" }, "source_text": "VERBATIM text copied from the input", "graphical_basis": null, "review_uses": [ "schedule_comparison", "cross_discipline_coordination", "code_review", "constructability_review" ], "confidence": "high | medium | low" } ], "unresolved_items": [] }
|
||||||
|
Optional sheet hint: {sheet_hint}
|
||||||
|
|
||||||
|
TEXT LAYER (segment ALL of it):
|
||||||
|
{text_layer}"""
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Stage 3a - assertion normalization (WIRED: normalizer.py)
|
# Stage 3a - assertion normalization (WIRED: normalizer.py)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -410,6 +435,9 @@ What is NOT a conflict:
|
|||||||
- Anything not supported with drawing evidence.
|
- Anything not supported with drawing evidence.
|
||||||
Be conservative:
|
Be conservative:
|
||||||
- Only flag genuine disagreements.
|
- Only flag genuine disagreements.
|
||||||
|
- When a value is marked DISPUTED (possible extraction misread), verify it against
|
||||||
|
the sheet images before relying on either reading; if the images do not resolve
|
||||||
|
it, do not assert a conflict from one reading alone.
|
||||||
- A clean cluster with no contradiction must return an empty conflicts array.
|
- A clean cluster with no contradiction must return an empty conflicts array.
|
||||||
- missing_element requires evidence that another discipline would reasonably be expected to show the missing item.
|
- missing_element requires evidence that another discipline would reasonably be expected to show the missing item.
|
||||||
For each conflict:
|
For each conflict:
|
||||||
@@ -447,6 +475,28 @@ Clustered assertions (evidence):
|
|||||||
{evidence}"""
|
{evidence}"""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Wave 5b - evidence verification (vision fact-check of cited sheet text)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
VERIFY_SYSTEM_PROMPT = """You are a meticulous construction document checker verifying machine-extracted evidence against the actual drawing sheet images.
|
||||||
|
For each evidence item you are given the sheet it was extracted from and the verbatim text the extractor claims appears there.
|
||||||
|
Judge each item against the images:
|
||||||
|
- confirmed: the text (or an obvious equivalent) appears on the cited sheet and means what the finding claims.
|
||||||
|
- corrected: the sheet shows a DIFFERENT value than the extracted text. Give the actual verbatim text.
|
||||||
|
- not_found: nothing like the extracted text appears on the cited sheet.
|
||||||
|
Be strict about numbers, quantities, and member sizes: "(2) 2x6" and "(5) 2x6" are different values. HSS16x4 and HSS16x16 are different values.
|
||||||
|
Use plain ASCII only.
|
||||||
|
Respond only with valid JSON."""
|
||||||
|
|
||||||
|
VERIFY_USER_INSTRUCTION = """Verify this finding's evidence against the attached sheet images.
|
||||||
|
Respond ONLY with a valid JSON object - no markdown fences, no explanation:
|
||||||
|
{ "verdicts": [ { "sheet": "string", "source_text": "the evidence text judged", "verdict": "confirmed | corrected | not_found", "actual_text": "verbatim sheet text when corrected, else null", "notes": "string or null" } ] }
|
||||||
|
Finding: {finding}
|
||||||
|
TEXT LAYER (deterministic page text extracted from the PDF - an oracle for alphanumeric content such as counts, dimensions, and member tags; when it disagrees with the extracted evidence, trust it and cite it as actual_text):
|
||||||
|
{text_layer}"""
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Stage 6 - senior architect full-set QAQC review (NOT WIRED YET)
|
# Stage 6 - senior architect full-set QAQC review (NOT WIRED YET)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -545,6 +595,12 @@ Flag:
|
|||||||
Rules:
|
Rules:
|
||||||
- Only flag issues supported by drawing evidence.
|
- Only flag issues supported by drawing evidence.
|
||||||
- Be specific about the location and why it is a constructability risk.
|
- Be specific about the location and why it is a constructability risk.
|
||||||
|
- Assertions are machine-extracted from sheet images and may contain misread values,
|
||||||
|
especially quantities and member sizes (e.g. "(2) 2x6" vs "(5) 2x6").
|
||||||
|
- When the cluster lists disputed_attributes, or two evidence items disagree on a
|
||||||
|
numeric value, do NOT assert a buildability conclusion from one reading. Report the
|
||||||
|
ambiguity itself (category "detail_gap", confidence "low") and state that the value
|
||||||
|
needs verification against the sheet.
|
||||||
- Use plain ASCII only.
|
- Use plain ASCII only.
|
||||||
Respond only with valid JSON."""
|
Respond only with valid JSON."""
|
||||||
|
|
||||||
@@ -554,7 +610,8 @@ Respond ONLY with a valid JSON object - no markdown fences, no explanation:
|
|||||||
If no constructability issues are found, return: { "issues": [] }
|
If no constructability issues are found, return: { "issues": [] }
|
||||||
Extracted assertions: {assertions}
|
Extracted assertions: {assertions}
|
||||||
Clusters: {clusters}
|
Clusters: {clusters}
|
||||||
Cross-discipline conflicts already found: {conflicts}"""
|
Cross-discipline conflicts already found: {conflicts}
|
||||||
|
Disputed extracted values in this cluster (possible vision misreads - treat as unverified): {disputes}"""
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -216,7 +216,7 @@ def finalize_review(job_id: str, out_dir: str) -> dict:
|
|||||||
rfis = _draft_rfis(kept)
|
rfis = _draft_rfis(kept)
|
||||||
|
|
||||||
report["validated_issues"] = kept
|
report["validated_issues"] = kept
|
||||||
report["suppressed_issues"] = suppressed
|
report["suppressed_issues"] = (report.get("suppressed_issues") or []) + suppressed
|
||||||
report["rfis"] = rfis
|
report["rfis"] = rfis
|
||||||
summary = report.setdefault("summary", {})
|
summary = report.setdefault("summary", {})
|
||||||
summary["agent_status"] = "complete"
|
summary["agent_status"] = "complete"
|
||||||
|
|||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""sheet_reconcile.py - deterministic sheet-list reconciliation (no LLM).
|
||||||
|
|
||||||
|
The cover sheet's own sheet index (SHEET LIST / DRAWING INDEX) declares which
|
||||||
|
sheets the set is SUPPOSED to contain. Comparing that declaration against the
|
||||||
|
sheets wave-1 actually identified answers two early questions:
|
||||||
|
|
||||||
|
- declared_not_in_set: sheets the index lists but we didn't identify - dark
|
||||||
|
pages, misidentification, or disciplines genuinely absent from this PDF.
|
||||||
|
- in_set_not_declared: sheet numbers we extracted that the index doesn't
|
||||||
|
list - misread title blocks or unlisted sheets.
|
||||||
|
|
||||||
|
Deterministic complement to the LLM sheet_index stage, which can only infer
|
||||||
|
from what extraction already found.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
|
||||||
|
# Markers that introduce the drawing set's own sheet index on a cover page.
|
||||||
|
_INDEX_MARKERS = (
|
||||||
|
"SHEET LIST",
|
||||||
|
"DRAWING INDEX",
|
||||||
|
"SHEET INDEX",
|
||||||
|
"DRAWING LIST",
|
||||||
|
"INDEX OF DRAWINGS",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Sheet ids: 1-2 letters, optional hyphen, 2-3 digits, optional decimal suffix.
|
||||||
|
# Covers S301, A102, LS101, C-001, C-001.1; excludes dates/project numbers
|
||||||
|
# (pure digits) and member marks (W12X26 - letter after digits).
|
||||||
|
_SHEET_TOKEN_RE = re.compile(r"\b([A-Z]{1,2}-?\d{2,3}(?:\.\d+)?)\b")
|
||||||
|
|
||||||
|
# Only cover-front pages carry the set index.
|
||||||
|
_MAX_INDEX_PAGE = 5
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_id(sheet_id: str) -> str:
|
||||||
|
return (sheet_id or "").upper().replace("-", "").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def declared_sheet_list(page_texts: Dict[int, Optional[str]]) -> List[str]:
|
||||||
|
"""Scrape the declared sheet list off the cover page's text layer.
|
||||||
|
|
||||||
|
page_texts: {page_number: text_layer_or_None}. Returns the ordered,
|
||||||
|
deduped list of declared sheet ids, or [] when no index marker exists.
|
||||||
|
Only the FIRST page containing a marker is parsed (later 'sheet list'
|
||||||
|
echoes in legends/schedules are ignored).
|
||||||
|
"""
|
||||||
|
for page_number in sorted(page_texts):
|
||||||
|
if page_number > _MAX_INDEX_PAGE:
|
||||||
|
break
|
||||||
|
text = page_texts.get(page_number) or ""
|
||||||
|
upper = text.upper()
|
||||||
|
marker_at = -1
|
||||||
|
for marker in _INDEX_MARKERS:
|
||||||
|
marker_at = upper.find(marker)
|
||||||
|
if marker_at >= 0:
|
||||||
|
break
|
||||||
|
if marker_at < 0:
|
||||||
|
continue
|
||||||
|
section = text[marker_at:]
|
||||||
|
declared: List[str] = []
|
||||||
|
for token in _SHEET_TOKEN_RE.findall(section):
|
||||||
|
if token not in declared:
|
||||||
|
declared.append(token)
|
||||||
|
return declared
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def reconcile_sheets(sheets: List[Dict], declared: List[str]) -> Dict:
|
||||||
|
"""Compare extracted sheet_numbers against the declared index.
|
||||||
|
|
||||||
|
Comparison is hyphen/case-normalized; output lists keep the declared /
|
||||||
|
extracted originals.
|
||||||
|
"""
|
||||||
|
found: List[str] = [str(s["sheet_number"]) for s in sheets or []
|
||||||
|
if s.get("sheet_number")]
|
||||||
|
found_norm = {_normalize_id(n) for n in found}
|
||||||
|
declared_norm = {_normalize_id(n) for n in declared}
|
||||||
|
|
||||||
|
declared_not_in_set = [n for n in declared if _normalize_id(n) not in found_norm]
|
||||||
|
# Preserve extraction order, dedupe, keep originals.
|
||||||
|
in_set_not_declared: List[str] = []
|
||||||
|
for n in found:
|
||||||
|
if _normalize_id(n) not in declared_norm and n not in in_set_not_declared:
|
||||||
|
in_set_not_declared.append(n)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"declared_total": len(declared),
|
||||||
|
"found_total": len(found),
|
||||||
|
"declared_not_in_set": declared_not_in_set,
|
||||||
|
"in_set_not_declared": in_set_not_declared,
|
||||||
|
}
|
||||||
@@ -0,0 +1,135 @@
|
|||||||
|
"""text_coverage.py - deterministic extraction-coverage measurement.
|
||||||
|
|
||||||
|
The coverage guarantee: for any page with a usable text layer, measure how
|
||||||
|
much of that layer ended up represented in extracted objects. Pages below
|
||||||
|
the floor route into the extraction retry ladder (agents/extractors.py and
|
||||||
|
pipeline/extractor.py). fallback_objects() is the last rung: stub objects
|
||||||
|
segmented straight from the text layer so no text-bearing page goes dark.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
|
||||||
|
MIN_LINE_CHARS = 12
|
||||||
|
_TICK_RE = re.compile(r"^[\d\s'\"/.,-]+$")
|
||||||
|
_WORD_RE = re.compile(r"[a-z0-9]+")
|
||||||
|
|
||||||
|
|
||||||
|
def _meaningful_lines(text: str) -> List[str]:
|
||||||
|
lines = []
|
||||||
|
for raw in (text or "").splitlines():
|
||||||
|
line = " ".join(raw.split())
|
||||||
|
if len(line) < MIN_LINE_CHARS or _TICK_RE.match(line):
|
||||||
|
continue
|
||||||
|
lines.append(line)
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def _norm(text: str) -> str:
|
||||||
|
return " ".join(_WORD_RE.findall((text or "").lower()))
|
||||||
|
|
||||||
|
|
||||||
|
def text_coverage(page_text: str, objects: List[Dict]) -> Dict:
|
||||||
|
"""Fraction of meaningful text-layer lines whose normalized form appears
|
||||||
|
in the concatenated normalized source_text of extracted objects."""
|
||||||
|
lines = _meaningful_lines(page_text)
|
||||||
|
if not lines:
|
||||||
|
return {"total_lines": 0, "covered_lines": 0, "ratio": 1.0}
|
||||||
|
haystack = " ".join(
|
||||||
|
_norm(str(o.get("source_text") or o.get("object_description")
|
||||||
|
or o.get("value") or ""))
|
||||||
|
for o in objects if isinstance(o, dict)
|
||||||
|
)
|
||||||
|
covered = sum(1 for ln in lines if _norm(ln) and _norm(ln) in haystack)
|
||||||
|
return {
|
||||||
|
"total_lines": len(lines),
|
||||||
|
"covered_lines": covered,
|
||||||
|
"ratio": covered / len(lines) if lines else 1.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def segment_text_layer(text: str) -> List[str]:
|
||||||
|
"""Segment a page text layer into note-sized blocks."""
|
||||||
|
segments: List[str] = []
|
||||||
|
buf: List[str] = []
|
||||||
|
number_re = re.compile(r"^(\d{1,2}[.)]?|[A-Z]\d{0,2}[.)]?)\s*$")
|
||||||
|
|
||||||
|
def flush():
|
||||||
|
joined = " ".join(buf).strip()
|
||||||
|
if len(joined) >= MIN_LINE_CHARS:
|
||||||
|
segments.append(joined)
|
||||||
|
buf.clear()
|
||||||
|
|
||||||
|
for raw in (text or "").splitlines():
|
||||||
|
line = raw.strip()
|
||||||
|
if not line:
|
||||||
|
flush()
|
||||||
|
continue
|
||||||
|
if number_re.match(line):
|
||||||
|
flush()
|
||||||
|
buf.append(line.rstrip(".)"))
|
||||||
|
continue
|
||||||
|
buf.append(line)
|
||||||
|
if line.endswith(".") and len(" ".join(buf)) > 120:
|
||||||
|
flush()
|
||||||
|
flush()
|
||||||
|
return segments
|
||||||
|
|
||||||
|
|
||||||
|
def fallback_objects(page_text: str, page_number: int,
|
||||||
|
max_objects: int = 200) -> List[Dict]:
|
||||||
|
"""Last-rung deterministic extraction: one stub object per text segment,
|
||||||
|
source_text verbatim from the text layer."""
|
||||||
|
objs = []
|
||||||
|
for idx, seg in enumerate(segment_text_layer(page_text)[:max_objects]):
|
||||||
|
objs.append({
|
||||||
|
"object_id": f"p{page_number}-tl{idx}",
|
||||||
|
"object_type": "general_note",
|
||||||
|
"category": "general",
|
||||||
|
"tag": None,
|
||||||
|
"name": seg[:80],
|
||||||
|
"description": seg,
|
||||||
|
"attributes": {},
|
||||||
|
"location_key": {},
|
||||||
|
"source_text": seg,
|
||||||
|
"graphical_basis": None,
|
||||||
|
"review_uses": ["code_review", "constructability_review"],
|
||||||
|
"confidence": "low",
|
||||||
|
"grounding": "text_layer_fallback",
|
||||||
|
})
|
||||||
|
return objs
|
||||||
|
|
||||||
|
|
||||||
|
def merge_objects(vision_objs: List[Dict], text_objs: List[Dict]) -> List[Dict]:
|
||||||
|
"""Union of vision and text-structured objects. Vision results come first
|
||||||
|
and are never dropped. Text objects are appended unless their normalized
|
||||||
|
source_text is already represented."""
|
||||||
|
merged = list(vision_objs or [])
|
||||||
|
seen = {_norm(str(o.get("source_text") or ""))
|
||||||
|
for o in merged if isinstance(o, dict)}
|
||||||
|
seen.discard("")
|
||||||
|
for obj in text_objs or []:
|
||||||
|
if not isinstance(obj, dict):
|
||||||
|
continue
|
||||||
|
key = _norm(str(obj.get("source_text") or ""))
|
||||||
|
if key and key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
merged.append(obj)
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
|
_SHEET_ID_RE = re.compile(r"\b([A-Z]{1,2}\d{2,3}(?:\.\d+)?)\b")
|
||||||
|
|
||||||
|
|
||||||
|
def recover_sheet_number(page_text: str) -> Optional[str]:
|
||||||
|
"""Deterministic sheet id from the text layer: prefer candidates in the
|
||||||
|
last ~15% of the page (title block lives at the drawing edge)."""
|
||||||
|
text = page_text or ""
|
||||||
|
cands = _SHEET_ID_RE.findall(text)
|
||||||
|
if not cands:
|
||||||
|
return None
|
||||||
|
tail = text[int(len(text) * 0.85):]
|
||||||
|
for cand in reversed(_SHEET_ID_RE.findall(tail)):
|
||||||
|
return cand
|
||||||
|
return cands[0]
|
||||||
@@ -0,0 +1,230 @@
|
|||||||
|
"""
|
||||||
|
text_layer.py - deterministic PDF text-layer extraction (PyMuPDF, no LLM).
|
||||||
|
|
||||||
|
Most CAD-produced drawing sets carry a real vector text layer. We extract it
|
||||||
|
once per job and feed it to the extractor (grounding), the grounding guard
|
||||||
|
(rescue tier), and the wave-5b verifier (text oracle + high-DPI evidence
|
||||||
|
crops). Pages below TEXT_LAYER_MIN_CHARS of text are treated as having no
|
||||||
|
text layer (scanned/raster sheets stay vision-only).
|
||||||
|
|
||||||
|
If PyMuPDF is unavailable the module degrades gracefully: every public
|
||||||
|
function returns empty/None, equivalent to TEXT_LAYER_ENABLED=false.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Dict, List, Optional, Tuple
|
||||||
|
|
||||||
|
from backend import config
|
||||||
|
|
||||||
|
try: # PyMuPDF >= 1.24 prefers the pymupdf name; fitz works everywhere.
|
||||||
|
import pymupdf as fitz
|
||||||
|
except ImportError: # pragma: no cover - older PyMuPDF
|
||||||
|
try:
|
||||||
|
import fitz
|
||||||
|
except ImportError: # pragma: no cover - PyMuPDF not installed
|
||||||
|
fitz = None
|
||||||
|
|
||||||
|
_warned_unavailable = False
|
||||||
|
|
||||||
|
# Word token normalization for evidence matching: lowercase alphanumeric only.
|
||||||
|
_TOKEN_RE = re.compile(r"[^a-z0-9]+")
|
||||||
|
# Fuzzy match floor: fraction of needle tokens that must align with the page's
|
||||||
|
# word sequence for a bbox to count as a confident evidence location.
|
||||||
|
_FUZZY_MIN_RATIO = 0.6
|
||||||
|
|
||||||
|
|
||||||
|
def _fitz_or_none():
|
||||||
|
"""Return the fitz module, logging once if PyMuPDF is missing."""
|
||||||
|
global _warned_unavailable
|
||||||
|
if fitz is None and not _warned_unavailable:
|
||||||
|
print("[TextLayer] PyMuPDF not available - text-layer grounding disabled")
|
||||||
|
_warned_unavailable = True
|
||||||
|
return fitz
|
||||||
|
|
||||||
|
|
||||||
|
def extract_text_layers(pdf_path: str) -> Dict[int, Dict]:
|
||||||
|
"""
|
||||||
|
Extract the text layer of every page. Returns {1-based page_number:
|
||||||
|
{"text": str, "words": [{"text", "bbox": (x0,y0,x1,y1)}, ...],
|
||||||
|
"has_text_layer": bool}}. Returns {} when disabled or unavailable.
|
||||||
|
"""
|
||||||
|
if not config.TEXT_LAYER_ENABLED:
|
||||||
|
return {}
|
||||||
|
f = _fitz_or_none()
|
||||||
|
if f is None:
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
doc = f.open(pdf_path)
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"[TextLayer] could not open {pdf_path}: {exc}")
|
||||||
|
return {}
|
||||||
|
layers: Dict[int, Dict] = {}
|
||||||
|
try:
|
||||||
|
for index in range(doc.page_count):
|
||||||
|
page = doc[index]
|
||||||
|
text = page.get_text("text") or ""
|
||||||
|
words = [
|
||||||
|
{"text": w[4], "bbox": (w[0], w[1], w[2], w[3])}
|
||||||
|
for w in (page.get_text("words") or [])
|
||||||
|
]
|
||||||
|
has_text_layer = len(text.strip()) >= config.TEXT_LAYER_MIN_CHARS
|
||||||
|
if not has_text_layer:
|
||||||
|
print(f"[TextLayer] Page {index + 1}: {len(text.strip())} chars "
|
||||||
|
f"(< TEXT_LAYER_MIN_CHARS={config.TEXT_LAYER_MIN_CHARS}) - "
|
||||||
|
f"vision-only")
|
||||||
|
layers[index + 1] = {
|
||||||
|
"text": text,
|
||||||
|
"words": words,
|
||||||
|
"has_text_layer": has_text_layer,
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
doc.close()
|
||||||
|
return layers
|
||||||
|
|
||||||
|
|
||||||
|
def attach_text_layers(
|
||||||
|
pdf_path: str,
|
||||||
|
pages: List[Dict],
|
||||||
|
text_dir: Optional[str] = None,
|
||||||
|
) -> Dict[int, List[Dict]]:
|
||||||
|
"""
|
||||||
|
Attach page["text_layer"] (text or None) to each converted page dict and
|
||||||
|
return the runner-local {page_number: words} map (kept off page dicts -
|
||||||
|
those get serialized). When text_dir is set, dump one .txt per page there
|
||||||
|
(plain file writes; ProjectMemory is a closed registry).
|
||||||
|
"""
|
||||||
|
layers = extract_text_layers(pdf_path)
|
||||||
|
page_words: Dict[int, List[Dict]] = {}
|
||||||
|
for page in pages:
|
||||||
|
layer = layers.get(page["page_number"]) or {}
|
||||||
|
page["text_layer"] = layer.get("text") if layer.get("has_text_layer") else None
|
||||||
|
page_words[page["page_number"]] = layer.get("words") or []
|
||||||
|
if text_dir and layers:
|
||||||
|
import os
|
||||||
|
os.makedirs(text_dir, exist_ok=True)
|
||||||
|
for page_number, layer in layers.items():
|
||||||
|
if not layer.get("has_text_layer"):
|
||||||
|
continue
|
||||||
|
with open(os.path.join(text_dir, f"page-{page_number:03d}.txt"),
|
||||||
|
"w", encoding="utf-8") as fh:
|
||||||
|
fh.write(layer.get("text") or "")
|
||||||
|
return page_words
|
||||||
|
|
||||||
|
|
||||||
|
def _tokens(text: str) -> List[str]:
|
||||||
|
return [t for t in _TOKEN_RE.split(text.lower()) if t]
|
||||||
|
|
||||||
|
|
||||||
|
def _union_bbox(boxes: List[Tuple[float, float, float, float]]):
|
||||||
|
return (
|
||||||
|
min(b[0] for b in boxes),
|
||||||
|
min(b[1] for b in boxes),
|
||||||
|
max(b[2] for b in boxes),
|
||||||
|
max(b[3] for b in boxes),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def find_evidence_bbox(
|
||||||
|
words: List[Dict],
|
||||||
|
needle: str,
|
||||||
|
) -> Optional[Tuple[float, float, float, float]]:
|
||||||
|
"""
|
||||||
|
Best-effort fuzzy substring match of an evidence source_text against the
|
||||||
|
page's word sequence. Returns the union bbox of the matched words, or
|
||||||
|
None when nothing aligns confidently.
|
||||||
|
|
||||||
|
Exact contiguous token runs win; otherwise the best-scoring window with
|
||||||
|
>= _FUZZY_MIN_RATIO token alignment is accepted (vision quotes imperfectly
|
||||||
|
but the value is real page text).
|
||||||
|
"""
|
||||||
|
if not words or not needle:
|
||||||
|
return None
|
||||||
|
needle_tokens = _tokens(str(needle))
|
||||||
|
if not needle_tokens:
|
||||||
|
return None
|
||||||
|
page_tokens = [_tokens(w.get("text") or "") for w in words]
|
||||||
|
# Flatten multi-token words, remembering which word each token came from.
|
||||||
|
flat: List[Tuple[str, int]] = []
|
||||||
|
for word_index, parts in enumerate(page_tokens):
|
||||||
|
for part in parts:
|
||||||
|
flat.append((part, word_index))
|
||||||
|
if not flat:
|
||||||
|
return None
|
||||||
|
|
||||||
|
n = len(needle_tokens)
|
||||||
|
best_span = None
|
||||||
|
best_score = 0.0
|
||||||
|
for start in range(0, len(flat)):
|
||||||
|
window = flat[start:start + n]
|
||||||
|
if not window:
|
||||||
|
break
|
||||||
|
score = sum(1 for i, tok in enumerate(needle_tokens)
|
||||||
|
if i < len(window) and window[i][0] == tok) / n
|
||||||
|
if score > best_score:
|
||||||
|
best_score = score
|
||||||
|
best_span = window
|
||||||
|
if best_score == 1.0:
|
||||||
|
break
|
||||||
|
if best_span is None or best_score < _FUZZY_MIN_RATIO:
|
||||||
|
return None
|
||||||
|
word_indexes = {word_index for _, word_index in best_span}
|
||||||
|
return _union_bbox([words[i]["bbox"] for i in sorted(word_indexes)])
|
||||||
|
|
||||||
|
|
||||||
|
def render_crop(
|
||||||
|
pdf_path: str,
|
||||||
|
page_number: int,
|
||||||
|
bbox: Tuple[float, float, float, float],
|
||||||
|
dpi: Optional[int] = None,
|
||||||
|
margin_pts: Optional[float] = None,
|
||||||
|
) -> Optional[bytes]:
|
||||||
|
"""
|
||||||
|
Render a clip of one page around bbox (+ margin, clamped to the page) at
|
||||||
|
the given DPI and return JPEG bytes, or None on any failure.
|
||||||
|
"""
|
||||||
|
f = _fitz_or_none()
|
||||||
|
if f is None:
|
||||||
|
return None
|
||||||
|
dpi = dpi or config.VERIFY_CROP_DPI
|
||||||
|
margin_pts = config.VERIFY_CROP_MARGIN_PTS if margin_pts is None else margin_pts
|
||||||
|
try:
|
||||||
|
doc = f.open(pdf_path)
|
||||||
|
try:
|
||||||
|
page = doc[page_number - 1]
|
||||||
|
rect = f.Rect(
|
||||||
|
bbox[0] - margin_pts,
|
||||||
|
bbox[1] - margin_pts,
|
||||||
|
bbox[2] + margin_pts,
|
||||||
|
bbox[3] + margin_pts,
|
||||||
|
) & page.rect
|
||||||
|
if rect.is_empty:
|
||||||
|
return None
|
||||||
|
pix = page.get_pixmap(clip=rect, dpi=dpi)
|
||||||
|
return pix.tobytes("jpeg")
|
||||||
|
finally:
|
||||||
|
doc.close()
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"[TextLayer] render_crop failed on page {page_number}: {exc}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def coverage_gaps(pages: List[Dict], sheets: List[Dict]) -> List[int]:
|
||||||
|
"""
|
||||||
|
Page numbers that have a text layer but whose extraction failed or
|
||||||
|
returned 0 objects - the silent extraction-loss signal. Logs one
|
||||||
|
[TextLayer] line per gap.
|
||||||
|
"""
|
||||||
|
by_page = {s.get("page_number"): s for s in sheets or []}
|
||||||
|
gaps: List[int] = []
|
||||||
|
for page in pages:
|
||||||
|
text = page.get("text_layer")
|
||||||
|
if not text:
|
||||||
|
continue
|
||||||
|
sheet = by_page.get(page["page_number"])
|
||||||
|
extracted = len(sheet.get("assertions") or []) if sheet else 0
|
||||||
|
if extracted == 0:
|
||||||
|
gaps.append(page["page_number"])
|
||||||
|
print(f"[TextLayer] Page {page['page_number']}: text layer present "
|
||||||
|
f"({len(text)} chars) but no objects extracted — possible "
|
||||||
|
f"extraction gap")
|
||||||
|
return gaps
|
||||||
@@ -0,0 +1,166 @@
|
|||||||
|
# Text-Layer Grounding — Design Spec
|
||||||
|
|
||||||
|
**Date:** 2026-08-12 · **Branch:** `agent-mode` · **Status:** approved by user (2026-08-12)
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The pipeline is vision-only for extraction, but most CAD-produced drawing sets
|
||||||
|
carry a real vector text layer. Two worst documented failure modes are text
|
||||||
|
problems being solved with pixels:
|
||||||
|
|
||||||
|
1. **Wave-1 text misreads propagate immutably** — e.g. job `959e16407573`:
|
||||||
|
vision read "(2) 2x6 STUD PACK" where the sheet says "(5)"; text-only
|
||||||
|
downstream specialists treated the misread as ground truth → confident
|
||||||
|
false-positive findings.
|
||||||
|
2. **Silent extraction loss** — failed/under-extracted pages are invisible
|
||||||
|
(job `475a6f184dd1`: 42% extraction loss), producing false
|
||||||
|
`missing_expected_sheets` warnings and missed conflicts.
|
||||||
|
|
||||||
|
Priority (user, 2026-08-12): reduce false positives **and** missed items;
|
||||||
|
more accurate conflicts.
|
||||||
|
|
||||||
|
## Approach
|
||||||
|
|
||||||
|
Extract the PDF text layer deterministically (PyMuPDF) once per job, and make
|
||||||
|
it a first-class citizen at three points: extractor grounding, the grounding
|
||||||
|
guard, and the wave-5b verifier (as text oracle + high-DPI evidence crops).
|
||||||
|
|
||||||
|
Inspired by `hamzaabduljabbar/construction-drawing-analyzer` (patterns only —
|
||||||
|
its license is source-available/no-resale; all code here is original).
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### 1. New module `backend/text_layer.py` (deterministic, no LLM)
|
||||||
|
|
||||||
|
- `extract_text_layers(pdf_path) -> Dict[int, dict]` — per 1-based page:
|
||||||
|
`{"text": str, "words": [{"text", "bbox": (x0,y0,x1,y1)}, ...],
|
||||||
|
"has_text_layer": bool}`. Pages with < `TEXT_LAYER_MIN_CHARS` of text are
|
||||||
|
`has_text_layer=False` (scanned/raster sheets stay vision-only; logged).
|
||||||
|
- `find_evidence_bbox(words, needle) -> bbox | None` — best-effort fuzzy
|
||||||
|
substring match of an evidence `source_text` against word sequence; returns
|
||||||
|
union rect of matched words.
|
||||||
|
- `render_crop(pdf_path, page_number, bbox, dpi, margin_pts) -> bytes` —
|
||||||
|
PyMuPDF `page.get_pixmap(clip=rect, dpi=dpi)` → JPEG bytes.
|
||||||
|
|
||||||
|
Both runners call `extract_text_layers` right after `convert_pdf_to_images`
|
||||||
|
and attach `page["text_layer"] = <text or None>` to each page dict. Word
|
||||||
|
positions stay in a separate `page_words: Dict[int, list]` runner-local map
|
||||||
|
(not attached to page dicts — they get serialized).
|
||||||
|
|
||||||
|
### 2. Extractor grounding (both pipelines)
|
||||||
|
|
||||||
|
- Static paragraph added to `_EXTRACTOR_SYSTEM_TEMPLATE` in
|
||||||
|
`backend/prompts.py` (no new placeholder): when a TEXT LAYER block is
|
||||||
|
present in the user message it is **authoritative for alphanumeric content**
|
||||||
|
(counts, dimensions, member tags, notes); the image is for geometry,
|
||||||
|
symbols, linework, and anything absent from the text layer.
|
||||||
|
- Text-layer content is **appended programmatically** at each extractor call
|
||||||
|
site (classic `extractor.py::_extract_one`, agent
|
||||||
|
`extractors.py::SheetExtractorAgent.run`) — NOT a new `{placeholder}` in the
|
||||||
|
shared template (two-render-path trap: `render()` silently leaves missing
|
||||||
|
keys as literals). Block capped at `TEXT_LAYER_MAX_CHARS`.
|
||||||
|
Format: `\n\nTEXT LAYER (authoritative for alphanumeric content — trust it
|
||||||
|
over the image for numbers, tags, and note text):\n<text>`
|
||||||
|
|
||||||
|
### 3. Grounding guard rescue tier (`pipeline/extractor.py::_normalize_sheet`)
|
||||||
|
|
||||||
|
Current guard drops an object when its primary value's digit-runs aren't in
|
||||||
|
its own `source_text`. New tier, only when a text layer exists for the page:
|
||||||
|
|
||||||
|
- digits ⊆ source_text → keep (unchanged)
|
||||||
|
- digits ⊆ page text layer but ⊄ source_text → keep, stamp
|
||||||
|
`grounding: "text_layer"` on the assertion (recall rescue — vision quoted
|
||||||
|
imperfectly but the value is real page text)
|
||||||
|
- otherwise → drop (unchanged)
|
||||||
|
|
||||||
|
`_is_grounded` gains an optional `page_text` param; existing callers/tests
|
||||||
|
unaffected. Dropped/ rescued counts logged per page.
|
||||||
|
|
||||||
|
### 4. Verifier: text oracle + high-DPI crops (wave 5b)
|
||||||
|
|
||||||
|
Wherever verify scopes are built (agent runner confirmed; classic runner to be
|
||||||
|
checked — integrate at both if present):
|
||||||
|
|
||||||
|
- Scope payload gains `text_layer_excerpt`: concatenated text of the finding's
|
||||||
|
cited sheets, capped at `VERIFY_TEXT_MAX_CHARS`. `VERIFY_USER_INSTRUCTION`
|
||||||
|
gains a `{text_layer}` placeholder with instructions to treat it as
|
||||||
|
deterministic page text (verdicts may cite it as `actual_text`). **Both
|
||||||
|
render sites** (agent verifier + any classic-path render) must substitute it
|
||||||
|
— grep the template name across `backend/agents/` and `backend/pipeline/`.
|
||||||
|
- When `VERIFY_HI_DPI_CROPS` and the page has words: for each evidence item,
|
||||||
|
`find_evidence_bbox` on the cited page's words; on hit, `render_crop` at
|
||||||
|
`VERIFY_CROP_DPI` with margin → crop images replace full-page images (up to
|
||||||
|
`AGENT_CONFLICT_MAX_IMAGES`). On any miss/failure → fall back to the current
|
||||||
|
full-page image. Zero-resolved-images ⇒ scope skipped (I2 guard preserved).
|
||||||
|
|
||||||
|
### 5. Coverage signal (recall)
|
||||||
|
|
||||||
|
After extraction in both runners: for each page with `has_text_layer=True`
|
||||||
|
whose extraction failed or returned 0 objects, log
|
||||||
|
`[TextLayer] Page N: text layer present (M chars) but no objects extracted —
|
||||||
|
possible extraction gap` and add the page to the existing gap-finding path
|
||||||
|
(agent: `orchestrator.stats.failed_scopes`-style finding; classic: log only).
|
||||||
|
|
||||||
|
## Config knobs (`backend/config.py`, env-overridable, documented in `.env.example`)
|
||||||
|
|
||||||
|
| Key | Default | Effect |
|
||||||
|
|-----|---------|--------|
|
||||||
|
| `TEXT_LAYER_ENABLED` | `true` | Master switch |
|
||||||
|
| `TEXT_LAYER_MIN_CHARS` | `20` | Below this per page → `has_text_layer=False` |
|
||||||
|
| `TEXT_LAYER_MAX_CHARS` | `12000` | Cap per sheet injected into extractor prompt |
|
||||||
|
| `VERIFY_TEXT_MAX_CHARS` | `8000` | Cap of text-layer excerpt in verify scope |
|
||||||
|
| `VERIFY_HI_DPI_CROPS` | `true` | Evidence-located crops in verifier |
|
||||||
|
| `VERIFY_CROP_DPI` | `300` | Crop render DPI |
|
||||||
|
| `VERIFY_CROP_MARGIN_PTS` | `36` | Padding around evidence bbox (PDF points) |
|
||||||
|
|
||||||
|
## Known traps (from project history — designed around)
|
||||||
|
|
||||||
|
- **Two render paths:** no new `{placeholder}` in extractor templates; the one
|
||||||
|
new placeholder (`{text_layer}` in VERIFY_USER_INSTRUCTION) substituted at
|
||||||
|
every render site; a render test asserts no `{...}` literals remain.
|
||||||
|
- **ProjectMemory closed registry:** no new memory keys. Text artifacts dump
|
||||||
|
via plain file writes under `outputs/<job>/text/` (agent: under `agent/`).
|
||||||
|
- **`slim_clusters`:** no new cluster fields — unchanged.
|
||||||
|
- **I2 zero-image path:** crops replace full-page images only on confident
|
||||||
|
bbox match; never reduce image count to zero.
|
||||||
|
- **Base64 hygiene:** page dicts already carry base64; `text_layer` strings
|
||||||
|
must not leak into `clusters.json` dumps — reuse `_without_base64` pattern
|
||||||
|
if assertions ever carry page refs (they don't today).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
`PyMuPDF>=1.23` added to `requirements.txt` (Docker image rebuild picks it up;
|
||||||
|
pdf2image/poppler unchanged).
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- `tests/test_text_layer.py` — build tiny PDFs with PyMuPDF in-test:
|
||||||
|
extraction, `has_text_layer` thresholds, `find_evidence_bbox` hit/miss,
|
||||||
|
`render_crop` dimensions.
|
||||||
|
- Extractor guard: rescue-tier unit tests (keep-with-flag, still-drop,
|
||||||
|
unchanged behavior without text layer).
|
||||||
|
- Prompt render test: extractor + verify instructions fully substituted at
|
||||||
|
every site (both pipelines).
|
||||||
|
- Runner-level (pattern from `tests/agents/test_wave5b_suppression.py`):
|
||||||
|
stubbed waves, assert text layer reaches extract scopes and verify scopes
|
||||||
|
(excerpt present, crop fallback on no-match), full `run_agent_pipeline`.
|
||||||
|
- Full `pytest tests/` green before push.
|
||||||
|
|
||||||
|
## Validation (post-deploy)
|
||||||
|
|
||||||
|
Re-run the Cypress set (source PDF persists at
|
||||||
|
`/app/backend/outputs/959e16407573/source.pdf` on sits-docker) per the
|
||||||
|
documented re-run workflow. Success criteria:
|
||||||
|
|
||||||
|
1. The "(2) vs (5)"-class findings are not generated, or are verifier-refuted
|
||||||
|
with text-layer evidence cited.
|
||||||
|
2. Coverage-gap log lines appear for any page with text but no objects.
|
||||||
|
3. No new `finish_reason=length` in waves 1/4; cost delta reported vs
|
||||||
|
baseline job.
|
||||||
|
|
||||||
|
## Out of scope (future PRs)
|
||||||
|
|
||||||
|
- Legend/symbol-library wave injected into extractor + critic prompts.
|
||||||
|
- Deterministic schedule-row recall pass (text-layer tables → assertions).
|
||||||
|
- pdf-markup export for the review UI.
|
||||||
|
- Takeoff/polygon geometry (belongs to AI_Takeoffs, not this product).
|
||||||
@@ -2,6 +2,7 @@ fastapi==0.115.0
|
|||||||
uvicorn[standard]==0.30.6
|
uvicorn[standard]==0.30.6
|
||||||
python-multipart==0.0.12
|
python-multipart==0.0.12
|
||||||
pdf2image==1.17.0
|
pdf2image==1.17.0
|
||||||
|
PyMuPDF>=1.23.0 # deterministic text-layer extraction (extractor grounding, verifier crops)
|
||||||
Pillow==10.4.0
|
Pillow==10.4.0
|
||||||
openai==1.51.0
|
openai==1.51.0
|
||||||
httpx==0.27.2 # openai 1.51 passes proxies= to httpx; >=0.28 dropped it
|
httpx==0.27.2 # openai 1.51 passes proxies= to httpx; >=0.28 dropped it
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
"""Classic pipeline path must also satisfy the {disputes} placeholder added to
|
||||||
|
CONSTRUCTABILITY_USER_INSTRUCTION (agent path substitutes it in construct_agent.py;
|
||||||
|
the classic stage builds its own subs dict)."""
|
||||||
|
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from backend.pipeline._stage import render
|
||||||
|
from backend.pipeline.constructability import constructability_review
|
||||||
|
from backend.prompts import CONSTRUCTABILITY_USER_INSTRUCTION
|
||||||
|
|
||||||
|
|
||||||
|
def _cluster_with_dispute():
|
||||||
|
return {
|
||||||
|
"key": "c1",
|
||||||
|
"assertions": [],
|
||||||
|
"disputed_attributes": [{
|
||||||
|
"attribute": "stud_pack_size",
|
||||||
|
"values": ["(2) 2x6 STUD PACK", "(5) 2x6 STUD PACK"],
|
||||||
|
"assertion_ids": ["a1", "a2"],
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_classic_constructability_supplies_disputes_sub():
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def fake_call_stage(system_prompt, user_instruction, subs=None, **kwargs):
|
||||||
|
captured["subs"] = subs or {}
|
||||||
|
return {"issues": []}
|
||||||
|
|
||||||
|
with patch("backend.pipeline.constructability.call_stage", fake_call_stage):
|
||||||
|
constructability_review([], [_cluster_with_dispute()], [])
|
||||||
|
|
||||||
|
assert "disputes" in captured["subs"], "classic path must substitute {disputes}"
|
||||||
|
rendered = render(CONSTRUCTABILITY_USER_INSTRUCTION, captured["subs"])
|
||||||
|
assert "{disputes}" not in rendered
|
||||||
|
assert "(5) 2x6 STUD PACK" in rendered
|
||||||
|
|
||||||
|
|
||||||
|
def test_classic_constructability_disputes_defaults_empty():
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def fake_call_stage(system_prompt, user_instruction, subs=None, **kwargs):
|
||||||
|
captured["subs"] = subs or {}
|
||||||
|
return {"issues": []}
|
||||||
|
|
||||||
|
with patch("backend.pipeline.constructability.call_stage", fake_call_stage):
|
||||||
|
constructability_review([], [{"key": "c2", "assertions": []}], [])
|
||||||
|
|
||||||
|
rendered = render(CONSTRUCTABILITY_USER_INSTRUCTION, captured["subs"])
|
||||||
|
assert "{disputes}" not in rendered
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
from backend.agents.disputes import annotate_clusters, find_disputes
|
||||||
|
|
||||||
|
|
||||||
|
def _a(id_, attribute, value):
|
||||||
|
return {"id": id_, "attribute": attribute, "value": value,
|
||||||
|
"source_text": value}
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_disputes_flags_same_attribute_different_values():
|
||||||
|
assertions = [
|
||||||
|
_a("a1", "stud_pack_size", "(2) 2x6 STUD PACK"),
|
||||||
|
_a("a2", "stud_pack_size", "(5) 2x6 STUD PACK"),
|
||||||
|
_a("a3", "beam_size", "HSS16X4X5/8"),
|
||||||
|
]
|
||||||
|
disputes = find_disputes(assertions)
|
||||||
|
assert len(disputes) == 1
|
||||||
|
assert disputes[0]["attribute"] == "stud_pack_size"
|
||||||
|
assert disputes[0]["values"] == ["(2) 2x6 STUD PACK", "(5) 2x6 STUD PACK"]
|
||||||
|
assert disputes[0]["assertion_ids"] == ["a1", "a2"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_disputes_ignores_agreeing_values_and_blanks():
|
||||||
|
assertions = [
|
||||||
|
_a("a1", "beam_size", "HSS16X4X5/8"),
|
||||||
|
_a("a2", "beam_size", " hss16x4x5/8 "), # same after normalize
|
||||||
|
_a("a3", "", "orphan"), # no attribute -> skipped
|
||||||
|
_a("a4", "beam_size", ""), # no value -> skipped
|
||||||
|
]
|
||||||
|
assert find_disputes(assertions) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_annotate_clusters_writes_disputed_attributes():
|
||||||
|
clusters = [
|
||||||
|
{"key": "c1", "assertions": [
|
||||||
|
_a("a1", "stud_pack_size", "(2) 2x6"),
|
||||||
|
_a("a2", "stud_pack_size", "(5) 2x6"),
|
||||||
|
]},
|
||||||
|
{"key": "c2", "assertions": [_a("a3", "x", "1"), _a("a4", "x", "1")]},
|
||||||
|
]
|
||||||
|
assert annotate_clusters(clusters) == 1
|
||||||
|
assert clusters[0]["disputed_attributes"][0]["attribute"] == "stud_pack_size"
|
||||||
|
assert "disputed_attributes" not in clusters[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_slim_clusters_preserves_disputed_attributes():
|
||||||
|
from backend.pipeline._serialize import slim_clusters
|
||||||
|
cluster = {"key": "c1", "assertions": [],
|
||||||
|
"disputed_attributes": [{"attribute": "a", "values": ["1", "2"],
|
||||||
|
"assertion_ids": ["x", "y"]}]}
|
||||||
|
slim = slim_clusters([cluster])[0]
|
||||||
|
assert slim["disputed_attributes"][0]["values"] == ["1", "2"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_disputes_handles_none_and_zero_values():
|
||||||
|
# None value/attribute -> skipped; numeric 0 is a real value, not blank
|
||||||
|
assertions = [
|
||||||
|
{"id": "a1", "attribute": "count", "value": 0},
|
||||||
|
{"id": "a2", "attribute": "count", "value": 1},
|
||||||
|
{"id": "a3", "attribute": None, "value": "x"},
|
||||||
|
{"id": "a4", "attribute": "count", "value": None},
|
||||||
|
]
|
||||||
|
disputes = find_disputes(assertions)
|
||||||
|
assert len(disputes) == 1
|
||||||
|
assert disputes[0]["values"] == ["0", "1"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_disputes_empty_input():
|
||||||
|
assert find_disputes([]) == []
|
||||||
|
assert annotate_clusters([]) == 0
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
from backend import config
|
||||||
|
from backend.agents.base import AgentScope, AgentUsage
|
||||||
|
from backend.agents.extractors import SheetExtractorAgent
|
||||||
|
from backend.prompts import TEXT_STRUCTURING_SYSTEM_PROMPT, TEXT_STRUCTURING_USER_INSTRUCTION
|
||||||
|
|
||||||
|
def test_text_structuring_prompt_demands_verbatim_and_completeness():
|
||||||
|
assert "verbatim" in TEXT_STRUCTURING_USER_INSTRUCTION.lower()
|
||||||
|
assert "every" in TEXT_STRUCTURING_USER_INSTRUCTION.lower()
|
||||||
|
assert "{text_layer}" in TEXT_STRUCTURING_USER_INSTRUCTION
|
||||||
|
|
||||||
|
|
||||||
|
def _page(n=8, text="1. \nALL SAWN LUMBER IN CONTACT WITH SOIL TO BE SOUTHERN PINE, PRESSURE TREATED.\n2. \nROOF SHEATHING: 5/8\" PLYWOOD, C-D GRADE, STRUCTURAL I."):
|
||||||
|
return {"page_number": n, "base64": "AAAA", "text_layer": text}
|
||||||
|
|
||||||
|
|
||||||
|
def _run(agent, page, hint=""):
|
||||||
|
scope = AgentScope(scope_id=f"sheet:{page['page_number']}",
|
||||||
|
payload={"page": page, "sheet_hint": hint})
|
||||||
|
result = agent.run(scope)
|
||||||
|
assert not result.error, result.error
|
||||||
|
return result.artifacts[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_ladder_falls_back_when_vision_returns_nothing(monkeypatch):
|
||||||
|
# vision pass returns 1 summary object that the guard drops;
|
||||||
|
# text-structuring disabled to exercise the deterministic rung
|
||||||
|
monkeypatch.setattr("backend.agents.extractors.call_json",
|
||||||
|
lambda **kw: [{"name": "general notes", "value": "notes"}])
|
||||||
|
monkeypatch.setattr("backend.config.EXTRACT_TEXT_RETRY_ENABLED", False)
|
||||||
|
agent = SheetExtractorAgent(AgentUsage())
|
||||||
|
sheet = _run(agent, _page())
|
||||||
|
assert sheet["assertions"], "dark sheet must be impossible with fallback enabled"
|
||||||
|
assert all(a.get("grounding") == "text_layer_fallback" for a in sheet["assertions"])
|
||||||
|
assert sheet["coverage"]["ratio"] >= 0.6
|
||||||
|
|
||||||
|
|
||||||
|
def test_ladder_merge_preserves_graphical_objects(monkeypatch):
|
||||||
|
# vision finds a graphical symbol; text rung adds notes.
|
||||||
|
# The graphical object MUST survive the merge.
|
||||||
|
calls = {"n": 0}
|
||||||
|
def fake_call_json(**kw):
|
||||||
|
calls["n"] += 1
|
||||||
|
if kw.get("images_b64"): # vision pass
|
||||||
|
return {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "g1", "object_type": "lighting_fixture",
|
||||||
|
"name": "pendant at grid C-4", "source_text": None,
|
||||||
|
"graphical_basis": "16in pendant symbol at grid C-4"}]}
|
||||||
|
return {"sheet": {}, "objects": [ # text-structuring pass
|
||||||
|
{"object_id": "t1", "object_type": "general_note",
|
||||||
|
"source_text": "ALL SAWN LUMBER IN CONTACT WITH SOIL TO BE SOUTHERN PINE, PRESSURE TREATED.",
|
||||||
|
"name": "lumber note"}]}
|
||||||
|
monkeypatch.setattr("backend.agents.extractors.call_json", fake_call_json)
|
||||||
|
agent = SheetExtractorAgent(AgentUsage())
|
||||||
|
sheet = _run(agent, _page())
|
||||||
|
assert calls["n"] >= 2, "text-structuring rung should have fired"
|
||||||
|
assert any(a.get("graphical_basis") for a in sheet["assertions"])
|
||||||
|
assert any("SAWN LUMBER" in (a.get("source_text") or "") for a in sheet["assertions"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_ladder_recovers_sheet_number_from_text_layer(monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"backend.agents.extractors.call_json",
|
||||||
|
lambda **kw: {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "o1", "name": "RCP note",
|
||||||
|
"source_text": "GYP. BD. CEILING 8'-11 3/8\" A.F.F. TYP. FOR ALL STOREFRONT",
|
||||||
|
"attributes": {"height": "8'-11 3/8\""}}]})
|
||||||
|
agent = SheetExtractorAgent(AgentUsage())
|
||||||
|
sheet = _run(agent, _page(18, "REFLECTED CEILING PLAN\nGYP. BD. CEILING 8'-11 3/8\" A.F.F. TYP. FOR ALL STOREFRONT\nA102"))
|
||||||
|
assert sheet["sheet_number"] == "A102"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ladder_skips_retry_when_coverage_healthy(monkeypatch):
|
||||||
|
# vision covers every meaningful text-layer line -> no rung 2/3 calls
|
||||||
|
calls = {"n": 0}
|
||||||
|
def fake_call_json(**kw):
|
||||||
|
calls["n"] += 1
|
||||||
|
return {"sheet": {"sheet_number": "A101"}, "objects": [
|
||||||
|
{"object_id": "o1", "object_type": "general_note", "name": "lumber note",
|
||||||
|
"source_text": "ALL SAWN LUMBER IN CONTACT WITH SOIL TO BE SOUTHERN PINE, PRESSURE TREATED."},
|
||||||
|
{"object_id": "o2", "object_type": "general_note", "name": "sheathing note",
|
||||||
|
"source_text": "ROOF SHEATHING: 5/8\" PLYWOOD, C-D GRADE, STRUCTURAL I."}]}
|
||||||
|
monkeypatch.setattr("backend.agents.extractors.call_json", fake_call_json)
|
||||||
|
agent = SheetExtractorAgent(AgentUsage())
|
||||||
|
sheet = _run(agent, _page())
|
||||||
|
assert sheet["coverage"]["ratio"] >= config.EXTRACT_COVERAGE_FLOOR
|
||||||
|
assert calls["n"] == 1
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
from backend.agents.base import AgentScope
|
||||||
|
from backend.agents.linker import build_link_scopes
|
||||||
|
|
||||||
|
|
||||||
|
def _sheet(number, page, level, assertions):
|
||||||
|
return {"sheet_number": number, "page_number": page,
|
||||||
|
"discipline": "Structural", "level": level,
|
||||||
|
"assertions": assertions}
|
||||||
|
|
||||||
|
|
||||||
|
def _assertion(id_, ref=None, tag=None, level=None):
|
||||||
|
return {"id": id_, "attribute": "stud_pack_size", "value": "(5) 2x6",
|
||||||
|
"source_text": "(5) 2x6 STUD PACK",
|
||||||
|
"location_key": {"detail_reference": ref, "tag": tag,
|
||||||
|
"level": level}}
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_joins_same_detail_reference_across_levels():
|
||||||
|
sheets = [
|
||||||
|
_sheet("S101", 10, "foundation", [_assertion("a1", ref="A/S205")]),
|
||||||
|
_sheet("S205", 20, "roof", [_assertion("a2", ref="A/S205")]),
|
||||||
|
_sheet("S401", 30, "roof", [_assertion("a3", ref="A/S205")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
xref = [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
assert xref, "expected a cross-level detail-reference scope"
|
||||||
|
ids = {a["id"] for s in xref for a in s.payload["assertions"]}
|
||||||
|
assert ids == {"a1", "a2", "a3"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_requires_two_distinct_sheets():
|
||||||
|
sheets = [
|
||||||
|
_sheet("S401", 30, "roof", [_assertion("a1", ref="A/S205"),
|
||||||
|
_assertion("a2", ref="A/S205")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
assert not [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_joins_shared_member_tag():
|
||||||
|
sheets = [
|
||||||
|
_sheet("S102", 5, "roof", [_assertion("a1", tag="HSS16X4X5/8")]),
|
||||||
|
_sheet("S401", 30, "unknown", [_assertion("a2", tag="HSS16X4X5/8")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
xref = [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
assert xref
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_joins_single_letter_member_mark():
|
||||||
|
# W-shapes (W12X26) are the most common steel marks and have one leading letter
|
||||||
|
sheets = [
|
||||||
|
_sheet("S102", 5, "roof", [_assertion("a1", tag="W12X26")]),
|
||||||
|
_sheet("S401", 30, "unknown", [_assertion("a2", tag="W12X26")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
xref = [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
assert xref, "single-letter member marks (W12X26) must join xref scopes"
|
||||||
|
|
||||||
|
|
||||||
|
def test_xref_scope_rechecks_sheet_diversity_after_cap(monkeypatch):
|
||||||
|
from backend import config
|
||||||
|
monkeypatch.setattr(config, "AGENT_LINK_MAX_ASSERTIONS", 2)
|
||||||
|
sheets = [
|
||||||
|
_sheet("S401", 30, "roof", [_assertion("a1", ref="A/S205"),
|
||||||
|
_assertion("a2", ref="A/S205")]),
|
||||||
|
_sheet("S205", 20, "roof", [_assertion("a3", ref="A/S205")]),
|
||||||
|
]
|
||||||
|
scopes = build_link_scopes(sheets)
|
||||||
|
xref = [s for s in scopes if s.scope_id.startswith("xref:")]
|
||||||
|
for scope in xref:
|
||||||
|
sheets_in_scope = {a["sheet_number"] for a in scope.payload["assertions"]}
|
||||||
|
assert len(sheets_in_scope) >= 2, \
|
||||||
|
"capped xref scope must still span two sheets"
|
||||||
@@ -68,9 +68,12 @@ def test_run_compact_retry_after_hard_failure():
|
|||||||
assert "COMPACT RETRY" in mock_call.call_args_list[1].kwargs["user_text"]
|
assert "COMPACT RETRY" in mock_call.call_args_list[1].kwargs["user_text"]
|
||||||
|
|
||||||
|
|
||||||
def test_run_fails_only_after_both_attempts_miss():
|
def test_run_returns_empty_sheet_after_both_attempts_miss():
|
||||||
agent = SheetExtractorAgent(usage=AgentUsage())
|
agent = SheetExtractorAgent(usage=AgentUsage())
|
||||||
with patch("backend.agents.extractors.call_json", return_value=None) as mock_call:
|
with patch("backend.agents.extractors.call_json", return_value=None) as mock_call:
|
||||||
result = agent.run(_scope())
|
result = agent.run(_scope())
|
||||||
assert result.error == "no structured extraction returned"
|
# Coverage ladder: no text layer to rescue the page -> empty sheet,
|
||||||
|
# but no hard failure (the ladder replaced the old raise).
|
||||||
|
assert not result.error
|
||||||
|
assert result.artifacts[0]["assertions"] == []
|
||||||
assert mock_call.call_count == 2
|
assert mock_call.call_count == 2
|
||||||
|
|||||||
@@ -0,0 +1,128 @@
|
|||||||
|
"""Runner-level text-layer flow: excerpt into verify scopes, hi-DPI crop
|
||||||
|
replacement with full-page fallback, and coverage-gap findings."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
fitz = pytest.importorskip("pymupdf")
|
||||||
|
|
||||||
|
import backend.agents.runner as runner_mod
|
||||||
|
from backend.agents.base import AgentResult
|
||||||
|
from backend.agents.runner import run_agent_pipeline
|
||||||
|
|
||||||
|
PAGE_TEXT = "(5) 2X6 STUD PACK AT BEARING"
|
||||||
|
|
||||||
|
|
||||||
|
def _make_pdf(path):
|
||||||
|
doc = fitz.open()
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
page.insert_text((72, 72), PAGE_TEXT, fontsize=11)
|
||||||
|
doc.save(str(path))
|
||||||
|
doc.close()
|
||||||
|
return str(path)
|
||||||
|
|
||||||
|
|
||||||
|
def _finding(sheets, evidence_text):
|
||||||
|
return {
|
||||||
|
"issue_id": "C1", "severity": "critical", "confidence": "high",
|
||||||
|
"source_stage": "constructability", "sheets": sheets,
|
||||||
|
"description": "stud pack conflict",
|
||||||
|
"evidence": [{"sheet": sheets[0], "source_text": evidence_text}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _stub_agent(artifacts):
|
||||||
|
return lambda usage: type("S", (), {
|
||||||
|
"name": "stub",
|
||||||
|
"run": lambda self, scope: AgentResult(
|
||||||
|
scope_id=scope.scope_id, artifacts=list(artifacts)),
|
||||||
|
})()
|
||||||
|
|
||||||
|
|
||||||
|
def _patch_pipeline(monkeypatch, finding, verify_sink):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
runner_mod, "convert_pdf_to_images",
|
||||||
|
lambda path: [{"page_number": 1, "base64": "QUJD"}])
|
||||||
|
monkeypatch.setattr(runner_mod, "SheetExtractorAgent", _stub_agent([
|
||||||
|
{"sheet_number": "S401", "page_number": 1, "level": "roof",
|
||||||
|
"discipline": "S", "assertions": [
|
||||||
|
{"text": "(5) 2X6 STUD PACK", "object_type": "framing"},
|
||||||
|
{"text": "HSS16X4 beam", "object_type": "framing"},
|
||||||
|
]},
|
||||||
|
]))
|
||||||
|
monkeypatch.setattr(runner_mod, "SheetIndexAgent", _stub_agent([{}]))
|
||||||
|
monkeypatch.setattr(runner_mod, "JurisdictionAgent", _stub_agent([{}]))
|
||||||
|
monkeypatch.setattr(runner_mod, "LinkerAgent", _stub_agent([
|
||||||
|
{"key": "c1", "location": "roof beam pocket", "assertions": []},
|
||||||
|
]))
|
||||||
|
monkeypatch.setattr(runner_mod, "ConflictCriticAgent", _stub_agent([]))
|
||||||
|
monkeypatch.setattr(runner_mod, "CodeAgent", _stub_agent([]))
|
||||||
|
monkeypatch.setattr(runner_mod, "ConstructabilityAgent",
|
||||||
|
_stub_agent([finding]))
|
||||||
|
monkeypatch.setattr(runner_mod, "CompletenessAgent", _stub_agent([]))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
runner_mod, "BrainAgent",
|
||||||
|
lambda usage: type("B", (), {
|
||||||
|
"run": lambda self, findings, sheet_index, jurisdiction:
|
||||||
|
(list(findings), [])})())
|
||||||
|
|
||||||
|
class _RecordingVerifier:
|
||||||
|
name = "verify"
|
||||||
|
|
||||||
|
def __init__(self, usage):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def run(self, scope):
|
||||||
|
verify_sink.append(scope.payload)
|
||||||
|
return AgentResult(scope_id=scope.scope_id, artifacts=[{
|
||||||
|
"finding_index": scope.payload["finding_index"],
|
||||||
|
"status": "confirmed",
|
||||||
|
"verdicts": [],
|
||||||
|
}])
|
||||||
|
|
||||||
|
monkeypatch.setattr(runner_mod, "EvidenceVerifierAgent",
|
||||||
|
lambda usage: _RecordingVerifier(usage))
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_scope_carries_text_excerpt_and_crop(monkeypatch, tmp_path):
|
||||||
|
"""Evidence text matches the page text layer -> excerpt present and the
|
||||||
|
full-page image is replaced by a hi-DPI crop."""
|
||||||
|
sink = []
|
||||||
|
_patch_pipeline(monkeypatch,
|
||||||
|
_finding(["S401"], "(5) 2X6 STUD PACK AT BEARING"), sink)
|
||||||
|
pdf = _make_pdf(tmp_path / "set.pdf")
|
||||||
|
run_agent_pipeline(pdf, out_dir=str(tmp_path), require_review=False)
|
||||||
|
assert len(sink) == 1
|
||||||
|
payload = sink[0]
|
||||||
|
assert "2X6 STUD PACK" in payload["text_layer_excerpt"]
|
||||||
|
assert payload["images_b64"], "crop must never drop all images"
|
||||||
|
assert payload["images_b64"][0] != "QUJD", "expected crop, not full page"
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_scope_falls_back_to_full_page(monkeypatch, tmp_path):
|
||||||
|
"""Evidence text not in the text layer -> keep the full-page image."""
|
||||||
|
sink = []
|
||||||
|
_patch_pipeline(monkeypatch,
|
||||||
|
_finding(["S401"], "PENTHOUSE EXHAUST FAN EF-9"), sink)
|
||||||
|
pdf = _make_pdf(tmp_path / "set.pdf")
|
||||||
|
run_agent_pipeline(pdf, out_dir=str(tmp_path), require_review=False)
|
||||||
|
assert len(sink) == 1
|
||||||
|
assert sink[0]["images_b64"] == ["QUJD"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_coverage_gap_becomes_gap_finding(monkeypatch, tmp_path):
|
||||||
|
"""Text layer present but zero objects extracted -> failed-scope gap
|
||||||
|
finding survives into the report."""
|
||||||
|
sink = []
|
||||||
|
_patch_pipeline(monkeypatch, _finding(["S401"], PAGE_TEXT), sink)
|
||||||
|
# Extractor returns a sheet with NO objects despite a real text layer.
|
||||||
|
monkeypatch.setattr(runner_mod, "SheetExtractorAgent", _stub_agent([
|
||||||
|
{"sheet_number": "S401", "page_number": 1, "level": "roof",
|
||||||
|
"discipline": "S", "assertions": []},
|
||||||
|
]))
|
||||||
|
pdf = _make_pdf(tmp_path / "set.pdf")
|
||||||
|
report = run_agent_pipeline(pdf, out_dir=str(tmp_path),
|
||||||
|
require_review=False)
|
||||||
|
gaps = [f for f in (report.get("validated_issues") or [])
|
||||||
|
if f.get("category") == "analysis_gap"]
|
||||||
|
assert any("extraction gap" in (g.get("description") or "")
|
||||||
|
for g in gaps)
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from backend.agents.base import AgentScope, AgentUsage
|
||||||
|
|
||||||
|
from backend.agents.verifier import (
|
||||||
|
EvidenceVerifierAgent, apply_verdicts, select_findings,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _finding(sev="critical", issue_id="i1", sheets=("S401",), cluster_key=None):
|
||||||
|
f = {"issue_id": issue_id, "severity": sev, "confidence": "high",
|
||||||
|
"source_stage": "constructability", "sheets": list(sheets),
|
||||||
|
"description": "HSS16x4 on (2) 2x6 STUD PACK is unbuildable",
|
||||||
|
"evidence": [{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
|
||||||
|
"asserted_value": "3-inch width"}]}
|
||||||
|
if cluster_key:
|
||||||
|
f["cluster_key"] = cluster_key
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_findings_by_severity_and_dispute():
|
||||||
|
findings = [_finding("critical"), _finding("low", "i2"),
|
||||||
|
_finding("medium", "i3", cluster_key="c9")]
|
||||||
|
clusters = [{"key": "c9", "disputed_attributes": [{"attribute": "a"}]}]
|
||||||
|
selected = select_findings(findings, clusters, max_checks=20,
|
||||||
|
severities={"critical", "high"})
|
||||||
|
assert [f["issue_id"] for f in selected] == ["i1", "i3"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_findings_respects_cap():
|
||||||
|
findings = [_finding("critical", f"i{n}") for n in range(30)]
|
||||||
|
selected = select_findings(findings, [], max_checks=5,
|
||||||
|
severities={"critical"})
|
||||||
|
assert len(selected) == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_attaches_verdicts_and_marks_refuted():
|
||||||
|
agent = EvidenceVerifierAgent(usage=AgentUsage())
|
||||||
|
scope = AgentScope(scope_id="verify:0", payload={
|
||||||
|
"finding_index": 0,
|
||||||
|
"finding": _finding(),
|
||||||
|
"images_b64": ["QUJD"],
|
||||||
|
})
|
||||||
|
verdicts = {"verdicts": [
|
||||||
|
{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
|
||||||
|
"verdict": "corrected", "actual_text": "(5) 2x6 STUD PACK",
|
||||||
|
"notes": "callout reads (5)"},
|
||||||
|
]}
|
||||||
|
with patch("backend.agents.verifier.call_json", return_value=verdicts):
|
||||||
|
result = agent.run(scope)
|
||||||
|
assert not result.error
|
||||||
|
artifact = result.artifacts[0]
|
||||||
|
assert artifact["finding_index"] == 0
|
||||||
|
assert artifact["status"] == "refuted" # no evidence confirmed
|
||||||
|
assert artifact["verdicts"][0]["actual_text"] == "(5) 2x6 STUD PACK"
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_verdicts_annotates_and_suppresses():
|
||||||
|
from backend.agents.base import AgentResult
|
||||||
|
findings = [_finding("critical", "i1"), _finding("high", "i2")]
|
||||||
|
results = [AgentResult(scope_id="verify:0", artifacts=[
|
||||||
|
{"finding_index": 0, "status": "refuted", "verdicts": []},
|
||||||
|
{"finding_index": 1, "status": "confirmed", "verdicts": []},
|
||||||
|
])]
|
||||||
|
suppressed = apply_verdicts(findings, results)
|
||||||
|
assert suppressed == [findings[0]]
|
||||||
|
assert findings[0]["verification"]["status"] == "refuted"
|
||||||
|
assert findings[1]["verification"]["status"] == "confirmed"
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
"""Runner-level wave-5b tests: suppression path and zero-image guard."""
|
||||||
|
|
||||||
|
import backend.agents.runner as runner_mod
|
||||||
|
from backend.agents.base import AgentResult
|
||||||
|
from backend.agents.runner import run_agent_pipeline
|
||||||
|
|
||||||
|
|
||||||
|
def _finding(sheets):
|
||||||
|
return {
|
||||||
|
"issue_id": "C1", "severity": "critical", "confidence": "high",
|
||||||
|
"source_stage": "constructability", "sheets": sheets,
|
||||||
|
"description": "HSS16x4 on (2) 2x6 STUD PACK is unbuildable",
|
||||||
|
"evidence": [{"sheet": sheets[0], "source_text": "(2) 2x6 STUD PACK"}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _stub_agent(artifacts):
|
||||||
|
return lambda usage: type("S", (), {
|
||||||
|
"name": "stub",
|
||||||
|
"run": lambda self, scope: AgentResult(
|
||||||
|
scope_id=scope.scope_id, artifacts=list(artifacts)),
|
||||||
|
})()
|
||||||
|
|
||||||
|
|
||||||
|
def _patch_pipeline(monkeypatch, finding):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
runner_mod, "convert_pdf_to_images",
|
||||||
|
lambda path: [{"page_number": 1, "base64": "QUJD"}])
|
||||||
|
monkeypatch.setattr(runner_mod, "SheetExtractorAgent", _stub_agent([
|
||||||
|
{"sheet_number": "S401", "page_number": 1, "level": "roof",
|
||||||
|
"discipline": "S", "assertions": [
|
||||||
|
{"text": "(2) 2x6 STUD PACK", "object_type": "framing"},
|
||||||
|
{"text": "HSS16X4 beam", "object_type": "framing"},
|
||||||
|
]},
|
||||||
|
]))
|
||||||
|
monkeypatch.setattr(runner_mod, "SheetIndexAgent", _stub_agent([{}]))
|
||||||
|
monkeypatch.setattr(runner_mod, "JurisdictionAgent", _stub_agent([{}]))
|
||||||
|
monkeypatch.setattr(runner_mod, "LinkerAgent", _stub_agent([
|
||||||
|
{"key": "c1", "location": "roof beam pocket", "assertions": []},
|
||||||
|
]))
|
||||||
|
monkeypatch.setattr(runner_mod, "ConflictCriticAgent", _stub_agent([]))
|
||||||
|
monkeypatch.setattr(runner_mod, "CodeAgent", _stub_agent([]))
|
||||||
|
monkeypatch.setattr(runner_mod, "ConstructabilityAgent", _stub_agent([finding]))
|
||||||
|
monkeypatch.setattr(runner_mod, "CompletenessAgent", _stub_agent([]))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
runner_mod, "BrainAgent",
|
||||||
|
lambda usage: type("B", (), {
|
||||||
|
"run": lambda self, findings, sheet_index, jurisdiction:
|
||||||
|
(list(findings), [])})())
|
||||||
|
|
||||||
|
|
||||||
|
def test_refuted_finding_is_suppressed_not_crash(monkeypatch, tmp_path):
|
||||||
|
"""Regression: memory.replace("suppressed", ...) must not KeyError."""
|
||||||
|
_patch_pipeline(monkeypatch, _finding(["S401"]))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"backend.agents.verifier.call_json",
|
||||||
|
lambda **kwargs: {"verdicts": [
|
||||||
|
{"sheet": "S401", "source_text": "(2) 2x6 STUD PACK",
|
||||||
|
"verdict": "corrected", "actual_text": "(5) 2x6 STUD PACK",
|
||||||
|
"notes": "callout reads (5)"},
|
||||||
|
]})
|
||||||
|
pdf = tmp_path / "dummy.pdf"
|
||||||
|
pdf.write_bytes(b"%PDF-1.4\n")
|
||||||
|
report = run_agent_pipeline(str(pdf), out_dir=str(tmp_path),
|
||||||
|
require_review=False)
|
||||||
|
assert [f["issue_id"] for f in report["suppressed_issues"]] == ["C1"]
|
||||||
|
assert report["suppressed_issues"][0]["verification"]["status"] == "refuted"
|
||||||
|
|
||||||
|
|
||||||
|
def test_zero_image_finding_is_not_suppressed(monkeypatch, tmp_path):
|
||||||
|
"""A finding whose sheets resolve to no page images must not be judged
|
||||||
|
(and must never be refuted) without pixels."""
|
||||||
|
_patch_pipeline(monkeypatch, _finding(["S999"])) # no such sheet
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"backend.agents.verifier.call_json",
|
||||||
|
lambda **kwargs: {"verdicts": [
|
||||||
|
{"sheet": "S999", "source_text": "(2) 2x6 STUD PACK",
|
||||||
|
"verdict": "not_found", "actual_text": None, "notes": None},
|
||||||
|
]})
|
||||||
|
pdf = tmp_path / "dummy.pdf"
|
||||||
|
pdf.write_bytes(b"%PDF-1.4\n")
|
||||||
|
report = run_agent_pipeline(str(pdf), out_dir=str(tmp_path),
|
||||||
|
require_review=False)
|
||||||
|
assert report["suppressed_issues"] == []
|
||||||
|
validated = report.get("validated_issues") or []
|
||||||
|
assert any(f.get("issue_id") == "C1" for f in validated)
|
||||||
@@ -85,6 +85,29 @@ def test_finalize_confirm_keeps_confirmed(monkeypatch, tmp_path):
|
|||||||
assert report["summary"]["agent_status"] == "complete"
|
assert report["summary"]["agent_status"] == "complete"
|
||||||
|
|
||||||
|
|
||||||
|
def test_finalize_preserves_verifier_suppressed(monkeypatch, tmp_path):
|
||||||
|
"""Wave-5b (verifier) suppressions must survive review finalization and
|
||||||
|
merge with review-rejected suppressions."""
|
||||||
|
monkeypatch.setattr("backend.review.finalizer._draft_rfis", lambda kept: [])
|
||||||
|
_write_job(
|
||||||
|
str(tmp_path),
|
||||||
|
prioritized=[{"issue_id": "AGENT-0001", "severity": "high"}],
|
||||||
|
queue=[_blocking_item("AGENT-0001")],
|
||||||
|
decisions=[{"review_item_id": "finding:AGENT-0001",
|
||||||
|
"decision": "reject", "reason_code": "not_a_contradiction"}],
|
||||||
|
)
|
||||||
|
path = os.path.join(str(tmp_path), "conflicts.json")
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
report = json.load(f)
|
||||||
|
report["suppressed_issues"] = [
|
||||||
|
{"issue_id": "C1", "verification": {"status": "refuted"}}]
|
||||||
|
with open(path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(report, f)
|
||||||
|
final = finalize_review("job1", str(tmp_path))
|
||||||
|
ids = [f["issue_id"] for f in final["suppressed_issues"]]
|
||||||
|
assert ids == ["C1", "AGENT-0001"]
|
||||||
|
|
||||||
|
|
||||||
def test_finalize_no_decision_keeps_unreviewed(monkeypatch, tmp_path):
|
def test_finalize_no_decision_keeps_unreviewed(monkeypatch, tmp_path):
|
||||||
"""Non-blocking (audit) items don't need a decision; issue stays unreviewed."""
|
"""Non-blocking (audit) items don't need a decision; issue stays unreviewed."""
|
||||||
monkeypatch.setattr("backend.review.finalizer._draft_rfis", lambda kept: [])
|
monkeypatch.setattr("backend.review.finalizer._draft_rfis", lambda kept: [])
|
||||||
|
|||||||
@@ -0,0 +1,84 @@
|
|||||||
|
"""Unit tests for the extraction_coverage block in the report summary.
|
||||||
|
|
||||||
|
Both the classic pipeline and the agent runner build their summary via
|
||||||
|
backend.pipeline.report.build_report, so unit tests on that function cover
|
||||||
|
every summary-producing path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from backend.pipeline.report import build_report
|
||||||
|
|
||||||
|
|
||||||
|
def _sheet(page, coverage=None, assertions=None):
|
||||||
|
sheet = {
|
||||||
|
"page_number": page,
|
||||||
|
"sheet_number": f"S{page:03d}",
|
||||||
|
"discipline": "S",
|
||||||
|
"assertions": assertions if assertions is not None else [
|
||||||
|
{"text": "NOTE ALPHA", "object_type": "note"},
|
||||||
|
{"text": "NOTE BETA", "object_type": "note"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
if coverage is not None:
|
||||||
|
sheet["coverage"] = coverage
|
||||||
|
return sheet
|
||||||
|
|
||||||
|
|
||||||
|
def _cov(total, covered):
|
||||||
|
return {
|
||||||
|
"total_lines": total,
|
||||||
|
"covered_lines": covered,
|
||||||
|
"ratio": covered / total if total else 0.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_extraction_coverage_omitted_without_coverage_data():
|
||||||
|
report = build_report(conflicts=[], sheets=[_sheet(1), _sheet(2)], clusters=[])
|
||||||
|
assert "extraction_coverage" not in report["summary"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_extraction_coverage_healthy():
|
||||||
|
sheets = [_sheet(1, _cov(100, 95)), _sheet(2, _cov(80, 76))]
|
||||||
|
report = build_report(conflicts=[], sheets=sheets, clusters=[])
|
||||||
|
cov = report["summary"]["extraction_coverage"]
|
||||||
|
assert cov["pages_measured"] == 2
|
||||||
|
assert cov["pages_below_floor"] == []
|
||||||
|
assert cov["fallback_pages"] == []
|
||||||
|
assert cov["mean_ratio"] == round((0.95 + 0.95) / 2, 3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_extraction_coverage_flags_below_floor():
|
||||||
|
sheets = [
|
||||||
|
_sheet(1, _cov(100, 95)),
|
||||||
|
_sheet(2, _cov(100, 40)), # ratio 0.4 < 0.6 floor
|
||||||
|
_sheet(3, _cov(100, 59)), # ratio 0.59 < 0.6 floor
|
||||||
|
]
|
||||||
|
report = build_report(conflicts=[], sheets=sheets, clusters=[])
|
||||||
|
cov = report["summary"]["extraction_coverage"]
|
||||||
|
assert cov["pages_measured"] == 3
|
||||||
|
assert cov["pages_below_floor"] == [2, 3]
|
||||||
|
assert cov["mean_ratio"] == round((0.95 + 0.4 + 0.59) / 3, 3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_extraction_coverage_flags_fallback_pages():
|
||||||
|
fallback_assertions = [
|
||||||
|
{"text": "NOTE ALPHA", "object_type": "note"},
|
||||||
|
{"text": "NOTE BETA", "object_type": "note",
|
||||||
|
"grounding": "text_layer_fallback"},
|
||||||
|
]
|
||||||
|
sheets = [
|
||||||
|
_sheet(1, _cov(100, 90), assertions=fallback_assertions),
|
||||||
|
_sheet(2, _cov(100, 90)),
|
||||||
|
]
|
||||||
|
report = build_report(conflicts=[], sheets=sheets, clusters=[])
|
||||||
|
cov = report["summary"]["extraction_coverage"]
|
||||||
|
assert cov["fallback_pages"] == [1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_extraction_coverage_mixed_sheets_only_counts_measured():
|
||||||
|
# Sheet 2 has no coverage dict (e.g. scanned page / older path).
|
||||||
|
sheets = [_sheet(1, _cov(100, 50)), _sheet(2)]
|
||||||
|
report = build_report(conflicts=[], sheets=sheets, clusters=[])
|
||||||
|
cov = report["summary"]["extraction_coverage"]
|
||||||
|
assert cov["pages_measured"] == 1
|
||||||
|
assert cov["pages_below_floor"] == [1]
|
||||||
|
assert cov["mean_ratio"] == 0.5
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
"""Coverage-driven extraction retry ladder — classic path (pipeline/extractor.py)."""
|
||||||
|
|
||||||
|
from backend import config
|
||||||
|
from backend.pipeline import extractor
|
||||||
|
|
||||||
|
PAGE_TEXT = ("1. \nALL SAWN LUMBER IN CONTACT WITH SOIL TO BE SOUTHERN PINE, "
|
||||||
|
"PRESSURE TREATED.\n2. \nROOF SHEATHING: 5/8\" PLYWOOD, C-D GRADE, "
|
||||||
|
"STRUCTURAL I.")
|
||||||
|
|
||||||
|
|
||||||
|
def _page(n=8, text=PAGE_TEXT):
|
||||||
|
return {"page_number": n, "base64": "AAAA", "text_layer": text}
|
||||||
|
|
||||||
|
|
||||||
|
def test_classic_fallback_when_vision_returns_nothing(monkeypatch):
|
||||||
|
"""Vision pass returns an unusable bare list; text retry disabled ->
|
||||||
|
deterministic fallback stubs make a dark sheet impossible."""
|
||||||
|
monkeypatch.setattr(extractor, "call_json",
|
||||||
|
lambda **kw: [{"name": "general notes", "value": "notes"}])
|
||||||
|
monkeypatch.setattr(config, "EXTRACT_TEXT_RETRY_ENABLED", False)
|
||||||
|
sheet = extractor._extract_one(_page())
|
||||||
|
assert sheet["assertions"], "dark sheet must be impossible with fallback enabled"
|
||||||
|
assert all(a.get("grounding") == "text_layer_fallback"
|
||||||
|
for a in sheet["assertions"])
|
||||||
|
assert sheet["coverage"]["ratio"] >= 0.6
|
||||||
|
|
||||||
|
|
||||||
|
def test_classic_merge_preserves_graphical_objects(monkeypatch):
|
||||||
|
"""Rung-2 merge must never drop vision-only graphical objects."""
|
||||||
|
def fake(**kw):
|
||||||
|
if kw.get("images_b64"):
|
||||||
|
return {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "g1", "object_type": "lighting_fixture",
|
||||||
|
"name": "pendant at grid C-4", "source_text": None,
|
||||||
|
"graphical_basis": "16in pendant symbol at grid C-4"}]}
|
||||||
|
return {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "t1", "object_type": "general_note",
|
||||||
|
"source_text": "ALL SAWN LUMBER IN CONTACT WITH SOIL TO BE "
|
||||||
|
"SOUTHERN PINE, PRESSURE TREATED.",
|
||||||
|
"name": "lumber note"}]}
|
||||||
|
|
||||||
|
monkeypatch.setattr(extractor, "call_json", fake)
|
||||||
|
sheet = extractor._extract_one(_page())
|
||||||
|
assert any(a.get("graphical_basis") for a in sheet["assertions"])
|
||||||
|
assert any("SAWN LUMBER" in (a.get("source_text") or "")
|
||||||
|
for a in sheet["assertions"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_classic_recovers_sheet_number(monkeypatch):
|
||||||
|
text = ("REFLECTED CEILING PLAN\n"
|
||||||
|
"GYP. BD. CEILING 8'-11 3/8\" A.F.F. TYP. FOR ALL STOREFRONT\n"
|
||||||
|
"LED TAPE LIGHT. SEE ELEC. SCONCE 8'-0\" A.F.F., SEE ELEC.\n"
|
||||||
|
"A102")
|
||||||
|
|
||||||
|
def fake(**kw):
|
||||||
|
if kw.get("images_b64"):
|
||||||
|
return {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "o1", "name": "RCP ceiling note",
|
||||||
|
"source_text": "GYP. BD. CEILING 8'-11 3/8\" A.F.F. TYP. "
|
||||||
|
"FOR ALL STOREFRONT",
|
||||||
|
"attributes": {"height": "8'-11 3/8\""}}]}
|
||||||
|
return {"sheet": {}, "objects": []}
|
||||||
|
|
||||||
|
monkeypatch.setattr(extractor, "call_json", fake)
|
||||||
|
sheet = extractor._extract_one(_page(18, text))
|
||||||
|
assert sheet["sheet_number"] == "A102"
|
||||||
|
|
||||||
|
|
||||||
|
def test_classic_skips_retry_when_coverage_healthy(monkeypatch):
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake(**kw):
|
||||||
|
calls.append(kw)
|
||||||
|
return {"sheet": {"sheet_number": "S202"}, "objects": [
|
||||||
|
{"object_id": "o1", "name": "lumber note",
|
||||||
|
"source_text": "ALL SAWN LUMBER IN CONTACT WITH SOIL TO BE "
|
||||||
|
"SOUTHERN PINE, PRESSURE TREATED.",
|
||||||
|
"attributes": {"species": "southern pine"}},
|
||||||
|
{"object_id": "o2", "name": "sheathing note",
|
||||||
|
"source_text": "ROOF SHEATHING: 5/8\" PLYWOOD, C-D GRADE, "
|
||||||
|
"STRUCTURAL I.",
|
||||||
|
"attributes": {"sheathing": "5/8 plywood"}}]}
|
||||||
|
|
||||||
|
monkeypatch.setattr(extractor, "call_json", fake)
|
||||||
|
sheet = extractor._extract_one(_page())
|
||||||
|
assert len(calls) == 1, "healthy coverage must not trigger the text-only rung"
|
||||||
|
assert sheet["coverage"]["ratio"] == 1.0
|
||||||
|
assert sheet["sheet_number"] == "S202"
|
||||||
|
|
||||||
|
|
||||||
|
def test_classic_scanned_page_keeps_failed_sheet_shape(monkeypatch):
|
||||||
|
"""No text layer (scanned page): total parse failure keeps the existing
|
||||||
|
'extraction failed' empty-sheet return — ladder is text-layer-only."""
|
||||||
|
monkeypatch.setattr(extractor, "call_json", lambda **kw: None)
|
||||||
|
sheet = extractor._extract_one({"page_number": 4, "base64": "AAAA",
|
||||||
|
"text_layer": None})
|
||||||
|
assert sheet["assertions"] == []
|
||||||
|
assert "extraction failed" in (sheet.get("sheet_title") or "")
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
"""Grounding-guard rescue tier, text-layer prompt block, and render hygiene."""
|
||||||
|
|
||||||
|
from backend import config
|
||||||
|
from backend.pipeline._stage import render
|
||||||
|
from backend.pipeline.extractor import (
|
||||||
|
_is_grounded,
|
||||||
|
_normalize_sheet,
|
||||||
|
_text_layer_block,
|
||||||
|
)
|
||||||
|
from backend.prompts import EXTRACTOR_USER_INSTRUCTION, VERIFY_USER_INSTRUCTION
|
||||||
|
|
||||||
|
PAGE_TEXT = "NOTES: (5) 2X6 STUD PACK AT BEARING. HSS16X4 BEAM. 7'-0\" AFF."
|
||||||
|
|
||||||
|
|
||||||
|
def _parsed(value, source_text):
|
||||||
|
return {
|
||||||
|
"sheet": {"sheet_number": "S401"},
|
||||||
|
"objects": [{
|
||||||
|
"object_id": "o1",
|
||||||
|
"object_type": "framing",
|
||||||
|
"name": "stud pack",
|
||||||
|
"attributes": {"count": value},
|
||||||
|
"source_text": source_text,
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_rescue_tier_keeps_and_stamps():
|
||||||
|
"""Digits absent from source_text but present in the page text layer:
|
||||||
|
kept, stamped grounding=text_layer (vision quoted imperfectly)."""
|
||||||
|
sheet = _normalize_sheet(_parsed("(2)", "(2) 2x6 STUD PACK"), 1,
|
||||||
|
page_text=PAGE_TEXT)
|
||||||
|
# "(2)" is not grounded by its own source_text alone? it is - use a value
|
||||||
|
# whose digits differ from the quote to exercise the rescue path.
|
||||||
|
sheet = _normalize_sheet(_parsed("5", "(2) 2x6 STUD PACK"), 1,
|
||||||
|
page_text=PAGE_TEXT)
|
||||||
|
assert len(sheet["assertions"]) == 1
|
||||||
|
assert sheet["assertions"][0]["grounding"] == "text_layer"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_rescue_without_page_text():
|
||||||
|
sheet = _normalize_sheet(_parsed("5", "(2) 2x6 STUD PACK"), 1)
|
||||||
|
assert sheet["assertions"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_still_dropped_when_digits_nowhere():
|
||||||
|
sheet = _normalize_sheet(_parsed("99", "(2) 2x6 STUD PACK"), 1,
|
||||||
|
page_text=PAGE_TEXT)
|
||||||
|
assert sheet["assertions"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_grounded_backward_compatible():
|
||||||
|
assert _is_grounded("(5)", "(5) 2x6 STUD PACK") is True
|
||||||
|
# Digit-run guard is a set check: "(3)" has no support anywhere.
|
||||||
|
assert _is_grounded("(3)", "(5) 2x6 STUD PACK") is False
|
||||||
|
assert _is_grounded("(3)", "(5) 2x6 STUD PACK",
|
||||||
|
page_text="(3) 2x6 STUD PACK") is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_text_layer_block_empty_without_layer():
|
||||||
|
assert _text_layer_block({"page_number": 1}) == ""
|
||||||
|
assert _text_layer_block({"page_number": 1, "text_layer": None}) == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_text_layer_block_appends_and_caps(monkeypatch):
|
||||||
|
block = _text_layer_block({"page_number": 1, "text_layer": PAGE_TEXT})
|
||||||
|
assert "TEXT LAYER" in block and "STUD PACK" in block
|
||||||
|
monkeypatch.setattr(config, "TEXT_LAYER_MAX_CHARS", 50)
|
||||||
|
block = _text_layer_block({"page_number": 1, "text_layer": "x" * 500})
|
||||||
|
assert len(block.split(":\n", 1)[1]) == 50
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_instruction_fully_rendered():
|
||||||
|
"""render() silently leaves missing keys as literals - both placeholders
|
||||||
|
must be substituted at the (single) verify render site."""
|
||||||
|
out = render(VERIFY_USER_INSTRUCTION,
|
||||||
|
{"finding": "FINDING_JSON", "text_layer": "PAGE_TEXT"})
|
||||||
|
assert "{finding}" not in out and "{text_layer}" not in out
|
||||||
|
assert "FINDING_JSON" in out and "PAGE_TEXT" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_extractor_instruction_fully_substituted():
|
||||||
|
page = {"page_number": 1, "text_layer": PAGE_TEXT}
|
||||||
|
out = (EXTRACTOR_USER_INSTRUCTION.replace("{sheet_hint}", "")
|
||||||
|
+ _text_layer_block(page))
|
||||||
|
assert "{sheet_hint}" not in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_vision_unverified_stamp_when_source_text_not_in_text_layer():
|
||||||
|
"""Digits ground the object against the page text, but its quoted
|
||||||
|
source_text is not actually present in the text layer: kept, stamped
|
||||||
|
vision_unverified (the wave-5b verifier consumes grounding stamps)."""
|
||||||
|
page_text = "WALL: 2X6 WD STUD @ 16\" O.C. WITH R-13 BATT INSULATION"
|
||||||
|
parsed = {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "x1", "name": "stud pack",
|
||||||
|
"source_text": "(5) 2X6 STUD PACK AT JAMB", # NOT in page text
|
||||||
|
"attributes": {"count": "5"}}]}
|
||||||
|
sheet = _normalize_sheet(parsed, 1, page_text=page_text)
|
||||||
|
assert len(sheet["assertions"]) == 1
|
||||||
|
assert sheet["assertions"][0]["grounding"] == "vision_unverified"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_unverified_stamp_when_source_text_in_text_layer():
|
||||||
|
page_text = "WALL: 2X6 WD STUD @ 16\" O.C. WITH R-13 BATT INSULATION"
|
||||||
|
parsed = {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "x1", "name": "stud note",
|
||||||
|
"source_text": "2X6 WD STUD @ 16\" O.C.",
|
||||||
|
"attributes": {"size": "2x6"}}]}
|
||||||
|
sheet = _normalize_sheet(parsed, 1, page_text=page_text)
|
||||||
|
assert len(sheet["assertions"]) == 1
|
||||||
|
assert "grounding" not in sheet["assertions"][0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_preset_grounding_stamp_survives_normalization():
|
||||||
|
"""Fallback/merge rungs stamp grounding upstream; normalization must
|
||||||
|
preserve a pre-set stamp instead of recomputing it away."""
|
||||||
|
parsed = {"sheet": {}, "objects": [
|
||||||
|
{"object_id": "f1", "name": "lumber note",
|
||||||
|
"source_text": "ALL LUMBER SOUTHERN PINE",
|
||||||
|
"grounding": "text_layer_fallback"}]}
|
||||||
|
sheet = _normalize_sheet(parsed, 1, page_text="ALL LUMBER SOUTHERN PINE")
|
||||||
|
assert sheet["assertions"][0]["grounding"] == "text_layer_fallback"
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
"""Deterministic sheet-list reconciliation: cover index vs extracted sheets."""
|
||||||
|
|
||||||
|
from backend.sheet_reconcile import declared_sheet_list, reconcile_sheets
|
||||||
|
|
||||||
|
COVER_TEXT = """VERIZON CYPRESS
|
||||||
|
SHEET LIST
|
||||||
|
SHEET NUMBER
|
||||||
|
SHEET NAME
|
||||||
|
G000
|
||||||
|
COVER
|
||||||
|
G001
|
||||||
|
GENERAL INFO
|
||||||
|
C-001
|
||||||
|
CIVIL COVER
|
||||||
|
C-001.1
|
||||||
|
ALTA SURVEY
|
||||||
|
L-101
|
||||||
|
LANDSCAPE PLAN
|
||||||
|
S101
|
||||||
|
FOUNDATION PLAN
|
||||||
|
S301
|
||||||
|
WALL SECTIONS
|
||||||
|
S401
|
||||||
|
PERSPECTIVE VIEW
|
||||||
|
A101
|
||||||
|
FLOOR PLAN
|
||||||
|
A102
|
||||||
|
REFLECTED CEILING PLAN
|
||||||
|
E400
|
||||||
|
ELECTRICAL SITE PLAN
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def test_declared_sheet_list_from_cover():
|
||||||
|
declared = declared_sheet_list({1: COVER_TEXT, 2: "symbols legend"})
|
||||||
|
assert declared[0] == "G000"
|
||||||
|
assert "C-001" in declared and "C-001.1" in declared # hyphenated ids kept
|
||||||
|
assert "L-101" in declared
|
||||||
|
assert "A102" in declared
|
||||||
|
assert declared.count("G000") == 1
|
||||||
|
assert len(declared) == 11
|
||||||
|
|
||||||
|
|
||||||
|
def test_declared_sheet_list_uses_first_index_page_only():
|
||||||
|
texts = {1: "no index here", 2: COVER_TEXT, 3: "SHEET LIST\nXX999\nBOGUS"}
|
||||||
|
declared = declared_sheet_list(texts)
|
||||||
|
assert "XX999" not in declared # only the first marker page is parsed
|
||||||
|
|
||||||
|
|
||||||
|
def test_declared_sheet_list_none_when_no_marker():
|
||||||
|
assert declared_sheet_list({1: "just notes", 2: "floor plan stuff"}) == []
|
||||||
|
|
||||||
|
|
||||||
|
def _sheets(*nums):
|
||||||
|
return [{"page_number": i + 1, "sheet_number": n}
|
||||||
|
for i, n in enumerate(nums)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_both_directions():
|
||||||
|
declared = declared_sheet_list({1: COVER_TEXT})
|
||||||
|
rec = reconcile_sheets(_sheets("G000", "G001", "S101", "S301", "S302", "A101"),
|
||||||
|
declared)
|
||||||
|
# declared but not extracted (civil/landscape not in this PDF + missing)
|
||||||
|
assert "C-001" in rec["declared_not_in_set"]
|
||||||
|
assert "A102" in rec["declared_not_in_set"]
|
||||||
|
assert "E400" in rec["declared_not_in_set"]
|
||||||
|
# extracted but not on the cover index (misread or unlisted sheet)
|
||||||
|
assert rec["in_set_not_declared"] == ["S302"]
|
||||||
|
assert rec["declared_total"] == 11
|
||||||
|
assert rec["found_total"] == 6
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_normalizes_hyphens():
|
||||||
|
declared = ["C-001", "S301"]
|
||||||
|
rec = reconcile_sheets(_sheets("C001", "S301"), declared)
|
||||||
|
assert rec["declared_not_in_set"] == []
|
||||||
|
assert rec["in_set_not_declared"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_ignores_unidentified_sheets():
|
||||||
|
rec = reconcile_sheets(
|
||||||
|
[{"page_number": 8, "sheet_number": None},
|
||||||
|
{"page_number": 9, "sheet_number": "S301"}],
|
||||||
|
["S301", "A102"])
|
||||||
|
assert rec["found_total"] == 1
|
||||||
|
assert rec["declared_not_in_set"] == ["A102"]
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
from backend.text_coverage import (text_coverage, segment_text_layer,
|
||||||
|
fallback_objects, merge_objects,
|
||||||
|
recover_sheet_number)
|
||||||
|
|
||||||
|
|
||||||
|
def test_coverage_full():
|
||||||
|
text = "NOTE 1\nALL LUMBER NO. 2 SOUTHERN PINE\nNOTE 2\nUSE 5/8\" PLYWOOD"
|
||||||
|
objects = [{"source_text": "ALL LUMBER NO. 2 SOUTHERN PINE"},
|
||||||
|
{"source_text": "USE 5/8\" PLYWOOD"}]
|
||||||
|
cov = text_coverage(text, objects)
|
||||||
|
assert cov["covered_lines"] == 2
|
||||||
|
assert cov["total_lines"] == 2
|
||||||
|
assert cov["ratio"] == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_coverage_zero_on_empty_objects():
|
||||||
|
cov = text_coverage("LINE ALPHA CONTENT\nLINE BETA CONTENT\nLINE GAMMA CONTENT", [])
|
||||||
|
assert cov["ratio"] == 0.0 and cov["total_lines"] == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_coverage_ignores_short_and_numeric_noise_lines():
|
||||||
|
text = "15\"\n19\"\nA\nB\nREAL NOTE ABOUT FRAMING HERE"
|
||||||
|
cov = text_coverage(text, [{"source_text": "REAL NOTE ABOUT FRAMING HERE"}])
|
||||||
|
assert cov["total_lines"] == 1 and cov["ratio"] == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_segment_notes_and_rows():
|
||||||
|
text = "WOOD CONSTRUCTION\n1. \nALL SAWN LUMBER TO BE SOUTHERN PINE.\n2. \nROOF SHEATHING 5/8\" PLYWOOD."
|
||||||
|
segs = segment_text_layer(text)
|
||||||
|
assert any("ALL SAWN LUMBER" in s for s in segs)
|
||||||
|
assert any("ROOF SHEATHING" in s for s in segs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fallback_objects_verbatim_and_stamped():
|
||||||
|
objs = fallback_objects("1. \nALL SAWN LUMBER TO BE SOUTHERN PINE.", page_number=8)
|
||||||
|
assert len(objs) == 1
|
||||||
|
assert objs[0]["source_text"] == "1 ALL SAWN LUMBER TO BE SOUTHERN PINE."
|
||||||
|
assert objs[0]["grounding"] == "text_layer_fallback"
|
||||||
|
assert objs[0]["confidence"] == "low"
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_objects_keeps_vision_and_unions_text():
|
||||||
|
vision = [
|
||||||
|
{"source_text": "2X6 WD STUD @ 16\" O.C.", "object_type": "wall"},
|
||||||
|
{"source_text": None, "graphical_basis": "light fixture symbol, grid C-4",
|
||||||
|
"object_type": "lighting_fixture"},
|
||||||
|
]
|
||||||
|
text = [
|
||||||
|
{"source_text": "2X6 WD STUD @ 16\" O.C.", "object_type": "wall"},
|
||||||
|
{"source_text": "ALL LUMBER NO. 2 SOUTHERN PINE", "object_type": "general_note"},
|
||||||
|
]
|
||||||
|
merged = merge_objects(vision, text)
|
||||||
|
assert len(merged) == 3
|
||||||
|
assert any(o.get("graphical_basis") for o in merged)
|
||||||
|
assert merged[0]["object_type"] == "wall"
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_objects_dedupes_by_normalized_text():
|
||||||
|
a = [{"source_text": "RTU-1: 5 TON, 1600 CFM"}]
|
||||||
|
b = [{"source_text": "rtu 1 5 ton 1600 cfm"}]
|
||||||
|
assert len(merge_objects(a, b)) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_recover_sheet_number_from_title_block():
|
||||||
|
text = ("WALL SECTIONS\n...\nSheet Information\nS301\n"
|
||||||
|
"Issue Date 05.29.26\nProject Number 25177")
|
||||||
|
assert recover_sheet_number(text) == "S301"
|
||||||
|
|
||||||
|
|
||||||
|
def test_recover_sheet_number_none_when_absent():
|
||||||
|
assert recover_sheet_number("just some notes about lumber") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_recover_prefers_discipline_pattern_over_dates():
|
||||||
|
text = "Issue Date 05.29.26\nProject Number 25177\nA102 REFLECTED CEILING PLAN"
|
||||||
|
assert recover_sheet_number(text) == "A102"
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
"""Text-layer extraction, evidence bbox matching, and crop rendering."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
fitz = pytest.importorskip("pymupdf")
|
||||||
|
|
||||||
|
from backend import config
|
||||||
|
from backend.text_layer import (
|
||||||
|
attach_text_layers,
|
||||||
|
coverage_gaps,
|
||||||
|
extract_text_layers,
|
||||||
|
find_evidence_bbox,
|
||||||
|
render_crop,
|
||||||
|
)
|
||||||
|
|
||||||
|
EVIDENCE = "(5) 2X6 STUD PACK @ 16 IN O.C."
|
||||||
|
|
||||||
|
|
||||||
|
def _make_pdf(path, pages):
|
||||||
|
"""pages: list of str ('' = effectively blank page)."""
|
||||||
|
doc = fitz.open()
|
||||||
|
for text in pages:
|
||||||
|
page = doc.new_page(width=612, height=792)
|
||||||
|
if text:
|
||||||
|
page.insert_text((72, 72), text, fontsize=11)
|
||||||
|
doc.save(str(path))
|
||||||
|
doc.close()
|
||||||
|
return str(path)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def text_pdf(tmp_path):
|
||||||
|
return _make_pdf(tmp_path / "set.pdf", [EVIDENCE, ""])
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_text_layers(text_pdf):
|
||||||
|
layers = extract_text_layers(text_pdf)
|
||||||
|
assert set(layers) == {1, 2}
|
||||||
|
assert layers[1]["has_text_layer"] is True
|
||||||
|
assert "2X6 STUD PACK" in layers[1]["text"]
|
||||||
|
assert layers[1]["words"], "expected word-level bboxes"
|
||||||
|
assert all("bbox" in w and len(w["bbox"]) == 4 for w in layers[1]["words"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_blank_page_below_min_chars(text_pdf):
|
||||||
|
layers = extract_text_layers(text_pdf)
|
||||||
|
assert layers[2]["has_text_layer"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_disabled_returns_empty(text_pdf, monkeypatch):
|
||||||
|
monkeypatch.setattr(config, "TEXT_LAYER_ENABLED", False)
|
||||||
|
assert extract_text_layers(text_pdf) == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_attach_text_layers(text_pdf, tmp_path):
|
||||||
|
pages = [{"page_number": 1}, {"page_number": 2}]
|
||||||
|
words = attach_text_layers(text_pdf, pages,
|
||||||
|
text_dir=str(tmp_path / "text"))
|
||||||
|
assert pages[0]["text_layer"] and "STUD PACK" in pages[0]["text_layer"]
|
||||||
|
assert pages[1]["text_layer"] is None
|
||||||
|
assert words[1] and not words[2]
|
||||||
|
assert os.path.isfile(tmp_path / "text" / "page-001.txt")
|
||||||
|
assert not os.path.exists(tmp_path / "text" / "page-002.txt")
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_evidence_bbox_exact(text_pdf):
|
||||||
|
words = extract_text_layers(text_pdf)[1]["words"]
|
||||||
|
bbox = find_evidence_bbox(words, EVIDENCE)
|
||||||
|
assert bbox is not None
|
||||||
|
assert bbox[2] > bbox[0] and bbox[3] > bbox[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_evidence_bbox_fuzzy(text_pdf):
|
||||||
|
# Vision quotes imperfectly: wrong count token, rest exact.
|
||||||
|
words = extract_text_layers(text_pdf)[1]["words"]
|
||||||
|
bbox = find_evidence_bbox(words, "(2) 2X6 STUD PACK @ 16 IN O.C.")
|
||||||
|
assert bbox is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_evidence_bbox_miss(text_pdf):
|
||||||
|
words = extract_text_layers(text_pdf)[1]["words"]
|
||||||
|
assert find_evidence_bbox(words, "PENTHOUSE EXHAUST FAN EF-9") is None
|
||||||
|
assert find_evidence_bbox([], EVIDENCE) is None
|
||||||
|
assert find_evidence_bbox(words, "") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_crop(text_pdf):
|
||||||
|
words = extract_text_layers(text_pdf)[1]["words"]
|
||||||
|
bbox = find_evidence_bbox(words, EVIDENCE)
|
||||||
|
crop = render_crop(text_pdf, 1, bbox)
|
||||||
|
assert crop is not None
|
||||||
|
# Decodes as an image of plausible size (margin around the text line).
|
||||||
|
doc = fitz.open(stream=crop, filetype="jpeg")
|
||||||
|
pix = doc[0].get_pixmap()
|
||||||
|
assert pix.width > 100 and pix.height > 20
|
||||||
|
doc.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_crop_bad_page(text_pdf):
|
||||||
|
assert render_crop(text_pdf, 99, (0, 0, 10, 10)) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_coverage_gaps():
|
||||||
|
pages = [{"page_number": 1, "text_layer": "some real text"},
|
||||||
|
{"page_number": 2, "text_layer": "more text"},
|
||||||
|
{"page_number": 3, "text_layer": None}]
|
||||||
|
sheets = [{"page_number": 1, "assertions": [{"id": "a"}]},
|
||||||
|
{"page_number": 2, "assertions": []}]
|
||||||
|
assert coverage_gaps(pages, sheets) == [2]
|
||||||
Reference in New Issue
Block a user