feat: refocus on drawings — code/ADA gated off, drawing-integrity wave, Brain-directed clarification
- ENABLE_CODE_REVIEW flag (default off): skips code/ADA/jurisdiction review path in both pipelines; nothing deleted, one env flag to restore. - Per-sheet Drawing Integrity QA wave (agent + classic, default on): dangling refs, on-sheet contradictions, dimension sanity, missing sheet essentials, tag hygiene. New DrawingIntegrityAgent + classic stage. - Broadened conflict critic: intra-sheet + same-discipline contradictions, not just cross-discipline. - Wave 6.5 Brain-directed clarification (bounded hub-and-spoke): Brain names uncertain findings, verify_evidence requests route through the wave-5b verifier; refuted findings suppressed. One planning call + capped verifies, single iteration. Shared _build_verify_scopes across 5b and 6.5. - Config knobs, .env.example, frontend copy, tests (182 passing).
This commit is contained in:
@@ -0,0 +1,715 @@
|
||||
# Extraction Coverage Guarantee — Implementation Plan
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Eliminate dark sheets (missed pages) and vision-misread content by making wave-1 extraction coverage-guaranteed: deterministic coverage measurement, a text-first retry ladder, deterministic fallback extraction, and text-layer sheet identity recovery.
|
||||
|
||||
**Architecture:** For text-bearing sheets the authoritative alphanumeric content already exists in the PyMuPDF text layer (backend/text_layer.py). Today the LLM transcribes from pixels and we merely *detect* failure post-hoc (coverage_gaps logs; nothing retries). This plan flips wave 1 to: run the vision pass (unchanged, always, on every page) → measure text coverage deterministically per page → if below floor, ADD a text-only structuring pass (LLM segments the text layer, no image, no misreads possible) and MERGE its objects into the vision results — vision keeps everything it found, text structuring fills what it missed → if still below floor, emit deterministic stub objects straight from the text layer so NO text-bearing page ever contributes zero objects. Sheet identity is recovered from the text layer when the LLM drops the header. Vision stays the only source for graphical content (symbols, geometry, line work) and the only path for scanned pages.
|
||||
|
||||
**Tech Stack:** Python 3.14, PyMuPDF (already a dep), existing call_json LLM plumbing, pytest.
|
||||
|
||||
---
|
||||
|
||||
## Root-Cause Diagnosis (why this keeps happening)
|
||||
|
||||
Confirmed against Cypress job 3e01d5baba32 (38-page Verizon set) and code:
|
||||
|
||||
**Missed sheets (pages 8 = S202 wood notes, 10 = S204 lap-splice tables, 18 = A102 REFLECTED CEILING PLAN — zero assertions each):**
|
||||
1. The extractor prompt (backend/prompts.py:277) is biased toward physical "construction objects" (rooms, doors, fixtures). Notes/table-dense sheets have few, so the model returns a bare array with ONE generic summary object (log: `wrapping bare objects array (1 items, no sheet header)`).
|
||||
2. The grounding guard (backend/pipeline/extractor.py:178 `_is_grounded`) drops that summary object as ungrounded → 0 objects.
|
||||
3. `_wrap_bare_list` (backend/agents/extractors.py:33) converts the 1-item bare array into a valid dict, so the compact retry (extractors.py:71) NEVER fires — it only triggers when parsing fully fails. A 1-object page counts as "success".
|
||||
4. `coverage_gaps()` (backend/text_layer.py:211) only LOGS the gap and adds a failed_scope note. No retry, no fallback. The page is silently dark for every downstream wave.
|
||||
5. Sheet identity comes ONLY from the LLM reading the title block in the image. 7/38 Cypress pages ended with `sheet_number=None` (4 of them WITH assertions: pages 22, 30, 31, 37), so they can't join sheet-keyed scopes and corrupt `missing_expected_sheets` downstream.
|
||||
|
||||
**Completely incorrect information:**
|
||||
1. Vision misreads of dense alphanumeric content (the "(2) vs (5) 2x6 STUD PACK" family). The wave-1.5 rescue tier catches invented numbers but is a SET subset test — it cannot catch SWAPPED numbers (documented in docs/superpowers/specs/2026-08-12-text-layer-grounding-design.md).
|
||||
2. Gemini thinking tokens count against max_tokens → `recovered truncated JSON` silently drops tail objects (bottom/right of sheet vanishes). Nothing flags the page as degraded.
|
||||
3. The model paraphrases `source_text`; the guard only checks digit-run/token overlap, so plausible-but-wrong values pass.
|
||||
4. `JSON parse error (giving up)` → classic path returns an empty "extraction failed" sheet (extractor.py:282-291); the page vanishes from analysis while `sheets_analyzed` still counts it.
|
||||
|
||||
**Cornerstone principles for the fix:**
|
||||
1. If a page has a text layer, the truth is already deterministic and free. The LLM's job on such pages is STRUCTURING, not TRANSCRIPTION. Every extracted alphanumeric claim must trace to the text layer; anything that can't is vision-only and gets stamped as such.
|
||||
2. The vision pass is never skipped and never replaced. These are construction documents: symbols, device/fixture locations, geometry, and line work exist only in the image. The text-only rung and the fallback rung are strictly ADDITIVE — they merge into the vision results (deduped by normalized source_text), so a rescue can only add coverage, never subtract graphical content.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Coverage metric module (backend/text_coverage.py)
|
||||
|
||||
**Objective:** Deterministic per-page coverage measurement: what fraction of the text layer is actually represented in extracted objects.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/text_coverage.py`
|
||||
- Test: `tests/test_text_coverage.py`
|
||||
|
||||
**Step 1: Write failing test**
|
||||
|
||||
```python
|
||||
# tests/test_text_coverage.py
|
||||
from backend.text_coverage import text_coverage, segment_text_layer, fallback_objects
|
||||
|
||||
def test_coverage_full():
|
||||
text = "NOTE 1\nALL LUMBER NO. 2 SOUTHERN PINE\nNOTE 2\nUSE 5/8\" PLYWOOD"
|
||||
objects = [{"source_text": "ALL LUMBER NO. 2 SOUTHERN PINE"},
|
||||
{"source_text": "USE 5/8\" PLYWOOD"}]
|
||||
cov = text_coverage(text, objects)
|
||||
assert cov["covered_lines"] == 2
|
||||
assert cov["total_lines"] == 2
|
||||
assert cov["ratio"] == 1.0
|
||||
|
||||
def test_coverage_zero_on_empty_objects():
|
||||
cov = text_coverage("LINE A\nLINE B\nLINE C", [])
|
||||
assert cov["ratio"] == 0.0 and cov["total_lines"] == 3
|
||||
|
||||
def test_coverage_ignores_short_and_numeric_noise_lines():
|
||||
text = "15\"\n19\"\nA\nB\nREAL NOTE ABOUT FRAMING HERE"
|
||||
cov = text_coverage(text, [{"source_text": "REAL NOTE ABOUT FRAMING HERE"}])
|
||||
# short/noise lines (< MIN_LINE_CHARS or pure dimension ticks) excluded
|
||||
assert cov["total_lines"] == 1 and cov["ratio"] == 1.0
|
||||
|
||||
def test_segment_notes_and_rows():
|
||||
text = "WOOD CONSTRUCTION\n1. \nALL SAWN LUMBER TO BE SOUTHERN PINE.\n2. \nROOF SHEATHING 5/8\" PLYWOOD."
|
||||
segs = segment_text_layer(text)
|
||||
assert any("ALL SAWN LUMBER" in s for s in segs)
|
||||
assert any("ROOF SHEATHING" in s for s in segs)
|
||||
|
||||
def test_fallback_objects_verbatim_and_stamped():
|
||||
objs = fallback_objects("1. \nALL SAWN LUMBER TO BE SOUTHERN PINE.", page_number=8)
|
||||
assert len(objs) == 1
|
||||
assert objs[0]["source_text"] == "ALL SAWN LUMBER TO BE SOUTHERN PINE."
|
||||
assert objs[0]["grounding"] == "text_layer_fallback"
|
||||
assert objs[0]["confidence"] == "low"
|
||||
|
||||
def test_merge_objects_keeps_vision_and_unions_text():
|
||||
vision = [
|
||||
{"source_text": "2X6 WD STUD @ 16\" O.C.", "object_type": "wall"},
|
||||
{"source_text": None, "graphical_basis": "light fixture symbol, grid C-4",
|
||||
"object_type": "lighting_fixture"}, # graphical: exists only in image
|
||||
]
|
||||
text = [
|
||||
{"source_text": "2X6 WD STUD @ 16\" O.C.", "object_type": "wall"}, # dup
|
||||
{"source_text": "ALL LUMBER NO. 2 SOUTHERN PINE", "object_type": "general_note"},
|
||||
]
|
||||
merged = merge_objects(vision, text)
|
||||
assert len(merged) == 3 # dup dropped, note added
|
||||
assert any(o.get("graphical_basis") for o in merged) # graphical kept
|
||||
assert merged[0]["object_type"] == "wall" # vision order preserved
|
||||
|
||||
def test_merge_objects_dedupes_by_normalized_text():
|
||||
a = [{"source_text": "RTU-1: 5 TON, 1600 CFM"}]
|
||||
b = [{"source_text": "rtu 1 5 ton 1600 cfm"}] # same content, different case/punct
|
||||
assert len(merge_objects(a, b)) == 1
|
||||
```
|
||||
|
||||
**Step 2: Run test to verify failure**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_text_coverage.py -v`
|
||||
Expected: FAIL — ModuleNotFoundError: backend.text_coverage
|
||||
|
||||
**Step 3: Implement `backend/text_coverage.py`**
|
||||
|
||||
```python
|
||||
"""text_coverage.py - deterministic extraction-coverage measurement.
|
||||
|
||||
The coverage guarantee: for any page with a usable text layer, measure how
|
||||
much of that layer ended up represented in extracted objects. Pages below
|
||||
the floor route into the extraction retry ladder (agents/extractors.py and
|
||||
pipeline/extractor.py). fallback_objects() is the last rung: stub objects
|
||||
segmented straight from the text layer so no text-bearing page goes dark.
|
||||
"""
|
||||
|
||||
import re
|
||||
from typing import Dict, List
|
||||
|
||||
# Lines below this many meaningful chars are noise (dimension ticks, grid
|
||||
# bubbles, single letters) and excluded from the coverage denominator.
|
||||
MIN_LINE_CHARS = 12
|
||||
# Pure dimension/elevation ticks like 15" or 8' - 0" carry no prose content.
|
||||
_TICK_RE = re.compile(r"^[\d\s'\"/.,-]+$")
|
||||
_WORD_RE = re.compile(r"[a-z0-9]+")
|
||||
|
||||
|
||||
def _meaningful_lines(text: str) -> List[str]:
|
||||
lines = []
|
||||
for raw in (text or "").splitlines():
|
||||
line = " ".join(raw.split())
|
||||
if len(line) < MIN_LINE_CHARS or _TICK_RE.match(line):
|
||||
continue
|
||||
lines.append(line)
|
||||
return lines
|
||||
|
||||
|
||||
def _norm(text: str) -> str:
|
||||
return " ".join(_WORD_RE.findall((text or "").lower()))
|
||||
|
||||
|
||||
def text_coverage(page_text: str, objects: List[Dict]) -> Dict:
|
||||
"""Fraction of meaningful text-layer lines whose normalized form appears
|
||||
in the concatenated normalized source_text of extracted objects."""
|
||||
lines = _meaningful_lines(page_text)
|
||||
if not lines:
|
||||
return {"total_lines": 0, "covered_lines": 0, "ratio": 1.0}
|
||||
haystack = " ".join(
|
||||
_norm(str(o.get("source_text") or o.get("object_description")
|
||||
or o.get("value") or ""))
|
||||
for o in objects if isinstance(o, dict)
|
||||
)
|
||||
covered = sum(1 for ln in lines if _norm(ln) and _norm(ln) in haystack)
|
||||
return {
|
||||
"total_lines": len(lines),
|
||||
"covered_lines": covered,
|
||||
"ratio": covered / len(lines) if lines else 1.0,
|
||||
}
|
||||
|
||||
|
||||
def segment_text_layer(text: str) -> List[str]:
|
||||
"""Segment a page text layer into note-sized blocks: numbered notes and
|
||||
contiguous prose runs. PyMuPDF emits each note number on its own line
|
||||
('1. ', '2. ') followed by wrapped text lines; rejoin number->body and
|
||||
merge continuation lines until the next number or blank-line break."""
|
||||
segments: List[str] = []
|
||||
buf: List[str] = []
|
||||
number_re = re.compile(r"^(\d{1,2}[.)]?|[A-Z]\d{0,2}[.)]?)\s*$")
|
||||
|
||||
def flush():
|
||||
joined = " ".join(buf).strip()
|
||||
if len(joined) >= MIN_LINE_CHARS:
|
||||
segments.append(joined)
|
||||
buf.clear()
|
||||
|
||||
for raw in (text or "").splitlines():
|
||||
line = raw.strip()
|
||||
if not line:
|
||||
flush()
|
||||
continue
|
||||
if number_re.match(line):
|
||||
flush()
|
||||
buf.append(line.rstrip(".)"))
|
||||
continue
|
||||
buf.append(line)
|
||||
# wrapped-note heuristic: a line starting a new sentence after a
|
||||
# period ends the segment
|
||||
if line.endswith(".") and len(" ".join(buf)) > 120:
|
||||
flush()
|
||||
flush()
|
||||
return segments
|
||||
|
||||
|
||||
def fallback_objects(page_text: str, page_number: int,
|
||||
max_objects: int = 200) -> List[Dict]:
|
||||
"""Last-rung deterministic extraction: one stub object per text segment,
|
||||
source_text verbatim from the text layer. confidence=low and
|
||||
grounding=text_layer_fallback make their provenance explicit downstream."""
|
||||
objs = []
|
||||
for idx, seg in enumerate(segment_text_layer(page_text)[:max_objects]):
|
||||
objs.append({
|
||||
"object_id": f"p{page_number}-tl{idx}",
|
||||
"object_type": "general_note",
|
||||
"category": "general",
|
||||
"tag": None,
|
||||
"name": seg[:80],
|
||||
"description": seg,
|
||||
"attributes": {},
|
||||
"location_key": {},
|
||||
"source_text": seg,
|
||||
"graphical_basis": None,
|
||||
"review_uses": ["code_review", "constructability_review"],
|
||||
"confidence": "low",
|
||||
"grounding": "text_layer_fallback",
|
||||
})
|
||||
return objs
|
||||
|
||||
|
||||
def merge_objects(vision_objs: List[Dict], text_objs: List[Dict]) -> List[Dict]:
|
||||
"""Union of vision and text-structured objects. Vision results come first
|
||||
and are never dropped (graphical_basis objects exist only in the image).
|
||||
Text objects are appended unless their normalized source_text is already
|
||||
represented. A merge can only add coverage, never subtract it."""
|
||||
merged = list(vision_objs or [])
|
||||
seen = {_norm(str(o.get("source_text") or ""))
|
||||
for o in merged if isinstance(o, dict)}
|
||||
seen.discard("")
|
||||
for obj in text_objs or []:
|
||||
if not isinstance(obj, dict):
|
||||
continue
|
||||
key = _norm(str(obj.get("source_text") or ""))
|
||||
if key and key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
merged.append(obj)
|
||||
return merged
|
||||
```
|
||||
|
||||
**Step 4: Run test to verify pass**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_text_coverage.py -v`
|
||||
Expected: 7 passed
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/text_coverage.py tests/test_text_coverage.py
|
||||
git commit -m "feat: deterministic text-layer coverage metric + fallback extraction"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Sheet identity recovery from the text layer
|
||||
|
||||
**Objective:** When the LLM drops/misreads the sheet header, recover `sheet_number` (and discipline via existing `discipline_from_sheet_number`) deterministically from the text layer instead of leaving None.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/text_coverage.py` (add `recover_sheet_number`)
|
||||
- Test: `tests/test_text_coverage.py` (add tests)
|
||||
|
||||
**Step 1: Write failing test**
|
||||
|
||||
```python
|
||||
def test_recover_sheet_number_from_title_block():
|
||||
text = ("WALL SECTIONS\n...\nSheet Information\nS301\n"
|
||||
"Issue Date 05.29.26\nProject Number 25177")
|
||||
assert recover_sheet_number(text) == "S301"
|
||||
|
||||
def test_recover_sheet_number_none_when_absent():
|
||||
assert recover_sheet_number("just some notes about lumber") is None
|
||||
|
||||
def test_recover_prefers_discipline_pattern_over_dates():
|
||||
# 05.29.26 and 25177 must never match
|
||||
text = "Issue Date 05.29.26\nProject Number 25177\nA102 REFLECTED CEILING PLAN"
|
||||
assert recover_sheet_number(text) == "A102"
|
||||
```
|
||||
|
||||
**Step 2: Run to verify failure**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_text_coverage.py::test_recover_sheet_number_from_title_block -v`
|
||||
Expected: FAIL — ImportError
|
||||
|
||||
**Step 3: Implement in `backend/text_coverage.py`**
|
||||
|
||||
```python
|
||||
# Sheet ids: 1-2 uppercase letters + 2-3 digits + optional decimal suffix
|
||||
# (S301, A102, M200, E500, LS101, P100, G000). Deliberately excludes pure
|
||||
# numbers (dates, project numbers) and long alphanumerics (member marks).
|
||||
_SHEET_ID_RE = re.compile(r"\b([A-Z]{1,2}\d{2,3}(?:\.\d+)?)\b")
|
||||
_TITLE_HINT_RE = re.compile(
|
||||
r"(?i)sheet\s*(?:information|no|number)?|"
|
||||
r"(floor plan|ceiling plan|elevations?|sections?|details?|schedule|"
|
||||
r"notes|legend|plan)")
|
||||
|
||||
def recover_sheet_number(page_text: str) -> Optional[str]:
|
||||
"""Deterministic sheet id from the text layer. Strategy: collect every
|
||||
sheet-id-shaped token, prefer ones appearing near title words or in the
|
||||
last ~15%% of the page (title block lives at the drawing edge)."""
|
||||
text = page_text or ""
|
||||
cands = _SHEET_ID_RE.findall(text)
|
||||
if not cands:
|
||||
return None
|
||||
tail = text[int(len(text) * 0.85):]
|
||||
for cand in reversed(_SHEET_ID_RE.findall(tail)):
|
||||
return cand
|
||||
return cands[0]
|
||||
```
|
||||
|
||||
(Add `from typing import Optional` to the imports.)
|
||||
|
||||
**Step 4: Run to verify pass**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_text_coverage.py -v`
|
||||
Expected: all pass (10 tests)
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/text_coverage.py tests/test_text_coverage.py
|
||||
git commit -m "feat: deterministic sheet-number recovery from text layer"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Text-only structuring prompt (no image)
|
||||
|
||||
**Objective:** Second rung of the ladder: give the LLM the raw text layer and ask it to segment EVERY note/row/callout into objects with verbatim source_text. No image = no vision misreads for alphanumerics; far cheaper than the vision pass.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/prompts.py` (append after EXTRACTOR_USER_INSTRUCTION, ~line 282)
|
||||
- Test: `tests/agents/test_extraction_ladder.py` (prompt-content assertions only; rendering tested in Task 4)
|
||||
|
||||
**Step 1: Write failing test**
|
||||
|
||||
```python
|
||||
# tests/agents/test_extraction_ladder.py
|
||||
from backend.prompts import TEXT_STRUCTURING_SYSTEM_PROMPT, TEXT_STRUCTURING_USER_INSTRUCTION
|
||||
|
||||
def test_text_structuring_prompt_demands_verbatim_and_completeness():
|
||||
assert "verbatim" in TEXT_STRUCTURING_USER_INSTRUCTION.lower()
|
||||
assert "every" in TEXT_STRUCTURING_USER_INSTRUCTION.lower()
|
||||
assert "{text_layer}" in TEXT_STRUCTURING_USER_INSTRUCTION
|
||||
```
|
||||
|
||||
**Step 2: Run to verify failure**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/agents/test_extraction_ladder.py -v`
|
||||
Expected: FAIL — ImportError
|
||||
|
||||
**Step 3: Append to `backend/prompts.py`**
|
||||
|
||||
```python
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stage 2b - text-only structuring (extraction retry ladder, rung 2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
TEXT_STRUCTURING_SYSTEM_PROMPT = """You are a construction document structuring engine.
|
||||
You receive the deterministic text layer extracted from one drawing sheet. It is complete and authoritative.
|
||||
Your ONLY job is to segment it into structured objects. You are NOT reading an image. You must NOT invent, complete, or correct any text.
|
||||
Rules:
|
||||
- Every numbered note, schedule row, callout, tag, legend entry, and title-block field becomes its own object.
|
||||
- source_text must be copied VERBATIM from the input, character-for-character. Never paraphrase.
|
||||
- Cover the ENTIRE input. Omitting a note is a failure. When unsure of an object's type, use general_note with confidence low.
|
||||
- Numbers, model numbers, dimensions, and tags must appear in source_text exactly as in the input.
|
||||
Respond only with valid JSON."""
|
||||
|
||||
TEXT_STRUCTURING_USER_INSTRUCTION = """Segment this sheet's text layer into structured construction objects.
|
||||
Respond ONLY with a valid JSON object - no markdown fences:
|
||||
{ "sheet": { "sheet_number": "string or null", "sheet_title": "string or null", "discipline": "string or null", "drawing_type": "string or null", "level": "string or null", "scale": "string or null" }, "objects": [ { "object_id": "string", "object_type": "room | door | window | wall | finish | ceiling | dimension | grid | callout | keynote | general_note | equipment | plumbing_fixture | mechanical_equipment | electrical_device | lighting_fixture | structural_element | schedule_reference | symbol | abbreviation", "category": "architectural | structural | mechanical | electrical | plumbing | code | general", "tag": "string or null", "name": "string or null", "description": "string or null", "attributes": { "attribute_name": "attribute_value" }, "location_key": { "room_number": "string or null", "grid": "string or null", "detail_reference": "string or null" }, "source_text": "VERBATIM text copied from the input", "graphical_basis": null, "review_uses": [ "schedule_comparison", "cross_discipline_coordination", "code_review", "constructability_review" ], "confidence": "high | medium | low" } ], "unresolved_items": [] }
|
||||
Optional sheet hint: {sheet_hint}
|
||||
|
||||
TEXT LAYER (segment ALL of it):
|
||||
{text_layer}"""
|
||||
```
|
||||
|
||||
NOTE the two render sites you will add in Tasks 4-5 substitute `{sheet_hint}` and `{text_layer}` with str.replace directly (NOT via render()/call_stage) — this matches the wave-1.5 pattern and avoids the classic-path literal-placeholder leak documented in the project pitfalls.
|
||||
|
||||
**Step 4: Run to verify pass**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/agents/test_extraction_ladder.py -v`
|
||||
Expected: 1 passed
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/prompts.py tests/agents/test_extraction_ladder.py
|
||||
git commit -m "feat: text-only structuring prompt for extraction retry ladder"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Retry ladder in the AGENT path (SheetExtractorAgent)
|
||||
|
||||
**Objective:** Replace the binary parse-fail retry with a coverage-driven ladder: vision pass → coverage check → text-only structuring pass → deterministic fallback. Also recover sheet identity and mark truncation-degraded pages.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/agents/extractors.py:63-86` (SheetExtractorAgent.run)
|
||||
- Modify: `backend/config.py` (new knobs, below)
|
||||
- Test: `tests/agents/test_extraction_ladder.py`
|
||||
|
||||
**New config knobs (backend/config.py, follow existing env pattern):**
|
||||
|
||||
```python
|
||||
EXTRACT_COVERAGE_FLOOR = float(os.getenv("EXTRACT_COVERAGE_FLOOR", "0.6"))
|
||||
EXTRACT_TEXT_RETRY_ENABLED = os.getenv("EXTRACT_TEXT_RETRY_ENABLED", "true").lower() == "true"
|
||||
EXTRACT_FALLBACK_ENABLED = os.getenv("EXTRACT_FALLBACK_ENABLED", "true").lower() == "true"
|
||||
EXTRACT_FALLBACK_MAX_OBJECTS = int(os.getenv("EXTRACT_FALLBACK_MAX_OBJECTS", "200"))
|
||||
```
|
||||
|
||||
**Step 1: Write failing test**
|
||||
|
||||
```python
|
||||
from backend.agents.extractors import SheetExtractorAgent
|
||||
from backend.agents.base import AgentScope, AgentUsage
|
||||
|
||||
def _page(n=8, text="1. \nALL SAWN LUMBER IN CONTACT WITH SOIL TO BE SOUTHERN PINE, PRESSURE TREATED.\n2. \nROOF SHEATHING: 5/8\" PLYWOOD, C-D GRADE, STRUCTURAL I."):
|
||||
return {"page_number": n, "base64": "AAAA", "text_layer": text}
|
||||
|
||||
def test_ladder_falls_back_when_vision_returns_nothing(agent_monkeypatch):
|
||||
# vision pass returns 1 summary object that the guard drops;
|
||||
# text-structuring disabled to exercise the deterministic rung
|
||||
agent_monkeypatch.setattr("backend.agents.extractors.call_json",
|
||||
lambda **kw: [{"name": "general notes", "value": "notes"}])
|
||||
agent_monkeypatch.setattr("backend.config.EXTRACT_TEXT_RETRY_ENABLED", False)
|
||||
agent = SheetExtractorAgent(AgentUsage())
|
||||
scope = AgentScope(scope_id="sheet:8", payload={"page": _page(), "sheet_hint": ""})
|
||||
result = agent.run(scope)
|
||||
sheet = result.artifacts[0]
|
||||
assert sheet["assertions"], "dark sheet must be impossible with fallback enabled"
|
||||
assert all(a.get("grounding") == "text_layer_fallback" for a in sheet["assertions"])
|
||||
assert sheet["coverage"]["ratio"] >= 0.6
|
||||
|
||||
def test_ladder_merge_preserves_graphical_objects(agent_monkeypatch):
|
||||
# vision finds a graphical symbol + misreads nothing; text rung adds notes.
|
||||
# The graphical object MUST survive the merge.
|
||||
calls = {"n": 0}
|
||||
def fake_call_json(**kw):
|
||||
calls["n"] += 1
|
||||
if kw.get("images_b64"): # vision pass
|
||||
return {"sheet": {}, "objects": [
|
||||
{"object_id": "g1", "object_type": "lighting_fixture",
|
||||
"name": "pendant at grid C-4", "source_text": None,
|
||||
"graphical_basis": "16in pendant symbol at grid C-4"}]}
|
||||
return {"sheet": {}, "objects": [ # text-structuring pass
|
||||
{"object_id": "t1", "object_type": "general_note",
|
||||
"source_text": "ALL SAWN LUMBER IN CONTACT WITH SOIL TO BE SOUTHERN PINE, PRESSURE TREATED.",
|
||||
"name": "lumber note"}]}
|
||||
agent_monkeypatch.setattr("backend.agents.extractors.call_json", fake_call_json)
|
||||
agent = SheetExtractorAgent(AgentUsage())
|
||||
scope = AgentScope(scope_id="sheet:8", payload={"page": _page(), "sheet_hint": ""})
|
||||
sheet = agent.run(scope).artifacts[0]
|
||||
assert any(a.get("graphical_basis") for a in sheet["assertions"])
|
||||
assert any("SAWN LUMBER" in (a.get("source_text") or "") for a in sheet["assertions"])
|
||||
|
||||
def test_ladder_recovers_sheet_number_from_text_layer(agent_monkeypatch):
|
||||
agent_monkeypatch.setattr(
|
||||
"backend.agents.extractors.call_json",
|
||||
lambda **kw: {"sheet": {}, "objects": [
|
||||
{"object_id": "o1", "name": "RCP note",
|
||||
"source_text": "GYP. BD. CEILING 8'-11 3/8\" A.F.F. TYP. FOR ALL STOREFRONT",
|
||||
"attributes": {"height": "8'-11 3/8\""}}]})
|
||||
agent = SheetExtractorAgent(AgentUsage())
|
||||
scope = AgentScope(scope_id="sheet:18",
|
||||
payload={"page": _page(18, "REFLECTED CEILING PLAN\nA102\nGYP. BD. CEILING 8'-11 3/8\" A.F.F. TYP. FOR ALL STOREFRONT"),
|
||||
"sheet_hint": ""})
|
||||
sheet = agent.run(scope).artifacts[0]
|
||||
assert sheet["sheet_number"] == "A102"
|
||||
```
|
||||
|
||||
(Monkeypatch fixture: plain `unittest.mock.patch` context or pytest `monkeypatch`; follow tests/agents/test_text_layer_flow.py patterns for scope/result construction — check AgentScope/AgentResult signatures in backend/agents/base.py before writing.)
|
||||
|
||||
**Step 2: Run to verify failure**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/agents/test_extraction_ladder.py -v`
|
||||
Expected: FAIL — assertions on coverage/sheet_number fail (ladder not implemented)
|
||||
|
||||
**Step 3: Implement the ladder in `backend/agents/extractors.py`**
|
||||
|
||||
Replace `SheetExtractorAgent.run` (lines 63-86) with:
|
||||
|
||||
```python
|
||||
def _text_structuring_call(self, page, sheet_hint):
|
||||
from backend.prompts import (TEXT_STRUCTURING_SYSTEM_PROMPT,
|
||||
TEXT_STRUCTURING_USER_INSTRUCTION)
|
||||
instruction = (TEXT_STRUCTURING_USER_INSTRUCTION
|
||||
.replace("{sheet_hint}", str(sheet_hint or ""))
|
||||
.replace("{text_layer}",
|
||||
(page.get("text_layer") or "")
|
||||
[:config.TEXT_LAYER_MAX_CHARS]))
|
||||
return call_json(
|
||||
system_prompt=TEXT_STRUCTURING_SYSTEM_PROMPT,
|
||||
user_text=instruction,
|
||||
images_b64=None,
|
||||
max_tokens=config.EXTRACT_MAX_TOKENS,
|
||||
model=config.AGENT_EXTRACT_MODEL,
|
||||
usage_tracker=self.usage,
|
||||
usage_stage="agent.extract_text",
|
||||
reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
|
||||
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
|
||||
)
|
||||
|
||||
def run(self, scope: AgentScope) -> AgentResult:
|
||||
from backend.text_coverage import (fallback_objects, merge_objects,
|
||||
recover_sheet_number, text_coverage)
|
||||
try:
|
||||
page = scope.payload["page"]
|
||||
hint = scope.payload.get("sheet_hint") or ""
|
||||
page_text = page.get("text_layer")
|
||||
instruction = EXTRACTOR_USER_INSTRUCTION.replace(
|
||||
"{sheet_hint}", str(hint)) + _text_layer_block(page)
|
||||
|
||||
# Rung 1: vision pass (unchanged behaviour, incl. compact retry)
|
||||
parsed = _wrap_bare_list(self._call(instruction, page),
|
||||
page["page_number"])
|
||||
if not isinstance(parsed, dict):
|
||||
print(f"[Extract] Page {page['page_number']}: full extraction "
|
||||
f"failed, retrying compact")
|
||||
parsed = _wrap_bare_list(
|
||||
self._call(instruction + _COMPACT_RETRY_SUFFIX, page),
|
||||
page["page_number"])
|
||||
if not isinstance(parsed, dict):
|
||||
parsed = {"sheet": {}, "objects": []}
|
||||
|
||||
sheet = _normalize_sheet(parsed, page["page_number"],
|
||||
page_text=page_text)
|
||||
cov = text_coverage(page_text or "", sheet["assertions"])
|
||||
sheet["coverage"] = cov
|
||||
|
||||
# Rung 2: text-only structuring when coverage is below floor.
|
||||
# MERGE, never replace: vision keeps every object it found
|
||||
# (graphical_basis content exists only in the image); the text
|
||||
# pass fills in the text content the vision pass missed.
|
||||
if (page_text and config.EXTRACT_TEXT_RETRY_ENABLED
|
||||
and cov["ratio"] < config.EXTRACT_COVERAGE_FLOOR):
|
||||
print(f"[Extract] Page {page['page_number']}: coverage "
|
||||
f"{cov['ratio']:.0%} < floor - text-only structuring pass")
|
||||
parsed2 = _wrap_bare_list(
|
||||
self._text_structuring_call(page, hint), page["page_number"])
|
||||
if isinstance(parsed2, dict):
|
||||
sheet2 = _normalize_sheet(parsed2, page["page_number"],
|
||||
page_text=page_text)
|
||||
before = len(sheet["assertions"])
|
||||
sheet["assertions"] = merge_objects(sheet["assertions"],
|
||||
sheet2["assertions"])
|
||||
# Fill header gaps the vision pass left null
|
||||
for key in ("sheet_number", "sheet_title", "discipline",
|
||||
"level", "scale", "drawing_type"):
|
||||
if not sheet.get(key) and sheet2.get(key):
|
||||
sheet[key] = sheet2[key]
|
||||
cov = text_coverage(page_text, sheet["assertions"])
|
||||
sheet["coverage"] = cov
|
||||
print(f"[Extract] Page {page['page_number']}: merged "
|
||||
f"{len(sheet['assertions']) - before} text-structured "
|
||||
f"object(s), coverage now {cov['ratio']:.0%}")
|
||||
|
||||
# Rung 3: deterministic fallback - dark sheets are impossible.
|
||||
# Also merged (deduped) so stub notes never double up with
|
||||
# objects the earlier rungs already captured.
|
||||
if (page_text and config.EXTRACT_FALLBACK_ENABLED
|
||||
and cov["ratio"] < config.EXTRACT_COVERAGE_FLOOR):
|
||||
stubs = fallback_objects(page_text, page["page_number"],
|
||||
config.EXTRACT_FALLBACK_MAX_OBJECTS)
|
||||
stubs = _normalize_sheet({"sheet": {}, "objects": stubs},
|
||||
page["page_number"],
|
||||
page_text=page_text)["assertions"]
|
||||
before = len(sheet["assertions"])
|
||||
sheet["assertions"] = merge_objects(sheet["assertions"], stubs)
|
||||
print(f"[Extract] Page {page['page_number']}: fallback merged "
|
||||
f"{len(sheet['assertions']) - before} text-layer stub(s)")
|
||||
sheet["coverage"] = text_coverage(page_text,
|
||||
sheet["assertions"])
|
||||
|
||||
# Identity recovery: never leave a text-bearing page sheet-less
|
||||
if not sheet.get("sheet_number") and page_text:
|
||||
recovered = recover_sheet_number(page_text)
|
||||
if recovered:
|
||||
sheet["sheet_number"] = recovered
|
||||
sheet["discipline"] = (
|
||||
__import__("backend.pipeline.extractor",
|
||||
fromlist=["discipline_from_sheet_number"])
|
||||
.discipline_from_sheet_number(recovered)
|
||||
or sheet.get("discipline") or "Unknown")
|
||||
print(f"[Extract] Page {page['page_number']}: sheet number "
|
||||
f"recovered from text layer -> {recovered}")
|
||||
|
||||
return AgentResult(scope_id=scope.scope_id, artifacts=[sheet])
|
||||
except Exception as exc:
|
||||
return failure(scope, exc)
|
||||
```
|
||||
|
||||
**Step 4: Run to verify pass**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/agents/test_extraction_ladder.py -v`
|
||||
Expected: all pass
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/agents/extractors.py backend/config.py tests/agents/test_extraction_ladder.py
|
||||
git commit -m "feat: coverage-driven extraction retry ladder (agent path)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Same ladder in the CLASSIC path (pipeline/extractor.py)
|
||||
|
||||
**Objective:** The classic pipeline (`_extract_one`, backend/pipeline/extractor.py:273-293) must get the identical ladder — two render paths share everything, per the documented project trap.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/pipeline/extractor.py:273-293`
|
||||
- Test: `tests/test_text_layer_flow.py` or new `tests/test_extraction_ladder_classic.py`
|
||||
|
||||
**Step 1: Write failing test** — mirror Task 4's tests against `_extract_one` directly (monkeypatch `backend.pipeline.extractor.call_json`).
|
||||
|
||||
**Step 2: Run to verify failure**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_extraction_ladder_classic.py -v`
|
||||
Expected: FAIL
|
||||
|
||||
**Step 3: Implement** — same ladder shape as Task 4 but inside `_extract_one`; the text-structuring call here uses default model (no `model=` kwarg, matching existing `_extract_one` call_json usage). Keep the existing "extraction failed" empty-sheet shape for pages with NO text layer (scanned pages stay vision-only and may legitimately return empty).
|
||||
|
||||
**Step 4: Run to verify pass**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_extraction_ladder_classic.py -v`
|
||||
Expected: all pass
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/pipeline/extractor.py tests/test_extraction_ladder_classic.py
|
||||
git commit -m "feat: coverage-driven extraction retry ladder (classic path)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Verbatim-source stamping upgrade
|
||||
|
||||
**Objective:** When a text layer exists, check each object's source_text against the page text with the existing fuzzy machinery; stamp `grounding="vision_unverified"` when it doesn't match so the wave-5b verifier prioritizes it. Cheap upgrade, reuses text_layer._tokens — no new call sites.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/pipeline/extractor.py` (`_normalize_sheet`, ~line 182 where `grounding` is stamped)
|
||||
- Test: extend `tests/test_extractor_text_grounding.py`
|
||||
|
||||
**Step 1: Failing test** — object whose source_text is NOT a fuzzy substring of the page text keeps the object (guard passes via digits) but gets stamped `vision_unverified`.
|
||||
|
||||
**Step 2:** Run, expect FAIL.
|
||||
|
||||
**Step 3: Implement** — in `_normalize_sheet`, when `page_text` is present and no `grounding` stamp yet: normalized source_text (via `backend.text_coverage._norm`) not substring of normalized page text → `grounding = "vision_unverified"` (counted in the existing log line as a third counter).
|
||||
|
||||
**Step 4:** Run, expect PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/pipeline/extractor.py tests/test_extractor_text_grounding.py
|
||||
git commit -m "feat: stamp vision-unverified source_text against text layer"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Surface coverage in the report
|
||||
|
||||
**Objective:** `report.summary` gains per-job extraction-quality visibility so "is extraction healthy?" is answerable without log spelunking.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/agents/runner.py` (where summary is assembled) and/or `backend/pipeline/report.py`
|
||||
- Test: extend the runner-level stub test (tests/agents/test_wave5b_suppression.py pattern)
|
||||
|
||||
**Step 1: Failing test** — runner-level: summary contains `extraction_coverage = {"pages_below_floor": [...], "mean_ratio": float, "fallback_pages": [...]}`.
|
||||
|
||||
**Step 2-4:** Implement by aggregating the `coverage` dicts Task 4/5 attach to each sheet; NO new ProjectMemory keys (closed registry trap) — compute at report assembly from the sheets list already in scope.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/agents/runner.py backend/pipeline/report.py tests/
|
||||
git commit -m "feat: extraction coverage summary in report"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Full suite + Cypress validation run
|
||||
|
||||
**Step 1:** `.venv/bin/python -m pytest tests/ -q` — expected: all pass (128 + new).
|
||||
|
||||
**Step 2:** Push branch, wait for Gitea Actions sha-<short> build, deploy per the skill's deploy runbook (compose pull + up -d --force-recreate).
|
||||
|
||||
**Step 3:** Resubmit the exact Cypress PDF (`docker cp`'d source.pdf preserved at /tmp/cypress-source.pdf on sits-docker):
|
||||
`curl -F file=@source.pdf -F pipeline_mode=agent https://conchecker.scoutitsystems.com/check`
|
||||
|
||||
**Step 4: Acceptance criteria (compare against job 3e01d5baba32):**
|
||||
- Zero text-bearing pages with 0 assertions (was: pages 8, 10, 18).
|
||||
- `sheet_number` present on >= 37/38 pages (was: 31/38).
|
||||
- A102 RCP content (ceiling heights, tape lights, sconces) present in assertions.
|
||||
- Spot-check: no regression in validated-issue quality — suppressed_issues and validated_issues counts within noise of the prior run; cost delta reported (expect +1 cheap text-only call per low-coverage page, ~$0 on healthy pages).
|
||||
- `report.summary.extraction_coverage.pages_below_floor` is empty or every entry is a genuinely scanned page.
|
||||
|
||||
---
|
||||
|
||||
## Files Touched (summary)
|
||||
|
||||
- Create: `backend/text_coverage.py`
|
||||
- Modify: `backend/prompts.py`, `backend/config.py`, `backend/agents/extractors.py`, `backend/pipeline/extractor.py`, `backend/agents/runner.py`, `backend/pipeline/report.py`
|
||||
- Tests: `tests/test_text_coverage.py`, `tests/agents/test_extraction_ladder.py`, `tests/test_extraction_ladder_classic.py`, extensions to `tests/test_extractor_text_grounding.py` and the runner-level stub test
|
||||
|
||||
## Risks, Tradeoffs, Open Questions
|
||||
|
||||
- **Fallback flood risk:** 200 low-confidence stubs/page could flood downstream scopes. Mitigations: EXTRACT_FALLBACK_MAX_OBJECTS cap, confidence=low (specialists already weight confidence), fallback only fires below the coverage floor (3/38 pages on Cypress). If Brain merge gets noisy, lower the cap or restrict fallback to pages where rungs 1+2 BOTH return 0 objects.
|
||||
- **Cost:** rung 2 adds one text-only call per low-coverage page (~12k input chars, no image) — negligible vs the 65k-token vision pass. Healthy pages skip it entirely.
|
||||
- **Sheet-id regex false positives:** member marks like W12X26 are excluded by the 2-3-digit shape, but "S301" inside a detail reference ("2/S301") will match. Tail-of-page preference mitigates; wrong-but-present sheet_number is still strictly better than None for scope keying (sheet_index wave can correct it).
|
||||
- **Open question:** should rung 2 route to the local model (aimax LM Studio) instead of the cloud extractor model to make retries free? Config knob `AGENT_EXTRACT_TEXT_MODEL` would allow it; not included in this plan (YAGNI until cost data from the validation run says otherwise).
|
||||
- **Explicit non-goal:** graphical-only content (symbol geometry, line work) stays vision-based — text-layer-first cannot see it. Scanned PDFs (no text layer) keep today's behaviour plus the existing failed_scopes gap note.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Plan: Brain-directed clarification pass (bounded hub-and-spoke)
|
||||
|
||||
Date: 2026-08-20
|
||||
Branch: agent-mode
|
||||
|
||||
## Goal
|
||||
Let the Brain actively chase weak/ambiguous findings instead of only judging
|
||||
the finished pile once. Bounded, traceable, reuses the wave-5b verifier as the
|
||||
"answer" channel. NOT a free agentic loop.
|
||||
|
||||
## Shape (agent pipeline)
|
||||
Insert **wave 6.5: Brain-directed clarification** between the wave-6 Brain merge
|
||||
and the review-gate / wave-7 branches, so BOTH paths benefit.
|
||||
|
||||
1. `BrainAgent.plan_clarifications(prioritized)` — one focused LLM call. Brain
|
||||
names findings it is unsure about and emits TYPED requests:
|
||||
`{issue_id, request_type, reason}`. v1 executes only `verify_evidence`;
|
||||
the router accepts other types but logs them as "planned, not executed"
|
||||
(extensible without a rewrite). Capped at `BRAIN_CLARIFY_MAX_REQUESTS`.
|
||||
Brain is told which findings already carry `verification` (from 5b) so it
|
||||
does not re-request them.
|
||||
2. Route `verify_evidence` requests → build verify scopes for exactly those
|
||||
findings (reuse the SAME scope builder as wave 5b: fresh page images +
|
||||
hi-DPI evidence crops + text-layer oracle) → `EvidenceVerifierAgent` →
|
||||
`apply_verdicts(prioritized, ...)`. Refuted findings are annotated,
|
||||
demoted, removed from `prioritized`, and pushed into `memory["suppressed"]`
|
||||
(existing key — no memory-registry crash). Clarify decisions recorded in
|
||||
`memory["decisions"]`.
|
||||
3. No second full Brain merge: Brain ASKED (step 1) and the verifier ANSWERED
|
||||
(step 2); the answer prunes/annotates the list. This keeps issue_ids stable
|
||||
for the review queue and adds at most 1 + N calls. One iteration only.
|
||||
|
||||
## Bounds / knobs (config.py, all env-overridable)
|
||||
- `ENABLE_BRAIN_CLARIFY` (_flag, default true)
|
||||
- `BRAIN_CLARIFY_MAX_REQUESTS` (default 8)
|
||||
- reuse `AGENT_VERIFY_CONCURRENCY`, `VERIFY_MAX_TOKENS`,
|
||||
`AGENT_VERIFY_REASONING_EFFORT`, `AGENT_CONFLICT_MAX_IMAGES`,
|
||||
`VERIFY_HI_DPI_CROPS`.
|
||||
|
||||
## Reuse / refactor
|
||||
- Extract the inline wave-5b verify-scope construction into
|
||||
`_build_verify_scopes(findings, targets, sheet_to_page, page_to_b64,
|
||||
page_to_text, page_words, pdf_path, prefix)` so wave 5b and wave 6.5 share
|
||||
it. Preserve wave-5b behavior exactly (its tests guard this).
|
||||
|
||||
## Classic pipeline
|
||||
Out of scope for v1 — the verifier/crops live only in the agent path. Classic
|
||||
keeps its single dedup_validate. Documented as agent-only.
|
||||
|
||||
## Tests
|
||||
- `plan_clarifications` parses/caps/skips-already-verified (stub call_json).
|
||||
- Router executes verify_evidence, ignores unknown types.
|
||||
- Runner smoke: a low-confidence finding Brain flags gets refuted → moves to
|
||||
suppressed_issues; stub verifier.call_json (no live calls).
|
||||
|
||||
## Pitfalls to respect
|
||||
- ProjectMemory keys are a closed registry — only use existing `suppressed` /
|
||||
`decisions`. (Skill defect C1.)
|
||||
- Stub `backend.agents.verifier.call_json` in runner tests or it hits the net.
|
||||
- Extractor stub sheets need >= 2 assertions or no clusters form.
|
||||
Reference in New Issue
Block a user