feat: refocus on drawings — code/ADA gated off, drawing-integrity wave, Brain-directed clarification
Docker Release / build-and-push (push) Successful in 1m8s
Docker Release / release (push) Skipped

- ENABLE_CODE_REVIEW flag (default off): skips code/ADA/jurisdiction review
  path in both pipelines; nothing deleted, one env flag to restore.
- Per-sheet Drawing Integrity QA wave (agent + classic, default on):
  dangling refs, on-sheet contradictions, dimension sanity, missing sheet
  essentials, tag hygiene. New DrawingIntegrityAgent + classic stage.
- Broadened conflict critic: intra-sheet + same-discipline contradictions,
  not just cross-discipline.
- Wave 6.5 Brain-directed clarification (bounded hub-and-spoke): Brain names
  uncertain findings, verify_evidence requests route through the wave-5b
  verifier; refuted findings suppressed. One planning call + capped verifies,
  single iteration. Shared _build_verify_scopes across 5b and 6.5.
- Config knobs, .env.example, frontend copy, tests (182 passing).
This commit is contained in:
2026-08-20 15:10:32 -05:00
parent bae608a505
commit d37ac8c1c7
22 changed files with 1917 additions and 63 deletions
+25
View File
@@ -27,6 +27,23 @@ AGENT_CONFLICT_CONCURRENCY=4
AGENT_SPECIALIST_CONCURRENCY=4
AGENT_RFI_CONCURRENCY=4
# -- Review focus toggles -------------------------------------------
# ENABLE_CODE_REVIEW: run the code/ADA/jurisdiction review path (both pipelines).
# Default OFF - the product focuses on drawing integrity and cross-discipline
# coordination, not code/accessibility compliance. Set to 1 to restore it.
ENABLE_CODE_REVIEW=false
# ENABLE_DRAWING_INTEGRITY: per-sheet Drawing Integrity QA wave (both pipelines).
# The drawing-focused pass - dangling references, on-sheet contradictions,
# dimension sanity, missing sheet essentials, tag hygiene. Default ON.
ENABLE_DRAWING_INTEGRITY=true
AGENT_INTEGRITY_MODEL=
AGENT_INTEGRITY_CONCURRENCY=4
AGENT_INTEGRITY_MAX_IMAGES=1
AGENT_INTEGRITY_MAX_ASSERTIONS=80
INTEGRITY_MAX_TOKENS=16384
# Skip sheets with fewer than this many extracted objects (too sparse to check)
INTEGRITY_MIN_ASSERTIONS=3
# Agent-mode human-review gate (pipeline stops after Brain until a human reviews)
AGENT_REQUIRE_REVIEW=true
# Max clean clusters added to the review queue as non-blocking spot-checks
@@ -79,6 +96,14 @@ AGENT_VERIFY_SEVERITIES=critical,high
AGENT_VERIFY_REASONING_EFFORT=low
VERIFY_MAX_TOKENS=8192
# Wave 6.5 Brain-directed clarification (bounded hub-and-spoke). After the Brain
# merge, the Brain names findings it is unsure about; verify_evidence requests
# route back through the wave-5b verifier. One planning call + at most
# BRAIN_CLARIFY_MAX_REQUESTS verifications, single iteration. Default ON.
ENABLE_BRAIN_CLARIFY=true
BRAIN_CLARIFY_MAX_REQUESTS=8
BRAIN_CLARIFY_MAX_TOKENS=4096
# Text-layer grounding (deterministic PDF text layer via PyMuPDF)
# TEXT_LAYER_ENABLED: master switch for text-layer extraction/grounding
# TEXT_LAYER_MIN_CHARS: below this per page the sheet stays vision-only
+72 -1
View File
@@ -6,7 +6,12 @@ from typing import Dict, List, Tuple
from backend import config
from backend.agents.base import AgentUsage
from backend.agents.prompts import BRAIN_SYSTEM_PROMPT, BRAIN_USER_PROMPT
from backend.agents.prompts import (
BRAIN_CLARIFY_SYSTEM_PROMPT,
BRAIN_CLARIFY_USER_PROMPT,
BRAIN_SYSTEM_PROMPT,
BRAIN_USER_PROMPT,
)
from backend.llm import call_json
from backend.pipeline._stage import collect_list, validate_issue
@@ -127,3 +132,69 @@ class BrainAgent:
issues.sort(key=lambda item: -int(item.get("risk_score") or 0))
decisions = parsed.get("decisions") or []
return issues, [item for item in decisions if isinstance(item, dict)]
def plan_clarifications(self, prioritized: List[Dict]) -> List[Dict]:
"""Wave 6.5 planning call: name kept findings the Brain wants to
double-check before publishing, as typed clarification requests.
Returns a capped list of {issue_id, request_type, reason}. Only
findings that carry an issue_id and do NOT already have a verification
result are offered to the model; anything the model names outside that
set, or with an unknown request_type, is dropped by the caller/router.
Never raises — a failed/empty plan just yields no requests.
"""
max_requests = config.BRAIN_CLARIFY_MAX_REQUESTS
if not prioritized or max_requests <= 0:
return []
candidates = [
{
"issue_id": f.get("issue_id"),
"severity": f.get("severity"),
"confidence": f.get("confidence"),
"source_stage": f.get("source_stage"),
"description": (f.get("description") or "")[:400],
"evidence": f.get("evidence") or [],
"already_verified": bool(f.get("verification")),
}
for f in prioritized
if f.get("issue_id") and not f.get("verification")
]
if not candidates:
return []
instruction = (
BRAIN_CLARIFY_USER_PROMPT
.replace("{max_requests}", str(max_requests))
.replace("{findings}", json.dumps(candidates, ensure_ascii=True))
)
try:
parsed = call_json(
system_prompt=BRAIN_CLARIFY_SYSTEM_PROMPT,
user_text=instruction,
max_tokens=config.BRAIN_CLARIFY_MAX_TOKENS,
model=config.AGENT_BRAIN_MODEL,
usage_tracker=self.usage,
usage_stage="agent.brain_clarify",
)
except Exception:
return []
raw = parsed.get("requests") if isinstance(parsed, dict) else parsed
if not isinstance(raw, list):
return []
valid_ids = {c["issue_id"] for c in candidates}
requests: List[Dict] = []
seen: set = set()
for item in raw:
if not isinstance(item, dict):
continue
issue_id = item.get("issue_id")
if issue_id not in valid_ids or issue_id in seen:
continue
requests.append({
"issue_id": issue_id,
"request_type": (item.get("request_type") or "verify_evidence").strip(),
"reason": (item.get("reason") or "").strip(),
})
seen.add(issue_id)
if len(requests) >= max_requests:
break
return requests
+111
View File
@@ -0,0 +1,111 @@
"""Per-sheet Drawing Integrity QA agent.
Reads ONE sheet's own extracted objects + sheet image + deterministic text
layer and flags defects internal to that single sheet: dangling detail/
callout/keynote references, schedule-vs-plan/legend disagreements on the same
sheet, dimension strings that do not sum, missing title-block/scale/north
essentials, and duplicate/inconsistent tags. This is the drawing-focused pass
that complements the cross-sheet conflict critic; it never does code/ADA or
cross-sheet coordination.
"""
from typing import Dict, List
from backend import config
from backend.agents.base import AgentResult, AgentScope, AgentUsage, failure
from backend.agents.prompts import (
DRAWING_INTEGRITY_SYSTEM_PROMPT,
DRAWING_INTEGRITY_USER_PROMPT,
)
from backend.llm import call_json
from backend.pipeline._serialize import dumps
from backend.pipeline._stage import collect_list, validate_issue
def _sheet_meta(sheet: Dict) -> Dict:
"""Compact title-block-ish descriptor of the sheet (no raw assertions)."""
return {
"sheet_number": sheet.get("sheet_number"),
"sheet_title": sheet.get("sheet_title"),
"discipline": sheet.get("discipline"),
"drawing_type": sheet.get("drawing_type"),
"level": sheet.get("level"),
"scale": sheet.get("scale"),
}
def build_integrity_scopes(
sheets: List[Dict], page_to_b64: Dict, page_to_text: Dict
) -> List[AgentScope]:
"""One scope per sheet that carries enough objects to judge internal
consistency. Sheets below INTEGRITY_MIN_ASSERTIONS are skipped as too
sparse for a meaningful single-sheet back-check."""
scopes: List[AgentScope] = []
for sheet in sheets:
assertions = sheet.get("assertions") or []
if len(assertions) < config.INTEGRITY_MIN_ASSERTIONS:
continue
page_number = sheet.get("page_number")
scopes.append(AgentScope(
scope_id=f"integrity:{page_number}",
payload={
"sheet": sheet,
"page_number": page_number,
"image_b64": page_to_b64.get(page_number),
"text_layer": page_to_text.get(page_number) or "",
},
))
return scopes
class DrawingIntegrityAgent:
name = "drawing_integrity"
def __init__(self, usage: AgentUsage) -> None:
self.usage = usage
def run(self, scope: AgentScope) -> AgentResult:
try:
sheet = dict(scope.payload["sheet"])
assertions = (
sheet.get("assertions") or []
)[:config.AGENT_INTEGRITY_MAX_ASSERTIONS]
text_layer = (scope.payload.get("text_layer") or "")[
:config.TEXT_LAYER_MAX_CHARS
]
image_b64 = scope.payload.get("image_b64")
images = [image_b64] if image_b64 else []
images = images[:config.AGENT_INTEGRITY_MAX_IMAGES]
instruction = DRAWING_INTEGRITY_USER_PROMPT
for key, value in {
"sheet_meta": dumps(_sheet_meta(sheet)),
"assertions": dumps(assertions),
"text_layer": text_layer,
}.items():
instruction = instruction.replace("{" + key + "}", value)
parsed = call_json(
system_prompt=DRAWING_INTEGRITY_SYSTEM_PROMPT,
user_text=instruction,
images_b64=images,
max_tokens=config.INTEGRITY_MAX_TOKENS,
model=config.AGENT_INTEGRITY_MODEL,
usage_tracker=self.usage,
usage_stage="agent.drawing_integrity",
reasoning_effort=config.EXTRACT_REASONING_EFFORT or None,
reasoning_max_tokens=config.EXTRACT_REASONING_MAX_TOKENS or None,
)
findings = collect_list(
parsed, "issues",
lambda item: validate_issue(item, "drawing_integrity"),
)
sheet_number = sheet.get("sheet_number")
for finding in findings:
finding.update(agent=self.name, scope_id=scope.scope_id)
# Anchor the finding to this sheet if the model left it blank.
if not finding.get("sheets") and sheet_number:
finding["sheets"] = [sheet_number]
return AgentResult(scope_id=scope.scope_id, artifacts=findings)
except Exception as exc:
return failure(scope, exc)
+53 -1
View File
@@ -12,6 +12,42 @@ Sheet index: {sheet_index}
Aggregate sheet summaries: {sheet_summaries}
Cluster summary: {cluster_summary}"""
DRAWING_INTEGRITY_SYSTEM_PROMPT = """You are a Senior Architect performing a single-sheet QAQC back-check of ONE construction drawing before the set is issued for bid, permit, or construction.
You are given the extracted construction objects for this one sheet, plus the sheet image and its deterministic PDF text layer.
Your job is to find problems INTERNAL TO THIS SHEET - defects a human checker would red-line on this drawing by itself, without needing any other sheet.
You are NOT performing code review. You are NOT checking ADA/accessibility. You are NOT doing cross-sheet coordination (a separate reviewer handles conflicts between sheets). You are NOT estimating cost. You are NOT redesigning anything.
What IS a drawing-integrity issue on this sheet:
- Dangling reference: a detail callout, section marker, elevation marker, keynote, or sheet reference that points to a target that does not exist on this sheet AND is not resolved by an explicit off-sheet reference (e.g. "SIM 5/A501" when this is A501 and it has no detail 5; a keynote number called out in the plan but absent from the keynote legend on the same sheet).
- On-sheet contradiction: the plan disagrees with a schedule or legend printed on the SAME sheet; two notes on the sheet contradict each other; a tag in the plan is not in the sheet's own schedule/legend (or vice versa); the title block discipline/level disagrees with the drawing content.
- Dimension sanity: a dimension string whose segments do not sum to the stated overall; an overall dimension that contradicts a repeated/typical dimension on the same sheet; obviously impossible or missing critical dimensions on a dimensioned plan.
- Missing sheet essentials: no scale, no north arrow on a plan that needs one, missing sheet number/title in the title block, a schedule with header columns but no rows, a legend referenced but not present.
- Label/tag hygiene: duplicate tags that should be unique on this sheet (two different doors both tagged 101A), a room shown with no room number/name where the sheet otherwise numbers rooms, inconsistent tag formatting that breaks a reference.
What is NOT a drawing-integrity issue:
- Anything requiring another sheet to judge (that is cross-sheet coordination, handled elsewhere).
- A code, ADA, or accessibility requirement.
- A design preference or cost concern.
- A value simply not repeated where repetition is optional.
- Anything you cannot support with text or a clear visual from THIS sheet.
Be conservative and evidence-bound:
- Only flag defects you can point to with verbatim source_text from this sheet or a clear description of what the image shows.
- Trust the TEXT LAYER for alphanumeric content (numbers, tags, note text, dimensions); use the image for geometry, symbols, linework, and whether a referenced target actually appears.
- When a value is marked DISPUTED (possible extraction misread), verify against the image before relying on it.
- If the sheet is internally clean, return an empty issues array.
Severity (use exactly one of critical, high, medium, low):
- high = a defect that would cause rework, a wrong build, or a stop at permit/bid if issued as-is (missing critical dimension, dangling reference to a nonexistent detail that drives construction).
- medium = a real drawing defect needing correction before issue.
- low = minor cleanup/clarification.
Use plain ASCII only. Respond only with valid JSON."""
DRAWING_INTEGRITY_USER_PROMPT = """Back-check this single sheet for internal drawing-integrity defects.
Respond ONLY with a valid JSON object - no markdown fences, no explanation:
{"issues":[{"issue_id":"string","source_stage":"drawing_integrity","category":"dangling_reference | on_sheet_contradiction | dimension_error | missing_sheet_essential | tag_or_label_error | other","severity":"critical | high | medium | low","confidence":"high | medium | low","location":"where on the sheet, e.g. 'Room 124 / detail callout 5' or 'door schedule'","disciplines":["string"],"sheets":["this sheet number"],"description":"senior architect explanation of the defect and why it matters","evidence":[{"discipline":"string","sheet":"string","source_text":"verbatim text from this sheet","asserted_value":"string"}],"recommended_resolution":"coordinate drawing | correct dimension | add missing detail | issue RFI | verify with architect | verify with engineer","code_reference":null}]}
If the sheet is internally clean, return {"issues":[]}.
Sheet: {sheet_meta}
Extracted objects on this sheet: {assertions}
TEXT LAYER (deterministic page text - authoritative for alphanumeric content):
{text_layer}"""
BRAIN_SYSTEM_PROMPT = """You are the central decision layer for a construction drawing
review. Merge duplicate specialist findings, reject vague or unsupported findings,
preserve verbatim evidence, and prioritize the kept issues. Do not create new issues.
@@ -19,7 +55,23 @@ Conflicts need drawing evidence; completeness findings may instead cite an expli
missing item from the sheet index. Return only valid JSON."""
BRAIN_USER_PROMPT = """Judge and consolidate these scoped specialist findings.
Return {"issues":[{"issue_id":"string","source_stage":"conflict | qaqc | code | constructability","category":"string","severity":"critical | high | medium | low","confidence":"high | medium | low","location":"string","disciplines":["string"],"sheets":["string"],"description":"string","evidence":[{"discipline":"string","sheet":"string","source_text":"string","asserted_value":"string"}],"recommended_resolution":"string","code_reference":"string or null","risk_score":1,"recommended_priority":"immediate | before_bid | before_permit | before_construction | track_only"}],"decisions":[{"finding_refs":["string"],"action":"kept | merged | dropped","reason":"string","kept_issue_id":"string or null"}]}.
Return {"issues":[{"issue_id":"string","source_stage":"conflict | drawing_integrity | qaqc | code | constructability","category":"string","severity":"critical | high | medium | low","confidence":"high | medium | low","location":"string","disciplines":["string"],"sheets":["string"],"description":"string","evidence":[{"discipline":"string","sheet":"string","source_text":"string","asserted_value":"string"}],"recommended_resolution":"string","code_reference":"string or null","risk_score":1,"recommended_priority":"immediate | before_bid | before_permit | before_construction | track_only"}],"decisions":[{"finding_refs":["string"],"action":"kept | merged | dropped","reason":"string","kept_issue_id":"string or null"}]}.
Sheet index: {sheet_index}
Jurisdiction summary: {jurisdiction}
Specialist findings: {findings}"""
BRAIN_CLARIFY_SYSTEM_PROMPT = """You are the central decision layer for a construction drawing review, deciding which of your kept findings you are NOT yet confident enough to publish.
You have already merged and prioritized the findings. Now, for the borderline ones, you may request ONE targeted clarification each before the report is finalized.
Request a clarification only when a finding's evidence is thin, ambiguous, possibly a misread of the drawing, or internally inconsistent - the kind of finding a senior reviewer would double-check against the sheet before signing off. Do NOT request clarification for findings that are already clearly supported by verbatim evidence, and do NOT re-request a finding that already carries a verification result.
The only request type available right now is:
- verify_evidence: re-check this finding's quoted evidence against the actual sheet images and deterministic text layer (catches wave-1 vision misreads such as "(2)" vs "(5)" and dangling references that do not actually appear on the sheet).
Be selective. Requesting everything wastes the budget and slows the review; request only the findings where a second look would actually change your decision.
Use plain ASCII only. Respond only with valid JSON."""
BRAIN_CLARIFY_USER_PROMPT = """Decide which of these kept findings you want to double-check before publishing.
You may request at most {max_requests} clarifications. Choose the findings where a second look at the sheet would most likely change your keep/drop/severity decision.
Respond ONLY with a valid JSON object - no markdown fences, no explanation:
{"requests":[{"issue_id":"the issue_id of the finding to check","request_type":"verify_evidence","reason":"one sentence: why this finding is uncertain"}]}
If every finding is already well supported, return {"requests":[]}.
Findings (each shows issue_id, severity, confidence, evidence, and whether it already has a verification result):
{findings}"""
+180 -38
View File
@@ -18,6 +18,9 @@ from backend.agents.extractors import (
SheetExtractorAgent,
SheetIndexAgent,
)
from backend.agents.integrity_agent import (
DrawingIntegrityAgent, build_integrity_scopes,
)
from backend.agents.linker import LinkerAgent, build_link_scopes, build_object_graph
from backend.agents.memory import ProjectMemory
from backend.agents.orchestrator import Orchestrator
@@ -174,11 +177,23 @@ def run_agent_pipeline(
memory.extend("findings", conflict_findings)
orchestrator.stage("Agent wave 5: scoped specialists")
code_results = orchestrator.run_scopes(
CodeAgent(usage),
build_code_scopes(sheets, jurisdiction, sheet_index),
config.AGENT_SPECIALIST_CONCURRENCY,
)
if config.ENABLE_CODE_REVIEW:
code_results = orchestrator.run_scopes(
CodeAgent(usage),
build_code_scopes(sheets, jurisdiction, sheet_index),
config.AGENT_SPECIALIST_CONCURRENCY,
)
else:
orchestrator.stage("[wave 5] code/ADA review disabled (ENABLE_CODE_REVIEW=0)")
code_results = []
if config.ENABLE_DRAWING_INTEGRITY:
integrity_results = orchestrator.run_scopes(
DrawingIntegrityAgent(usage),
build_integrity_scopes(sheets, page_to_b64, page_to_text),
config.AGENT_INTEGRITY_CONCURRENCY,
)
else:
integrity_results = []
construct_results = orchestrator.run_scopes(
ConstructabilityAgent(usage),
build_construct_scopes(clusters, conflict_findings),
@@ -197,7 +212,8 @@ def run_agent_pipeline(
)
specialist_findings = [
artifact
for result in code_results + construct_results + completeness_results
for result in (code_results + integrity_results
+ construct_results + completeness_results)
for artifact in result.artifacts
]
@@ -210,38 +226,12 @@ def run_agent_pipeline(
severities=config.AGENT_VERIFY_SEVERITIES,
)
target_indexes = {id(f): i for i, f in enumerate(specialist_findings)}
verify_scopes = []
for finding in verify_targets:
cited_pages = [
sheet_to_page[str(name)]
for name in (finding.get("sheets") or [])
if sheet_to_page.get(str(name)) in page_to_b64
]
images = [
page_to_b64[p]
for p in cited_pages[:config.AGENT_CONFLICT_MAX_IMAGES]
]
if not images:
continue # never judge evidence against images we could not load
# Text oracle: concatenated text layer of the cited sheets, capped.
excerpt = "\n\n".join(
f"--- Page {p} ---\n{page_to_text[p]}"
for p in cited_pages
if page_to_text.get(p)
)[:config.VERIFY_TEXT_MAX_CHARS]
if config.VERIFY_HI_DPI_CROPS:
images = _evidence_crops(finding, cited_pages, sheet_to_page,
page_words, page_to_b64, pdf_path,
fallback=images)
verify_scopes.append(AgentScope(
scope_id=f"verify:{target_indexes[id(finding)]}",
payload={
"finding_index": target_indexes[id(finding)],
"finding": finding,
"images_b64": images,
"text_layer_excerpt": excerpt,
},
))
verify_scopes = _build_verify_scopes(
verify_targets,
index_for=lambda f: target_indexes[id(f)],
sheet_to_page=sheet_to_page, page_to_b64=page_to_b64,
page_to_text=page_to_text, page_words=page_words, pdf_path=pdf_path,
)
verify_results = orchestrator.run_scopes(
EvidenceVerifierAgent(usage), verify_scopes, config.AGENT_VERIFY_CONCURRENCY)
suppressed = apply_verdicts(specialist_findings, verify_results)
@@ -286,6 +276,18 @@ def run_agent_pipeline(
1 for decision in decisions if decision.get("action") == "merged"
)
# Wave 6.5 — Brain-directed clarification (bounded hub-and-spoke). The Brain
# names kept findings it is unsure about; verify_evidence requests route
# back through the wave-5b verifier (fresh images + text-layer oracle).
# Refuted findings are demoted, dropped from `prioritized`, and moved to
# memory["suppressed"]. One planning call, one bounded verify wave, no loop.
if config.ENABLE_BRAIN_CLARIFY and prioritized:
prioritized = _brain_clarification_pass(
orchestrator, usage, memory, prioritized,
sheet_to_page=sheet_to_page, page_to_b64=page_to_b64,
page_to_text=page_to_text, page_words=page_words, pdf_path=pdf_path,
)
if require_review:
orchestrator.stage("Agent review gate: build human-review queue")
memory_snapshot = memory.snapshot()
@@ -331,6 +333,10 @@ def run_agent_pipeline(
1 for item in specialist_findings
if item.get("source_stage") == "code"
),
"drawing_integrity": sum(
1 for item in specialist_findings
if item.get("source_stage") == "drawing_integrity"
),
"constructability": sum(
1 for item in specialist_findings
if item.get("source_stage") == "constructability"
@@ -404,6 +410,10 @@ def run_agent_pipeline(
1 for item in specialist_findings
if item.get("source_stage") == "code"
),
"drawing_integrity": sum(
1 for item in specialist_findings
if item.get("source_stage") == "drawing_integrity"
),
"constructability": sum(
1 for item in specialist_findings
if item.get("source_stage") == "constructability"
@@ -440,6 +450,138 @@ def _dump(out_dir: str, name: str, value) -> None:
json.dump(value, f, indent=2)
def _brain_clarification_pass(
orchestrator,
usage,
memory,
prioritized,
sheet_to_page,
page_to_b64,
page_to_text,
page_words,
pdf_path,
):
"""Wave 6.5: let the Brain request targeted clarifications, execute the
verify_evidence ones through the wave-5b verifier, and prune refuted
findings out of `prioritized` into memory["suppressed"].
Bounded and non-looping: one Brain planning call, at most
BRAIN_CLARIFY_MAX_REQUESTS verifications, a single pass. Returns the
(possibly shortened) prioritized list. Any request type other than
verify_evidence is logged as planned-but-not-executed and left untouched.
"""
requests = BrainAgent(usage).plan_clarifications(prioritized)
if not requests:
return prioritized
by_id = {f.get("issue_id"): f for f in prioritized}
verify_findings = []
unsupported = 0
for req in requests:
if req.get("request_type") != "verify_evidence":
unsupported += 1
continue
finding = by_id.get(req.get("issue_id"))
if finding is not None and finding not in verify_findings:
verify_findings.append(finding)
orchestrator.stage(
f"Agent wave 6.5: Brain-directed clarification "
f"({len(verify_findings)} verify, {unsupported} other)"
)
if unsupported:
for req in requests:
if req.get("request_type") != "verify_evidence":
orchestrator.stats.failed_scopes.append(
f"brain_clarify:{req.get('issue_id')}: "
f"request_type '{req.get('request_type')}' planned, "
f"not executed (v1 supports verify_evidence only)"
)
if not verify_findings:
return prioritized
index_of = {id(f): i for i, f in enumerate(prioritized)}
verify_scopes = _build_verify_scopes(
verify_findings,
index_for=lambda f: index_of[id(f)],
sheet_to_page=sheet_to_page, page_to_b64=page_to_b64,
page_to_text=page_to_text, page_words=page_words, pdf_path=pdf_path,
)
if not verify_scopes:
return prioritized
verify_results = orchestrator.run_scopes(
EvidenceVerifierAgent(usage), verify_scopes,
config.AGENT_VERIFY_CONCURRENCY,
)
suppressed = apply_verdicts(prioritized, verify_results)
if suppressed:
suppressed_ids = {id(f) for f in suppressed}
prioritized = [f for f in prioritized if id(f) not in suppressed_ids]
existing = memory.snapshot().get("suppressed") or []
memory.replace("suppressed", existing + suppressed)
memory.extend("decisions", [
{
"finding_refs": [f.get("issue_id")],
"action": "dropped",
"reason": "Brain-directed clarification: evidence refuted on re-check",
"kept_issue_id": None,
}
for f in suppressed
])
return prioritized
def _build_verify_scopes(
targets,
index_for,
sheet_to_page,
page_to_b64,
page_to_text,
page_words,
pdf_path,
):
"""Build EvidenceVerifierAgent scopes for a set of findings.
Shared by wave 5b (severity-gated) and wave 6.5 (Brain-directed): each
finding's cited sheets are mapped to page images (hi-DPI evidence crops
when enabled, else full pages) plus a capped text-layer oracle. Findings
whose sheets resolve to NO loadable image are skipped (I2 guard) — never
judge evidence against images we could not load. index_for(finding) yields
the finding_index the verifier echoes back for apply_verdicts alignment.
"""
scopes = []
for finding in targets:
cited_pages = [
sheet_to_page[str(name)]
for name in (finding.get("sheets") or [])
if sheet_to_page.get(str(name)) in page_to_b64
]
images = [
page_to_b64[p]
for p in cited_pages[:config.AGENT_CONFLICT_MAX_IMAGES]
]
if not images:
continue # never judge evidence against images we could not load
# Text oracle: concatenated text layer of the cited sheets, capped.
excerpt = "\n\n".join(
f"--- Page {p} ---\n{page_to_text[p]}"
for p in cited_pages
if page_to_text.get(p)
)[:config.VERIFY_TEXT_MAX_CHARS]
if config.VERIFY_HI_DPI_CROPS:
images = _evidence_crops(finding, cited_pages, sheet_to_page,
page_words, page_to_b64, pdf_path,
fallback=images)
finding_index = index_for(finding)
scopes.append(AgentScope(
scope_id=f"verify:{finding_index}",
payload={
"finding_index": finding_index,
"finding": finding,
"images_b64": images,
"text_layer_excerpt": excerpt,
},
))
return scopes
def _evidence_crops(
finding: Dict,
cited_pages: list,
+9
View File
@@ -36,6 +36,15 @@ def _valid_verdict(item):
def _status(verdicts):
"""Roll per-evidence verdicts up to a finding-level status.
NOTE on "corrected": it is deliberately NON-confirming. The canonical case
(job 959e16407573) is evidence quoting "(2) 2x6 STUD PACK" against a sheet
that reads "(5)" — the text exists but the VALUE the finding rests on was a
wave-1 misread, so the finding's basis is gone. Hence refuted = zero
CONFIRMED verdicts, not zero not_found ones. Do not "fix" this to treat
corrected as supporting; see tests/agents/test_verifier.py.
"""
if not verdicts:
return "unverified"
confirmed = sum(1 for v in verdicts if v["verdict"] == "confirmed")
+51 -4
View File
@@ -12,6 +12,15 @@ load_dotenv(os.path.join(os.path.dirname(os.path.abspath(__file__)), ".env"))
_BASE_DIR = os.path.dirname(os.path.abspath(__file__))
_TRUTHY = ("1", "true", "yes", "on")
def _flag(name: str, default: str) -> bool:
"""Parse a boolean env knob. Accepts 1/true/yes/on (case-insensitive) so a
knob set to "1" behaves the same as one set to "true" — mixing bare
`== "true"` comparisons with this set silently disabled features."""
return os.getenv(name, default).strip().lower() in _TRUTHY
# -- AI Backend (OpenRouter) ----------------------------------------
# One multimodal model does both extraction (Stage 1) and conflict
# reasoning (Stage 3). Override MODEL per-stage if you ever split them.
@@ -47,6 +56,33 @@ AGENT_CONFLICT_CONCURRENCY = int(os.getenv("AGENT_CONFLICT_CONCURRENCY", "4"))
AGENT_SPECIALIST_CONCURRENCY = int(os.getenv("AGENT_SPECIALIST_CONCURRENCY", "4"))
AGENT_RFI_CONCURRENCY = int(os.getenv("AGENT_RFI_CONCURRENCY", "4"))
# -- Review focus toggles -------------------------------------------
# ENABLE_CODE_REVIEW gates the code/ADA/jurisdiction review path in BOTH
# pipelines. Default OFF: the product's focus is drawing-integrity and
# cross-discipline coordination, not code/accessibility compliance. When
# False the CodeAgent wave (agent) and the Code/ADA stage (classic) are
# skipped entirely, by_stage.code reports 0, and nothing in the ADA corpus
# or jurisdiction meta is deleted so the path can be re-enabled with one env
# flag. Set ENABLE_CODE_REVIEW=1 to restore code/ADA findings.
ENABLE_CODE_REVIEW = _flag("ENABLE_CODE_REVIEW", "false")
# Per-sheet Drawing Integrity QA wave (agent + classic). This is the
# drawing-focused pass: it reads ONE sheet's own objects + image + text layer
# and flags problems internal to that sheet -- dangling detail/callout/keynote
# references, schedule-vs-plan or legend disagreements on the same sheet,
# dimension strings that do not sum, missing title-block/scale/north-arrow,
# and notes that contradict each other. It complements (does not replace) the
# cross-sheet conflict critic. Default ON.
ENABLE_DRAWING_INTEGRITY = _flag("ENABLE_DRAWING_INTEGRITY", "true")
AGENT_INTEGRITY_MODEL = os.getenv("AGENT_INTEGRITY_MODEL", "") or MODEL
AGENT_INTEGRITY_CONCURRENCY = int(os.getenv("AGENT_INTEGRITY_CONCURRENCY", "4"))
AGENT_INTEGRITY_MAX_IMAGES = int(os.getenv("AGENT_INTEGRITY_MAX_IMAGES", "1"))
AGENT_INTEGRITY_MAX_ASSERTIONS = int(os.getenv("AGENT_INTEGRITY_MAX_ASSERTIONS", "80"))
INTEGRITY_MAX_TOKENS = int(os.getenv("INTEGRITY_MAX_TOKENS", "16384"))
# Skip sheets with fewer than this many extracted objects -- too sparse for a
# meaningful internal-consistency pass (avoids burning a call on near-empty pages).
INTEGRITY_MIN_ASSERTIONS = int(os.getenv("INTEGRITY_MIN_ASSERTIONS", "3"))
# Wave 5b evidence verification (vision fact-check of cited sheet text)
AGENT_VERIFY_MODEL = os.getenv("AGENT_VERIFY_MODEL", "") or MODEL
AGENT_VERIFY_CONCURRENCY = int(os.getenv("AGENT_VERIFY_CONCURRENCY", "4"))
@@ -59,6 +95,17 @@ AGENT_VERIFY_SEVERITIES = {
AGENT_VERIFY_REASONING_EFFORT = os.getenv("AGENT_VERIFY_REASONING_EFFORT", "low").strip()
VERIFY_MAX_TOKENS = int(os.getenv("VERIFY_MAX_TOKENS", "8192"))
# Wave 6.5 Brain-directed clarification. After the Brain merge, the Brain may
# name findings it is unsure about and emit typed clarification requests; v1
# executes verify_evidence requests by routing them back through the wave-5b
# EvidenceVerifierAgent (fresh page images + hi-DPI evidence crops + text-layer
# oracle). Bounded: one planning call, at most BRAIN_CLARIFY_MAX_REQUESTS
# verifications, a single iteration. Reuses AGENT_VERIFY_* / VERIFY_* knobs for
# the verification calls. Default ON.
ENABLE_BRAIN_CLARIFY = _flag("ENABLE_BRAIN_CLARIFY", "true")
BRAIN_CLARIFY_MAX_REQUESTS = int(os.getenv("BRAIN_CLARIFY_MAX_REQUESTS", "8"))
BRAIN_CLARIFY_MAX_TOKENS = int(os.getenv("BRAIN_CLARIFY_MAX_TOKENS", "4096"))
# -- Text-layer grounding (deterministic PDF text layer via PyMuPDF) ----
# The vector text layer is extracted once per job and grounds the extractor,
# rescues misquoted-but-real values in the grounding guard, and serves the
@@ -118,8 +165,8 @@ EXTRACT_REASONING_MAX_TOKENS = int(os.getenv("EXTRACT_REASONING_MAX_TOKENS", "20
# structuring pass (rung 2), then deterministic text-layer fallback stubs
# (rung 3) so no text-bearing page goes dark.
EXTRACT_COVERAGE_FLOOR = float(os.getenv("EXTRACT_COVERAGE_FLOOR", "0.6"))
EXTRACT_TEXT_RETRY_ENABLED = os.getenv("EXTRACT_TEXT_RETRY_ENABLED", "true").lower() == "true"
EXTRACT_FALLBACK_ENABLED = os.getenv("EXTRACT_FALLBACK_ENABLED", "true").lower() == "true"
EXTRACT_TEXT_RETRY_ENABLED = _flag("EXTRACT_TEXT_RETRY_ENABLED", "true")
EXTRACT_FALLBACK_ENABLED = _flag("EXTRACT_FALLBACK_ENABLED", "true")
EXTRACT_FALLBACK_MAX_OBJECTS = int(os.getenv("EXTRACT_FALLBACK_MAX_OBJECTS", "200"))
REASON_MAX_TOKENS = int(os.getenv("REASON_MAX_TOKENS", "4096"))
@@ -191,5 +238,5 @@ SMTP_PORT = int(os.getenv("SMTP_PORT", "587"))
SMTP_USER = os.getenv("SMTP_USER", "")
SMTP_PASSWORD = os.getenv("SMTP_PASSWORD", "")
SMTP_FROM = os.getenv("SMTP_FROM", "")
SMTP_USE_TLS = os.getenv("SMTP_USE_TLS", "true").lower() == "true"
SMTP_USE_SSL = os.getenv("SMTP_USE_SSL", "false").lower() == "true"
SMTP_USE_TLS = _flag("SMTP_USE_TLS", "true")
SMTP_USE_SSL = _flag("SMTP_USE_SSL", "false")
+88
View File
@@ -0,0 +1,88 @@
"""
drawing_integrity.py - Per-sheet Drawing Integrity QA (LLM, classic pipeline).
The drawing-focused pass: reads ONE sheet's own extracted objects + sheet image
+ deterministic text layer and flags defects internal to that single sheet
(dangling detail/callout/keynote references, schedule-vs-plan/legend
disagreements on the same sheet, dimension strings that do not sum, missing
title-block/scale essentials, duplicate/inconsistent tags). It complements the
cross-sheet conflict checker; it never does code/ADA or cross-sheet
coordination. Emits the canonical issue schema. Returns [] on failure.
Gated by config.ENABLE_DRAWING_INTEGRITY. Runs sheets concurrently, one call
per sheet, skipping sheets below INTEGRITY_MIN_ASSERTIONS.
"""
from concurrent.futures import ThreadPoolExecutor
from typing import Dict, List
from backend import config
from backend.agents.prompts import (
DRAWING_INTEGRITY_SYSTEM_PROMPT,
DRAWING_INTEGRITY_USER_PROMPT,
)
from backend.pipeline._serialize import dumps
from backend.pipeline._stage import call_stage, collect_list, validate_issue
def _sheet_meta(sheet: Dict) -> Dict:
return {
"sheet_number": sheet.get("sheet_number"),
"sheet_title": sheet.get("sheet_title"),
"discipline": sheet.get("discipline"),
"drawing_type": sheet.get("drawing_type"),
"level": sheet.get("level"),
"scale": sheet.get("scale"),
}
def _review_sheet(sheet: Dict, page_to_b64: Dict, page_to_text: Dict) -> List[Dict]:
page_number = sheet.get("page_number")
assertions = (sheet.get("assertions") or [])[
:config.AGENT_INTEGRITY_MAX_ASSERTIONS
]
text_layer = (page_to_text.get(page_number) or "")[
:config.TEXT_LAYER_MAX_CHARS
]
image = page_to_b64.get(page_number)
images = [image][:config.AGENT_INTEGRITY_MAX_IMAGES] if image else []
parsed = call_stage(
DRAWING_INTEGRITY_SYSTEM_PROMPT,
DRAWING_INTEGRITY_USER_PROMPT,
subs={
"sheet_meta": dumps(_sheet_meta(sheet)),
"assertions": dumps(assertions),
"text_layer": text_layer,
},
images_b64=images,
max_tokens=config.INTEGRITY_MAX_TOKENS,
)
issues = collect_list(
parsed, "issues", lambda c: validate_issue(c, "drawing_integrity")
)
sheet_number = sheet.get("sheet_number")
for issue in issues:
if not issue.get("sheets") and sheet_number:
issue["sheets"] = [sheet_number]
return issues
def drawing_integrity_review(sheets: List[Dict], pages: List[Dict]) -> List[Dict]:
"""One LLM call per non-sparse sheet, run concurrently."""
if not config.ENABLE_DRAWING_INTEGRITY:
return []
page_to_b64 = {p["page_number"]: p.get("base64") for p in pages}
page_to_text = {p["page_number"]: p.get("text_layer") for p in pages}
targets = [
s for s in sheets
if len(s.get("assertions") or []) >= config.INTEGRITY_MIN_ASSERTIONS
]
issues: List[Dict] = []
if targets:
with ThreadPoolExecutor(max_workers=config.AGENT_INTEGRITY_CONCURRENCY) as pool:
for res in pool.map(
lambda s: _review_sheet(s, page_to_b64, page_to_text), targets
):
issues.extend(res)
print(f"[DrawingIntegrity] {len(issues)} issue(s) across {len(targets)} sheet(s)")
return issues
+14 -3
View File
@@ -39,6 +39,7 @@ from backend.agents.disputes import annotate_clusters
from backend.pipeline.conflict_checker import check_conflicts
from backend.pipeline.qaqc_review import senior_review
from backend.pipeline.code_review import code_review
from backend.pipeline.drawing_integrity import drawing_integrity_review
from backend.pipeline.constructability import constructability_review
from backend.pipeline.validator import dedup_validate
from backend.pipeline.risk import score_and_prioritize
@@ -158,15 +159,23 @@ def _run_stages(
stage("Full-set QAQC review")
qaqc_issues = senior_review(sheets, clusters, conflicts, sheet_index)
stage("Code / ADA review")
code_issues = code_review(jurisdiction, sheets, sheet_index)
if config.ENABLE_CODE_REVIEW:
stage("Code / ADA review")
code_issues = code_review(jurisdiction, sheets, sheet_index)
else:
print("[Code] code/ADA review disabled (ENABLE_CODE_REVIEW=0)")
code_issues = []
stage("Drawing integrity (per-sheet QA)")
integrity_issues = drawing_integrity_review(sheets, pages)
stage("Constructability review")
construct_issues = constructability_review(sheets, clusters, conflicts)
stage("Validate & deduplicate")
conflict_issues = [v for v in (validate_issue(c, "conflict") for c in conflicts) if v]
all_issues = conflict_issues + qaqc_issues + code_issues + construct_issues
all_issues = (conflict_issues + integrity_issues + qaqc_issues
+ code_issues + construct_issues)
validated = dedup_validate(all_issues)
stage("Risk scoring & prioritization")
@@ -189,6 +198,7 @@ def _run_stages(
"conflicts": len(conflicts),
"qaqc": len(qaqc_issues),
"code": len(code_issues),
"drawing_integrity": len(integrity_issues),
"constructability": len(construct_issues),
"validated": len(validated),
"rfis": len(rfis),
@@ -213,6 +223,7 @@ def _run_stages(
_dump(out_dir, "project_intelligence.json", project_intel)
_dump(out_dir, "qaqc_issues.json", qaqc_issues)
_dump(out_dir, "code_issues.json", code_issues)
_dump(out_dir, "drawing_integrity.json", integrity_issues)
_dump(out_dir, "constructability.json", construct_issues)
_dump(out_dir, "validated_issues.json", prioritized)
_dump(out_dir, "rfis.json", rfis)
+10 -10
View File
@@ -410,24 +410,24 @@ Normalized assertions: {normalized_assertions}"""
# ---------------------------------------------------------------------------
CONFLICT_SYSTEM_PROMPT = """You are a Senior Architect and construction-drawing coordination reviewer doing a back-check of a drawing set BEFORE it is issued for bid, permit, or construction.
You are given clustered facts that multiple disciplines have asserted about the same location or element.
Decide whether these disciplines GENUINELY CONTRADICT each other - the kind of issue a human coordinator would issue as a QAQC comment or RFI before the set goes out.
You are given clustered facts asserted about the same location or element. Those facts may come from MULTIPLE disciplines, from a SINGLE discipline across several sheets, or from ONE sheet (plan vs schedule vs detail vs keynote on that sheet).
Decide whether these facts GENUINELY CONTRADICT each other - the kind of issue a human coordinator would issue as a QAQC comment or RFI before the set goes out. A contradiction between two facts is a conflict whether or not the two facts come from different disciplines.
You are NOT performing code review in this stage. You are NOT checking ADA in this stage. You are NOT estimating cost or scope. You are NOT rewriting the drawings.
What IS a conflict:
- Two disciplines state different values for the same physical quantity at the same place.
- An element is shown in different locations by different disciplines.
- A schedule disagrees with what is drawn on the plan.
- Two facts state different values for the same physical quantity at the same place (across disciplines, across sheets of one discipline, or on the same sheet).
- An element is shown in different locations by different facts.
- A schedule disagrees with what is drawn on the plan (even on the same sheet).
- A detail disagrees with the plan.
- A keynote disagrees with a schedule, plan, or detail.
- A keynote or general note disagrees with a schedule, plan, detail, or legend - including a keynote/legend mismatch on a single sheet.
- A callout, detail reference, section marker, or tag references something that does not exist (a dangling reference).
- The same room, door, equipment, wall, or utility is labeled or dimensioned inconsistently across sheets or within one sheet.
- An element required by one discipline has no counterpart where another discipline should show it.
- A duct, pipe, conduit, or piece of equipment conflicts with structure, ceiling height, rated wall, or required clearance.
- Equipment shown by one discipline lacks required power, plumbing, ventilation, access, or support in another discipline.
- Demolition drawings remove something that new work drawings keep without explanation.
- A callout, keynote, or tag references something that does not exist.
- The same room, door, equipment, wall, or utility is labeled inconsistently across sheets.
What is NOT a conflict:
- Two disciplines describing different, compatible aspects of the same place.
- A value shown on one discipline and simply not repeated on another, unless that discipline is expected to show it.
- Two facts describing different, compatible aspects of the same place.
- A value shown once and simply not repeated elsewhere, unless another sheet or discipline is expected to show it.
- Rounding or representation differences that resolve to the same real value.
- A possible code issue.
- A design preference.
+6 -1
View File
@@ -119,7 +119,12 @@ def merge_objects(vision_objs: List[Dict], text_objs: List[Dict]) -> List[Dict]:
return merged
_SHEET_ID_RE = re.compile(r"\b([A-Z]{1,2}\d{2,3}(?:\.\d+)?)\b")
# Sheet ids: 1-2 letters, OPTIONAL HYPHEN, 2-3 digits, optional decimal suffix.
# The hyphen matters: civil/landscape sets number sheets C-001 / L-101, and a
# regex without it leaves those pages sheet_number=None, which then shows up as
# a false "declared but not in set" in sheet_reconcile. Kept in sync with
# sheet_reconcile._SHEET_TOKEN_RE.
_SHEET_ID_RE = re.compile(r"\b([A-Z]{1,2}-?\d{2,3}(?:\.\d+)?)\b")
def recover_sheet_number(page_text: str) -> Optional[str]: