John Wilganowski
1c1d2ff21b
Agent web jobs now stop after Brain consolidation and enter needs_review with a persisted review queue (blocking: high-severity, low-confidence, sensitive-category findings; audit sample of clean clusters). Humans decide confirm/reject/unsure/needs_clarification via new review API and frontend queue; a finalizer applies decisions (rejections suppressed with reason codes), performs bounded targeted reruns for clarifications, drafts RFIs only for kept issues, and only then marks the job done and sends the final email. Two-phase email (review-required, then final report), per-decision feedback labels with redacted aggregate metrics, restart recovery from job artifacts, and CLI --no-review bypass. Classic pipeline unchanged. 65 non-LLM tests.
254 lines
10 KiB
Markdown
254 lines
10 KiB
Markdown
# Conflict Checker
|
|
|
|
Cross-discipline **design contradiction** detector for construction drawing sets.
|
|
Built for architects to back-check a set (Arch / Struct / Mech / Elec / Plumb / FP / …)
|
|
**before** it goes out to a GC for bid. It flags information that *disagrees between
|
|
disciplines* — e.g. the RCP ceiling height vs the mechanical duct elevation, a door
|
|
schedule count vs the floor plan, a column that lands in a corridor.
|
|
|
|
> This is **not** the IronBid scope-ownership conflict checker (who-owns-what for bidding).
|
|
> That stays in IronBid; this tool is a separate, design-QA pipeline.
|
|
|
|
## How it works (hybrid extract-then-compare)
|
|
|
|
```
|
|
PDF -> page images -> per-sheet grounded facts -> cluster by location -> reason -> report
|
|
Stage 0 Stage 1 (vision) Stage 2 (no LLM) Stage 3 Stage 4
|
|
```
|
|
|
|
1. **Stage 0** `pdf_processor` — PDF pages to base64 JPEGs (100 DPI, 2400px cap).
|
|
2. **Stage 1** `extractor` — one vision call per sheet extracts discrete, **grounded**
|
|
assertions (each carries verbatim `source_text` and a `location_key`). A grounding
|
|
guard drops any assertion whose number isn't in its own source text.
|
|
3. **Stage 2** `clusterer` — deterministic: groups assertions that refer to the same
|
|
grid / room / tag across **≥2 disciplines** (or a schedule-vs-plan within one). This
|
|
keeps Stage 3 calls small and few.
|
|
4. **Stage 3** `conflict_checker` — one reasoning call per cluster (with the relevant
|
|
sheet images) decides whether the disciplines genuinely contradict, and classifies it.
|
|
5. **Stage 4** `report` — `conflicts.json` + a readable `report.md`.
|
|
|
|
The model is **`google/gemini-2.5-pro`** via OpenRouter (configurable). **All prompt
|
|
tuning lives in `backend/prompts.py`** — the system prompts, the attribute vocabulary,
|
|
and the conflict taxonomy.
|
|
|
|
## Setup
|
|
|
|
### Docker (recommended)
|
|
|
|
```bash
|
|
cp backend/.env.example backend/.env # set AI_API_KEY (and SMTP if you want email)
|
|
docker compose up -d --build
|
|
# -> http://localhost:8099
|
|
```
|
|
|
|
Check health: `curl http://localhost:8099/health`
|
|
|
|
Logs: `docker compose logs -f app`
|
|
|
|
CLI inside the container (mount your PDF read-only):
|
|
|
|
```bash
|
|
docker compose run --rm \
|
|
-v "$(pwd)/samples/your_set.pdf:/data/set.pdf:ro" \
|
|
app python cli/run_check.py /data/set.pdf --out /app/backend/outputs/cli-run
|
|
```
|
|
|
|
Persistent data lives in Docker volumes (`uploads`, `outputs`, `llm_cache`). To bind
|
|
mount host directories instead, replace the named volumes in `docker-compose.yml`.
|
|
|
|
For hybrid mode (local vLLM on the host), set `LOCAL_BASE_URL=http://host.docker.internal:8000/v1`
|
|
in `backend/.env`. Compose already maps `host.docker.internal` to the host gateway on Linux.
|
|
|
|
### Local Python
|
|
|
|
```bash
|
|
python3 -m venv .venv && . .venv/bin/activate
|
|
pip install -r requirements.txt # needs system poppler-utils for pdf2image
|
|
cp backend/.env.example backend/.env # then set AI_API_KEY to your OpenRouter key
|
|
```
|
|
|
|
## Repository (Gitea)
|
|
|
|
Source of truth is Scout IT's Gitea instance:
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Remote** | `https://gitea.scoutitsystems.com/woogi/Conflict_Checker.git` |
|
|
| **Web UI** | https://gitea.scoutitsystems.com/woogi/Conflict_Checker |
|
|
|
|
Clone:
|
|
|
|
```bash
|
|
git clone https://gitea.scoutitsystems.com/woogi/Conflict_Checker.git
|
|
cd Conflict_Checker
|
|
```
|
|
|
|
This repo's git config uses a dedicated credential store at
|
|
`~/.config/git/gitea-credentials` (not your global GitHub credentials). If push/pull
|
|
prompts for auth, log in via the Gitea web UI and create a personal access token, then
|
|
store it there or run one authenticated push so the helper saves it.
|
|
|
|
## CI/CD (Gitea Actions → Container Registry)
|
|
|
|
Develop locally, push to Gitea, and let a runner build/publish the Docker image.
|
|
|
|
### One-time setup
|
|
|
|
1. **Enable Actions** on the repo: Settings → Actions → Enable Repository Actions.
|
|
2. **Runner** — an `act_runner` registered against `gitea.scoutitsystems.com`
|
|
with the `self-hosted` label and access to a Docker daemon (typically
|
|
`/var/run/docker.sock` mounted into the runner). This instance uses
|
|
`sits-docker-runner` (`self-hosted`, `linux`, `docker`).
|
|
3. **Registry secret** — create a Personal Access Token on Gitea with at least
|
|
`write:package` (and `write:release` if you use version tags). Add it as a
|
|
repository secret named **`REGISTRY_TOKEN`** (Settings → Actions → Secrets).
|
|
|
|
### What runs automatically
|
|
|
|
| Trigger | Result |
|
|
|---------|--------|
|
|
| Push to `main` | Image pushed as `:latest` and `:sha-<commit>` |
|
|
| Push tag `v*` (e.g. `v0.1.0`) | Image tagged with the release + Gitea Release created |
|
|
|
|
Image location:
|
|
|
|
```text
|
|
gitea.scoutitsystems.com/woogi/conflict-checker:<tag>
|
|
```
|
|
|
|
Workflow file: `.gitea/workflows/docker-release.yml`
|
|
|
|
### Local dev → deploy loop
|
|
|
|
```bash
|
|
# 1. Develop and test locally (venv or docker compose build)
|
|
git add -A && git commit -m "your change"
|
|
git push origin main # CI builds :latest
|
|
|
|
# 2. Cut a release when ready
|
|
git tag v0.1.0 && git push origin v0.1.0
|
|
|
|
# 3. On the deploy host — pull and run the published image
|
|
docker login gitea.scoutitsystems.com
|
|
IMAGE_TAG=v0.1.0 docker compose -f docker-compose.prod.yml up -d
|
|
# or: IMAGE_TAG=latest for the newest main-branch build
|
|
```
|
|
|
|
Copy `backend/.env` to the deploy host separately (never commit it). Use
|
|
`docker-compose.prod.yml` for production; keep `docker-compose.yml` for local
|
|
builds from source.
|
|
|
|
## Run
|
|
|
|
CLI (the fast tuning loop — also dumps `assertions.json` / `clusters.json` for inspection):
|
|
|
|
```bash
|
|
python cli/run_check.py samples/your_set.pdf --out out/your_set
|
|
# -> out/your_set/report.md + conflicts.json
|
|
```
|
|
|
|
The Classic pipeline remains the recommended default. The experimental Agent
|
|
fork runs in the same image and can be selected in the web UI or from the CLI:
|
|
|
|
```bash
|
|
python cli/run_check.py samples/your_set.pdf --mode agent --out out/agent-run
|
|
```
|
|
|
|
Agent mode uses OpenRouter for every model call and runs bounded specialist
|
|
waves: one-sheet extraction, sheet-index/jurisdiction orientation, semantic
|
|
linkers partitioned by level and object family, per-cluster conflict critics,
|
|
batched code review, cluster-scoped constructability, summary-only completeness,
|
|
central Brain consolidation, and one-finding RFI writers. It returns the same
|
|
`conflicts`, `validated_issues`, `rfis`, and `summary` fields as Classic.
|
|
|
|
Agent artifacts are also written under `<output>/agent/`, including wave
|
|
snapshots and the final Project Memory. `summary.agent_stats`,
|
|
`summary.cost_by_stage`, and `summary.models_used` are job-local, so concurrent
|
|
Agent jobs do not share accounting.
|
|
|
|
Optional `AGENT_*_MODEL` variables select an OpenRouter model per specialist.
|
|
The `AGENT_*_CONCURRENCY` and scope-cap variables in `backend/.env.example`
|
|
bound fan-out and prompt size. Agent mode intentionally ignores the hybrid/local
|
|
text option in v1.
|
|
|
|
### Agent mode: required human review
|
|
|
|
By default (`AGENT_REQUIRE_REVIEW=true`) an Agent run **stops after the Brain
|
|
consolidation wave** and waits for a human before anything ships:
|
|
|
|
```
|
|
Brain merge -> needs_review -> review UI (/?job=<id>) -> finalize -> final report
|
|
```
|
|
|
|
The job lifecycle adds review states: `needs_review` (queue built, waiting),
|
|
`reviewing` (decisions submitted), `finalizing` (targeted reruns + RFI writers
|
|
running), then `done` — or `finalization_error` if finalization fails. Open the
|
|
job in the web UI to work the queue: blocking items (high/critical severity,
|
|
low confidence, sensitive categories) must be decided; clean-cluster items are
|
|
non-blocking spot-checks.
|
|
|
|
Email is **two-phase**: a "review required" notice goes out when the job enters
|
|
`needs_review` (with a link to the review UI); the final conflict report email
|
|
is only sent after finalization completes. The unreviewed report never leaves
|
|
the server.
|
|
|
|
**Privacy boundary:** all review artifacts (queue, decisions, final report) are
|
|
job-local under `outputs/<job_id>/review/`. Cross-job review-feedback
|
|
aggregation, when built, excludes verbatim `source_text`, images, and comments
|
|
unless `REVIEW_AGGREGATE_INCLUDE_TEXT=true`.
|
|
|
|
Config knobs (see `backend/.env.example`):
|
|
|
|
| Key | Default | Effect |
|
|
|-----|---------|--------|
|
|
| `AGENT_REQUIRE_REVIEW` | `true` | `false` = Agent jobs skip the gate entirely (old behavior: RFIs, final report, one email) |
|
|
| `AGENT_REVIEW_AUDIT_SAMPLE` | `5` | Max clean clusters added to the queue as spot-checks |
|
|
| `REVIEW_AGGREGATE_INCLUDE_TEXT` | `false` | Allow future aggregate feedback to include source text/images/comments |
|
|
|
|
From the CLI, `--no-review` bypasses the gate for that run (it overrides
|
|
`AGENT_REQUIRE_REVIEW=true`):
|
|
|
|
```bash
|
|
python cli/run_check.py samples/your_set.pdf --mode agent --no-review --out out/agent-run
|
|
```
|
|
|
|
**Deployment note:** the review endpoints (`/jobs/{id}/review-decisions`,
|
|
`/jobs/{id}/finalize-review`) are **state-changing and sensitive** — they accept
|
|
human decisions that alter the final report. Do **not** expose the UI/API
|
|
publicly without reverse-proxy auth or a shared access token in front of it.
|
|
|
|
Web UI (upload + view):
|
|
|
|
```bash
|
|
uvicorn backend.main:app --reload --port 8099 # open http://127.0.0.1:8099
|
|
```
|
|
|
|
Or use Docker: `docker compose up -d` (see **Setup** above).
|
|
|
|
The standard Docker image contains both pipelines; no additional queue,
|
|
database, or model service is required. Set `AI_API_KEY` in `backend/.env` and
|
|
choose Agent mode per request. Treat Agent output as experimental and compare it
|
|
against a reviewed golden set before using it for issuance decisions.
|
|
|
|
## Conflict categories
|
|
|
|
`dimensional_disagreement`, `elevation_disagreement`, `location_mismatch`,
|
|
`missing_element`, `schedule_vs_plan_mismatch`, `tag_or_reference_inconsistency`,
|
|
`spatial_clash`, `note_or_spec_contradiction`.
|
|
|
|
## Tuning
|
|
|
|
- Edit prompts/vocab/taxonomy in `backend/prompts.py`.
|
|
- Inspect `out/<set>/assertions.json` to see what Stage 1 extracted, and
|
|
`out/<set>/clusters.json` to see what got compared. Most false negatives are a
|
|
clustering/extraction miss; most false positives are a Stage-3 prompt issue.
|
|
- Build a golden set: hand-verify `conflicts.json` for a known set and diff future runs.
|
|
|
|
## Known limitations (v1)
|
|
|
|
- A location appearing in only **one** discipline never forms a cluster, so pure
|
|
"missing element" gaps aren't caught deterministically (Stage 3 still catches missing
|
|
counterparts when the location is co-located).
|
|
- Stage 3 currently sends **full pages**, not crops. If accuracy/cost needs it, add
|
|
crop-by-grid later.
|