Add required human review gate to the Agent pipeline #1

Open
woogi wants to merge 28 commits from agent-mode into main
11 changed files with 699 additions and 150 deletions
Showing only changes of commit f7e1b6bb7c - Show all commits
+160
View File
@@ -0,0 +1,160 @@
# Session Notes — Conflict Checker
Orientation for a new coding session. Setup and Docker details live in [README.md](README.md). This file tracks what the code actually does and what tends to waste time.
**Last updated:** 2026-08-02 · `agent-mode` branch (merged `main` tip `bf508bf`)
## What this is
Cross-discipline **design contradiction / senior-architect QAQC** for construction drawing PDFs (Arch, Struct, Mech, Elec, Plumb, FP, etc.). Flags disagreements *between disciplines* before a set goes to bid/permit.
**Not** IronBids scope-ownership conflict checker.
Source of truth: Scout IT Gitea — `gitea.scoutitsystems.com/woogi/Conflict_Checker`.
## Stack
| Layer | Detail |
|-------|--------|
| API | Python 3.12, FastAPI, Uvicorn ([backend/main.py](backend/main.py)) |
| UI | Single static file [frontend/index.html](frontend/index.html), served by FastAPI |
| Pipelines | **Classic:** [backend/pipeline/runner.py](backend/pipeline/runner.py) (shared by web + CLI). **Agent:** [backend/agents/runner.py](backend/agents/runner.py) — experimental scoped specialist agents, selected per job (`pipeline_mode`) |
| Review gate | Agent jobs stop at `needs_review` for human decisions before the report emails ([backend/review/](backend/review/)) |
| LLM | OpenRouter via `openai` SDK; default `google/gemini-2.5-pro`. Vision always OpenRouter; text stages can use local vLLM (classic/hybrid only) |
| PDF | `pdf2image` + system `poppler-utils` → JPEG page images |
| Jobs | In-memory threads ([backend/jobs.py](backend/jobs.py)) — no Redis/DB |
| Tests | `pytest tests/` (~82 tests; see Quick start) |
| Deploy | Docker Compose; app on port **8099**; public URL `https://conchecker.scoutitsystems.com` |
## Live pipeline (authoritative)
README still describes an older 5-stage extract-then-compare loop. **Trust `runner.py`.** Actual flow:
```
PDF → images → extract → sheet index → jurisdiction
→ normalize → project intelligence (GOIDs)
→ cluster → conflict reason
→ QAQC / code / constructability
→ dedup-validate → risk → RFIs → report
```
| Runner stage | Module | Notes |
|--------------|--------|--------|
| PDF → images | `pdf_processor` | Rasterize |
| Extract assertions | `extractor` | Vision, per sheet |
| Classify sheet index | `sheet_index` | LLM |
| Jurisdiction profile | `jurisdiction` | After cover meta |
| Normalize | `normalizer` | LLM batches |
| Project intelligence | `normalizer.build_project_intelligence` | GOIDs + relationships |
| Cluster | `llm_clusterer` or `clusterer` | Default `CLUSTERER=llm` |
| Conflicts | `conflict_checker` | Per-cluster vision reason |
| QAQC / code / constructability | `qaqc_review`, `code_review`, `constructability` | Full-set / batched |
| Validate & dedup | `validator` | Merges conflict + QAQC + code + construct issues |
| Risk / RFIs | `risk`, `rfi` | Text-only |
| Report | `report` | `conflicts.json` + `report.md` (+ stage JSON dumps when `out_dir` set) |
Design notes for Stage 2/3 engines also live under `Changes/*.docx`.
**Agent pipeline** (`pipeline_mode=agent`): scoped specialist agents in [backend/agents/](backend/agents/) (extractors, linker, brain, critics, RFI writer) run through `Orchestrator` + `ProjectMemory`; artifacts under `outputs/<job_id>/agent/`. Agent mode is OpenRouter-only (no hybrid) and, when `AGENT_REQUIRE_REVIEW=true`, stops at `needs_review` until a human saves decisions and finalizes via the review endpoints. Design docs: `docs/superpowers/`.
## HTTP API (current)
| Method | Path | Purpose |
|--------|------|---------|
| GET | `/health` | Liveness + `model` / `text_model` + `version` / `build` + key/email flags |
| GET | `/models` | OpenRouter catalog split into `vision[]` / `text[]` + `defaults`, with per-1M-token pricing (cached ~1h; **502** when OpenRouter is unreachable) |
| POST | `/check` | Upload PDF; returns `{job_id}` immediately |
| GET | `/jobs/{id}` | Status poll. Running: `stage` + `log_tail`. Done/error/needs_review/finalization_error: `report` and/or `error` + full `log` |
| GET | `/jobs/{id}/log` | Full run log as `text/plain` (404 when no log file) |
| GET | `/jobs/{id}/review` | Review queue + progress + saved decisions (agent jobs) |
| POST | `/jobs/{id}/review-decisions` | Save reviewer decisions (409 outside needs_review/reviewing) |
| POST | `/jobs/{id}/finalize-review` | Background finalize + send report (409 unless review gate passed) |
| GET | `/jobs/{id}/sheet-image/{page}` | JPEG of source PDF page for the sheet viewer |
| GET | `/` | Serves `frontend/index.html` |
### `/check` form fields
- Required: `file` (PDF)
- Optional: `notification_email`, `project_name`, `address`, `occupancy`, `work_type`
- Pipeline: `pipeline_mode` (`classic` default, or `agent`)
- Compute: `text_local` (`true` = hybrid local text; forced off for agent mode)
- Models: `vision_model`, `text_model` (OpenRouter ids; blank = config defaults)
## Job logs
Pipeline `print()` is teed for the job thread ([backend/job_log.py](backend/job_log.py)):
- Live: `GET /jobs/{id}``log_tail` (last 80 lines)
- Done/error/needs_review/finalization_error: same payload includes full `log`
- Disk: `backend/outputs/<job_id>/job.log` (survives restart; status registry does not)
- API: `GET /jobs/{id}/log``text/plain`
- UI: “Run log” panel updates while running; stays visible after finish/fail/review
- Failures append the **full traceback** to the log; each run starts with a header line (job id, mode, models, start time)
- `outputs/<job_id>/job.json` (written at start) carries email/mode/models so the disk fallback can rebuild a job after restart
## Vision vs text models
Two models, not one:
| Role | Config | Stages | Backend |
|------|--------|--------|---------|
| Vision | `MODEL` | Extract, conflict reason (images) | Always OpenRouter |
| Text | `TEXT_MODEL` (falls back to `MODEL`) | Sheet index, jurisdiction, normalize, cluster(LLM), QAQC, code, construct, validate, risk, RFI | OpenRouter, or local when hybrid |
UI: two dropdowns (with per-1M pricing) filled from `GET /models` ([backend/models.py](backend/models.py)), shown only for OpenRouter compute. Per-run picks go through `set_model_overrides(vision, text)` in [backend/llm.py](backend/llm.py): classic runs pass them as `run_pipeline` kwargs (runner clears in `finally`); agent runs set them module-level around `run_agent_pipeline`. UI picks beat per-call agent `AGENT_*_MODEL` args but **never name the hybrid local model** — local stays on `LOCAL_TEXT_MODEL`; the text pick only covers the cloud fallback.
## Where to change what
| Concern | File |
|---------|------|
| Prompts, vocab, conflict taxonomy | [backend/prompts.py](backend/prompts.py) |
| Env knobs | [backend/config.py](backend/config.py), [backend/.env.example](backend/.env.example) |
| HTTP API | [backend/main.py](backend/main.py) |
| UI (upload, models, live log, results) | [frontend/index.html](frontend/index.html) |
| CLI tuning loop | [cli/run_check.py](cli/run_check.py) |
| LLM client, cache, cost, model overrides | [backend/llm.py](backend/llm.py) |
| OpenRouter catalog + pricing + vision/text split | [backend/models.py](backend/models.py) |
| Agent-mode pipeline | [backend/agents/](backend/agents/) (`runner.py` entry; agents call `llm.call_json` with per-call model args) |
| Human review gate (queue, decisions, finalize) | [backend/review/](backend/review/) |
| Job registry + stdout tee log | [backend/jobs.py](backend/jobs.py), [backend/job_log.py](backend/job_log.py) |
| Stage helpers (prompt render, issue validate) | [backend/pipeline/_stage.py](backend/pipeline/_stage.py) |
| Code text corpus (Stage 7) | [backend/code_corpus/](backend/code_corpus/) |
| Hybrid local LLM helper | [scripts/setup_vllm.sh](scripts/setup_vllm.sh) |
Older prompt snapshot: `backend/prompts.py.v1`.
## Gotchas
1. **Prompt placeholders** — Use `str.replace` via `_stage.render`, never `str.format`. Prompts contain literal `{` JSON braces.
2. **Clustering** — Default is LLM (`CLUSTERER=llm`); empty LLM result falls back to deterministic. Deterministic clusters need ≥2 disciplines (or schedule-vs-plan); single-discipline “missing” gaps are a known limit.
3. **Jobs are in-memory** — Process restart clears job status; `outputs/<job_id>/` (report + `job.log` + `job.json` + `source.pdf`) still reload via disk fallback, including `needs_review` recovery.
4. **Dependency pin**`httpx==0.27.2` with `openai==1.51.0`. httpx ≥0.28 breaks openais `proxies=` kwarg.
5. **Code corpus licensing** — Only `ada_2010.txt` is shipped. Do not paste IBC/IFC/IECC without a license (see `backend/code_corpus/README.md`).
6. **Dual assertion schema** — Newer `{sheet, objects[]}` is mapped to legacy `{assertions[]}` with `attribute`/`value` for older stages.
7. **Grounding guard** — Extractor drops objects whose numeric claims are not in `source_text` (graphical-only objects allowed).
8. **Module globals** — LLM cost counters, stdout tee, and model overrides are process-global; overlapping jobs can interleave (single-user tool assumption).
9. **Samples**`samples/*.pdf` are gitignored; drop PDFs locally for CLI runs.
10. **README drift** — Treat README for setup/CI; treat this file + `runner.py` for pipeline truth. `prompts.py` header may still say some prompts are unwired — they are wired through the runner.
11. **Git identity** — This box has no `user.name` / `user.email`; commits need `GIT_AUTHOR_*` / `GIT_COMMITTER_*` env vars (do not `git config`). Remote push to Gitea works.
12. **No local Python deps on host** — App is meant to run in Docker; bare `python3` imports may miss `dotenv` / `httpx`. Prefer `docker compose`. For tests on this box: venv + requirements, but unpin Pillow (`Pillow>=11`) — 10.4.0 doesn't build on Python 3.14 (Docker uses 3.12, where the pin is fine).
13. **Agent mode constraints** — OpenRouter-only (hybrid disabled in UI and forced off server-side); review gate statuses are `needs_review → reviewing → finalizing → done` (`finalization_error` on finalize failure); only terminal states include the full `log` in polls.
## Quick start pointers
- Full setup: [README.md](README.md) (`docker compose up -d --build` → http://localhost:8099).
- Local CLI: `python cli/run_check.py samples/your_set.pdf --out out/your_set`.
- Prompt iteration: set `LLM_CACHE=true` in `backend/.env` so unchanged stages replay for free; clear with `rm -rf backend/.llm_cache`.
- Artifacts per job: `assertions.json`, `clusters.json`, per-stage JSON, `conflicts.json`, `report.md`, `job.log`, `job.json`, `source.pdf` under `backend/outputs/<job_id>/`.
- Tests: `python -m pytest tests/` (needs the deps from `requirements.txt` + `pytest`; on this box use a venv, see gotcha #12).
## Recent work (2026-08-02, agent-mode)
Merged `main` tip (`a6b0c8f` + `bf508bf`) into `agent-mode`, reconciling with this branch's own earlier implementations:
- **Two model dropdowns** — main's vision/text split ported onto this branch's priced catalog (`models.py`); pickers stay OpenRouter-compute-only, and UI picks never override `LOCAL_TEXT_MODEL` (main's hybrid footgun avoided).
- **Better run logs** — main's timestamped line-splitting tee, `log_tail` polls, terminal-state full log, and log-only disk recovery merged with this branch's header line, `job.json` metadata, and review-gate states. Failed runs now also append the traceback to `job.log`.
- `backend/models_catalog.py` (main's unpriced catalog) intentionally dropped in favor of `models.py`.
## Conflict categories (taxonomy)
Defined in `backend/prompts.py`: `dimensional_disagreement`, `elevation_disagreement`, `location_mismatch`, `missing_element`, `schedule_vs_plan_mismatch`, `detail_vs_plan_mismatch`, `tag_or_reference_inconsistency`, `spatial_clash`, `note_or_spec_contradiction`.
+93
View File
@@ -0,0 +1,93 @@
"""
job_log.py - Capture pipeline stdout/stderr into a per-job log.
The pipeline already prints stage progress via print(). For a reviewable
post-run log we tee those lines into memory + outputs/<job_id>/job.log
without rewriting every call site.
"""
import sys
import time
from contextlib import contextmanager
from typing import Callable, Iterator, List, Optional, TextIO
class _LineSplitter:
"""Accumulate write() chunks and emit complete lines."""
def __init__(self, on_line: Callable[[str], None]):
self._buf = ""
self._on_line = on_line
def write(self, s: str) -> None:
if not s:
return
self._buf += s
while "\n" in self._buf:
line, self._buf = self._buf.split("\n", 1)
# Strip trailing CR from Windows-ish streams; keep content intact.
self._on_line(line.rstrip("\r"))
def flush_remainder(self) -> None:
if self._buf:
self._on_line(self._buf.rstrip("\r"))
self._buf = ""
class _Tee:
"""Mirror writes to the original stream and a line callback."""
def __init__(self, stream: TextIO, on_line: Callable[[str], None]):
self._stream = stream
self._lines = _LineSplitter(on_line)
def write(self, s: str) -> int:
n = self._stream.write(s)
self._stream.flush()
self._lines.write(s)
return n
def flush(self) -> None:
self._stream.flush()
def flush_remainder(self) -> None:
self._lines.flush_remainder()
def __getattr__(self, name: str):
return getattr(self._stream, name)
def stamp_line(line: str, t: Optional[float] = None) -> str:
"""Prefix a log line with HH:MM:SS."""
ts = time.strftime("%H:%M:%S", time.localtime(t if t is not None else time.time()))
return f"[{ts}] {line}"
@contextmanager
def capture_stdio(on_line: Callable[[str], None]) -> Iterator[None]:
"""
Tee sys.stdout and sys.stderr into on_line(raw_line) for the duration.
Safe for the single-job-at-a-time usage of this app; overlapping jobs
would interleave (same limitation as the LLM cost counters).
"""
old_out, old_err = sys.stdout, sys.stderr
tee_out = _Tee(old_out, on_line)
tee_err = _Tee(old_err, on_line)
sys.stdout = tee_out # type: ignore[assignment]
sys.stderr = tee_err # type: ignore[assignment]
try:
yield
finally:
tee_out.flush_remainder()
tee_err.flush_remainder()
sys.stdout = old_out
sys.stderr = old_err
def read_log_file(path: str) -> List[str]:
try:
with open(path, encoding="utf-8") as f:
return [ln.rstrip("\n") for ln in f]
except OSError:
return []
+143 -78
View File
@@ -9,61 +9,33 @@ for the completion email.
State is in-memory (fine for a single-user tool); the report is also persisted State is in-memory (fine for a single-user tool); the report is also persisted
to outputs/<job_id>/ so results survive a restart even though live status does to outputs/<job_id>/ so results survive a restart even though live status does
not. No external queue/DB. not. No external queue/DB.
A teed stdout/stderr log is kept in memory and written to outputs/<job_id>/job.log
so failed or suspicious runs can be reviewed after the fact.
""" """
import json import json
import contextlib
import os import os
import sys
import time import time
import traceback
import uuid import uuid
import shutil import shutil
import threading import threading
from typing import Dict, Optional from typing import Dict, List, Optional
from backend import config from backend import config
from backend import llm from backend import llm
from backend.job_log import capture_stdio, read_log_file, stamp_line
from backend.agents.runner import run_agent_pipeline from backend.agents.runner import run_agent_pipeline
from backend.pipeline.runner import run_pipeline from backend.pipeline.runner import run_pipeline
from backend.email_sender import send_conflict_report, send_review_required from backend.email_sender import send_conflict_report, send_review_required
_jobs: Dict[str, Dict] = {} _jobs: Dict[str, Dict] = {}
_lock = threading.Lock() _lock = threading.Lock()
_LOG_TAIL = 80
PIPELINE_MODES = {"classic", "agent"} PIPELINE_MODES = {"classic", "agent"}
# States where the job will produce no more log output; polls get the full log.
_TERMINAL_STATES = {"done", "error", "needs_review", "finalization_error"}
class _Tee:
"""Write to both the real stream and the job log file."""
def __init__(self, stream, log_file) -> None:
self._stream = stream
self._log = log_file
def write(self, data):
self._stream.write(data)
self._log.write(data)
def flush(self):
self._stream.flush()
self._log.flush()
@contextlib.contextmanager
def _tee_log(log_path: str, header: str):
"""Mirror stdout/stderr into a per-job log file for the duration of a run.
sys.stdout is process-global, so two concurrent jobs would interleave in
each other's logs - acceptable for this single-user tool (same tradeoff as
the LLM cost globals in llm.py).
"""
with open(log_path, "a", encoding="utf-8") as log_file:
log_file.write(header + "\n")
real_out, real_err = sys.stdout, sys.stderr
sys.stdout, sys.stderr = _Tee(real_out, log_file), _Tee(real_err, log_file)
try:
yield
finally:
sys.stdout, sys.stderr = real_out, real_err
def _set(job_id: str, **fields) -> None: def _set(job_id: str, **fields) -> None:
@@ -71,16 +43,39 @@ def _set(job_id: str, **fields) -> None:
_jobs[job_id].update(fields) _jobs[job_id].update(fields)
def create_job(pdf_path: str, source_filename: str, email: Optional[str] = None, def _append_log(job_id: str, raw_line: str, log_path: str) -> None:
project_input: Optional[Dict] = None, text_local: bool = False, """Stamp, store, and append one captured stdout/stderr line."""
pipeline_mode: str = "classic", model: Optional[str] = None) -> str: entry = stamp_line(raw_line)
with _lock:
job = _jobs.get(job_id)
if job is not None:
job.setdefault("log", []).append(entry)
try:
os.makedirs(os.path.dirname(log_path), exist_ok=True)
with open(log_path, "a", encoding="utf-8") as f:
f.write(entry + "\n")
except OSError:
# Don't fail the job over log I/O; avoid print() here — it would
# re-enter the stdio tee while a job is capturing.
pass
def create_job(
pdf_path: str,
source_filename: str,
email: Optional[str] = None,
project_input: Optional[Dict] = None,
text_local: bool = False,
pipeline_mode: str = "classic",
vision_model: Optional[str] = None,
text_model: Optional[str] = None,
) -> str:
"""Register a job and kick off its background thread. Returns the job_id.""" """Register a job and kick off its background thread. Returns the job_id."""
pipeline_mode = pipeline_mode.strip().lower() pipeline_mode = pipeline_mode.strip().lower()
if pipeline_mode not in PIPELINE_MODES: if pipeline_mode not in PIPELINE_MODES:
raise ValueError(f"Unsupported pipeline mode: {pipeline_mode!r}") raise ValueError(f"Unsupported pipeline mode: {pipeline_mode!r}")
# Agent mode v1 is OpenRouter-only. # Agent mode v1 is OpenRouter-only.
text_local = bool(text_local and pipeline_mode == "classic") text_local = bool(text_local and pipeline_mode == "classic")
model = (model or "").strip() or None
job_id = uuid.uuid4().hex[:12] job_id = uuid.uuid4().hex[:12]
with _lock: with _lock:
_jobs[job_id] = { _jobs[job_id] = {
@@ -91,35 +86,61 @@ def create_job(pdf_path: str, source_filename: str, email: Optional[str] = None,
"project_input": project_input or {}, "project_input": project_input or {},
"text_local": text_local, "text_local": text_local,
"pipeline_mode": pipeline_mode, "pipeline_mode": pipeline_mode,
"model": model, "vision_model": (vision_model or "").strip() or None,
"text_model": (text_model or "").strip() or None,
"stage": None, "stage": None,
"created_at": time.time(), "created_at": time.time(),
"finished_at": None, "finished_at": None,
"report": None, "report": None,
"error": None, "error": None,
"log": [],
} }
threading.Thread(target=_run, args=( threading.Thread(
job_id, pdf_path, project_input, text_local, pipeline_mode, model, target=_run,
), args=(job_id, pdf_path, project_input, text_local, pipeline_mode,
daemon=True).start() vision_model, text_model),
daemon=True,
).start()
return job_id return job_id
def _run(job_id: str, pdf_path: str, project_input: Optional[Dict] = None, def _run(
text_local: bool = False, pipeline_mode: str = "classic", job_id: str,
model: Optional[str] = None) -> None: pdf_path: str,
project_input: Optional[Dict] = None,
text_local: bool = False,
pipeline_mode: str = "classic",
vision_model: Optional[str] = None,
text_model: Optional[str] = None,
) -> None:
out_dir = os.path.join(config.OUTPUT_DIR, job_id) out_dir = os.path.join(config.OUTPUT_DIR, job_id)
log_path = os.path.join(out_dir, "job.log")
try: try:
_set(job_id, status="running") _set(job_id, status="running")
# Keep a copy of the source PDF so its sheets can be viewed later.
os.makedirs(out_dir, exist_ok=True) os.makedirs(out_dir, exist_ok=True)
# Truncate any leftover log if job_id somehow collided (shouldn't).
with open(log_path, "w", encoding="utf-8"):
pass
header = (f"=== Job {job_id} | {pipeline_mode} | {_jobs[job_id].get('source')} | " header = (f"=== Job {job_id} | {pipeline_mode} | {_jobs[job_id].get('source')} | "
f"model={model or 'default'} | " f"vision={vision_model or 'default'} text={text_model or 'default'} | "
f"started {time.strftime('%Y-%m-%d %H:%M:%S %Z', time.gmtime())} UTC ===") f"started {time.strftime('%Y-%m-%d %H:%M:%S %Z', time.gmtime())} UTC ===")
with _tee_log(os.path.join(out_dir, "job.log"), header): _append_log(job_id, header, log_path)
def on_line(raw: str) -> None:
_append_log(job_id, raw, log_path)
with capture_stdio(on_line):
_run_pipeline(job_id, pdf_path, out_dir, project_input, text_local, _run_pipeline(job_id, pdf_path, out_dir, project_input, text_local,
pipeline_mode, model) pipeline_mode, vision_model, text_model)
except Exception as e: except Exception as e:
# Land the failure AND its traceback in the job log so failed runs can
# be diagnosed from the log alone (the stdio tee is already torn down).
try:
_append_log(job_id, f"[Jobs] Job {job_id} failed: {e}", log_path)
for ln in traceback.format_exc().rstrip().splitlines():
_append_log(job_id, ln, log_path)
except Exception:
pass
print(f"[Jobs] Job {job_id} failed: {e}") print(f"[Jobs] Job {job_id} failed: {e}")
_set(job_id, status="error", error=str(e), finished_at=time.time()) _set(job_id, status="error", error=str(e), finished_at=time.time())
_notify_error(job_id) _notify_error(job_id)
@@ -132,7 +153,8 @@ def _run(job_id: str, pdf_path: str, project_input: Optional[Dict] = None,
def _run_pipeline(job_id: str, pdf_path: str, out_dir: str, def _run_pipeline(job_id: str, pdf_path: str, out_dir: str,
project_input: Optional[Dict], text_local: bool, project_input: Optional[Dict], text_local: bool,
pipeline_mode: str, model: Optional[str]) -> None: pipeline_mode: str, vision_model: Optional[str],
text_model: Optional[str]) -> None:
"""The body of a job run; executes inside the job's tee'd log capture.""" """The body of a job run; executes inside the job's tee'd log capture."""
# Persist minimal job metadata so the disk fallback in get_job can # Persist minimal job metadata so the disk fallback in get_job can
# recover the recipient email / pipeline mode after a server restart # recover the recipient email / pipeline mode after a server restart
@@ -143,7 +165,10 @@ def _run_pipeline(job_id: str, pdf_path: str, out_dir: str,
"email": _jobs[job_id].get("email"), "email": _jobs[job_id].get("email"),
"pipeline_mode": pipeline_mode, "pipeline_mode": pipeline_mode,
"source": _jobs[job_id].get("source"), "source": _jobs[job_id].get("source"),
"vision_model": vision_model,
"text_model": text_model,
}, f, indent=2) }, f, indent=2)
# Keep a copy of the source PDF so its sheets can be viewed later.
shutil.copy2(pdf_path, os.path.join(out_dir, "source.pdf")) shutil.copy2(pdf_path, os.path.join(out_dir, "source.pdf"))
runner = run_agent_pipeline if pipeline_mode == "agent" else run_pipeline runner = run_agent_pipeline if pipeline_mode == "agent" else run_pipeline
runner_kwargs = { runner_kwargs = {
@@ -153,17 +178,22 @@ def _run_pipeline(job_id: str, pdf_path: str, out_dir: str,
"source_name": _jobs[job_id].get("source"), "source_name": _jobs[job_id].get("source"),
} }
if pipeline_mode == "classic": if pipeline_mode == "classic":
# run_pipeline takes the picks as params and clears them in finally.
runner_kwargs["text_local"] = text_local runner_kwargs["text_local"] = text_local
runner_kwargs["vision_model"] = vision_model
runner_kwargs["text_model"] = text_model
else: else:
runner_kwargs["require_review"] = config.AGENT_REQUIRE_REVIEW runner_kwargs["require_review"] = config.AGENT_REQUIRE_REVIEW
if model: # The agent runner has no override params; set them module-level.
print(f"[Jobs] Model override for this run: {model}") if vision_model or text_model:
llm.set_model_override(model) print(f"[Jobs] Model overrides for this run: "
f"vision={vision_model or '(default)'} text={text_model or '(default)'}")
llm.set_model_overrides(vision_model, text_model)
try: try:
report = runner(pdf_path, **runner_kwargs) report = runner(pdf_path, **runner_kwargs)
finally: finally:
if model: if pipeline_mode == "agent":
llm.set_model_override(None) llm.set_model_overrides(None, None)
report.setdefault("summary", {})["pipeline_mode"] = pipeline_mode report.setdefault("summary", {})["pipeline_mode"] = pipeline_mode
if report["summary"].get("agent_status") == "needs_review": if report["summary"].get("agent_status") == "needs_review":
# Human-review gate: hold the job, don't email the unreviewed report. # Human-review gate: hold the job, don't email the unreviewed report.
@@ -197,11 +227,6 @@ def _notify_error(job_id: str) -> None:
email = job.get("email") email = job.get("email")
if not email: if not email:
return return
# Reuse the report mailer with a minimal error-shaped payload.
err_report = {
"source": job.get("source", ""),
"summary": {"conflicts_found": 0, "by_severity": {}, "disciplines": []},
}
try: try:
from backend.email_sender import _smtp_ready, _send from backend.email_sender import _smtp_ready, _send
from email.message import EmailMessage from email.message import EmailMessage
@@ -216,6 +241,7 @@ def _notify_error(job_id: str) -> None:
"Your conflict check did not complete.\n\n" "Your conflict check did not complete.\n\n"
f"Drawing set: {job.get('source','')}\n" f"Drawing set: {job.get('source','')}\n"
f"Error: {job.get('error','unknown')}\n\n" f"Error: {job.get('error','unknown')}\n\n"
f"Review the run log at: {config.APP_BASE_URL.rstrip('/')}/?job={job_id}\n\n"
"Generated by Conflict Checker" "Generated by Conflict Checker"
) )
_send(msg) _send(msg)
@@ -223,28 +249,63 @@ def _notify_error(job_id: str) -> None:
print(f"[Email] Failed to send error notice: {e}") print(f"[Email] Failed to send error notice: {e}")
def _log_from_disk(job_id: str) -> List[str]:
return read_log_file(os.path.join(config.OUTPUT_DIR, job_id, "job.log"))
def get_job_log(job_id: str) -> Optional[List[str]]:
"""Full job log lines, from memory or disk. None if job unknown."""
with _lock:
job = _jobs.get(job_id)
if job is not None:
return list(job.get("log") or [])
log = _log_from_disk(job_id)
# Job exists on disk if we have a log or a report artifact.
report_path = os.path.join(config.OUTPUT_DIR, job_id, "conflicts.json")
if log or os.path.isfile(report_path):
return log
return None
def get_job(job_id: str) -> Optional[Dict]: def get_job(job_id: str) -> Optional[Dict]:
"""Public job view. Includes the full report only when done. """Public job view. Includes the full report only when done.
Falls back to the on-disk conflicts.json when the job isn't in the Falls back to the on-disk artifacts (conflicts.json / job.log) when the
in-memory registry (e.g. after a server restart). job isn't in the in-memory registry (e.g. after a server restart).
""" """
with _lock: with _lock:
job = _jobs.get(job_id) job = _jobs.get(job_id)
if job: if job:
return dict(job) out = dict(job)
log = list(job.get("log") or [])
out["log_tail"] = log[-_LOG_TAIL:]
# Full log on terminal states so the UI can show it without a
# second fetch; keep polls light while running.
if out.get("status") in _TERMINAL_STATES:
out["log"] = log
else:
out.pop("log", None)
return out
# Try loading from disk # Try loading from disk
report_path = os.path.join(config.OUTPUT_DIR, job_id, "conflicts.json") report_path = os.path.join(config.OUTPUT_DIR, job_id, "conflicts.json")
if not os.path.isfile(report_path): log = _log_from_disk(job_id)
if not os.path.isfile(report_path) and not log:
return None return None
try: try:
with open(report_path, encoding="utf-8") as f: report = None
report = json.load(f) if os.path.isfile(report_path):
summary = report.get("summary", {}) with open(report_path, encoding="utf-8") as f:
# Recover the job's real state: a job that stopped at the review gate report = json.load(f)
# must come back as needs_review (not done) or it can never finalize. summary = (report or {}).get("summary", {})
status = "needs_review" if summary.get("agent_status") == "needs_review" else "done" if report is None:
# Crashed before writing a report; the log is the only artifact.
status = "error"
else:
# Recover the job's real state: a job that stopped at the review gate
# must come back as needs_review (not done) or it can never finalize.
status = "needs_review" if summary.get("agent_status") == "needs_review" else "done"
# job.json (written at job start) carries the recipient email and # job.json (written at job start) carries the recipient email and
# pipeline mode so the final notification still fires after a restart. # pipeline mode so the final notification still fires after a restart.
# Missing/corrupt job.json degrades to the previous derivations. # Missing/corrupt job.json degrades to the previous derivations.
@@ -261,16 +322,20 @@ def get_job(job_id: str) -> Optional[Dict]:
job = { job = {
"job_id": job_id, "job_id": job_id,
"status": status, "status": status,
"source": meta.get("source") or report.get("source", os.path.basename(report_path)), "source": meta.get("source") or (report or {}).get("source", os.path.basename(report_path)),
"email": meta.get("email"), "email": meta.get("email"),
"project_input": report.get("project_input", {}), "project_input": (report or {}).get("project_input", {}),
"text_local": summary.get("text_backend") == "local", "text_local": summary.get("text_backend") == "local",
"pipeline_mode": meta.get("pipeline_mode") or summary.get("pipeline_mode", "classic"), "pipeline_mode": meta.get("pipeline_mode") or summary.get("pipeline_mode", "classic"),
"vision_model": meta.get("vision_model"),
"text_model": meta.get("text_model"),
"stage": None, "stage": None,
"created_at": os.path.getmtime(source_pdf) if os.path.isfile(source_pdf) else None, "created_at": os.path.getmtime(source_pdf) if os.path.isfile(source_pdf) else None,
"finished_at": os.path.getmtime(report_path), "finished_at": os.path.getmtime(report_path) if os.path.isfile(report_path) else None,
"report": report, "report": report,
"error": None, "error": None if report is not None else "Report missing; see job log",
"log": log,
"log_tail": log[-_LOG_TAIL:],
} }
# Hydrate the in-memory registry so _set(...) transitions (reviewing, # Hydrate the in-memory registry so _set(...) transitions (reviewing,
# finalizing, done) work for restart-recovered jobs. # finalizing, done) work for restart-recovered jobs.
+22 -12
View File
@@ -23,16 +23,13 @@ _clients: Dict[str, OpenAI] = {}
# call_json when routing a no-image (text) call. Module-global mirrors the # call_json when routing a no-image (text) call. Module-global mirrors the
# set_stage/cost pattern (single-user tool). # set_stage/cost pattern (single-user tool).
_text_local = False _text_local = False
# Per-job model overrides (user picked models in the UI). Same module-global
# Per-job model override (user picked a model in the UI). Same module-global
# pattern: set by the job runner before the pipeline starts, cleared after. # pattern: set by the job runner before the pipeline starts, cleared after.
_model_override: Optional[str] = None # Vision applies to image calls, text to no-image calls on OpenRouter (and to
# the local->cloud fallback). The LOCAL endpoint's model name is never taken
# from these overrides - hybrid local keeps LOCAL_TEXT_MODEL.
def set_model_override(model: Optional[str]) -> None: _vision_model_override: Optional[str] = None
"""Override the model for all OpenRouter calls (vision + text), or None to clear.""" _text_model_override: Optional[str] = None
global _model_override
_model_override = (model or "").strip() or None
def set_text_backend(local: bool) -> None: def set_text_backend(local: bool) -> None:
@@ -40,6 +37,13 @@ def set_text_backend(local: bool) -> None:
global _text_local global _text_local
_text_local = bool(local) _text_local = bool(local)
def set_model_overrides(vision: Optional[str] = None, text: Optional[str] = None) -> None:
"""Per-run OpenRouter vision/text model picks. None/blank clears to defaults."""
global _vision_model_override, _text_model_override
_vision_model_override = (vision or "").strip() or None
_text_model_override = (text or "").strip() or None
# --- per-job cost accounting ------------------------------------------------- # --- per-job cost accounting -------------------------------------------------
# OpenRouter returns the real USD cost of each call when we request usage # OpenRouter returns the real USD cost of each call when we request usage
# accounting. We accumulate it in a module-level counter; the runner resets it # accounting. We accumulate it in a module-level counter; the runner resets it
@@ -177,17 +181,22 @@ def _resolve_backend(has_images: bool, model_override: Optional[str]) -> Dict[st
return { return {
"base_url": config.LOCAL_BASE_URL, "base_url": config.LOCAL_BASE_URL,
"api_key": config.LOCAL_API_KEY, "api_key": config.LOCAL_API_KEY,
# Local model name comes from per-call args or LOCAL_TEXT_MODEL —
# never the UI's OpenRouter picks, which a local server won't serve.
"model": model_override or config.LOCAL_TEXT_MODEL or config.TEXT_MODEL, "model": model_override or config.LOCAL_TEXT_MODEL or config.TEXT_MODEL,
"usage": False, # local has no OpenRouter usage accounting "usage": False, # local has no OpenRouter usage accounting
"local": True, "local": True,
} }
# Vision, or text-on-OpenRouter (default / fallback). A per-job override # Vision, or text-on-OpenRouter (default / fallback). A per-job override
# (user's UI model pick) wins over per-call and env defaults. # (user's UI model pick) wins over per-call and env defaults.
default_model = config.MODEL if has_images else config.TEXT_MODEL if has_images:
model = _vision_model_override or model_override or config.MODEL
else:
model = _text_model_override or model_override or config.TEXT_MODEL
return { return {
"base_url": config.AI_BASE_URL, "base_url": config.AI_BASE_URL,
"api_key": config.AI_API_KEY, "api_key": config.AI_API_KEY,
"model": _model_override or model_override or default_model, "model": model,
"usage": True, "usage": True,
"local": False, "local": False,
} }
@@ -362,7 +371,8 @@ def call_json(
_models["text_local"].add(be["model"]) _models["text_local"].add(be["model"])
_models["fallback_count"] += 1 _models["fallback_count"] += 1
be = {"base_url": config.AI_BASE_URL, "api_key": config.AI_API_KEY, be = {"base_url": config.AI_BASE_URL, "api_key": config.AI_API_KEY,
"model": config.TEXT_MODEL, "usage": True, "local": False} "model": _text_model_override or config.TEXT_MODEL,
"usage": True, "local": False}
cache_key = None # don't cache fallback under the local-model key cache_key = None # don't cache fallback under the local-model key
fell_back = True fell_back = True
continue continue
+15 -5
View File
@@ -35,6 +35,7 @@ _FRONTEND_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "
@app.get("/health") @app.get("/health")
def health(): def health():
return {"status": "ok", "model": config.MODEL, return {"status": "ok", "model": config.MODEL,
"text_model": config.TEXT_MODEL,
"version": config.APP_VERSION, "version": config.APP_VERSION,
"build": config.APP_BUILD, "build": config.APP_BUILD,
"key_configured": bool(config.AI_API_KEY), "key_configured": bool(config.AI_API_KEY),
@@ -43,13 +44,15 @@ def health():
@app.get("/models") @app.get("/models")
def list_models(): def list_models():
"""Available OpenRouter models with per-1M-token pricing for the UI picker.""" """Vision/text OpenRouter model lists with pricing for the UI dropdowns."""
from backend.models import fetch_models from backend.models import fetch_models, split_vision_text
models = fetch_models() models = fetch_models()
if models is None: if models is None:
raise HTTPException(status_code=502, raise HTTPException(status_code=502,
detail="Could not fetch the model list from OpenRouter") detail="Could not fetch the model list from OpenRouter")
return {"models": models, "default": config.MODEL, "default_text": config.TEXT_MODEL} vision, text = split_vision_text(models)
return {"vision": vision, "text": text,
"defaults": {"vision": config.MODEL, "text": config.TEXT_MODEL}}
@app.get("/jobs/{job_id}/log") @app.get("/jobs/{job_id}/log")
@@ -72,7 +75,8 @@ async def check(
work_type: Optional[str] = Form(None), work_type: Optional[str] = Form(None),
text_local: bool = Form(False), text_local: bool = Form(False),
pipeline_mode: str = Form("classic"), pipeline_mode: str = Form("classic"),
model: Optional[str] = Form(None), vision_model: Optional[str] = Form(None),
text_model: Optional[str] = Form(None),
): ):
""" """
Accept a PDF, start a background conflict check, and return a job_id Accept a PDF, start a background conflict check, and return a job_id
@@ -81,6 +85,9 @@ async def check(
Optional intake fields (project_name/address/occupancy/work_type) feed the Optional intake fields (project_name/address/occupancy/work_type) feed the
Stage 0 jurisdiction profile; anything left blank is derived from the cover Stage 0 jurisdiction profile; anything left blank is derived from the cover
sheet. sheet.
vision_model / text_model override the configured defaults for this run
(vision always OpenRouter; text follows the OpenRouter vs hybrid choice).
""" """
if not file.filename.lower().endswith(".pdf"): if not file.filename.lower().endswith(".pdf"):
raise HTTPException(status_code=400, detail="Please upload a PDF.") raise HTTPException(status_code=400, detail="Please upload a PDF.")
@@ -106,9 +113,12 @@ async def check(
}.items() }.items()
if v and v.strip() if v and v.strip()
} }
v_model = (vision_model or "").strip() or None
t_model = (text_model or "").strip() or None
job_id = create_job(tmp_path, source_filename=file.filename, email=email, job_id = create_job(tmp_path, source_filename=file.filename, email=email,
project_input=project_input, text_local=text_local, project_input=project_input, text_local=text_local,
pipeline_mode=pipeline_mode, model=model) pipeline_mode=pipeline_mode, vision_model=v_model,
text_model=t_model)
return JSONResponse({ return JSONResponse({
"job_id": job_id, "job_id": job_id,
"status": "queued", "status": "queued",
+27 -2
View File
@@ -3,11 +3,13 @@ models.py - Fetch the available OpenRouter model list with pricing (cached).
The /models endpoint is public (no API key needed). Results are normalized to The /models endpoint is public (no API key needed). Results are normalized to
per-1M-token USD costs for display and cached in memory for an hour; callers per-1M-token USD costs for display and cached in memory for an hour; callers
degrade gracefully when OpenRouter is unreachable. degrade gracefully when OpenRouter is unreachable. Each entry also carries a
vision flag (accepts image input) so the UI can offer separate vision/text
model dropdowns.
""" """
import time import time
from typing import List, Optional from typing import List, Optional, Tuple
import httpx import httpx
@@ -25,6 +27,18 @@ def _per_mtok(rate) -> float:
return 0.0 return 0.0
def _is_vision(item: dict) -> bool:
"""True when the model accepts image input and produces text output."""
arch = item.get("architecture") or {}
inputs = arch.get("input_modalities") or []
outputs = arch.get("output_modalities") or []
# Legacy string form: "text+image->text"
modality = (arch.get("modality") or "").lower()
has_image_in = ("image" in inputs) or ("image" in modality.split("->")[0])
has_text_out = ("text" in outputs) or ("->text" in modality) or (not outputs and not modality)
return has_image_in and has_text_out
def _fetch_openrouter_models() -> Optional[List[dict]]: def _fetch_openrouter_models() -> Optional[List[dict]]:
"""Raw GET of the OpenRouter model list; None on any failure.""" """Raw GET of the OpenRouter model list; None on any failure."""
try: try:
@@ -55,6 +69,7 @@ def fetch_models(force: bool = False) -> Optional[List[dict]]:
"prompt_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("prompt")), "prompt_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("prompt")),
"completion_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("completion")), "completion_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("completion")),
"context_length": item.get("context_length"), "context_length": item.get("context_length"),
"vision": _is_vision(item),
} }
for item in data for item in data
if item.get("id") if item.get("id")
@@ -63,3 +78,13 @@ def fetch_models(force: bool = False) -> Optional[List[dict]]:
_cache["models"] = models _cache["models"] = models
_cache["at"] = time.time() _cache["at"] = time.time()
return models return models
def split_vision_text(models: List[dict]) -> Tuple[List[dict], List[dict]]:
"""Partition the normalized catalog into (vision, text) lists for the UI.
Every catalog model takes text in/out, so vision models appear in both
lists (same dicts, pricing included).
"""
vision = [m for m in models if m.get("vision")]
return vision, list(models)
+30 -1
View File
@@ -42,7 +42,9 @@ from backend.pipeline.risk import score_and_prioritize
from backend.pipeline.rfi import generate_rfis from backend.pipeline.rfi import generate_rfis
from backend.pipeline.report import build_report, to_markdown from backend.pipeline.report import build_report, to_markdown
from backend.pipeline._stage import validate_issue from backend.pipeline._stage import validate_issue
from backend.llm import reset_cost, get_cost, set_stage, set_text_backend from backend.llm import (
reset_cost, get_cost, set_stage, set_text_backend, set_model_overrides,
)
def run_pipeline( def run_pipeline(
@@ -52,6 +54,8 @@ def run_pipeline(
project_input: Optional[Dict] = None, project_input: Optional[Dict] = None,
source_name: Optional[str] = None, source_name: Optional[str] = None,
text_local: bool = False, text_local: bool = False,
vision_model: Optional[str] = None,
text_model: Optional[str] = None,
) -> Dict: ) -> Dict:
""" """
Run the full QAQC pipeline on one PDF and return the report dict. Run the full QAQC pipeline on one PDF and return the report dict.
@@ -59,6 +63,9 @@ def run_pipeline(
project_input: optional intake fields (project_name, address, occupancy, project_input: optional intake fields (project_name, address, occupancy,
work_type). Cover-sheet-derived values fill any gaps; intake fields win. work_type). Cover-sheet-derived values fill any gaps; intake fields win.
vision_model / text_model: optional per-run OpenRouter (or local text)
model overrides from the UI. Blank/None keeps config defaults.
If out_dir is given, writes conflicts.json, report.md, and the intermediate If out_dir is given, writes conflicts.json, report.md, and the intermediate
artifacts (assertions.json, clusters.json, and one json per QAQC stage). artifacts (assertions.json, clusters.json, and one json per QAQC stage).
""" """
@@ -70,7 +77,29 @@ def run_pipeline(
reset_cost() reset_cost()
set_text_backend(text_local) set_text_backend(text_local)
set_model_overrides(vision_model, text_model)
if vision_model or text_model:
print(f"[Runner] model overrides: vision={vision_model or '(default)'} "
f"text={text_model or '(default)'}")
try:
return _run_stages(
pdf_path, out_dir, stage, project_input, source_name, text_local,
)
finally:
# Don't leak per-run picks into a later overlapping/CLI call.
set_model_overrides(None, None)
set_text_backend(False)
def _run_stages(
pdf_path: str,
out_dir: Optional[str],
stage: Callable[[str], None],
project_input: Optional[Dict],
source_name: Optional[str],
text_local: bool,
) -> Dict:
stage("PDF -> images") stage("PDF -> images")
pages = convert_pdf_to_images(pdf_path) pages = convert_pdf_to_images(pdf_path)
+104 -27
View File
@@ -60,7 +60,15 @@
border:1px solid var(--line); background:#0c0e13; color:var(--text); font-size:16px; outline:none; } border:1px solid var(--line); background:#0c0e13; color:var(--text); font-size:16px; outline:none; }
.email-card input[type=email]::placeholder { color:#6b7280; } .email-card input[type=email]::placeholder { color:#6b7280; }
.email-card input[type=email]:focus { border-color:var(--accent); box-shadow:0 0 0 3px rgba(91,140,255,.18); } .email-card input[type=email]:focus { border-color:var(--accent); box-shadow:0 0 0 3px rgba(91,140,255,.18); }
.email-card select { width:100%; padding:10px 12px; border-radius:8px; margin-top:6px;
border:1px solid var(--line); background:#0c0e13; color:var(--text); font-size:14px; }
.email-card .field { margin-top:12px; }
.email-card .field > span { display:block; font-size:13px; color:var(--muted); margin-bottom:2px; }
.btn.full { width:100%; padding:14px; font-size:15px; margin-top:0; } .btn.full { width:100%; padding:14px; font-size:15px; margin-top:0; }
.logbox { background:#0c0e13; border:1px solid var(--line); border-radius:8px; padding:12px 14px;
margin-top:10px; max-height:320px; overflow:auto; font:12px/1.45 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;
color:#c6cdd8; white-space:pre-wrap; word-break:break-word; }
.logbox .empty-log { color:var(--muted); }
.note { background:var(--panel); border:1px solid var(--line); border-radius:10px; .note { background:var(--panel); border:1px solid var(--line); border-radius:10px;
padding:16px 18px; margin:18px 0; } padding:16px 18px; margin:18px 0; }
.note b { color:var(--text); } .note b { color:var(--text); }
@@ -121,12 +129,22 @@
<label style="display:block;font-weight:400;margin-top:6px"> <label style="display:block;font-weight:400;margin-top:6px">
<input type="radio" name="compute" value="local"> Hybrid &mdash; text stages on local LLM (cheaper, slower)</label> <input type="radio" name="compute" value="local"> Hybrid &mdash; text stages on local LLM (cheaper, slower)</label>
<div id="modelPick" style="margin-top:10px"> <div id="modelPick" style="margin-top:10px">
<label for="model" style="font-weight:400">Model <span class="opt" id="modelNote">loading...</span></label> <div class="field">
<select id="model" style="width:100%;margin-top:6px;padding:10px;border-radius:8px;border:1px solid var(--line);background:#0c0e13;color:var(--text)"></select> <span>Vision model <span class="opt">(image stages)</span> <span class="opt" id="modelNote">loading...</span></span>
<select id="vision_model" disabled><option value="">Loading models&hellip;</option></select>
</div>
<div class="field">
<span>Text model <span class="opt">(non-image stages)</span></span>
<select id="text_model" disabled><option value="">Loading models&hellip;</option></select>
</div>
</div> </div>
</div> </div>
<button class="btn full" id="run" disabled>Run conflict check</button> <button class="btn full" id="run" disabled>Run conflict check</button>
<div class="status" id="status"></div> <div class="status" id="status"></div>
<div id="liveLog" style="display:none" class="note">
<b>Run log</b> <span class="opt" id="logHint">(updates live)</span>
<pre class="logbox" id="logBox"><span class="empty-log">Waiting for output&hellip;</span></pre>
</div>
<div id="results"></div> <div id="results"></div>
</main> </main>
<div id="viewer"> <div id="viewer">
@@ -144,9 +162,71 @@
const drop=document.getElementById('drop'), fileInput=document.getElementById('file'), const drop=document.getElementById('drop'), fileInput=document.getElementById('file'),
runBtn=document.getElementById('run'), statusEl=document.getElementById('status'), runBtn=document.getElementById('run'), statusEl=document.getElementById('status'),
results=document.getElementById('results'), dropLabel=document.getElementById('dropLabel'), results=document.getElementById('results'), dropLabel=document.getElementById('dropLabel'),
emailEl=document.getElementById('email'); emailEl=document.getElementById('email'),
visionSel=document.getElementById('vision_model'),
textSel=document.getElementById('text_model'),
liveLog=document.getElementById('liveLog'),
logBox=document.getElementById('logBox'),
logHint=document.getElementById('logHint');
let chosen=null, polling=null, currentJobId=null, sheetPage={}, viewerZoom=1, reviewDirty=false; let chosen=null, polling=null, currentJobId=null, sheetPage={}, viewerZoom=1, reviewDirty=false;
function modelLabel(m){
// Include per-1M-token pricing when the catalog provides it.
let s=m.name||m.id;
if(m.prompt_usd_per_mtok!=null)
s+=' — $'+m.prompt_usd_per_mtok+' / $'+m.completion_usd_per_mtok+' per 1M tok';
return s;
}
function fillSelect(sel, items, preferred){
sel.innerHTML='';
(items||[]).forEach(m=>{
const opt=document.createElement('option');
opt.value=m.id; opt.textContent=modelLabel(m);
if(m.id===preferred) opt.selected=true;
sel.appendChild(opt);
});
if(!sel.options.length){
const opt=document.createElement('option');
opt.value=preferred||''; opt.textContent=preferred||'(no models)';
sel.appendChild(opt);
}
sel.disabled=false;
}
let modelsLoaded=false;
async function loadModels(){
const note=document.getElementById('modelNote');
try{
const res=await fetch('/models');
if(!res.ok) throw new Error('models HTTP '+res.status);
const data=await res.json();
const defs=data.defaults||{};
fillSelect(visionSel, data.vision, defs.vision);
fillSelect(textSel, data.text, defs.text);
modelsLoaded=true;
note.textContent='('+(data.text||[]).length+' text / '+(data.vision||[]).length+' vision available)';
}catch(err){
visionSel.innerHTML='<option value="">(default)</option>';
textSel.innerHTML='<option value="">(default)</option>';
visionSel.disabled=false; textSel.disabled=false;
note.textContent='using configured defaults (list unavailable)';
console.warn('Could not load models:', err);
}
}
function showLog(lines, live){
liveLog.style.display='block';
logHint.textContent=live?'(updates live)':'(saved with this job)';
const arr=lines||[];
if(!arr.length){
logBox.innerHTML='<span class="empty-log">No log lines yet&hellip;</span>';
return;
}
logBox.textContent=arr.join('\n');
logBox.scrollTop=logBox.scrollHeight;
}
function setFile(f){ chosen=f; dropLabel.textContent=f?('Selected: '+f.name):'Drop a PDF drawing set here, or click to choose'; function setFile(f){ chosen=f; dropLabel.textContent=f?('Selected: '+f.name):'Drop a PDF drawing set here, or click to choose';
runBtn.disabled=!f; } runBtn.disabled=!f; }
dropLabel.addEventListener('click',()=>fileInput.click()); dropLabel.addEventListener('click',()=>fileInput.click());
@@ -160,6 +240,7 @@ runBtn.addEventListener('click',async e=>{
if(!chosen) return; if(!chosen) return;
runBtn.disabled=true; results.innerHTML=''; runBtn.disabled=true; results.innerHTML='';
statusEl.innerHTML='<span class="spinner"></span>Uploading...'; statusEl.innerHTML='<span class="spinner"></span>Uploading...';
showLog([], true);
const fd=new FormData(); fd.append('file',chosen); const fd=new FormData(); fd.append('file',chosen);
const email=(emailEl.value||'').trim(); if(email) fd.append('notification_email',email); const email=(emailEl.value||'').trim(); if(email) fd.append('notification_email',email);
['project_name','address','occupancy','work_type'].forEach(id=>{ ['project_name','address','occupancy','work_type'].forEach(id=>{
@@ -168,8 +249,9 @@ runBtn.addEventListener('click',async e=>{
const compute=(document.querySelector('input[name="compute"]:checked')||{}).value; const compute=(document.querySelector('input[name="compute"]:checked')||{}).value;
fd.append('text_local', compute==='local' ? 'true' : 'false'); fd.append('text_local', compute==='local' ? 'true' : 'false');
if(compute==='openrouter'){ if(compute==='openrouter'){
const modelSel=document.getElementById('model'); // Model picks only apply to OpenRouter compute; hybrid keeps its local model.
if(modelSel.value) fd.append('model', modelSel.value); if(visionSel.value) fd.append('vision_model', visionSel.value);
if(textSel.value) fd.append('text_model', textSel.value);
} }
const pipelineMode=(document.querySelector('input[name="pipeline_mode"]:checked')||{}).value||'classic'; const pipelineMode=(document.querySelector('input[name="pipeline_mode"]:checked')||{}).value||'classic';
fd.append('pipeline_mode',pipelineMode); fd.append('pipeline_mode',pipelineMode);
@@ -196,21 +278,30 @@ function poll(jobId){
const res=await fetch('/jobs/'+jobId); const res=await fetch('/jobs/'+jobId);
if(!res.ok) throw new Error('job not found'); if(!res.ok) throw new Error('job not found');
const job=await res.json(); const job=await res.json();
const live=['running','queued','finalizing'].includes(job.status);
if(job.log_tail && job.log_tail.length) showLog(job.log_tail, live);
if(job.status==='running'||job.status==='queued'){ if(job.status==='running'||job.status==='queued'){
statusEl.innerHTML='<span class="spinner"></span>'+esc(job.stage||'Working...')+ statusEl.innerHTML='<span class="spinner"></span>'+esc(job.stage||'Working...')+
' &middot; you can leave this page'; ' &middot; you can leave this page';
} else if(job.status==='done'){ } else if(job.status==='done'){
clearInterval(polling); polling=null; runBtn.disabled=false; render(job.report); clearInterval(polling); polling=null; runBtn.disabled=false;
if(job.log && job.log.length) showLog(job.log, false);
render(job.report);
} else if(job.status==='needs_review'||job.status==='reviewing'){ } else if(job.status==='needs_review'||job.status==='reviewing'){
clearInterval(polling); polling=null; runBtn.disabled=false; renderReview(job); clearInterval(polling); polling=null; runBtn.disabled=false;
if(job.log && job.log.length) showLog(job.log, false);
renderReview(job);
} else if(job.status==='finalizing'){ } else if(job.status==='finalizing'){
statusEl.innerHTML='<span class="spinner"></span>Finalizing reviewed report...'; statusEl.innerHTML='<span class="spinner"></span>Finalizing reviewed report...';
} else if(job.status==='finalization_error'){ } else if(job.status==='finalization_error'){
clearInterval(polling); polling=null; runBtn.disabled=false; clearInterval(polling); polling=null; runBtn.disabled=false;
statusEl.textContent='Finalization failed: '+(job.error||'unknown error'); statusEl.textContent='Finalization failed: '+(job.error||'unknown error');
if(job.log && job.log.length) showLog(job.log, false);
} else if(job.status==='error'){ } else if(job.status==='error'){
clearInterval(polling); polling=null; runBtn.disabled=false; clearInterval(polling); polling=null; runBtn.disabled=false;
statusEl.textContent='Run failed: '+(job.error||'unknown error'); statusEl.textContent='Run failed: '+(job.error||'unknown error');
if(job.log && job.log.length) showLog(job.log, false);
else if(job.log_tail && job.log_tail.length) showLog(job.log_tail, false);
} }
}catch(err){ clearInterval(polling); polling=null; runBtn.disabled=false; }catch(err){ clearInterval(polling); polling=null; runBtn.disabled=false;
statusEl.textContent='Error: '+err.message; } statusEl.textContent='Error: '+err.message; }
@@ -230,29 +321,11 @@ function syncPipelineOptions(){
document.querySelectorAll('input[name="pipeline_mode"]').forEach(el=>el.addEventListener('change',syncPipelineOptions)); document.querySelectorAll('input[name="pipeline_mode"]').forEach(el=>el.addEventListener('change',syncPipelineOptions));
syncPipelineOptions(); syncPipelineOptions();
// --- model picker (OpenRouter compute only) --- // --- model pickers (OpenRouter compute only) ---
let modelList=null;
async function loadModels(){
const note=document.getElementById('modelNote'), sel=document.getElementById('model');
try{
const res=await fetch('/models');
if(!res.ok) throw new Error('list unavailable');
const data=await res.json();
modelList=data.models||[];
sel.innerHTML=modelList.map(m=>
'<option value="'+escAttr(m.id)+'"'+(m.id===data.default?' selected':'')+'>'+
esc(m.name||m.id)+' &mdash; $'+esc(m.prompt_usd_per_mtok)+' / $'+esc(m.completion_usd_per_mtok)+
' per 1M tok</option>').join('');
note.textContent='('+modelList.length+' available)';
}catch(e){
sel.innerHTML='';
note.textContent='using configured default (list unavailable)';
}
}
function syncCompute(){ function syncCompute(){
const openrouter=(document.querySelector('input[name="compute"]:checked')||{}).value==='openrouter'; const openrouter=(document.querySelector('input[name="compute"]:checked')||{}).value==='openrouter';
document.getElementById('modelPick').style.display=openrouter?'block':'none'; document.getElementById('modelPick').style.display=openrouter?'block':'none';
if(openrouter&&!modelList) loadModels(); if(openrouter&&!modelsLoaded) loadModels();
} }
document.querySelectorAll('input[name="compute"]').forEach(el=>el.addEventListener('change',syncCompute)); document.querySelectorAll('input[name="compute"]').forEach(el=>el.addEventListener('change',syncCompute));
syncCompute(); syncCompute();
@@ -394,6 +467,10 @@ function render(rep){
html+='</details>'; html+='</details>';
} }
html+='<div class="meta" style="margin-top:14px">Full run log: <a href="/jobs/'+
esc(currentJobId)+'/log?plain=1" target="_blank" rel="noopener">/jobs/'+
esc(currentJobId)+'/log</a> (also saved as job.log on the server)</div>';
results.innerHTML=html; results.innerHTML=html;
} }
function stat(v,l){ return '<div class="stat"><b>'+esc(v)+'</b><span>'+esc(l)+'</span></div>'; } function stat(v,l){ return '<div class="stat"><b>'+esc(v)+'</b><span>'+esc(l)+'</span></div>'; }
+45 -7
View File
@@ -60,18 +60,56 @@ def test_job_log_endpoint_serves_log_and_404s(job_env, monkeypatch):
assert client.get("/jobs/nope/log").status_code == 404 assert client.get("/jobs/nope/log").status_code == 404
def test_model_override_set_and_cleared_around_run(job_env, monkeypatch): def test_model_overrides_passed_to_classic_runner(job_env, monkeypatch):
from backend import llm """Classic mode: per-run picks travel as run_pipeline kwargs (the runner
sets and clears llm.set_model_overrides itself)."""
seen = {} seen = {}
def fake_runner(pdf_path, **kwargs): def fake_runner(pdf_path, **kwargs):
seen["override"] = llm._model_override seen.update(kwargs)
return {"source": "set.pdf", "summary": {}} return {"source": "set.pdf", "summary": {}}
monkeypatch.setattr("backend.jobs.run_pipeline", fake_runner) monkeypatch.setattr("backend.jobs.run_pipeline", fake_runner)
jobs.create_job(str(job_env / "set.pdf"), "set.pdf", jobs.create_job(str(job_env / "set.pdf"), "set.pdf",
pipeline_mode="classic", model="openai/gpt-4o") pipeline_mode="classic",
vision_model="openai/gpt-4o", text_model="openai/gpt-4o-mini")
assert seen["override"] == "openai/gpt-4o" assert seen["vision_model"] == "openai/gpt-4o"
assert llm._model_override is None # cleared after the run assert seen["text_model"] == "openai/gpt-4o-mini"
def test_model_overrides_set_and_cleared_around_agent_run(job_env, monkeypatch):
"""Agent mode: the agent runner has no override params, so jobs.py sets
them module-level for the duration of the run."""
from backend import llm
seen = {}
def fake_agent_runner(pdf_path, **kwargs):
seen["vision"] = llm._vision_model_override
seen["text"] = llm._text_model_override
return {"source": "set.pdf", "summary": {}}
monkeypatch.setattr("backend.jobs.run_agent_pipeline", fake_agent_runner)
jobs.create_job(str(job_env / "set.pdf"), "set.pdf",
pipeline_mode="agent",
vision_model="openai/gpt-4o", text_model="openai/gpt-4o-mini")
assert seen["vision"] == "openai/gpt-4o"
assert seen["text"] == "openai/gpt-4o-mini"
assert llm._vision_model_override is None # cleared after the run
assert llm._text_model_override is None
def test_failed_run_logs_traceback(job_env, monkeypatch):
"""A crashed job must leave the traceback in job.log, not just str(e)."""
def boom(pdf_path, **kwargs):
raise RuntimeError("kaboom-stage-failure")
monkeypatch.setattr("backend.jobs.run_pipeline", boom)
job_id = jobs.create_job(str(job_env / "set.pdf"), "set.pdf", pipeline_mode="classic")
assert jobs._jobs[job_id]["status"] == "error"
content = (job_env / job_id / "job.log").read_text()
assert "Traceback (most recent call last)" in content
assert "RuntimeError: kaboom-stage-failure" in content
+27 -3
View File
@@ -11,12 +11,23 @@ _PAYLOAD = {
"name": "GPT-4o", "name": "GPT-4o",
"pricing": {"prompt": "0.0000025", "completion": "0.00001"}, "pricing": {"prompt": "0.0000025", "completion": "0.00001"},
"context_length": 128000, "context_length": 128000,
"architecture": {"input_modalities": ["text", "image"],
"output_modalities": ["text"]},
}, },
{ {
"id": "google/gemini-2.5-pro", "id": "google/gemini-2.5-pro",
"name": "Gemini 2.5 Pro", "name": "Gemini 2.5 Pro",
"pricing": {"prompt": "0.00000125", "completion": "0.00001"}, "pricing": {"prompt": "0.00000125", "completion": "0.00001"},
"context_length": 1000000, "context_length": 1000000,
"architecture": {"modality": "text+image->text"},
},
{
"id": "meta-llama/llama-3.1-70b-instruct",
"name": "Llama 3.1 70B Instruct",
"pricing": {"prompt": "0.0000005", "completion": "0.0000008"},
"context_length": 131072,
"architecture": {"input_modalities": ["text"],
"output_modalities": ["text"]},
}, },
] ]
} }
@@ -34,14 +45,27 @@ def test_models_endpoint_normalizes_pricing(monkeypatch):
response = client.get("/models") response = client.get("/models")
assert response.status_code == 200 assert response.status_code == 200
body = response.json() body = response.json()
assert body["default"] == config.MODEL assert body["defaults"] == {"vision": config.MODEL, "text": config.TEXT_MODEL}
assert body["default_text"] == config.TEXT_MODEL by_id = {m["id"]: m for m in body["text"]}
by_id = {m["id"]: m for m in body["models"]}
assert by_id["openai/gpt-4o"]["prompt_usd_per_mtok"] == 2.5 assert by_id["openai/gpt-4o"]["prompt_usd_per_mtok"] == 2.5
assert by_id["openai/gpt-4o"]["completion_usd_per_mtok"] == 10.0 assert by_id["openai/gpt-4o"]["completion_usd_per_mtok"] == 10.0
assert by_id["openai/gpt-4o"]["context_length"] == 128000 assert by_id["openai/gpt-4o"]["context_length"] == 128000
def test_models_endpoint_splits_vision_and_text(monkeypatch):
_reset_cache()
monkeypatch.setattr(models, "_fetch_openrouter_models", lambda: _PAYLOAD["data"])
client = TestClient(app)
body = client.get("/models").json()
vision_ids = {m["id"] for m in body["vision"]}
text_ids = {m["id"] for m in body["text"]}
# Both modality shapes (structured and legacy string) are recognized.
assert vision_ids == {"openai/gpt-4o", "google/gemini-2.5-pro"}
# Text list is the full catalog; vision models appear in both.
assert text_ids == {"openai/gpt-4o", "google/gemini-2.5-pro",
"meta-llama/llama-3.1-70b-instruct"}
def test_models_endpoint_caches(monkeypatch): def test_models_endpoint_caches(monkeypatch):
_reset_cache() _reset_cache()
calls = [] calls = []
+33 -15
View File
@@ -1,26 +1,44 @@
from backend import config from backend import config
from backend.llm import _resolve_backend, set_model_override from backend.llm import _resolve_backend, set_model_overrides, set_text_backend
def test_override_wins_for_vision_and_text(): def teardown_function():
set_model_override("openai/gpt-4o") set_model_overrides(None, None)
try: set_text_backend(False)
assert _resolve_backend(has_images=True, model_override=None)["model"] == "openai/gpt-4o"
assert _resolve_backend(has_images=False, model_override=None)["model"] == "openai/gpt-4o"
finally: def test_vision_override_wins_for_vision_only():
set_model_override(None) set_model_overrides(vision="openai/gpt-4o", text=None)
assert _resolve_backend(has_images=True, model_override=None)["model"] == "openai/gpt-4o"
assert _resolve_backend(has_images=False, model_override=None)["model"] == config.TEXT_MODEL
def test_text_override_wins_for_text_only():
set_model_overrides(vision=None, text="anthropic/claude-sonnet-4")
assert _resolve_backend(has_images=False, model_override=None)["model"] == "anthropic/claude-sonnet-4"
assert _resolve_backend(has_images=True, model_override=None)["model"] == config.MODEL
def test_override_beats_per_call_model_arg(): def test_override_beats_per_call_model_arg():
set_model_override("openai/gpt-4o") set_model_overrides(vision="openai/gpt-4o", text="openai/gpt-4o-mini")
try: # Agents pass their AGENT_*_MODEL per call; the user's job pick wins.
# Agents pass their AGENT_*_MODEL per call; the user's job pick wins. assert _resolve_backend(has_images=True, model_override="other/model")["model"] == "openai/gpt-4o"
assert _resolve_backend(has_images=False, model_override="other/model")["model"] == "openai/gpt-4o" assert _resolve_backend(has_images=False, model_override="other/model")["model"] == "openai/gpt-4o-mini"
finally:
set_model_override(None)
def test_no_override_keeps_defaults(): def test_no_override_keeps_defaults():
set_model_override(None) set_model_overrides(None, None)
assert _resolve_backend(has_images=True, model_override=None)["model"] == config.MODEL assert _resolve_backend(has_images=True, model_override=None)["model"] == config.MODEL
assert _resolve_backend(has_images=False, model_override=None)["model"] == config.TEXT_MODEL assert _resolve_backend(has_images=False, model_override=None)["model"] == config.TEXT_MODEL
def test_ui_picks_never_name_the_local_model(monkeypatch):
"""Hybrid runs keep LOCAL_TEXT_MODEL; OpenRouter picks must not leak into
the local endpoint (a vLLM server won't serve OpenRouter model ids)."""
monkeypatch.setattr(config, "LOCAL_BASE_URL", "http://localhost:8000/v1")
monkeypatch.setattr(config, "LOCAL_TEXT_MODEL", "qwen/local-instruct")
set_text_backend(True)
set_model_overrides(vision="openai/gpt-4o", text="anthropic/claude-sonnet-4")
be = _resolve_backend(has_images=False, model_override=None)
assert be["local"] is True
assert be["model"] == "qwen/local-instruct"