diff --git a/NOTES.md b/NOTES.md new file mode 100644 index 0000000..1b2daf9 --- /dev/null +++ b/NOTES.md @@ -0,0 +1,160 @@ +# Session Notes — Conflict Checker + +Orientation for a new coding session. Setup and Docker details live in [README.md](README.md). This file tracks what the code actually does and what tends to waste time. + +**Last updated:** 2026-08-02 · `agent-mode` branch (merged `main` tip `bf508bf`) + +## What this is + +Cross-discipline **design contradiction / senior-architect QAQC** for construction drawing PDFs (Arch, Struct, Mech, Elec, Plumb, FP, etc.). Flags disagreements *between disciplines* before a set goes to bid/permit. + +**Not** IronBid’s scope-ownership conflict checker. + +Source of truth: Scout IT Gitea — `gitea.scoutitsystems.com/woogi/Conflict_Checker`. + +## Stack + +| Layer | Detail | +|-------|--------| +| API | Python 3.12, FastAPI, Uvicorn ([backend/main.py](backend/main.py)) | +| UI | Single static file [frontend/index.html](frontend/index.html), served by FastAPI | +| Pipelines | **Classic:** [backend/pipeline/runner.py](backend/pipeline/runner.py) (shared by web + CLI). **Agent:** [backend/agents/runner.py](backend/agents/runner.py) — experimental scoped specialist agents, selected per job (`pipeline_mode`) | +| Review gate | Agent jobs stop at `needs_review` for human decisions before the report emails ([backend/review/](backend/review/)) | +| LLM | OpenRouter via `openai` SDK; default `google/gemini-2.5-pro`. Vision always OpenRouter; text stages can use local vLLM (classic/hybrid only) | +| PDF | `pdf2image` + system `poppler-utils` → JPEG page images | +| Jobs | In-memory threads ([backend/jobs.py](backend/jobs.py)) — no Redis/DB | +| Tests | `pytest tests/` (~82 tests; see Quick start) | +| Deploy | Docker Compose; app on port **8099**; public URL `https://conchecker.scoutitsystems.com` | + +## Live pipeline (authoritative) + +README still describes an older 5-stage extract-then-compare loop. **Trust `runner.py`.** Actual flow: + +``` +PDF → images → extract → sheet index → jurisdiction + → normalize → project intelligence (GOIDs) + → cluster → conflict reason + → QAQC / code / constructability + → dedup-validate → risk → RFIs → report +``` + +| Runner stage | Module | Notes | +|--------------|--------|--------| +| PDF → images | `pdf_processor` | Rasterize | +| Extract assertions | `extractor` | Vision, per sheet | +| Classify sheet index | `sheet_index` | LLM | +| Jurisdiction profile | `jurisdiction` | After cover meta | +| Normalize | `normalizer` | LLM batches | +| Project intelligence | `normalizer.build_project_intelligence` | GOIDs + relationships | +| Cluster | `llm_clusterer` or `clusterer` | Default `CLUSTERER=llm` | +| Conflicts | `conflict_checker` | Per-cluster vision reason | +| QAQC / code / constructability | `qaqc_review`, `code_review`, `constructability` | Full-set / batched | +| Validate & dedup | `validator` | Merges conflict + QAQC + code + construct issues | +| Risk / RFIs | `risk`, `rfi` | Text-only | +| Report | `report` | `conflicts.json` + `report.md` (+ stage JSON dumps when `out_dir` set) | + +Design notes for Stage 2/3 engines also live under `Changes/*.docx`. + +**Agent pipeline** (`pipeline_mode=agent`): scoped specialist agents in [backend/agents/](backend/agents/) (extractors, linker, brain, critics, RFI writer) run through `Orchestrator` + `ProjectMemory`; artifacts under `outputs//agent/`. Agent mode is OpenRouter-only (no hybrid) and, when `AGENT_REQUIRE_REVIEW=true`, stops at `needs_review` until a human saves decisions and finalizes via the review endpoints. Design docs: `docs/superpowers/`. + +## HTTP API (current) + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/health` | Liveness + `model` / `text_model` + `version` / `build` + key/email flags | +| GET | `/models` | OpenRouter catalog split into `vision[]` / `text[]` + `defaults`, with per-1M-token pricing (cached ~1h; **502** when OpenRouter is unreachable) | +| POST | `/check` | Upload PDF; returns `{job_id}` immediately | +| GET | `/jobs/{id}` | Status poll. Running: `stage` + `log_tail`. Done/error/needs_review/finalization_error: `report` and/or `error` + full `log` | +| GET | `/jobs/{id}/log` | Full run log as `text/plain` (404 when no log file) | +| GET | `/jobs/{id}/review` | Review queue + progress + saved decisions (agent jobs) | +| POST | `/jobs/{id}/review-decisions` | Save reviewer decisions (409 outside needs_review/reviewing) | +| POST | `/jobs/{id}/finalize-review` | Background finalize + send report (409 unless review gate passed) | +| GET | `/jobs/{id}/sheet-image/{page}` | JPEG of source PDF page for the sheet viewer | +| GET | `/` | Serves `frontend/index.html` | + +### `/check` form fields + +- Required: `file` (PDF) +- Optional: `notification_email`, `project_name`, `address`, `occupancy`, `work_type` +- Pipeline: `pipeline_mode` (`classic` default, or `agent`) +- Compute: `text_local` (`true` = hybrid local text; forced off for agent mode) +- Models: `vision_model`, `text_model` (OpenRouter ids; blank = config defaults) + +## Job logs + +Pipeline `print()` is teed for the job thread ([backend/job_log.py](backend/job_log.py)): + +- Live: `GET /jobs/{id}` → `log_tail` (last 80 lines) +- Done/error/needs_review/finalization_error: same payload includes full `log` +- Disk: `backend/outputs//job.log` (survives restart; status registry does not) +- API: `GET /jobs/{id}/log` → `text/plain` +- UI: “Run log” panel updates while running; stays visible after finish/fail/review +- Failures append the **full traceback** to the log; each run starts with a header line (job id, mode, models, start time) +- `outputs//job.json` (written at start) carries email/mode/models so the disk fallback can rebuild a job after restart + +## Vision vs text models + +Two models, not one: + +| Role | Config | Stages | Backend | +|------|--------|--------|---------| +| Vision | `MODEL` | Extract, conflict reason (images) | Always OpenRouter | +| Text | `TEXT_MODEL` (falls back to `MODEL`) | Sheet index, jurisdiction, normalize, cluster(LLM), QAQC, code, construct, validate, risk, RFI | OpenRouter, or local when hybrid | + +UI: two dropdowns (with per-1M pricing) filled from `GET /models` ([backend/models.py](backend/models.py)), shown only for OpenRouter compute. Per-run picks go through `set_model_overrides(vision, text)` in [backend/llm.py](backend/llm.py): classic runs pass them as `run_pipeline` kwargs (runner clears in `finally`); agent runs set them module-level around `run_agent_pipeline`. UI picks beat per-call agent `AGENT_*_MODEL` args but **never name the hybrid local model** — local stays on `LOCAL_TEXT_MODEL`; the text pick only covers the cloud fallback. + +## Where to change what + +| Concern | File | +|---------|------| +| Prompts, vocab, conflict taxonomy | [backend/prompts.py](backend/prompts.py) | +| Env knobs | [backend/config.py](backend/config.py), [backend/.env.example](backend/.env.example) | +| HTTP API | [backend/main.py](backend/main.py) | +| UI (upload, models, live log, results) | [frontend/index.html](frontend/index.html) | +| CLI tuning loop | [cli/run_check.py](cli/run_check.py) | +| LLM client, cache, cost, model overrides | [backend/llm.py](backend/llm.py) | +| OpenRouter catalog + pricing + vision/text split | [backend/models.py](backend/models.py) | +| Agent-mode pipeline | [backend/agents/](backend/agents/) (`runner.py` entry; agents call `llm.call_json` with per-call model args) | +| Human review gate (queue, decisions, finalize) | [backend/review/](backend/review/) | +| Job registry + stdout tee log | [backend/jobs.py](backend/jobs.py), [backend/job_log.py](backend/job_log.py) | +| Stage helpers (prompt render, issue validate) | [backend/pipeline/_stage.py](backend/pipeline/_stage.py) | +| Code text corpus (Stage 7) | [backend/code_corpus/](backend/code_corpus/) | +| Hybrid local LLM helper | [scripts/setup_vllm.sh](scripts/setup_vllm.sh) | + +Older prompt snapshot: `backend/prompts.py.v1`. + +## Gotchas + +1. **Prompt placeholders** — Use `str.replace` via `_stage.render`, never `str.format`. Prompts contain literal `{` JSON braces. +2. **Clustering** — Default is LLM (`CLUSTERER=llm`); empty LLM result falls back to deterministic. Deterministic clusters need ≥2 disciplines (or schedule-vs-plan); single-discipline “missing” gaps are a known limit. +3. **Jobs are in-memory** — Process restart clears job status; `outputs//` (report + `job.log` + `job.json` + `source.pdf`) still reload via disk fallback, including `needs_review` recovery. +4. **Dependency pin** — `httpx==0.27.2` with `openai==1.51.0`. httpx ≥0.28 breaks openai’s `proxies=` kwarg. +5. **Code corpus licensing** — Only `ada_2010.txt` is shipped. Do not paste IBC/IFC/IECC without a license (see `backend/code_corpus/README.md`). +6. **Dual assertion schema** — Newer `{sheet, objects[]}` is mapped to legacy `{assertions[]}` with `attribute`/`value` for older stages. +7. **Grounding guard** — Extractor drops objects whose numeric claims are not in `source_text` (graphical-only objects allowed). +8. **Module globals** — LLM cost counters, stdout tee, and model overrides are process-global; overlapping jobs can interleave (single-user tool assumption). +9. **Samples** — `samples/*.pdf` are gitignored; drop PDFs locally for CLI runs. +10. **README drift** — Treat README for setup/CI; treat this file + `runner.py` for pipeline truth. `prompts.py` header may still say some prompts are unwired — they are wired through the runner. +11. **Git identity** — This box has no `user.name` / `user.email`; commits need `GIT_AUTHOR_*` / `GIT_COMMITTER_*` env vars (do not `git config`). Remote push to Gitea works. +12. **No local Python deps on host** — App is meant to run in Docker; bare `python3` imports may miss `dotenv` / `httpx`. Prefer `docker compose`. For tests on this box: venv + requirements, but unpin Pillow (`Pillow>=11`) — 10.4.0 doesn't build on Python 3.14 (Docker uses 3.12, where the pin is fine). +13. **Agent mode constraints** — OpenRouter-only (hybrid disabled in UI and forced off server-side); review gate statuses are `needs_review → reviewing → finalizing → done` (`finalization_error` on finalize failure); only terminal states include the full `log` in polls. + +## Quick start pointers + +- Full setup: [README.md](README.md) (`docker compose up -d --build` → http://localhost:8099). +- Local CLI: `python cli/run_check.py samples/your_set.pdf --out out/your_set`. +- Prompt iteration: set `LLM_CACHE=true` in `backend/.env` so unchanged stages replay for free; clear with `rm -rf backend/.llm_cache`. +- Artifacts per job: `assertions.json`, `clusters.json`, per-stage JSON, `conflicts.json`, `report.md`, `job.log`, `job.json`, `source.pdf` under `backend/outputs//`. +- Tests: `python -m pytest tests/` (needs the deps from `requirements.txt` + `pytest`; on this box use a venv, see gotcha #12). + +## Recent work (2026-08-02, agent-mode) + +Merged `main` tip (`a6b0c8f` + `bf508bf`) into `agent-mode`, reconciling with this branch's own earlier implementations: + +- **Two model dropdowns** — main's vision/text split ported onto this branch's priced catalog (`models.py`); pickers stay OpenRouter-compute-only, and UI picks never override `LOCAL_TEXT_MODEL` (main's hybrid footgun avoided). +- **Better run logs** — main's timestamped line-splitting tee, `log_tail` polls, terminal-state full log, and log-only disk recovery merged with this branch's header line, `job.json` metadata, and review-gate states. Failed runs now also append the traceback to `job.log`. +- `backend/models_catalog.py` (main's unpriced catalog) intentionally dropped in favor of `models.py`. + +## Conflict categories (taxonomy) + +Defined in `backend/prompts.py`: `dimensional_disagreement`, `elevation_disagreement`, `location_mismatch`, `missing_element`, `schedule_vs_plan_mismatch`, `detail_vs_plan_mismatch`, `tag_or_reference_inconsistency`, `spatial_clash`, `note_or_spec_contradiction`. diff --git a/backend/job_log.py b/backend/job_log.py new file mode 100644 index 0000000..a69078d --- /dev/null +++ b/backend/job_log.py @@ -0,0 +1,93 @@ +""" +job_log.py - Capture pipeline stdout/stderr into a per-job log. + +The pipeline already prints stage progress via print(). For a reviewable +post-run log we tee those lines into memory + outputs//job.log +without rewriting every call site. +""" + +import sys +import time +from contextlib import contextmanager +from typing import Callable, Iterator, List, Optional, TextIO + + +class _LineSplitter: + """Accumulate write() chunks and emit complete lines.""" + + def __init__(self, on_line: Callable[[str], None]): + self._buf = "" + self._on_line = on_line + + def write(self, s: str) -> None: + if not s: + return + self._buf += s + while "\n" in self._buf: + line, self._buf = self._buf.split("\n", 1) + # Strip trailing CR from Windows-ish streams; keep content intact. + self._on_line(line.rstrip("\r")) + + def flush_remainder(self) -> None: + if self._buf: + self._on_line(self._buf.rstrip("\r")) + self._buf = "" + + +class _Tee: + """Mirror writes to the original stream and a line callback.""" + + def __init__(self, stream: TextIO, on_line: Callable[[str], None]): + self._stream = stream + self._lines = _LineSplitter(on_line) + + def write(self, s: str) -> int: + n = self._stream.write(s) + self._stream.flush() + self._lines.write(s) + return n + + def flush(self) -> None: + self._stream.flush() + + def flush_remainder(self) -> None: + self._lines.flush_remainder() + + def __getattr__(self, name: str): + return getattr(self._stream, name) + + +def stamp_line(line: str, t: Optional[float] = None) -> str: + """Prefix a log line with HH:MM:SS.""" + ts = time.strftime("%H:%M:%S", time.localtime(t if t is not None else time.time())) + return f"[{ts}] {line}" + + +@contextmanager +def capture_stdio(on_line: Callable[[str], None]) -> Iterator[None]: + """ + Tee sys.stdout and sys.stderr into on_line(raw_line) for the duration. + + Safe for the single-job-at-a-time usage of this app; overlapping jobs + would interleave (same limitation as the LLM cost counters). + """ + old_out, old_err = sys.stdout, sys.stderr + tee_out = _Tee(old_out, on_line) + tee_err = _Tee(old_err, on_line) + sys.stdout = tee_out # type: ignore[assignment] + sys.stderr = tee_err # type: ignore[assignment] + try: + yield + finally: + tee_out.flush_remainder() + tee_err.flush_remainder() + sys.stdout = old_out + sys.stderr = old_err + + +def read_log_file(path: str) -> List[str]: + try: + with open(path, encoding="utf-8") as f: + return [ln.rstrip("\n") for ln in f] + except OSError: + return [] diff --git a/backend/jobs.py b/backend/jobs.py index 53c9d79..d199541 100644 --- a/backend/jobs.py +++ b/backend/jobs.py @@ -9,61 +9,33 @@ for the completion email. State is in-memory (fine for a single-user tool); the report is also persisted to outputs// so results survive a restart even though live status does not. No external queue/DB. + +A teed stdout/stderr log is kept in memory and written to outputs//job.log +so failed or suspicious runs can be reviewed after the fact. """ import json -import contextlib import os -import sys import time +import traceback import uuid import shutil import threading -from typing import Dict, Optional +from typing import Dict, List, Optional from backend import config from backend import llm +from backend.job_log import capture_stdio, read_log_file, stamp_line from backend.agents.runner import run_agent_pipeline from backend.pipeline.runner import run_pipeline from backend.email_sender import send_conflict_report, send_review_required _jobs: Dict[str, Dict] = {} _lock = threading.Lock() +_LOG_TAIL = 80 PIPELINE_MODES = {"classic", "agent"} - - -class _Tee: - """Write to both the real stream and the job log file.""" - - def __init__(self, stream, log_file) -> None: - self._stream = stream - self._log = log_file - - def write(self, data): - self._stream.write(data) - self._log.write(data) - - def flush(self): - self._stream.flush() - self._log.flush() - - -@contextlib.contextmanager -def _tee_log(log_path: str, header: str): - """Mirror stdout/stderr into a per-job log file for the duration of a run. - - sys.stdout is process-global, so two concurrent jobs would interleave in - each other's logs - acceptable for this single-user tool (same tradeoff as - the LLM cost globals in llm.py). - """ - with open(log_path, "a", encoding="utf-8") as log_file: - log_file.write(header + "\n") - real_out, real_err = sys.stdout, sys.stderr - sys.stdout, sys.stderr = _Tee(real_out, log_file), _Tee(real_err, log_file) - try: - yield - finally: - sys.stdout, sys.stderr = real_out, real_err +# States where the job will produce no more log output; polls get the full log. +_TERMINAL_STATES = {"done", "error", "needs_review", "finalization_error"} def _set(job_id: str, **fields) -> None: @@ -71,16 +43,39 @@ def _set(job_id: str, **fields) -> None: _jobs[job_id].update(fields) -def create_job(pdf_path: str, source_filename: str, email: Optional[str] = None, - project_input: Optional[Dict] = None, text_local: bool = False, - pipeline_mode: str = "classic", model: Optional[str] = None) -> str: +def _append_log(job_id: str, raw_line: str, log_path: str) -> None: + """Stamp, store, and append one captured stdout/stderr line.""" + entry = stamp_line(raw_line) + with _lock: + job = _jobs.get(job_id) + if job is not None: + job.setdefault("log", []).append(entry) + try: + os.makedirs(os.path.dirname(log_path), exist_ok=True) + with open(log_path, "a", encoding="utf-8") as f: + f.write(entry + "\n") + except OSError: + # Don't fail the job over log I/O; avoid print() here — it would + # re-enter the stdio tee while a job is capturing. + pass + + +def create_job( + pdf_path: str, + source_filename: str, + email: Optional[str] = None, + project_input: Optional[Dict] = None, + text_local: bool = False, + pipeline_mode: str = "classic", + vision_model: Optional[str] = None, + text_model: Optional[str] = None, +) -> str: """Register a job and kick off its background thread. Returns the job_id.""" pipeline_mode = pipeline_mode.strip().lower() if pipeline_mode not in PIPELINE_MODES: raise ValueError(f"Unsupported pipeline mode: {pipeline_mode!r}") # Agent mode v1 is OpenRouter-only. text_local = bool(text_local and pipeline_mode == "classic") - model = (model or "").strip() or None job_id = uuid.uuid4().hex[:12] with _lock: _jobs[job_id] = { @@ -91,35 +86,61 @@ def create_job(pdf_path: str, source_filename: str, email: Optional[str] = None, "project_input": project_input or {}, "text_local": text_local, "pipeline_mode": pipeline_mode, - "model": model, + "vision_model": (vision_model or "").strip() or None, + "text_model": (text_model or "").strip() or None, "stage": None, "created_at": time.time(), "finished_at": None, "report": None, "error": None, + "log": [], } - threading.Thread(target=_run, args=( - job_id, pdf_path, project_input, text_local, pipeline_mode, model, - ), - daemon=True).start() + threading.Thread( + target=_run, + args=(job_id, pdf_path, project_input, text_local, pipeline_mode, + vision_model, text_model), + daemon=True, + ).start() return job_id -def _run(job_id: str, pdf_path: str, project_input: Optional[Dict] = None, - text_local: bool = False, pipeline_mode: str = "classic", - model: Optional[str] = None) -> None: +def _run( + job_id: str, + pdf_path: str, + project_input: Optional[Dict] = None, + text_local: bool = False, + pipeline_mode: str = "classic", + vision_model: Optional[str] = None, + text_model: Optional[str] = None, +) -> None: out_dir = os.path.join(config.OUTPUT_DIR, job_id) + log_path = os.path.join(out_dir, "job.log") try: _set(job_id, status="running") - # Keep a copy of the source PDF so its sheets can be viewed later. os.makedirs(out_dir, exist_ok=True) + # Truncate any leftover log if job_id somehow collided (shouldn't). + with open(log_path, "w", encoding="utf-8"): + pass header = (f"=== Job {job_id} | {pipeline_mode} | {_jobs[job_id].get('source')} | " - f"model={model or 'default'} | " + f"vision={vision_model or 'default'} text={text_model or 'default'} | " f"started {time.strftime('%Y-%m-%d %H:%M:%S %Z', time.gmtime())} UTC ===") - with _tee_log(os.path.join(out_dir, "job.log"), header): + _append_log(job_id, header, log_path) + + def on_line(raw: str) -> None: + _append_log(job_id, raw, log_path) + + with capture_stdio(on_line): _run_pipeline(job_id, pdf_path, out_dir, project_input, text_local, - pipeline_mode, model) + pipeline_mode, vision_model, text_model) except Exception as e: + # Land the failure AND its traceback in the job log so failed runs can + # be diagnosed from the log alone (the stdio tee is already torn down). + try: + _append_log(job_id, f"[Jobs] Job {job_id} failed: {e}", log_path) + for ln in traceback.format_exc().rstrip().splitlines(): + _append_log(job_id, ln, log_path) + except Exception: + pass print(f"[Jobs] Job {job_id} failed: {e}") _set(job_id, status="error", error=str(e), finished_at=time.time()) _notify_error(job_id) @@ -132,7 +153,8 @@ def _run(job_id: str, pdf_path: str, project_input: Optional[Dict] = None, def _run_pipeline(job_id: str, pdf_path: str, out_dir: str, project_input: Optional[Dict], text_local: bool, - pipeline_mode: str, model: Optional[str]) -> None: + pipeline_mode: str, vision_model: Optional[str], + text_model: Optional[str]) -> None: """The body of a job run; executes inside the job's tee'd log capture.""" # Persist minimal job metadata so the disk fallback in get_job can # recover the recipient email / pipeline mode after a server restart @@ -143,7 +165,10 @@ def _run_pipeline(job_id: str, pdf_path: str, out_dir: str, "email": _jobs[job_id].get("email"), "pipeline_mode": pipeline_mode, "source": _jobs[job_id].get("source"), + "vision_model": vision_model, + "text_model": text_model, }, f, indent=2) + # Keep a copy of the source PDF so its sheets can be viewed later. shutil.copy2(pdf_path, os.path.join(out_dir, "source.pdf")) runner = run_agent_pipeline if pipeline_mode == "agent" else run_pipeline runner_kwargs = { @@ -153,17 +178,22 @@ def _run_pipeline(job_id: str, pdf_path: str, out_dir: str, "source_name": _jobs[job_id].get("source"), } if pipeline_mode == "classic": + # run_pipeline takes the picks as params and clears them in finally. runner_kwargs["text_local"] = text_local + runner_kwargs["vision_model"] = vision_model + runner_kwargs["text_model"] = text_model else: runner_kwargs["require_review"] = config.AGENT_REQUIRE_REVIEW - if model: - print(f"[Jobs] Model override for this run: {model}") - llm.set_model_override(model) + # The agent runner has no override params; set them module-level. + if vision_model or text_model: + print(f"[Jobs] Model overrides for this run: " + f"vision={vision_model or '(default)'} text={text_model or '(default)'}") + llm.set_model_overrides(vision_model, text_model) try: report = runner(pdf_path, **runner_kwargs) finally: - if model: - llm.set_model_override(None) + if pipeline_mode == "agent": + llm.set_model_overrides(None, None) report.setdefault("summary", {})["pipeline_mode"] = pipeline_mode if report["summary"].get("agent_status") == "needs_review": # Human-review gate: hold the job, don't email the unreviewed report. @@ -197,11 +227,6 @@ def _notify_error(job_id: str) -> None: email = job.get("email") if not email: return - # Reuse the report mailer with a minimal error-shaped payload. - err_report = { - "source": job.get("source", ""), - "summary": {"conflicts_found": 0, "by_severity": {}, "disciplines": []}, - } try: from backend.email_sender import _smtp_ready, _send from email.message import EmailMessage @@ -216,6 +241,7 @@ def _notify_error(job_id: str) -> None: "Your conflict check did not complete.\n\n" f"Drawing set: {job.get('source','')}\n" f"Error: {job.get('error','unknown')}\n\n" + f"Review the run log at: {config.APP_BASE_URL.rstrip('/')}/?job={job_id}\n\n" "Generated by Conflict Checker" ) _send(msg) @@ -223,28 +249,63 @@ def _notify_error(job_id: str) -> None: print(f"[Email] Failed to send error notice: {e}") +def _log_from_disk(job_id: str) -> List[str]: + return read_log_file(os.path.join(config.OUTPUT_DIR, job_id, "job.log")) + + +def get_job_log(job_id: str) -> Optional[List[str]]: + """Full job log lines, from memory or disk. None if job unknown.""" + with _lock: + job = _jobs.get(job_id) + if job is not None: + return list(job.get("log") or []) + + log = _log_from_disk(job_id) + # Job exists on disk if we have a log or a report artifact. + report_path = os.path.join(config.OUTPUT_DIR, job_id, "conflicts.json") + if log or os.path.isfile(report_path): + return log + return None + + def get_job(job_id: str) -> Optional[Dict]: """Public job view. Includes the full report only when done. - Falls back to the on-disk conflicts.json when the job isn't in the - in-memory registry (e.g. after a server restart). + Falls back to the on-disk artifacts (conflicts.json / job.log) when the + job isn't in the in-memory registry (e.g. after a server restart). """ with _lock: job = _jobs.get(job_id) if job: - return dict(job) + out = dict(job) + log = list(job.get("log") or []) + out["log_tail"] = log[-_LOG_TAIL:] + # Full log on terminal states so the UI can show it without a + # second fetch; keep polls light while running. + if out.get("status") in _TERMINAL_STATES: + out["log"] = log + else: + out.pop("log", None) + return out # Try loading from disk report_path = os.path.join(config.OUTPUT_DIR, job_id, "conflicts.json") - if not os.path.isfile(report_path): + log = _log_from_disk(job_id) + if not os.path.isfile(report_path) and not log: return None try: - with open(report_path, encoding="utf-8") as f: - report = json.load(f) - summary = report.get("summary", {}) - # Recover the job's real state: a job that stopped at the review gate - # must come back as needs_review (not done) or it can never finalize. - status = "needs_review" if summary.get("agent_status") == "needs_review" else "done" + report = None + if os.path.isfile(report_path): + with open(report_path, encoding="utf-8") as f: + report = json.load(f) + summary = (report or {}).get("summary", {}) + if report is None: + # Crashed before writing a report; the log is the only artifact. + status = "error" + else: + # Recover the job's real state: a job that stopped at the review gate + # must come back as needs_review (not done) or it can never finalize. + status = "needs_review" if summary.get("agent_status") == "needs_review" else "done" # job.json (written at job start) carries the recipient email and # pipeline mode so the final notification still fires after a restart. # Missing/corrupt job.json degrades to the previous derivations. @@ -261,16 +322,20 @@ def get_job(job_id: str) -> Optional[Dict]: job = { "job_id": job_id, "status": status, - "source": meta.get("source") or report.get("source", os.path.basename(report_path)), + "source": meta.get("source") or (report or {}).get("source", os.path.basename(report_path)), "email": meta.get("email"), - "project_input": report.get("project_input", {}), + "project_input": (report or {}).get("project_input", {}), "text_local": summary.get("text_backend") == "local", "pipeline_mode": meta.get("pipeline_mode") or summary.get("pipeline_mode", "classic"), + "vision_model": meta.get("vision_model"), + "text_model": meta.get("text_model"), "stage": None, "created_at": os.path.getmtime(source_pdf) if os.path.isfile(source_pdf) else None, - "finished_at": os.path.getmtime(report_path), + "finished_at": os.path.getmtime(report_path) if os.path.isfile(report_path) else None, "report": report, - "error": None, + "error": None if report is not None else "Report missing; see job log", + "log": log, + "log_tail": log[-_LOG_TAIL:], } # Hydrate the in-memory registry so _set(...) transitions (reviewing, # finalizing, done) work for restart-recovered jobs. diff --git a/backend/llm.py b/backend/llm.py index 4e2151d..7871b67 100644 --- a/backend/llm.py +++ b/backend/llm.py @@ -23,16 +23,13 @@ _clients: Dict[str, OpenAI] = {} # call_json when routing a no-image (text) call. Module-global mirrors the # set_stage/cost pattern (single-user tool). _text_local = False - -# Per-job model override (user picked a model in the UI). Same module-global +# Per-job model overrides (user picked models in the UI). Same module-global # pattern: set by the job runner before the pipeline starts, cleared after. -_model_override: Optional[str] = None - - -def set_model_override(model: Optional[str]) -> None: - """Override the model for all OpenRouter calls (vision + text), or None to clear.""" - global _model_override - _model_override = (model or "").strip() or None +# Vision applies to image calls, text to no-image calls on OpenRouter (and to +# the local->cloud fallback). The LOCAL endpoint's model name is never taken +# from these overrides - hybrid local keeps LOCAL_TEXT_MODEL. +_vision_model_override: Optional[str] = None +_text_model_override: Optional[str] = None def set_text_backend(local: bool) -> None: @@ -40,6 +37,13 @@ def set_text_backend(local: bool) -> None: global _text_local _text_local = bool(local) + +def set_model_overrides(vision: Optional[str] = None, text: Optional[str] = None) -> None: + """Per-run OpenRouter vision/text model picks. None/blank clears to defaults.""" + global _vision_model_override, _text_model_override + _vision_model_override = (vision or "").strip() or None + _text_model_override = (text or "").strip() or None + # --- per-job cost accounting ------------------------------------------------- # OpenRouter returns the real USD cost of each call when we request usage # accounting. We accumulate it in a module-level counter; the runner resets it @@ -177,17 +181,22 @@ def _resolve_backend(has_images: bool, model_override: Optional[str]) -> Dict[st return { "base_url": config.LOCAL_BASE_URL, "api_key": config.LOCAL_API_KEY, + # Local model name comes from per-call args or LOCAL_TEXT_MODEL — + # never the UI's OpenRouter picks, which a local server won't serve. "model": model_override or config.LOCAL_TEXT_MODEL or config.TEXT_MODEL, "usage": False, # local has no OpenRouter usage accounting "local": True, } # Vision, or text-on-OpenRouter (default / fallback). A per-job override # (user's UI model pick) wins over per-call and env defaults. - default_model = config.MODEL if has_images else config.TEXT_MODEL + if has_images: + model = _vision_model_override or model_override or config.MODEL + else: + model = _text_model_override or model_override or config.TEXT_MODEL return { "base_url": config.AI_BASE_URL, "api_key": config.AI_API_KEY, - "model": _model_override or model_override or default_model, + "model": model, "usage": True, "local": False, } @@ -362,7 +371,8 @@ def call_json( _models["text_local"].add(be["model"]) _models["fallback_count"] += 1 be = {"base_url": config.AI_BASE_URL, "api_key": config.AI_API_KEY, - "model": config.TEXT_MODEL, "usage": True, "local": False} + "model": _text_model_override or config.TEXT_MODEL, + "usage": True, "local": False} cache_key = None # don't cache fallback under the local-model key fell_back = True continue diff --git a/backend/main.py b/backend/main.py index d3f2201..d4232a1 100644 --- a/backend/main.py +++ b/backend/main.py @@ -35,6 +35,7 @@ _FRONTEND_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", " @app.get("/health") def health(): return {"status": "ok", "model": config.MODEL, + "text_model": config.TEXT_MODEL, "version": config.APP_VERSION, "build": config.APP_BUILD, "key_configured": bool(config.AI_API_KEY), @@ -43,13 +44,15 @@ def health(): @app.get("/models") def list_models(): - """Available OpenRouter models with per-1M-token pricing for the UI picker.""" - from backend.models import fetch_models + """Vision/text OpenRouter model lists with pricing for the UI dropdowns.""" + from backend.models import fetch_models, split_vision_text models = fetch_models() if models is None: raise HTTPException(status_code=502, detail="Could not fetch the model list from OpenRouter") - return {"models": models, "default": config.MODEL, "default_text": config.TEXT_MODEL} + vision, text = split_vision_text(models) + return {"vision": vision, "text": text, + "defaults": {"vision": config.MODEL, "text": config.TEXT_MODEL}} @app.get("/jobs/{job_id}/log") @@ -72,7 +75,8 @@ async def check( work_type: Optional[str] = Form(None), text_local: bool = Form(False), pipeline_mode: str = Form("classic"), - model: Optional[str] = Form(None), + vision_model: Optional[str] = Form(None), + text_model: Optional[str] = Form(None), ): """ Accept a PDF, start a background conflict check, and return a job_id @@ -81,6 +85,9 @@ async def check( Optional intake fields (project_name/address/occupancy/work_type) feed the Stage 0 jurisdiction profile; anything left blank is derived from the cover sheet. + + vision_model / text_model override the configured defaults for this run + (vision always OpenRouter; text follows the OpenRouter vs hybrid choice). """ if not file.filename.lower().endswith(".pdf"): raise HTTPException(status_code=400, detail="Please upload a PDF.") @@ -106,9 +113,12 @@ async def check( }.items() if v and v.strip() } + v_model = (vision_model or "").strip() or None + t_model = (text_model or "").strip() or None job_id = create_job(tmp_path, source_filename=file.filename, email=email, project_input=project_input, text_local=text_local, - pipeline_mode=pipeline_mode, model=model) + pipeline_mode=pipeline_mode, vision_model=v_model, + text_model=t_model) return JSONResponse({ "job_id": job_id, "status": "queued", diff --git a/backend/models.py b/backend/models.py index d9e09c6..596710b 100644 --- a/backend/models.py +++ b/backend/models.py @@ -3,11 +3,13 @@ models.py - Fetch the available OpenRouter model list with pricing (cached). The /models endpoint is public (no API key needed). Results are normalized to per-1M-token USD costs for display and cached in memory for an hour; callers -degrade gracefully when OpenRouter is unreachable. +degrade gracefully when OpenRouter is unreachable. Each entry also carries a +vision flag (accepts image input) so the UI can offer separate vision/text +model dropdowns. """ import time -from typing import List, Optional +from typing import List, Optional, Tuple import httpx @@ -25,6 +27,18 @@ def _per_mtok(rate) -> float: return 0.0 +def _is_vision(item: dict) -> bool: + """True when the model accepts image input and produces text output.""" + arch = item.get("architecture") or {} + inputs = arch.get("input_modalities") or [] + outputs = arch.get("output_modalities") or [] + # Legacy string form: "text+image->text" + modality = (arch.get("modality") or "").lower() + has_image_in = ("image" in inputs) or ("image" in modality.split("->")[0]) + has_text_out = ("text" in outputs) or ("->text" in modality) or (not outputs and not modality) + return has_image_in and has_text_out + + def _fetch_openrouter_models() -> Optional[List[dict]]: """Raw GET of the OpenRouter model list; None on any failure.""" try: @@ -55,6 +69,7 @@ def fetch_models(force: bool = False) -> Optional[List[dict]]: "prompt_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("prompt")), "completion_usd_per_mtok": _per_mtok((item.get("pricing") or {}).get("completion")), "context_length": item.get("context_length"), + "vision": _is_vision(item), } for item in data if item.get("id") @@ -63,3 +78,13 @@ def fetch_models(force: bool = False) -> Optional[List[dict]]: _cache["models"] = models _cache["at"] = time.time() return models + + +def split_vision_text(models: List[dict]) -> Tuple[List[dict], List[dict]]: + """Partition the normalized catalog into (vision, text) lists for the UI. + + Every catalog model takes text in/out, so vision models appear in both + lists (same dicts, pricing included). + """ + vision = [m for m in models if m.get("vision")] + return vision, list(models) diff --git a/backend/pipeline/runner.py b/backend/pipeline/runner.py index fdae24f..519733d 100644 --- a/backend/pipeline/runner.py +++ b/backend/pipeline/runner.py @@ -42,7 +42,9 @@ from backend.pipeline.risk import score_and_prioritize from backend.pipeline.rfi import generate_rfis from backend.pipeline.report import build_report, to_markdown from backend.pipeline._stage import validate_issue -from backend.llm import reset_cost, get_cost, set_stage, set_text_backend +from backend.llm import ( + reset_cost, get_cost, set_stage, set_text_backend, set_model_overrides, +) def run_pipeline( @@ -52,6 +54,8 @@ def run_pipeline( project_input: Optional[Dict] = None, source_name: Optional[str] = None, text_local: bool = False, + vision_model: Optional[str] = None, + text_model: Optional[str] = None, ) -> Dict: """ Run the full QAQC pipeline on one PDF and return the report dict. @@ -59,6 +63,9 @@ def run_pipeline( project_input: optional intake fields (project_name, address, occupancy, work_type). Cover-sheet-derived values fill any gaps; intake fields win. + vision_model / text_model: optional per-run OpenRouter (or local text) + model overrides from the UI. Blank/None keeps config defaults. + If out_dir is given, writes conflicts.json, report.md, and the intermediate artifacts (assertions.json, clusters.json, and one json per QAQC stage). """ @@ -70,7 +77,29 @@ def run_pipeline( reset_cost() set_text_backend(text_local) + set_model_overrides(vision_model, text_model) + if vision_model or text_model: + print(f"[Runner] model overrides: vision={vision_model or '(default)'} " + f"text={text_model or '(default)'}") + try: + return _run_stages( + pdf_path, out_dir, stage, project_input, source_name, text_local, + ) + finally: + # Don't leak per-run picks into a later overlapping/CLI call. + set_model_overrides(None, None) + set_text_backend(False) + + +def _run_stages( + pdf_path: str, + out_dir: Optional[str], + stage: Callable[[str], None], + project_input: Optional[Dict], + source_name: Optional[str], + text_local: bool, +) -> Dict: stage("PDF -> images") pages = convert_pdf_to_images(pdf_path) diff --git a/frontend/index.html b/frontend/index.html index fdb4c9e..7a9f63b 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -60,7 +60,15 @@ border:1px solid var(--line); background:#0c0e13; color:var(--text); font-size:16px; outline:none; } .email-card input[type=email]::placeholder { color:#6b7280; } .email-card input[type=email]:focus { border-color:var(--accent); box-shadow:0 0 0 3px rgba(91,140,255,.18); } + .email-card select { width:100%; padding:10px 12px; border-radius:8px; margin-top:6px; + border:1px solid var(--line); background:#0c0e13; color:var(--text); font-size:14px; } + .email-card .field { margin-top:12px; } + .email-card .field > span { display:block; font-size:13px; color:var(--muted); margin-bottom:2px; } .btn.full { width:100%; padding:14px; font-size:15px; margin-top:0; } + .logbox { background:#0c0e13; border:1px solid var(--line); border-radius:8px; padding:12px 14px; + margin-top:10px; max-height:320px; overflow:auto; font:12px/1.45 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace; + color:#c6cdd8; white-space:pre-wrap; word-break:break-word; } + .logbox .empty-log { color:var(--muted); } .note { background:var(--panel); border:1px solid var(--line); border-radius:10px; padding:16px 18px; margin:18px 0; } .note b { color:var(--text); } @@ -121,12 +129,22 @@
- - +
+ Vision model (image stages) loading... + +
+
+ Text model (non-image stages) + +
+
@@ -144,9 +162,71 @@ const drop=document.getElementById('drop'), fileInput=document.getElementById('file'), runBtn=document.getElementById('run'), statusEl=document.getElementById('status'), results=document.getElementById('results'), dropLabel=document.getElementById('dropLabel'), - emailEl=document.getElementById('email'); + emailEl=document.getElementById('email'), + visionSel=document.getElementById('vision_model'), + textSel=document.getElementById('text_model'), + liveLog=document.getElementById('liveLog'), + logBox=document.getElementById('logBox'), + logHint=document.getElementById('logHint'); let chosen=null, polling=null, currentJobId=null, sheetPage={}, viewerZoom=1, reviewDirty=false; +function modelLabel(m){ + // Include per-1M-token pricing when the catalog provides it. + let s=m.name||m.id; + if(m.prompt_usd_per_mtok!=null) + s+=' — $'+m.prompt_usd_per_mtok+' / $'+m.completion_usd_per_mtok+' per 1M tok'; + return s; +} + +function fillSelect(sel, items, preferred){ + sel.innerHTML=''; + (items||[]).forEach(m=>{ + const opt=document.createElement('option'); + opt.value=m.id; opt.textContent=modelLabel(m); + if(m.id===preferred) opt.selected=true; + sel.appendChild(opt); + }); + if(!sel.options.length){ + const opt=document.createElement('option'); + opt.value=preferred||''; opt.textContent=preferred||'(no models)'; + sel.appendChild(opt); + } + sel.disabled=false; +} + +let modelsLoaded=false; +async function loadModels(){ + const note=document.getElementById('modelNote'); + try{ + const res=await fetch('/models'); + if(!res.ok) throw new Error('models HTTP '+res.status); + const data=await res.json(); + const defs=data.defaults||{}; + fillSelect(visionSel, data.vision, defs.vision); + fillSelect(textSel, data.text, defs.text); + modelsLoaded=true; + note.textContent='('+(data.text||[]).length+' text / '+(data.vision||[]).length+' vision available)'; + }catch(err){ + visionSel.innerHTML=''; + textSel.innerHTML=''; + visionSel.disabled=false; textSel.disabled=false; + note.textContent='using configured defaults (list unavailable)'; + console.warn('Could not load models:', err); + } +} + +function showLog(lines, live){ + liveLog.style.display='block'; + logHint.textContent=live?'(updates live)':'(saved with this job)'; + const arr=lines||[]; + if(!arr.length){ + logBox.innerHTML='No log lines yet…'; + return; + } + logBox.textContent=arr.join('\n'); + logBox.scrollTop=logBox.scrollHeight; +} + function setFile(f){ chosen=f; dropLabel.textContent=f?('Selected: '+f.name):'Drop a PDF drawing set here, or click to choose'; runBtn.disabled=!f; } dropLabel.addEventListener('click',()=>fileInput.click()); @@ -160,6 +240,7 @@ runBtn.addEventListener('click',async e=>{ if(!chosen) return; runBtn.disabled=true; results.innerHTML=''; statusEl.innerHTML='Uploading...'; + showLog([], true); const fd=new FormData(); fd.append('file',chosen); const email=(emailEl.value||'').trim(); if(email) fd.append('notification_email',email); ['project_name','address','occupancy','work_type'].forEach(id=>{ @@ -168,8 +249,9 @@ runBtn.addEventListener('click',async e=>{ const compute=(document.querySelector('input[name="compute"]:checked')||{}).value; fd.append('text_local', compute==='local' ? 'true' : 'false'); if(compute==='openrouter'){ - const modelSel=document.getElementById('model'); - if(modelSel.value) fd.append('model', modelSel.value); + // Model picks only apply to OpenRouter compute; hybrid keeps its local model. + if(visionSel.value) fd.append('vision_model', visionSel.value); + if(textSel.value) fd.append('text_model', textSel.value); } const pipelineMode=(document.querySelector('input[name="pipeline_mode"]:checked')||{}).value||'classic'; fd.append('pipeline_mode',pipelineMode); @@ -196,21 +278,30 @@ function poll(jobId){ const res=await fetch('/jobs/'+jobId); if(!res.ok) throw new Error('job not found'); const job=await res.json(); + const live=['running','queued','finalizing'].includes(job.status); + if(job.log_tail && job.log_tail.length) showLog(job.log_tail, live); if(job.status==='running'||job.status==='queued'){ statusEl.innerHTML=''+esc(job.stage||'Working...')+ ' · you can leave this page'; } else if(job.status==='done'){ - clearInterval(polling); polling=null; runBtn.disabled=false; render(job.report); + clearInterval(polling); polling=null; runBtn.disabled=false; + if(job.log && job.log.length) showLog(job.log, false); + render(job.report); } else if(job.status==='needs_review'||job.status==='reviewing'){ - clearInterval(polling); polling=null; runBtn.disabled=false; renderReview(job); + clearInterval(polling); polling=null; runBtn.disabled=false; + if(job.log && job.log.length) showLog(job.log, false); + renderReview(job); } else if(job.status==='finalizing'){ statusEl.innerHTML='Finalizing reviewed report...'; } else if(job.status==='finalization_error'){ clearInterval(polling); polling=null; runBtn.disabled=false; statusEl.textContent='Finalization failed: '+(job.error||'unknown error'); + if(job.log && job.log.length) showLog(job.log, false); } else if(job.status==='error'){ clearInterval(polling); polling=null; runBtn.disabled=false; statusEl.textContent='Run failed: '+(job.error||'unknown error'); + if(job.log && job.log.length) showLog(job.log, false); + else if(job.log_tail && job.log_tail.length) showLog(job.log_tail, false); } }catch(err){ clearInterval(polling); polling=null; runBtn.disabled=false; statusEl.textContent='Error: '+err.message; } @@ -230,29 +321,11 @@ function syncPipelineOptions(){ document.querySelectorAll('input[name="pipeline_mode"]').forEach(el=>el.addEventListener('change',syncPipelineOptions)); syncPipelineOptions(); -// --- model picker (OpenRouter compute only) --- -let modelList=null; -async function loadModels(){ - const note=document.getElementById('modelNote'), sel=document.getElementById('model'); - try{ - const res=await fetch('/models'); - if(!res.ok) throw new Error('list unavailable'); - const data=await res.json(); - modelList=data.models||[]; - sel.innerHTML=modelList.map(m=> - '').join(''); - note.textContent='('+modelList.length+' available)'; - }catch(e){ - sel.innerHTML=''; - note.textContent='using configured default (list unavailable)'; - } -} +// --- model pickers (OpenRouter compute only) --- function syncCompute(){ const openrouter=(document.querySelector('input[name="compute"]:checked')||{}).value==='openrouter'; document.getElementById('modelPick').style.display=openrouter?'block':'none'; - if(openrouter&&!modelList) loadModels(); + if(openrouter&&!modelsLoaded) loadModels(); } document.querySelectorAll('input[name="compute"]').forEach(el=>el.addEventListener('change',syncCompute)); syncCompute(); @@ -394,6 +467,10 @@ function render(rep){ html+=''; } + html+='
Full run log: /jobs/'+ + esc(currentJobId)+'/log (also saved as job.log on the server)
'; + results.innerHTML=html; } function stat(v,l){ return '
'+esc(v)+''+esc(l)+'
'; } diff --git a/tests/api/test_job_log.py b/tests/api/test_job_log.py index 94da9f4..e9bf9a0 100644 --- a/tests/api/test_job_log.py +++ b/tests/api/test_job_log.py @@ -60,18 +60,56 @@ def test_job_log_endpoint_serves_log_and_404s(job_env, monkeypatch): assert client.get("/jobs/nope/log").status_code == 404 -def test_model_override_set_and_cleared_around_run(job_env, monkeypatch): - from backend import llm - +def test_model_overrides_passed_to_classic_runner(job_env, monkeypatch): + """Classic mode: per-run picks travel as run_pipeline kwargs (the runner + sets and clears llm.set_model_overrides itself).""" seen = {} def fake_runner(pdf_path, **kwargs): - seen["override"] = llm._model_override + seen.update(kwargs) return {"source": "set.pdf", "summary": {}} monkeypatch.setattr("backend.jobs.run_pipeline", fake_runner) jobs.create_job(str(job_env / "set.pdf"), "set.pdf", - pipeline_mode="classic", model="openai/gpt-4o") + pipeline_mode="classic", + vision_model="openai/gpt-4o", text_model="openai/gpt-4o-mini") - assert seen["override"] == "openai/gpt-4o" - assert llm._model_override is None # cleared after the run + assert seen["vision_model"] == "openai/gpt-4o" + assert seen["text_model"] == "openai/gpt-4o-mini" + + +def test_model_overrides_set_and_cleared_around_agent_run(job_env, monkeypatch): + """Agent mode: the agent runner has no override params, so jobs.py sets + them module-level for the duration of the run.""" + from backend import llm + + seen = {} + + def fake_agent_runner(pdf_path, **kwargs): + seen["vision"] = llm._vision_model_override + seen["text"] = llm._text_model_override + return {"source": "set.pdf", "summary": {}} + + monkeypatch.setattr("backend.jobs.run_agent_pipeline", fake_agent_runner) + jobs.create_job(str(job_env / "set.pdf"), "set.pdf", + pipeline_mode="agent", + vision_model="openai/gpt-4o", text_model="openai/gpt-4o-mini") + + assert seen["vision"] == "openai/gpt-4o" + assert seen["text"] == "openai/gpt-4o-mini" + assert llm._vision_model_override is None # cleared after the run + assert llm._text_model_override is None + + +def test_failed_run_logs_traceback(job_env, monkeypatch): + """A crashed job must leave the traceback in job.log, not just str(e).""" + def boom(pdf_path, **kwargs): + raise RuntimeError("kaboom-stage-failure") + + monkeypatch.setattr("backend.jobs.run_pipeline", boom) + job_id = jobs.create_job(str(job_env / "set.pdf"), "set.pdf", pipeline_mode="classic") + + assert jobs._jobs[job_id]["status"] == "error" + content = (job_env / job_id / "job.log").read_text() + assert "Traceback (most recent call last)" in content + assert "RuntimeError: kaboom-stage-failure" in content diff --git a/tests/api/test_models_api.py b/tests/api/test_models_api.py index d093a74..afb0da2 100644 --- a/tests/api/test_models_api.py +++ b/tests/api/test_models_api.py @@ -11,12 +11,23 @@ _PAYLOAD = { "name": "GPT-4o", "pricing": {"prompt": "0.0000025", "completion": "0.00001"}, "context_length": 128000, + "architecture": {"input_modalities": ["text", "image"], + "output_modalities": ["text"]}, }, { "id": "google/gemini-2.5-pro", "name": "Gemini 2.5 Pro", "pricing": {"prompt": "0.00000125", "completion": "0.00001"}, "context_length": 1000000, + "architecture": {"modality": "text+image->text"}, + }, + { + "id": "meta-llama/llama-3.1-70b-instruct", + "name": "Llama 3.1 70B Instruct", + "pricing": {"prompt": "0.0000005", "completion": "0.0000008"}, + "context_length": 131072, + "architecture": {"input_modalities": ["text"], + "output_modalities": ["text"]}, }, ] } @@ -34,14 +45,27 @@ def test_models_endpoint_normalizes_pricing(monkeypatch): response = client.get("/models") assert response.status_code == 200 body = response.json() - assert body["default"] == config.MODEL - assert body["default_text"] == config.TEXT_MODEL - by_id = {m["id"]: m for m in body["models"]} + assert body["defaults"] == {"vision": config.MODEL, "text": config.TEXT_MODEL} + by_id = {m["id"]: m for m in body["text"]} assert by_id["openai/gpt-4o"]["prompt_usd_per_mtok"] == 2.5 assert by_id["openai/gpt-4o"]["completion_usd_per_mtok"] == 10.0 assert by_id["openai/gpt-4o"]["context_length"] == 128000 +def test_models_endpoint_splits_vision_and_text(monkeypatch): + _reset_cache() + monkeypatch.setattr(models, "_fetch_openrouter_models", lambda: _PAYLOAD["data"]) + client = TestClient(app) + body = client.get("/models").json() + vision_ids = {m["id"] for m in body["vision"]} + text_ids = {m["id"] for m in body["text"]} + # Both modality shapes (structured and legacy string) are recognized. + assert vision_ids == {"openai/gpt-4o", "google/gemini-2.5-pro"} + # Text list is the full catalog; vision models appear in both. + assert text_ids == {"openai/gpt-4o", "google/gemini-2.5-pro", + "meta-llama/llama-3.1-70b-instruct"} + + def test_models_endpoint_caches(monkeypatch): _reset_cache() calls = [] diff --git a/tests/test_llm_model_override.py b/tests/test_llm_model_override.py index 6502eb6..ee6432a 100644 --- a/tests/test_llm_model_override.py +++ b/tests/test_llm_model_override.py @@ -1,26 +1,44 @@ from backend import config -from backend.llm import _resolve_backend, set_model_override +from backend.llm import _resolve_backend, set_model_overrides, set_text_backend -def test_override_wins_for_vision_and_text(): - set_model_override("openai/gpt-4o") - try: - assert _resolve_backend(has_images=True, model_override=None)["model"] == "openai/gpt-4o" - assert _resolve_backend(has_images=False, model_override=None)["model"] == "openai/gpt-4o" - finally: - set_model_override(None) +def teardown_function(): + set_model_overrides(None, None) + set_text_backend(False) + + +def test_vision_override_wins_for_vision_only(): + set_model_overrides(vision="openai/gpt-4o", text=None) + assert _resolve_backend(has_images=True, model_override=None)["model"] == "openai/gpt-4o" + assert _resolve_backend(has_images=False, model_override=None)["model"] == config.TEXT_MODEL + + +def test_text_override_wins_for_text_only(): + set_model_overrides(vision=None, text="anthropic/claude-sonnet-4") + assert _resolve_backend(has_images=False, model_override=None)["model"] == "anthropic/claude-sonnet-4" + assert _resolve_backend(has_images=True, model_override=None)["model"] == config.MODEL def test_override_beats_per_call_model_arg(): - set_model_override("openai/gpt-4o") - try: - # Agents pass their AGENT_*_MODEL per call; the user's job pick wins. - assert _resolve_backend(has_images=False, model_override="other/model")["model"] == "openai/gpt-4o" - finally: - set_model_override(None) + set_model_overrides(vision="openai/gpt-4o", text="openai/gpt-4o-mini") + # Agents pass their AGENT_*_MODEL per call; the user's job pick wins. + assert _resolve_backend(has_images=True, model_override="other/model")["model"] == "openai/gpt-4o" + assert _resolve_backend(has_images=False, model_override="other/model")["model"] == "openai/gpt-4o-mini" def test_no_override_keeps_defaults(): - set_model_override(None) + set_model_overrides(None, None) assert _resolve_backend(has_images=True, model_override=None)["model"] == config.MODEL assert _resolve_backend(has_images=False, model_override=None)["model"] == config.TEXT_MODEL + + +def test_ui_picks_never_name_the_local_model(monkeypatch): + """Hybrid runs keep LOCAL_TEXT_MODEL; OpenRouter picks must not leak into + the local endpoint (a vLLM server won't serve OpenRouter model ids).""" + monkeypatch.setattr(config, "LOCAL_BASE_URL", "http://localhost:8000/v1") + monkeypatch.setattr(config, "LOCAL_TEXT_MODEL", "qwen/local-instruct") + set_text_backend(True) + set_model_overrides(vision="openai/gpt-4o", text="anthropic/claude-sonnet-4") + be = _resolve_backend(has_images=False, model_override=None) + assert be["local"] is True + assert be["model"] == "qwen/local-instruct"