Conflict_Checker/docs/superpowers/specs/2026-07-28-agent-human-review-design.md

# Required Human Review for Agent Pipeline Design

**Date:** 2026-07-28  
**Status:** Approved  
**Owner:** Conflict Checker Agent pipeline

## Goal

Make Agent mode produce higher-quality findings by requiring structured human review before final RFIs/reports are issued, and by turning review decisions into usable feedback for future prompt, rule, threshold, and evaluation improvements.

## Background

Agent mode is intended to replace Classic mode. Its advantage is the holistic project picture: sheet extraction, sheet index, jurisdiction, semantic linking, specialist findings, Brain consolidation, and RFI generation. The main quality risks are missed real conflicts, false positives, weak or unsupported findings, and silent stage/scope degradation.

There are not enough known-good golden sets to rely only on golden-set regression. Human review becomes the feedback mechanism. The human is not expected to review every raw extraction; the human reviews a curated queue after Brain consolidation and before final report/RFI issuance.

## Requirements

### Functional requirements

1. Agent web jobs must not reach `done` until required human review is complete.
2. The Agent pipeline runs through Brain, then enters `needs_review`.
3. RFI generation happens only after review finalization.
4. Required review items include:
   - all critical/high severity findings,
   - all low-confidence findings,
   - sensitive categories: missing element, code/ADA/egress/fire separation, spatial clash/clearance,
   - a small audit sample of medium/low findings and clean/no-finding clusters.
5. Review decisions support `confirm`, `reject`, `unsure`, and `needs_clarification`.
6. Rejections require a reason code.
7. Review progress persists to disk and survives server restart.
8. Rejected findings are suppressed, not deleted.
9. Clarifications are stored as first-class artifacts.
10. Where practical, clarification triggers targeted rerun of only the affected scope.
11. Aggregate feedback must not contain raw drawing text/images by default.
12. Classic mode remains unchanged.

### Non-functional requirements

- No automatic prompt mutation from human labels.
- No final email before review completion.
- Review endpoints must be treated as state-changing and sensitive.
- Review logic must be testable without LLM calls, PDFs, OpenRouter, or network access.
- Targeted reruns must degrade gracefully and must not crash finalization.

## Architecture

Add three small components.

### ReviewGate

Runs after Brain and before RFI/report finalization.

Consumes:

- `ProjectMemory` snapshot
- Brain prioritized issues
- Brain decisions
- review policy

Produces:

- `review/review_queue.json`
- candidate report with `summary.agent_status = "needs_review"`
- job transition to `needs_review`

### ReviewStore

Owns review persistence under the job output directory.

Stores:

- `review/review_queue.json`
- `review/review_decisions.json`
- `review/review_progress.json`

Writes must be atomic using a temporary file plus `os.replace`, matching the existing LLM cache/report artifact style.

### ReviewFinalizer

Runs after required decisions are submitted.

Responsibilities:

- validate completeness,
- apply decisions,
- perform bounded targeted reruns for clarification where supported,
- re-run Brain only for affected findings,
- draft RFIs only for kept/confirmed issues,
- write final artifacts,
- transition to `done`,
- send final email.

## Job lifecycle

Current lifecycle:

`queued -> running -> done -> email`

New Agent lifecycle:

`queued -> running -> needs_review -> reviewing -> finalizing -> done -> email`

Additional failure state:

- `finalization_error`

If the server restarts while a job is in `needs_review` or `reviewing`, the backend rebuilds state from `outputs/<job_id>/conflicts.json`, `outputs/<job_id>/review/review_queue.json`, and `outputs/<job_id>/review/review_decisions.json`.

## Email behavior

If email is enabled, Agent mode sends two emails:

1. **Review required** when the job enters `needs_review`.
2. **Final report** only after review finalization.

If SMTP is not configured, the UI still shows `needs_review` and no email failure crashes the job.

## Review queue policy

Blocking review items are findings that meet any of these rules:

- severity is `critical` or `high`,
- confidence is `low`,
- category is `missing_element`,
- source stage is `code`,
- category is in `ada`, `tas_tdlr`, `egress`, `fire_separation`, `occupancy`, `spatial_clash`, `clearance_conflict`, or `penetration_conflict`.

Audit sample items are selected deterministically from:

- medium/low findings not already blocking,
- clean clusters with no findings,
- no-finding scopes when available.

Default audit sample size is 5 items.

## Review decision schema

```json
{
  "review_item_id": "finding:AGENT-0007",
  "decision": "reject",
  "reason_code": "same_value_different_representation",
  "category_correction": null,
  "severity_correction": null,
  "comment": "9'-0\" AFF and 108 inches are the same value here.",
  "clarification_answer": null,
  "reviewed_at": "2026-07-28T12:00:00Z"
}
```

Allowed reason codes:

- `wrong_cluster_link`
- `same_value_different_representation`
- `not_a_contradiction`
- `missing_evidence`
- `extraction_misread`
- `code_path_not_applicable`
- `duplicate`
- `severity_too_high`
- `severity_too_low`
- `other`

## Finalization rules

- All blocking review items must have a valid decision before finalization.
- Confirmed findings become final `validated_issues`.
- Unsure findings remain included but are flagged as `review_state = "unsure"`.
- Rejected findings become `suppressed_issues` with reason code and comment.
- Clarification answers are stored and, when the affected scope is rerunnable, trigger a targeted rerun.
- Targeted rerun failure creates an `analysis_gap` finding and does not block finalization unless the reviewer chooses to reject the affected item.
- RFIs are drafted only for final kept issues.

## Feedback labels

Every decision emits a label artifact for metrics:

```json
{
  "review_item_id": "finding:AGENT-0007",
  "job_id": "abc123",
  "pipeline_mode": "agent",
  "source_stage": "conflict",
  "category": "elevation_disagreement",
  "severity": "high",
  "confidence": "medium",
  "decision": "reject",
  "reason_code": "same_value_different_representation",
  "location": "Room 204 / Level 2",
  "disciplines": ["Architectural", "Mechanical"],
  "sheets": ["A2.1", "M2.1"],
  "drawing_type": "floor_plan",
  "models_used": ["google/gemini-2.5-pro"],
  "created_at": "2026-07-28T12:00:00Z"
}
```

Default aggregate metrics exclude `source_text`, images, raw sheet content, and reviewer free-text comments.

## API shape

- `GET /jobs/{job_id}` includes `needs_review`, `reviewing`, `finalizing`, `done`, `error`, or `finalization_error` plus review progress.
- `GET /jobs/{job_id}/review` returns `{ "queue": [...], "progress": {...} }`.
- `POST /jobs/{job_id}/review-decisions` saves one or more decisions.
- `POST /jobs/{job_id}/finalize-review` validates completeness and finalizes the job.

## Security and privacy

Review endpoints are more sensitive than read-only report endpoints because they mutate job state and expose evidence. Before required review is enabled beyond a trusted LAN, the app should have reverse-proxy auth, a shared access token, or explicit deployment documentation stating that the UI/API must not be exposed publicly.

Review artifacts stay job-local by default. Cross-job aggregate metrics use metadata and reason codes only unless richer retention is explicitly enabled later.

## Testing strategy

Tests must not require PDFs, LLMs, OpenRouter, or network access.

Cover:

- required-review trigger policy,
- review queue construction,
- decision validation and reason codes,
- finalization behavior for confirm/reject/unsure/clarification,
- restart recovery from review artifacts,
- targeted rerun failure degradation,
- metrics redaction,
- API state transitions,
- email flow blocking until finalization.

## Rollout

- Classic mode is unchanged.
- Agent web jobs default to required human review.
- CLI supports an explicit bypass flag, `--no-review`, for tuning/debug runs.
- Review state and decisions are always written to job artifacts.
- Aggregate feedback is metadata-only by default.

## Acceptance criteria

- An Agent web job cannot reach `done` or send the final email while required review items are undecided.
- Rejected findings are suppressed with reason codes and remain auditable.
- Review progress survives server restart.
- Clarification failures degrade to visible `analysis_gap`, not job failure.
- Aggregate feedback contains no raw drawing text/images by default.
- New tests cover the review gate without requiring LLM calls.