Conflict_Checker/docs/superpowers/specs/2026-07-28-agent-human-review-design.md

Required Human Review for Agent Pipeline Design

Date: 2026-07-28
Status: Approved
Owner: Conflict Checker Agent pipeline

Goal

Make Agent mode produce higher-quality findings by requiring structured human review before final RFIs/reports are issued, and by turning review decisions into usable feedback for future prompt, rule, threshold, and evaluation improvements.

Background

Agent mode is intended to replace Classic mode. Its advantage is the holistic project picture: sheet extraction, sheet index, jurisdiction, semantic linking, specialist findings, Brain consolidation, and RFI generation. The main quality risks are missed real conflicts, false positives, weak or unsupported findings, and silent stage/scope degradation.

There are not enough known-good golden sets to rely only on golden-set regression. Human review becomes the feedback mechanism. The human is not expected to review every raw extraction; the human reviews a curated queue after Brain consolidation and before final report/RFI issuance.

Requirements

Functional requirements

1. Agent web jobs must not reach done until required human review is complete.
2. The Agent pipeline runs through Brain, then enters needs_review.
3. RFI generation happens only after review finalization.
4. Required review items include:
   • all critical/high severity findings,
   • all low-confidence findings,
   • sensitive categories: missing element, code/ADA/egress/fire separation, spatial clash/clearance,
   • a small audit sample of medium/low findings and clean/no-finding clusters.
5. Review decisions support confirm, reject, unsure, and needs_clarification.
6. Rejections require a reason code.
7. Review progress persists to disk and survives server restart.
8. Rejected findings are suppressed, not deleted.
9. Clarifications are stored as first-class artifacts.
10. Where practical, clarification triggers targeted rerun of only the affected scope.
11. Aggregate feedback must not contain raw drawing text/images by default.
12. Classic mode remains unchanged.

Non-functional requirements

• No automatic prompt mutation from human labels.
• No final email before review completion.
• Review endpoints must be treated as state-changing and sensitive.
• Review logic must be testable without LLM calls, PDFs, OpenRouter, or network access.
• Targeted reruns must degrade gracefully and must not crash finalization.

Architecture

Add three small components.

ReviewGate

Runs after Brain and before RFI/report finalization.

Consumes:

• ProjectMemory snapshot
• Brain prioritized issues
• Brain decisions
• review policy

Produces:

• review/review_queue.json
• candidate report with summary.agent_status = "needs_review"
• job transition to needs_review

ReviewStore

Owns review persistence under the job output directory.

Stores:

• review/review_queue.json
• review/review_decisions.json
• review/review_progress.json

Writes must be atomic using a temporary file plus os.replace, matching the existing LLM cache/report artifact style.

ReviewFinalizer

Runs after required decisions are submitted.

Responsibilities:

• validate completeness,
• apply decisions,
• perform bounded targeted reruns for clarification where supported,
• re-run Brain only for affected findings,
• draft RFIs only for kept/confirmed issues,
• write final artifacts,
• transition to done,
• send final email.

Job lifecycle

Current lifecycle:

queued -> running -> done -> email

New Agent lifecycle:

queued -> running -> needs_review -> reviewing -> finalizing -> done -> email

Additional failure state:

• finalization_error

If the server restarts while a job is in needs_review or reviewing, the backend rebuilds state from outputs/<job_id>/conflicts.json, outputs/<job_id>/review/review_queue.json, and outputs/<job_id>/review/review_decisions.json.

Email behavior

If email is enabled, Agent mode sends two emails:

1. Review required when the job enters needs_review.
2. Final report only after review finalization.

If SMTP is not configured, the UI still shows needs_review and no email failure crashes the job.

Review queue policy

Blocking review items are findings that meet any of these rules:

• severity is critical or high,
• confidence is low,
• category is missing_element,
• source stage is code,
• category is in ada, tas_tdlr, egress, fire_separation, occupancy, spatial_clash, clearance_conflict, or penetration_conflict.

Audit sample items are selected deterministically from:

• medium/low findings not already blocking,
• clean clusters with no findings,
• no-finding scopes when available.

Default audit sample size is 5 items.

Review decision schema

{
  "review_item_id": "finding:AGENT-0007",
  "decision": "reject",
  "reason_code": "same_value_different_representation",
  "category_correction": null,
  "severity_correction": null,
  "comment": "9'-0\" AFF and 108 inches are the same value here.",
  "clarification_answer": null,
  "reviewed_at": "2026-07-28T12:00:00Z"
}

Allowed reason codes:

• wrong_cluster_link
• same_value_different_representation
• not_a_contradiction
• missing_evidence
• extraction_misread
• code_path_not_applicable
• duplicate
• severity_too_high
• severity_too_low
• other

Finalization rules

• All blocking review items must have a valid decision before finalization.
• Confirmed findings become final validated_issues.
• Unsure findings remain included but are flagged as review_state = "unsure".
• Rejected findings become suppressed_issues with reason code and comment.
• Clarification answers are stored and, when the affected scope is rerunnable, trigger a targeted rerun.
• Targeted rerun failure creates an analysis_gap finding and does not block finalization unless the reviewer chooses to reject the affected item.
• RFIs are drafted only for final kept issues.

Feedback labels

Every decision emits a label artifact for metrics:

{
  "review_item_id": "finding:AGENT-0007",
  "job_id": "abc123",
  "pipeline_mode": "agent",
  "source_stage": "conflict",
  "category": "elevation_disagreement",
  "severity": "high",
  "confidence": "medium",
  "decision": "reject",
  "reason_code": "same_value_different_representation",
  "location": "Room 204 / Level 2",
  "disciplines": ["Architectural", "Mechanical"],
  "sheets": ["A2.1", "M2.1"],
  "drawing_type": "floor_plan",
  "models_used": ["google/gemini-2.5-pro"],
  "created_at": "2026-07-28T12:00:00Z"
}

Default aggregate metrics exclude source_text, images, raw sheet content, and reviewer free-text comments.

API shape

• GET /jobs/{job_id} includes needs_review, reviewing, finalizing, done, error, or finalization_error plus review progress.
• GET /jobs/{job_id}/review returns { "queue": [...], "progress": {...} }.
• POST /jobs/{job_id}/review-decisions saves one or more decisions.
• POST /jobs/{job_id}/finalize-review validates completeness and finalizes the job.

Security and privacy

Review endpoints are more sensitive than read-only report endpoints because they mutate job state and expose evidence. Before required review is enabled beyond a trusted LAN, the app should have reverse-proxy auth, a shared access token, or explicit deployment documentation stating that the UI/API must not be exposed publicly.

Review artifacts stay job-local by default. Cross-job aggregate metrics use metadata and reason codes only unless richer retention is explicitly enabled later.

Testing strategy

Tests must not require PDFs, LLMs, OpenRouter, or network access.

Cover:

• required-review trigger policy,
• review queue construction,
• decision validation and reason codes,
• finalization behavior for confirm/reject/unsure/clarification,
• restart recovery from review artifacts,
• targeted rerun failure degradation,
• metrics redaction,
• API state transitions,
• email flow blocking until finalization.

Rollout

• Classic mode is unchanged.
• Agent web jobs default to required human review.
• CLI supports an explicit bypass flag, --no-review, for tuning/debug runs.
• Review state and decisions are always written to job artifacts.
• Aggregate feedback is metadata-only by default.

Acceptance criteria

• An Agent web job cannot reach done or send the final email while required review items are undecided.
• Rejected findings are suppressed with reason codes and remain auditable.
• Review progress survives server restart.
• Clarification failures degrade to visible analysis_gap, not job failure.
• Aggregate feedback contains no raw drawing text/images by default.
• New tests cover the review gate without requiring LLM calls.