Skip to content

ARGUS-404 Acceptance-Fidelity Review

Problem

Argus currently marks a PR-backed task verified when deterministic evidence says the PR merged, CI was green, and the deploy completed. That boundary is valuable: it is tamper-resistant, machine-checkable, and should stay the only writer of verified=true.

It does not answer a different question: did the merged PR actually satisfy the task acceptance? ARGUS-385/403 proved the gap. A PR could honestly say Closes ARGUS-385, pass CI, deploy, and verify the task while only addressing a narrow sub-path and leaving the real Grafana alert-formatting acceptance unmet.

Design

Add an advisory acceptance-fidelity review at land time. It is not a blocking verifier. Put another way: this is not a blocking verifier, never sets verified, and never blocks verified. The deterministic checker remains the tamper boundary.

When the landing flow has a task id and PR metadata, enqueue a single LLM-as-judge call with these inputs:

  • task title, why, and task acceptance
  • PR title
  • PR body
  • PR diff, truncated by files and hunks with tests kept preferentially
  • deterministic evidence summary: merged commit, CI status, deploy status

The judge returns strict JSON:

{
  "verdict": "met | partial | unmet",
  "confidence": 0.0,
  "rationale": "one-line rationale",
  "missing_acceptance": ["short unmet acceptance clause"]
}

The review scale is exactly met / partial / unmet, paired with confidence and a one-line rationale so humans can scan the result without opening the raw prompt.

Record the result as a task event with an event name such as acceptance_fidelity_review. Store the raw structured result in event detail, including model, prompt version, token counts, cost, and source PR number. Also derive a lightweight flag for task surfaces:

  • met at high confidence: informational event only.
  • partial or unmet: show in inbox/review as needs human review.
  • low confidence: show in inbox/review without claiming failure.
  • unmet at high confidence: push a notification with the one-line rationale.

Prompt And Model

Route through LiteLLM using the smart/reasoning tier. This is a judgment task, not a fast classification. The prompt should force conservative semantics:

  • Compare only the task acceptance to the PR diff/title/body.
  • Treat tests and code changes as evidence, but do not assume unstated behavior.
  • If the PR addresses only some acceptance clauses, return partial.
  • If the diff is too small, unrelated, or impossible to inspect, return low confidence rather than inventing certainty.
  • Output only strict JSON matching the schema.

The model call should be wrapped like other LiteLLM calls so Langfuse/cost data is captured where available. The first implementation can use a static prompt version (acceptance-fidelity-v1) and a single retry only for transport errors.

False-Positive Tolerance

The signal is advisory, so the system can tolerate false positives that call for human review. It should avoid false negatives: a bad met is more dangerous than a noisy partial.

Recommended policy:

  • Bias uncertain or partial evidence toward partial with low or medium confidence.
  • Never block the landing queue, deployment, or deterministic auto-verify.
  • Keep review rows dismissible by a human so the signal can be tuned without fighting the existing Approve path.
  • Track outcomes over time: human dismissed, human agreed, PR follow-up filed.

Verification Boundary

Do not gate auto-verify behind this signal in the first implementation. The checker should keep verifying exactly as it does today: merged PR reference, green CI, deploy. The acceptance-fidelity review composes beside it as a task event/flag. That preserves the current tamper-resistant contract while adding an intelligent smoke alarm for misaligned Closes X PRs.

If later data shows the signal is precise enough, a separate design can consider "verified but flagged" policy changes. That is explicitly out of scope here.

Cost

Expected cost is one smart/reasoning LiteLLM chat completion per landed PR that references a task. Most Argus PRs are small enough for a diff summary plus selected hunks. For large PRs:

  • cap input tokens;
  • include file list and PR body first;
  • keep tests and task-relevant files before generated/static files;
  • fall back to low confidence if truncation hides material evidence.

The event should record token counts and estimated cost so noisy or expensive reviews are visible in the runs/landing audit trail.

Human Approve Composition

For verify=approval tasks, the same advisory review can run when an approval references a PR or artifact, but it must not replace Aaron's Approve button. The human Approve path remains authoritative for approval-backed tasks. A partial or unmet result should appear next to the approval request as context; the human can approve anyway, reject, or file follow-up work.

For PR-backed tasks, the review should surface in the same inbox/review area so the operator sees "verified but acceptance-fidelity is partial/unmet" without having to inspect every landed PR manually.

Deep-Research Input

This spike follows the existing task-system design from docs/research/2026-06-14-task-system-research.md: tasks need explicit why and acceptance, the board is the unit of work, and verification should be deterministic where possible. The ARGUS-404 incident shows the remaining class that deterministic verification intentionally cannot solve: semantic acceptance matching. I did not find a separate ARGUS-404 deep-research artifact exposed by the current board tools; the implementation follow-up should attach one if it exists or rerun the research before final prompt tuning.

Follow-Up

Filed follow-up task: ARGUS-431, "Implement advisory acceptance-fidelity review".

Acceptance for that task should require:

  • schema/migration support for an advisory task event or flag;
  • landing-time review trigger for PRs that close a task;
  • LiteLLM smart/reasoning judge with strict JSON parsing;
  • inbox/review surfacing for low-confidence, partial, and unmet results;
  • push notification for high-confidence unmet results;
  • tests proving the signal never sets verified and never blocks verified.