Skip to content

Resuming interrupted skill workflows (ARGUS-300)

The last open gap from the 2026-07-04 restart-resilience audit (docs/restart-resilience.md): when the reaper (ARGUS-298) requeues an agent run whose host died mid-skill, the retry restarts the skill FROM ZERO. For deep-research that throws away 20-45 minutes of fan-out work whose results are, in fact, still on disk -- the in-session Workflow journals every completed sub-agent and supports resumeFromRunId replay.

This document is the investigation + design; the drill in scripts/workflow-resume-drill.sh is the working prototype.

Facts established (all verified on this host, 2026-07-04)

  1. The CLI has every primitive needed. claude -p --session-id <uuid> pins a headless session's id at spawn; claude -p --resume <uuid> resumes that session with a new prompt turn appended to its intact context. (--fork-session exists too; not needed here.)
  2. The workflow journal survives a kill -9 of the whole process group. It lives inside the session's transcript directory: ~/.claude/projects/<cwd-slug>/<session-id>/subagents/workflows/<wf-run-id>/journal.jsonl with one agent-*.jsonl transcript per completed sub-agent alongside it. Verified by the drill: SIGKILL to the process group mid-workflow, journal and completed-agent transcripts intact.
  3. resumeFromRunId is same-session only -- and a --resumed session IS the same session. The resumed turn sees the prior context, including the original Workflow tool result carrying the persisted scriptPath and runId, so a one-line resume prompt ("re-invoke the Workflow with resumeFromRunId from your earlier tool result") needs no state threaded from outside. Completed agent() calls return their cached results instantly; only the interrupted suffix runs live.
  4. Drill verdict: see the transcript block at the bottom -- the killed 6-step workflow resumed with the pre-kill agents served from cache and only the remainder executed live.

Design: the breadcrumb is just the session id

Because the resumed session already knows its own workflow run id and script path (fact 3), the runner does NOT need to parse journals or track workflow run ids. One column, one flag, one branch:

  • Migration: agent_runs ADD COLUMN session_id uuid (null = never spawned / pre-feature rows).
  • Spawn (fresh): the runner generates a uuid, reports it to the hub (POST /runs/agent/{id}/session {"session_id": "..."} -- same shape as the progress heartbeat), and passes --session-id <uuid> to claude -p.
  • Spawn (retry): the claim response already carries the run row; when attempts > 0 && session_id != null, the runner spawns claude -p --resume <session_id> with the standard resume prompt instead of the original skill prompt. The heartbeat, timeout, and completion paths are unchanged -- a resumed run is just a run.
  • Fallback, fail-open to fresh: if the resume spawn errors ("No conversation found with session ID", pruned transcripts -- cleanupPeriodDays defaults to 30 days, so only long-dead runs), the runner clears the breadcrumb and re-runs fresh with a NEW session id within the same attempt. Resume is an optimization; correctness stays with the ARGUS-298 attempts cap (2) and the visible-failure path.

What this deliberately does not do:

  • No journal parsing in Go. The session's own context is the source of truth for the workflow run id; duplicating it hub-side invites drift.
  • No resume for chat turns. A chat turn is one short claude -p call (2m sandbox); RecoverStale (ARGUS-299) already re-runs it whole, and there is nothing expensive to salvage.
  • No change to the deterministic loops. The 301 audit table covers them: they are idempotent sweeps over durable state and restart clean.

Cost/benefit

A reaped deep-research run today re-spends the full fan-out (~100+ sub-agent calls on run #13's shape). With resume, the retry re-spends only the interrupted suffix plus one orchestrator turn. The breadcrumb write is one tiny POST per run. Worst case (stale session, failed resume) costs one extra spawn attempt before the fresh fallback -- bounded and visible.

Implementation status

Implemented (ARGUS-308): migration 0064 adds agent_runs.session_id; the runner mints and breadcrumbs a session id before every fresh spawn (POST /runs/agent/{id}/session), a requeued attempt with a breadcrumb spawns claude -p --resume, and a resume whose session is gone falls back fail-open to a fresh run within the same attempt. The drill script stays in scripts/ and doubles as the regression check: run it after any claude CLI major upgrade to confirm the resume contract still holds.

Drill transcript (2026-07-04, claude-sonnet-5)

== drill session 595f9ee3-6a62-41b0-87f8-0f1770b33762 (model claude-sonnet-5)
== killed pgid 620212 with 3 agents completed
== pre-kill: 3 agent transcripts; journal: .../595f9ee3.../subagents/workflows/wf_d7bee57c-205/journal.jsonl
== resumed OK in 27s: all 6 steps present
== agents: 3 cached from before the kill (untouched: 3 of 3), 4 ran live after resume
== PASS: resume replayed the cached prefix instead of restarting from zero

The 4 (not 3) live agents after resume: the step that was MID-FLIGHT at the kill has a transcript file but no journaled result -- only completed agent() calls cache, so the interrupted one re-runs. That is the correct boundary: cached prefix replayed, interrupted work redone, nothing lost.