Restart resilience — audited per loop (ARGUS-301)

What actually happens to each moving part when its host dies mid-work. Written after the 2026-07-04 incident: two LXC restarts on .96 orphaned an in-flight deep-research run and a chat turn invisibly, and the keepalive crons relaunched both daemons with a broken PATH that failed 9 runs over 8 hours. Every claim below cites the code or test that proves it; "believed safe" without evidence is exactly what this document exists to kill.

The system's restart posture rests on one principle: durable state in Postgres + idempotent sweeps over it. Anything holding work only in process memory is a gap by definition.

Verdict table

Loop	Restart-safe?	Mechanism	Evidence
Chat worker daemon	yes	keepalive cron every 3m + @reboot	crontab on .96
— in-flight chat turn	yes (since ARGUS-299)	stale-running sweep: pending once, visible error at cap	RecoverStale (chat_turns.go), TestChatTurnRecoverStale
Agent-runner daemon	yes	keepalive cron every cycle + @reboot	crontab on .96
— self-swap on a staged rebuild	yes (since ARGUS-541)	binary mtime/size fingerprinted at start; re-checked after every completed run and every idle poll; a change exits 0 at that safe point (never mid-run) for the keepalive cron to respawn onto	SelfSwapWatcher, TestClaimLaneIdleExitsOnStagedBinary, TestClaimLaneDefersSwapUntilRunCompletes
— in-flight agent run	yes (since ARGUS-298)	heartbeat-based reaper: requeue ≤2 attempts, then visible fail + ledger close + push	ReapStale (agent_runs.go), TestReapStale
— in-run skill progress	yes (since ARGUS-308)	session-id breadcrumb at spawn; a requeued attempt spawns claude -p --resume and the journal's cached prefix replays; fresh fallback if the session is gone	TestDefaultResearchRunnerSessionModes, TestAgentRunSessionBreadcrumb, drill in docs/workflow-resume.md
Push notifier	yes	durable notified_at watermark; stamp-after-attempt	PushNotifier.Tick, TestPushNotifierChatTurns (no re-push case)
Runs retention	yes	idempotent DELETE over durable rows; boot + 24h; self-reports a ledger row	PruneRoutineRuns, prune loop in cmd/hub/main.go
Resilience sweep itself	yes	stateless; boot + every 1m; FOR UPDATE SKIP LOCKED never races live workers	reap loop in cmd/hub/main.go
Watchdog	yes	boot sweep + 5m ticks; expectations re-read from config.ini every sweep	watchdog wiring in cmd/hub/main.go ("armed from second zero", ARGUS observability-ops-5)
Ofelia scheduled ticks	yes (by detection)	fire-and-forget; a tick missed during downtime is NOT replayed — it surfaces as a watchdog no-show instead	design: spec F4; no-show path ActiveNoShows
Digest scheduler	yes (skips, never spins)	next-fire recomputed in memory from now() at boot; a missed window is skipped, not replayed; send failure advances next-fire	maybeSend (digest_scheduler.go)
Capture/recall hooks	yes	stateless per invocation; capture keeps a per-session byte offset on disk and re-seeks (shrunken file resets)	cmd/capture-hook/main.go loadOffset
Hub/memory containers	yes	restart: unless-stopped + docker healthchecks; migrations idempotent at boot	compose.yaml
Claude interactive session	yes (since 2026-07-04)	tmux launcher respawn loop with --continue after any death (the 8G cgroup OOM case)	/usr/local/bin/start-claude-tmux.sh on .96

Incident findings (2026-07-04) and their fixes

1. Keepalive restarts carried cron's minimal PATH — claude (in ~/.local/bin) was invisible to both daemons after an LXC reboot; 9 runs failed executable file not found in $PATH over 8 hours. The daemons had only ever been started manually from login shells, masking it. FIXED: both ~/.argus/bin/*-run.sh scripts export PATH themselves. Residual hardening (spawn-time binary re-resolution + consecutive-failure escalation) is ARGUS-302.
2. Orphaned in-flight work was invisible — a running agent run or chat turn stayed running forever after a host death. FIXED: ARGUS-298 (heartbeat reaper) + ARGUS-299 (turn recovery), both bounded so a repeat-victim fails loudly instead of cycling.
3. The OOM that caused the restarts — the interactive claude process ballooned to ~7.5G running a multi-agent Workflow in-session and the LXC's 8G cgroup killed it; the kill logs ONLY in the Proxmox host's dmesg, never inside the container. Do not run large Workflow fan-outs in the long-lived interactive session; route them through the agent-runner.
4. Failure pushes fired but did not cut through — each of the 9 failures pushed individually overnight; nobody acts on nine separate 3am pings. Escalation-on-consecutive-failures is the ARGUS-302 remedy.

Known residual gaps (tracked)

• (none from the in-run-progress gap: ARGUS-308 implemented the ARGUS-300 design -- docs/workflow-resume.md.)
• OAuth usage endpoint 401 after restarts (observed 06:23 2026-07-04): the ccusage fallback gate takes over by design, so admission stays gated; self-heals when the interactive CLI refreshes credentials. Watch for persistence; not yet filed as a task.

Deploying a new agent-runner build (ARGUS-541)

The runner cannot replace its own executing process, so the deploy flow is deliberately dumb: build the new binary straight onto the same path the running process was started from, then walk away.

cd ~/projects/argus-build && git pull
PATH=~/sdk/go/bin:$PATH go build -o /path/to/agent-runner ./cmd/agent-runner

No kill, no manual restart, no coordinating with whatever the runner happens to be doing. At startup the runner fingerprints (mtime + size) the binary at its own resolved executable path (os.Executable(), symlinks followed). It re-checks that fingerprint at every SAFE point in its claim loop -- right after a run completes and is reported, and on every idle poll that claimed nothing -- never while a run is actually executing and never leaving a claimed row unreported. The moment the fingerprint differs it logs staged binary detected; exiting for respawn and exits 0; the keepalive cron (Agent-runner daemon row above) notices the dead process and relaunches it from the same path within its normal cycle, now running the new build. With ARGUS_RUNNER_LANES > 1 every lane shares one watcher and independently notices the same staged file, so the whole process still exits promptly without any cross-lane coordination.

How to re-verify this document

Each "yes" row's evidence is a store method + test pair or a main.go wiring block; go test ./internal/hub/ -run 'TestReapStale|TestChatTurnRecoverStale|TestPushNotifier' exercises the recovery paths directly. For a live drill: kill the runner mid-run (kill $(pgrep -f agent-runner)) and watch the run requeue within ~6 minutes, then exhaust the cap and watch it fail with a push.