Restart resilience — audited per loop (ARGUS-301)

What actually happens to each moving part when its host dies mid-work. Written after the 2026-07-04 incident: two LXC restarts on .96 orphaned an in-flight deep-research run and a chat turn invisibly, and the keepalive crons relaunched both daemons with a broken PATH that failed 9 runs over 8 hours. Every claim below cites the code or test that proves it; "believed safe" without evidence is exactly what this document exists to kill.

The system's restart posture rests on one principle: durable state in Postgres + idempotent sweeps over it. Anything holding work only in process memory is a gap by definition.

Verdict table

Loop	Restart-safe?	Mechanism	Evidence
Chat worker daemon	yes	keepalive cron every 3m + @reboot	crontab on .96
— in-flight chat turn	yes (since ARGUS-299)	stale-running sweep: pending once, visible error at cap	RecoverStale (chat_turns.go), TestChatTurnRecoverStale
Agent-runner daemon	yes	keepalive cron every cycle + @reboot	crontab on .96
— self-swap on a staged rebuild	yes (since ARGUS-541, deploy-safe since ARGUS-768)	binary mtime/size fingerprinted at start; re-checked after every completed run and every idle poll; a change exits 0 at that safe point (never mid-run, and never while the deploy-lane's own in-flight deploy has not yet reported its outcome) for the keepalive cron to respawn onto	SelfSwapWatcher, DeploySafeSwapWatcher, TestClaimLaneIdleExitsOnStagedBinary, TestClaimLaneDefersSwapUntilRunCompletes, TestDeploySafeSwapWatcherDefersWhileDeployInFlight
— in-flight agent run	yes (since ARGUS-298)	heartbeat-based reaper: requeue ≤2 attempts, then visible fail + ledger close + push	ReapStale (agent_runs.go), TestReapStale
chat-worker/chat-mcp/skill-sync build freshness	yes (since ARGUS-1416; agent-runner since ARGUS-749)	deploy/changed-services.sh tracks all four host binaries; deploy/argus.sh stages each onto its $ARGUS_DEPLOY_*_BIN path when its build inputs change	TestDeployChangedServices, TestArgusDeployHookStagesChatWorkerAndChatMCPBinaries, TestArgusDeployHookStagesSkillSyncBinary, TestArgusDeployHookStagesAgentRunnerBinary
— stale-binary visibility	yes (since ARGUS-1416)	each staged host binary best-effort POSTs its build stamp to the hub; GET /host-binaries.json reads it back	HostBinaryBuildStore, TestArgusDeployHookReportsHostBinaryBuildStamp, TestHandleHostBinaryBuildReportRecordsAllowlistedBinary
Satellite runner build freshness (mac, pve-builder)	mechanism shipped (ARGUS-1464); host wiring pending (ARGUS-1525)	cmd/satellite-runner-update, cron/launchd-driven, stages onto the SAME path SelfSwapWatcher already fingerprints; .commit sidecar records the installed sha	TestNeedsUpdate, TestReadInstalledSHA, TestWriteInstalledSHA, TestSidecarPath
— build-input gate (only rebuild for a diff that actually touches the binary)	yes (since ARGUS-1529)	a tracked sidecar sha diffs the changed paths against AGENT_RUNNER_PREFIXES's mirror before deciding to rebuild cmd/agent-runner; a missing/never-tracked sidecar always rebuilds	TestAffectsAgentRunner, TestRunGatesRebuildByChangedPaths
— updater self-rebuild	yes (since ARGUS-1540)	-self-bin (defaults to the running executable's own path) rebuilds+atomically-replaces cmd/satellite-runner-update's OWN binary, gated independently by its own build-input prefixes (cmd/satellite-runner-update/, go.mod, go.sum), so a fix to the updater itself (e.g. ARGUS-1529) reaches a satellite the same automatic way an agent-runner fix does, instead of needing install-*-runner.sh re-run by hand; darwin re-signs with its own -self-codesign-identifier, distinct from agent-runner's	TestAffectsSelfUpdater, TestRunSelfRebuildsOnOwnDiff, TestRunCodesignsSelfOnDarwinBeforeInstall, TestInstallMacRunnerUpdatePlistPassesSelfUpdateFlags, TestInstallMacRunnerCodesignsUpdateBinAtInstallTime
— runner_capabilities poll-interval wire format	yes (since ARGUS-1464)	RunnerHeartbeat JSON now reports poll_interval_seconds/stale_after_seconds as whole seconds, not time.Duration's raw nanosecond count	TestRunnerHeartbeatJSONPollIntervalIsSeconds
— in-run skill progress	yes (since ARGUS-308)	session-id breadcrumb at spawn; a requeued attempt spawns claude -p --resume and the journal's cached prefix replays; fresh fallback if the session is gone	TestDefaultResearchRunnerSessionModes, TestAgentRunSessionBreadcrumb, drill in docs/workflow-resume.md
Push notifier	yes	durable notified_at watermark; stamp-after-attempt	PushNotifier.Tick, TestPushNotifierChatTurns (no re-push case)
Runs retention	yes	idempotent DELETE over durable rows; boot + 24h; self-reports a ledger row	PruneRoutineRuns, prune loop in cmd/hub/main.go
Resilience sweep itself	yes	stateless; boot + every 1m; FOR UPDATE SKIP LOCKED never races live workers	reap loop in cmd/hub/main.go
Watchdog	yes	boot sweep + 5m ticks; expectations re-read from config.ini every sweep	watchdog wiring in cmd/hub/main.go ("armed from second zero", ARGUS observability-ops-5)
Ofelia scheduled ticks	yes (by detection)	fire-and-forget; a tick missed during downtime is NOT replayed — it surfaces as a watchdog no-show instead	design: spec F4; no-show path ActiveNoShows
Digest scheduler	yes (skips, never spins)	next-fire recomputed in memory from now() at boot; a missed window is skipped, not replayed; send failure advances next-fire	maybeSend (digest_scheduler.go)
Capture/recall hooks	yes	stateless per invocation; capture keeps a per-session byte offset on disk and re-seeks (shrunken file resets)	cmd/capture-hook/main.go loadOffset
Hub/memory containers	yes	restart: unless-stopped + docker healthchecks; migrations idempotent at boot	compose.yaml
Claude interactive session	yes (since 2026-07-04)	tmux launcher respawn loop with --continue after any death (the 8G cgroup OOM case)	/usr/local/bin/start-claude-tmux.sh on .96

Incident findings (2026-07-04) and their fixes

1. Keepalive restarts carried cron's minimal PATH — claude (in ~/.local/bin) was invisible to both daemons after an LXC reboot; 9 runs failed executable file not found in $PATH over 8 hours. The daemons had only ever been started manually from login shells, masking it. FIXED: both ~/.argus/bin/*-run.sh scripts export PATH themselves. Residual hardening (spawn-time binary re-resolution + consecutive-failure escalation) is ARGUS-302.
2. Orphaned in-flight work was invisible — a running agent run or chat turn stayed running forever after a host death. FIXED: ARGUS-298 (heartbeat reaper) + ARGUS-299 (turn recovery), both bounded so a repeat-victim fails loudly instead of cycling.
3. The OOM that caused the restarts — the interactive claude process ballooned to ~7.5G running a multi-agent Workflow in-session and the LXC's 8G cgroup killed it; the kill logs ONLY in the Proxmox host's dmesg, never inside the container. Do not run large Workflow fan-outs in the long-lived interactive session; route them through the agent-runner.
4. Failure pushes fired but did not cut through — each of the 9 failures pushed individually overnight; nobody acts on nine separate 3am pings. Escalation-on-consecutive-failures is the ARGUS-302 remedy.

Known residual gaps (tracked)

• (none from the in-run-progress gap: ARGUS-308 implemented the ARGUS-300 design -- docs/workflow-resume.md.)
• OAuth usage endpoint 401 after restarts (observed 06:23 2026-07-04): the ccusage fallback gate takes over by design, so admission stays gated; self-heals when the interactive CLI refreshes credentials. Watch for persistence; not yet filed as a task.
• Satellite runner-update wiring not yet applied to the live hosts (ARGUS-1464 shipped the mechanism; ARGUS-1525 tracks applying it): the Mac launchd job needs a human at the keyboard (codesign + launchctl under the logged-in session) and pve-builder CT113's cron needs scripts/install-pve-builder-runner.sh run on that host -- neither is something a Linux build-box PR can do by itself. Until both are applied, the two satellite runners stay on whatever binary was last hand-built.
• chat-worker has no in-process self-swap watcher (ARGUS-1416): unlike agent-runner, staging a new chat-worker binary does not by itself make the already-running process pick it up -- it needs the host's existing keepalive cron to actually cycle the process (a crash, or a manual bounce) before the new code takes effect. Mirroring SelfSwapWatcher into chat-worker's own poll loop (cmd/chat-worker/main.go) is a natural follow-up, not yet built. chat-mcp has no such gap: it is spawned fresh per chat turn, so a staged binary is picked up by the very next turn with no restart needed at all.

Regression note: a schema rename's blast radius includes host binaries, not just hub (ARGUS-1416)

2026-08-19: the app chat page had been silently dead since the telegram_channels->chat_channels rename because ~/.argus/bin/chat-worker (a host binary, cron-keepalive, built Jul 20) was not in deploy/argus.sh's build set -- only agent-runner got rebuilt+self-swapped automatically. Every deploy shipped a current hub while chat-worker silently queried the dropped table; every chat turn errored load session: channel session: relation telegram_channels does not exist. cmd/skill-sync rotted the identical way on an earlier rename (telegram_enabled->command_enabled), caught only by a manual rebuild on 2026-08-16 -- and, as of this fix, is tracked mechanically instead of relying on the next rename to catch it by hand again.

The lesson for any future table/column rename (or any other breaking internal/hub API change): its blast radius is every binary that imports package hub, not just the containerized hub/memory services deploy/argus.sh already rebuilt automatically. As of ARGUS-1416, deploy/changed-services.sh tracks agent-runner, chat-worker, chat-mcp, and skill-sync as a group (all four import package hub directly, so any of them changing under internal/hub/ rebuilds all four) -- but a FUTURE host binary added under cmd/ still needs to be added to that script's prefix lists and to deploy/argus.sh's staging blocks by hand; nothing detects a new cmd/* directory automatically. When adding a new host binary that imports package hub, update both scripts in the same PR that adds the binary, not as a follow-up.

Stale-binary visibility (also ARGUS-1416): every host binary deploy/argus.sh stages now best-effort POSTs its build stamp to the hub (POST /internal/host-binary-build, opt-in via ARGUS_DEPLOY_HUB_URL/ ARGUS_DEPLOY_CAPTURE_TOKEN), readable back via GET /host-binaries.json (internal/hub/host_binary_builds.go) -- so "is what's actually running stale" is answerable by checking that endpoint instead of waiting for a live error, or SSHing into the host to check a binary's mtime by hand.

Deploying a new host binary build (agent-runner: ARGUS-541/ARGUS-749; chat-worker/chat-mcp/skill-sync: ARGUS-1416)

The runner cannot replace its own executing process, so the deploy flow is deliberately dumb: build the new binary straight onto the same path the running process was started from, then walk away. The same pattern now covers all four host binaries deploy/argus.sh knows about -- agent-runner, chat-worker, chat-mcp, and skill-sync -- each staged onto its own $ARGUS_DEPLOY_*_BIN path (defaults ~/.argus/bin/agent-runner, ~/.argus/bin/chat-worker, ~/.argus/bin/chat-mcp, ~/.argus/bin/skill-sync).

This staging step is automatic: deploy/argus.sh runs it as part of every deploy whenever deploy/changed-services.sh reports the binary changed (any commit under its own cmd/.../ directory, or internal/hub/, which all four import directly). Before ARGUS-749, nothing ever rebuilt agent-runner automatically -- a runner-affecting merge could go green and sit un-deployed indefinitely, which is exactly how ARGUS-626 (the run-id-in-env/prompt fix milestone posting depends on) sat un-deployed on the live host for several days: verified live against the running binary, whose strings output still had the old "ARGUS_AGENT_RUN=1" placeholder and no ARGUS_RUN_ID. chat-worker, chat-mcp, and skill-sync had the identical gap until ARGUS-1416 -- see the regression note above for that incident.

Unlike agent-runner, none of chat-worker/chat-mcp/skill-sync have an in-process self-swap watcher (see "Known residual gaps" above): chat-mcp and skill-sync don't need one (chat-mcp is spawned fresh per chat turn, skill-sync is a short-lived cron invocation -- both simply pick up a staged binary on their next natural run), and chat-worker's is a documented follow-up, not yet built. The rest of this section (fingerprinting, DeployInFlight deferral) is agent-runner-specific.

The manual command below is now only a break-glass fallback (host down for the deploy poll, deploy pipeline itself broken):

cd ~/projects/argus-build && git pull
PATH=~/sdk/go/bin:$PATH go build -o /path/to/agent-runner ./cmd/agent-runner

No kill, no manual restart, no coordinating with whatever the runner happens to be doing -- true whether the binary was staged by the automated hook or by hand. At startup the runner fingerprints (mtime + size) the binary at its own resolved executable path (os.Executable(), symlinks followed). It re-checks that fingerprint at every SAFE point in its claim loop -- right after a run completes and is reported, and on every idle poll that claimed nothing -- never while a run is actually executing and never leaving a claimed row unreported. Since ARGUS-768, "safe" also requires no deploy to be in flight on the separate deploy lane (RunDeployLoop, ARGUS-517): a DeployInFlight counter is held from the moment a deploy is claimed until its completion report has landed (including any completeDeployWithRetry attempts), and DeploySafeSwapWatcher defers the exit while it is set -- otherwise a PR that touches cmd/agent-runner stages its own replacement binary as a build side effect, and the bare fingerprint check would exit the whole process (killing every goroutine, including that same deploy's in-flight completion report) before the deploy it just staged could report its own outcome, exactly what happened to deploy 298/ARGUS-766. The moment the fingerprint differs AND no deploy is in flight, it logs staged binary detected; exiting for respawn and exits 0; the keepalive cron (Agent-runner daemon row above) notices the dead process and relaunches it from the same path within its normal cycle, now running the new build. With ARGUS_RUNNER_SLOTS > 1 (ARGUS-1040; formerly ARGUS_RUNNER_LANES, ARGUS-534) every slot shares one watcher and independently notices the same staged file, so the whole process still exits promptly without any cross-slot coordination.

Deploying a satellite runner build (mac, pve-builder: ARGUS-1464)

The .96 host's staging half (deploy/argus.sh + changed-services.sh) only ever runs on .96 -- the Mac launchd runner and pve-builder CT113 had no equivalent, so every agent-runner fix since their last hand-build simply never reached them, and their runner_capabilities.poll_interval_seconds stayed 0 (undeclared) long after ARGUS-1421 shipped elsewhere. cmd/satellite-runner-update is the satellite-side staging half: given -checkout <repo> and -bin <installed path>, it fetches the checkout's origin/main, compares its sha against a .commit sidecar file recorded beside the binary (e.g. ~/.argus/bin/agent-runner.commit), and on a mismatch resets the checkout, rebuilds cmd/agent-runner, atomically replaces the binary (os.Rename, same filesystem), and updates the sidecar. Identical shas is a no-op -- nothing rebuilt, nothing touched.

ARGUS-1540: the tool also rebuilds ITSELF when the diff touches its own package (cmd/satellite-runner-update/) or go.mod/go.sum, via a second, independently-gated -self-bin path (defaulting to the currently running executable's own path via os.Executable, so no install-script change is required just to opt in) -- before this, a fix to the updater's own code (such as ARGUS-1529's build-input gate above) never reached a satellite host until someone re-ran the install script by hand, the same rot class ARGUS-749/ARGUS-1416 already found for other host binaries.

It deliberately reuses the SAME safe-point mechanism as the .96 host instead of inventing its own: SelfSwapWatcher inside cmd/agent-runner is already generic and already armed on both satellite lanes (confirmed live in each runner's own startup log: self-swap watcher armed path=.../agent-runner), so the atomic rename above is by itself enough to trigger it -- the tool never needs to know whether a run is in flight, and never restarts anything directly. The existing OS-level keepalive (launchd KeepAlive, or pve-builder's keepalive cron) relaunches the process from the same path once it exits.

Wiring, one per satellite:

• Mac: scripts/install-mac-runner.sh builds satellite-runner-update alongside agent-runner, codesigns BOTH with the persistent "Argus Runner" identity ($UPDATE_BIN with its own $UPDATE_IDENTIFIER, distinct from agent-runner's $IDENTIFIER, ARGUS-1540), and installs a SECOND launchd job (com.runonyourown.argus-mac-runner-update, StartInterval=300) that runs it against the checkout the script itself was run from, passing -self-bin/-self-codesign-identifier so a future self-rebuild re-signs with that same persistent identity. The runner job's own plist also now sets ARGUS_RUNNER_NAME=mac (closing a previously-undocumented gap where a reinstall could re-collide the Mac lane onto runner_capabilities' "host" row) and ARGUS_RUNNER_POLL=15s (matching pve-builder's existing declared cadence).
• pve-builder: scripts/install-pve-builder-runner.sh (new; this host's run/keepalive wrapper scripts were never repo-tracked before ARGUS-1464) builds both binaries, writes the run/keepalive wrappers, and installs a THIRD cron line (*/5 * * * *) alongside the existing keepalive cron that runs satellite-runner-update (passing -self-bin, ARGUS-1540; no codesign flags -- this host has no TCC to protect) against the checkout the script was run from.

Neither script has been re-run against the live physical Mac or pve-builder CT113 as part of shipping this mechanism (ARGUS-1525 tracks that + the live poll_interval_seconds=15 / no-false-STALE confirmation) -- a Linux build box has no access to the former and applying a new cron entry to the latter without review is a live production mutation outside a single PR's diff.

How to re-verify this document

Each "yes" row's evidence is a store method + test pair or a main.go wiring block; go test ./internal/hub/ -run 'TestReapStale|TestChatTurnRecoverStale|TestPushNotifier' exercises the recovery paths directly. For a live drill: kill the runner mid-run (kill $(pgrep -f agent-runner)) and watch the run requeue within ~6 minutes, then exhaust the cap and watch it fail with a push.