Skip to content

Dispatch, claim-lease and heartbeats

Two separate claim mechanisms exist on top of each other: the run-level queue (agent_runs, claimed by a runner process) and the task-level claim (task_items.lifecycle, claimed by the skill running inside that process). This page traces both, plus the two independent watchdogs that recover a stuck run or a stuck task.

flowchart TD
    T["Dispatch.Run poll tick\nor Kick()"] --> P["pollOnce: per-project gate"]
    P -->|"not armed"| SKIP["skip project"]
    P -->|armed| PD["PromoteReadyDependents"]
    PD --> DH["dispatchHub"]

    DH --> RC{"ready count == 0?"}
    RC -->|yes| CLR["clear defer state, return"]
    RC -->|no| BACK{"no-claim backoff\nstreak exceeded?"}
    BACK -->|yes| CLR2["back off, log ledger row"]
    BACK -->|no| LIVE{"live session holds\nAND defer window not elapsed?"}

    LIVE -->|yes| DEFER["armDeferRetry\nself-Kick at expiry"]
    LIVE -->|no| INFL{"builder already\nqueued/running?"}
    INFL -->|yes| WAIT["return, serialize one builder"]
    INFL -->|no| ENQ["AgentRunStore.Enqueue"]

    ENQ --> QROW[("agent_runs\nstatus=queued")]
    QROW --> CLAIM["Runner: ClaimWithOptions\nPOST /runs/agent/claim"]
    CLAIM --> CLAIMDB["Claim: UPDATE ... SKIP LOCKED\nstatus=running"]
    CLAIMDB --> TCLAIM["Skill calls\nPOST /tasks/{id}/lifecycle"]
    TCLAIM --> TROW[("task_items\nlifecycle=in_progress")]

    TROW --> HB["1-min ticker:\nProgress(heartbeat=true)"]
    HB --> HBDB["Heartbeat:\nUPDATE progress_at=now()"]
    HBDB -.-> HB

    TROW --> DONE["Skill returns"]
    DONE --> COMPLETE["Complete:\nPOST /runs/agent/{id}/complete"]
    COMPLETE --> DSTAT[("agent_runs\nstatus=done|failed")]

    QROW --> REAPER{"ReapStale:\nno progress_at for 5m?"}
    REAPER -->|attempts left| REQ["status=queued\nsession_id preserved"]
    REQ -.->|"resume with --resume"| CLAIM
    REAPER -->|attempts exhausted| FAILED[("agent_runs\nstatus=failed")]

    TROW --> LEASE{"ClaimLeaseSweeper:\nlease window elapsed\nAND no evidence\nAND no live linked run?"}
    LEASE -->|yes| RELEASE["release:\nlifecycle=ready"]
    RELEASE -.->|"task_events: claim_released\n+ alert"| T
    LEASE -->|no, run still live| TROW

Node annotations

Dispatch poll tick / Kick()

internal/hub/dispatch.go:Dispatch.Run (L403) ticks on d.interval; Dispatch.Kick (L425) fires the same pass immediately on task-ready, builder-free, and landing events, rather than waiting for the next tick.

pollOnce: per-project gate

internal/hub/dispatch.go:pollOnce (L529). Checks projectDispatchArmed (per-project windows in dispatch_window_rule.go, else the legacy off/night/always schedule via dispatchArmed, L494). A project outside its window is skipped entirely for this tick.

PromoteReadyDependents

Called from pollOnce before every dispatch pass -- moves a backlog task to ready once its dependencies verify (ARGUS-393); see the task lifecycle page.

dispatchHub

internal/hub/dispatch.go:dispatchHub (L560). Reads ReadyTaskCountByOwner(project, "hub-agent") and applies, in order: the ready-count-zero short-circuit (which also clears the defer anchor below), the no-claim-backoff streak guard (noClaimBackoffAt), the live-session defer window (hubLiveDefer, L133, computed from the persisted defer-clock anchor so a long-running interactive session doesn't starve the board forever), and the one-in-flight guard (runs.PendingBySkill(hubBuildSkill), serializing one builder per project). Mac-agent work is routed separately via routeMacWork (L749), using the same anchor mechanism for its lease window.

dispatchDeferSince (defer-clock anchor, ARGUS-723)

internal/hub/dispatch.go:dispatchDeferSince, backed by the dispatch_defer_anchor table (migration 0154). OldestReadyByOwner (ARGUS-400) is durable against a hub restart, but it is RECOMPUTED from the current ready set's latest ready-transition event on every poll -- any ready-set churn (a stale-claim release, a task cycling out of and back into eligibility) can slide its own value forward and make an already-running window look freshly re-armed (the 2026-07-17 overnight incident: two 90+ minute stalls, each cleared only by a manual force-dispatch). The anchor is latched ONCE per ready streak -- bootstrapped from OldestReadyByOwner if no anchor exists yet, else now -- and then held fixed on every later call regardless of ready-set churn, until ClearDeferAnchor runs (ready count back to zero), so the next streak latches its own fresh anchor. The defer ledger row names the computed deadline (defer until HH:MM UTC) so a parked queue is visible from the ledger alone.

armDeferRetry

internal/hub/dispatch.go:armDeferRetry (L437). Self-arms a Kick() for the moment the defer window expires, so a deferred dispatch isn't lost to the next slow poll tick.

AgentRunStore.Enqueue

internal/hub/agent_runs.go:Enqueue (L216). Inserts an agent_runs row (status='queued', task_id set) built from buildPrompt (dispatch.go:726). The task's own lifecycle is still ready here -- the run being queued and the task being claimed are two separate writes, correlated only by task_id/prompt text.

Runner claims (ClaimWithOptions)

Host process cmd/agent-runner/main.go (L194) runs RunClaimLanes (agent_runner.go:262) -> claimLane (L280) -> ProcessAgentRun (L141), which calls Claim/ClaimWithOptions -> POST /runs/agent/claim (agent_runs_api.go:handleAgentClaim, L107).

Claim: UPDATE ... SKIP LOCKED

internal/hub/agent_runs.go:Claim (L289): UPDATE agent_runs SET status='running', started_at=now() WHERE id = (SELECT ... status='queued' ... FOR UPDATE SKIP LOCKED) RETURNING .... No queued row -> the runner loop just polls again (default 10s). A claimed row also upserts an agent_sessions presence row and opens a ledger row.

Skill calls task-lifecycle endpoint

This is the actual task-board claim: the spawned claude -p process running the hub-build skill calls POST /tasks/{id}/lifecycle (tasks_api.go:handleSetLifecycle -> items.go:SetLifecycleBlocked), flipping task_items.lifecycle ready -> in_progress and setting owner.

Heartbeat ticker

internal/hub/agent_runner.go:ProcessAgentRun (L181-202) spawns a background goroutine on a 1-minute ticker calling Progress(ctx, runID, note, heartbeat=true) -> POST /runs/agent/{id}/progress (agent_runs_api.go:handleAgentProgress, L172). heartbeat=true -> AgentRunStore.Heartbeat (agent_runs.go:658, a no-op error if the run isn't running); heartbeat=false (a genuine milestone note) -> RecordMilestone (agent_runs.go:676), appended to progress_trail jsonb.

Complete

ProcessAgentRun calls Complete -> POST /runs/agent/{id}/complete (agent_runs_api.go:handleAgentComplete, L362), setting status='done'|'failed'. A UsageCapError instead completes as failed with a pauseUntil, pausing the runner's claim loop until the usage window resets.

ReapStale (agent_runs watchdog)

cmd/hub/main.go:700 runs AgentRunStore.ReapStale(5*time.Minute, maxAttempts=2) (agent_runs.go:482) every minute: any running row whose progress_at/started_at/created_at is older than 5 minutes is requeued (status='queued', session_id kept for resume) if attempts remain, else marked failed. ProcessAgentRun (L210) detects Attempts>0 && SessionID!="" and resumes via claude -p --resume.

ClaimLeaseSweeper (task_items watchdog)

internal/hub/watchdog.go:315, invoked from Watchdog.Sweep (L205) -> ClaimLeaseSweeper.Sweep (claim_lease.go:62). Selects in_progress tasks owned by a *-agent, unchanged past the lease window (default 20m, hub_settings.claim_lease_window), with no task_evidence yet. For each, latestLinkedRun (claim_lease.go:169) checks whether a live (queued or running) agent_runs row is still correlated to the task; if so it's skipped as a false positive. Otherwise release (L197) flips the task back to ready, writes a claim_released task_events row, records a claim-lease alert, and Watchdog.notifyClaimReleased pushes a notification -- the task re-enters dispatch at the top of this page.

Key tables and endpoints

  • agent_runs (queued/running/done/failed, progress_at, progress_trail, task_id, session_id, attempts) -- the run-level queue/lease.
  • task_items.lifecycle -- the board-level claim (see task lifecycle).
  • hub_settings.claim_lease_window -- runtime-overridable lease duration (migrations 0101, 0112, 0131).
  • migrations/0097_project_dispatch_windows.sql -- per-project dispatch windows.
  • Endpoints (internal/hub/server.go:744-753): POST /runs/agent, POST /runs/agent/claim, POST /runs/agent/{id}/progress, POST /runs/agent/{id}/session, POST /runs/agent/{id}/release, POST /runs/agent/{id}/complete, GET /runs/agent/forced, POST /tasks/{id}/lifecycle.