Skip to content

Landing queue: rebase, CI, merge, evidence

The landing queue serializes one merge at a time per repo so two independently-green PRs never combine into a red main. It runs a small state machine (queued -> rebasing -> ci -> merging -> landed) and is the only door through which a task's PR evidence gets attached.

Note: the acceptance criteria for this scenario names a "renumber" step. The only renumbering that actually exists is a migration-filename collision fixup that runs post-rebase, pre-CI -- not a task-renumbering step, and not related to supersession.go (a same-named-sounding but functionally separate task-graph dedup mechanism). The diagram below reflects the real sequence, not the assumed one.

flowchart TD
    R["POST /landing/{owner}/{name}/ready"] --> Q[("landing_queue\nstatus=queued")]
    Q --> PR["ProcessRepo: promote oldest\nqueued row (SKIP LOCKED)"]
    PR --> REB["stageRebase:\nforge.UpdateBranch"]

    REB -->|"merged outside queue"| LANDED
    REB -->|"PR closed"| FAIL1["failed: PR closed\nwithout merging"]
    REB -->|"409 conflict"| FAIL2["failed: rebase conflict --\nresolve and re-signal ready"]
    REB -->|"10m timeout"| FAIL3["failed: persistent\nforge error"]
    REB -->|"success"| RENUM["renumberMigrations:\nfilename collision -> max+1"]

    RENUM -->|"best effort, logs\nonly on failure"| CI["stageCI:\nCombinedStatus"]

    CI -->|"head moved mid-wait\n(bounded 3 attempts)"| REB
    CI -->|"pending past 2m grace"| KICK["KickCI: close/reopen\nto mint a real event"]
    KICK --> CI
    CI -->|"pending past 45m"| FAIL4["failed: CI timeout"]
    CI -->|"failure, flake-shaped"| RETRY["retryFlake:\none automatic re-check"]
    RETRY --> CI
    CI -->|"failure, not flake"| FAILDET["fail:\nPR comment + onFailed hook"]
    CI -->|"success"| MERGE["stageMerge: forge.Merge\n+ re-verify via PRHead"]

    MERGE -->|"merged=false"| FAIL5["failed: merge\ndid not take"]
    MERGE -->|"confirmed merged"| LANDED["finish(landed)"]

    LANDED --> EV["AttachPREvidence\nsatisfied=true"]
    EV --> DEPHOOK["deploy-hook enqueue\n(ARGUS-343)"]
    DEPHOOK --> AUTOLINK["AutoLinkPRs\nrepo catch-up"]
    AUTOLINK --> SWEEP["items.VerifySweep"]
    SWEEP --> RESOLVE["alert auto-resolve +\nacceptance-fidelity review +\nowner notification"]

    LANDED --> POSTCI["watchPostCI:\nCombinedStatus on merge SHA"]
    POSTCI -->|green| POSTGREEN["onPostGreen:\nauto-resolve alert"]
    POSTCI -->|red| PAGE["onPostRed:\nPushAlert MAIN IS RED\n+ post_merge_break alert"]

    FAILDET -.->|"PR later closed unmerged\non an otherwise-verified task"| SUPER["reconcileSuperseded:\nstatus=withdrawn"]

Node annotations

POST /landing/{owner}/{name}/ready

internal/hub/landing_api.go:handleLandingReady (L34) -> internal/hub/landing.go:LandingQueue.Ready (L197). Idempotent per live (repo, PR): a unique-violation on insert just returns the existing live entry rather than erroring.

ProcessRepo: promote oldest queued row

internal/hub/landing.go:ProcessRepo (L438), driven by Run's loop (processAll, L400, up to 10 stage-advances per repo per tick). Uses FOR UPDATE SKIP LOCKED plus a landing_one_inflight unique index, so a double-promotion attempt is a no-op rather than a race.

stageRebase

internal/hub/landing.go:stageRebase (L479) calls forge.UpdateBranch -> Forgejo POST /pulls/{n}/update?style=rebase (landing_forge.go:91). Branch outcomes: already merged, closed, a genuine 409 conflict (fail with the exact runbook-matching string), a 10-minute timeout, or success.

renumberMigrations

internal/hub/landing.go:renumberMigrations (L539), called from inside stageRebase immediately after a clean rebase and before CI. Diffs migration filenames added on the branch against main; a numeric-prefix collision is renamed to max+1 and committed onto the branch via forge.RenumberMigrations. Best-effort: a failure here only logs a warning (L511-515) and CI still runs.

stageCI

internal/hub/landing.go:stageCI (L587). Re-reads the head SHA first (if the branch moved mid-wait, it bounces back to rebasing, bounded by landingMaxAttempts=3), otherwise reads forge.CombinedStatus.

KickCI

Pending status past landingCIKickGrace (2 minutes) triggers one close/reopen of the PR to force a fresh CI event out of Forgejo, rather than waiting indefinitely on a status that may never post.

retryFlake / fail

On CI failure, ciFailureDiagnosis (L718) enriches the failure via ForgeSource (failing check name + a log excerpt, best-effort, ARGUS-572). classifyCIFailure decides flake-shaped vs real: flake-shaped and not yet retried -> retryFlake (L749, one automatic re-check); otherwise fail (L1418) -- failed status, a PR comment via forge.Comment, and the onFailed hook (push notification + ci_red alert signal).

stageMerge

internal/hub/landing.go:stageMerge (L655) calls forge.Merge (Forgejo POST /pulls/{n}/merge) then re-verifies via PRHead rather than trusting the merge response alone -- a merged=false readback fails the stage with "merge did not take".

finish(landed)

landing.go:671. Triggers the onLanded hook wired in cmd/hub/main.go:264, run in this exact order: AttachPREvidence, deploy hook enqueue, AutoLinkPRs, items.VerifySweep.

AttachPREvidence

internal/hub/landing_evidence.go:AttachPREvidence (L23). Inserts an already-satisfied task_evidence(kind='pr') row -- by the time this runs, CI-green and merge are already proven facts, not something to re-check.

items.VerifySweep

Re-derives verified from all evidence immediately (cmd/hub/main.go:291); see the task lifecycle page for the full sweep logic and the binding-anchor rule that guards it (verify_cirun_binding_test.go, verify_artifact_binding_test.go, verify_grandfather_test.go for the pre-ARGUS-450 evidence carve-out). verify_rollup.go:rollupVerify (L68) then propagates verification up to epics/features once decomposed_at is set.

watchPostCI

internal/hub/landing.go:watchPostCI (L783), a separate per-tick loop: fetches the merge SHA and watches CombinedStatus on it specifically (not the PR branch, which no longer exists post-merge).

onPostRed

Pages Aaron directly: push.PushAlert (cmd/hub/main.go:354, "MAIN IS RED after landing ... see docs/landing-runbook.md") plus a post_merge_break alert -- see native alert pipeline. Manual revert procedure lives in docs/runbook/landing-runbook.md.

reconcileSuperseded

internal/hub/landing.go:reconcileSuperseded (L858), each tick: a failed entry whose PR later closed unmerged, on a task that later got verified some other way, is flipped to withdrawn. Unrelated to migration renumbering despite the naming similarity.

Failure surfacing

Every CI-red / rebase-conflict / stage failure produces a best-effort PR comment plus the onFailed hook: a push notification (see notification fan-out) and a ci_red alert-store signal. A post-merge break is a separate, more severe page -- the one failure mode the serial queue cannot prevent, since it only shows up once two independently-green changes combine.

Key tables and endpoints

  • landing_queue (migrations/0069_landing_queue.sql).
  • POST /landing/{owner}/{name}/ready, POST /landing/{owner}/{name}/rerun-ci/{pr}, GET /landing/{owner}/{name}/status, GET /landing/stats.json, DELETE /landing/{owner}/{name}/ready/{pr}.