Skip to content

Native alert pipeline: signal to push to resolve

This is a first-class, dedicated subsystem (US-11/US-36, ARGUS-596..599), distinct from the general notification fan-out -- it decides whether and when an alert-worthy event should page Aaron, with its own dedup, debounce, and storm-suppression logic, before handing off to the same push plumbing everything else uses. A legacy Grafana-relay path (alerts.go/alert_webhook.go/telegram.go) still exists in parallel and is being phased out -- it is not part of this diagram.

flowchart TD
    SIG["SignalXxx builders\n(alert_detectors.go)"] --> ENG["AlertEngine.Apply\n(pure, DB-free)"]

    ENG --> DEDUP{"matching open alert\nfor this dedup key?"}
    DEDUP -->|"none, firing"| CREATE["created: new alert row"]
    DEDUP -->|"exists, same/lower severity"| EXT["extended: absorbed\nas child line"]
    DEDUP -->|"exists, higher severity"| ESC["escalated: resets\npush-cycle bookkeeping"]
    DEDUP -->|"resolved, refires <75s"| REOPEN["reopened: same row"]
    DEDUP -->|"clearing signal"| RESOLVEAUTO["auto-resolved"]

    ENG --> STORM{">10 distinct dedup keys\nwithin 5 minutes?"}
    STORM -->|yes| ABSORB["absorbed: folded into one\nsynthetic critical storm alert"]

    CREATE --> STORE["AlertStore.Ingest:\ntx commit"]
    EXT --> STORE
    ESC --> STORE
    REOPEN --> STORE
    ABSORB --> STORE

    STORE -->|"post-commit,\nbest-effort"| ROUTE["PushNotifier.RouteAlert"]

    ROUTE --> SEV{severity}
    SEV -->|critical| CRIT["routeCritical:\nbypass quiet hours,\nrenotify +15m/+2h, cap 3"]
    SEV -->|warning| WARN["routeWarning:\none push ever,\nheld during quiet window"]
    SEV -->|info| INFO["routeInfo: always live"]

    CRIT --> FANOUT["fanOut -> Pusher.Push\n(APNs, ES256 JWT)"]
    WARN -->|"quiet window ends"| FANOUT
    INFO --> FANOUT

    FANOUT -->|"410 Unregistered"| GONE["device row deleted\n(ErrDeviceGone)"]
    FANOUT -->|"other error"| LOGFAIL["logged + fail row\nin runs ledger"]
    FANOUT -->|success| DELIVERED["delivered to device"]

    RESOLVEAUTO --> DONE1["resolved: auto"]
    DELIVERED --> ACK{"human action"}
    ACK -->|"POST /alerts/{id}/ack"| ACKED["acked"]
    ACK -->|"POST /alerts/{id}/dismiss"| DISMISSED["resolved: dismissed"]

Node annotations

SignalXxx builders

internal/hub/alert_detectors.go. Pure SignalXxx(...) functions (no DB or IO) mapping a real hub event to a Signal{DedupKey, Severity, Firing, Detail, DeepLink, At}: SignalRunFailure, SignalNoShowFiring/ SignalNoShowResolved, SignalWatchdogBlind, SignalPostMergeBreak/ Green, SignalCIRed/Resolved, SignalCaptureStale/Fresh, SignalDeployFailure/Recovered. Called from internal/hub/server.go:handleReportRun (L926), internal/hub/watchdog.go (L252, L287), and cmd/hub/main.go (L298, L329, L358, L366, L678, L795, L818) -- landing CI-red/post-merge, deploy failure, watchdog-blind dead-man's-switch, ingest heartbeat.

AlertEngine.Apply

internal/hub/alert_engine.go:Apply(alerts, sig, cfg) (L180). Pure decision core, unit-tested without a database. Order of checks: a clearing signal resolves the matching open alert or no-ops; a matching open alert absorbs a same/lower-severity signal as a child line ("extended") or escalates on higher severity (resetting push-cycle bookkeeping, ARGUS-604); a resolved alert re-firing within cfg.Debounce (75s) reopens the same row instead of creating a new one; more than 10 distinct dedup keys within 5 minutes fold everything into one synthetic critical "storm" alert (absorbIntoStorm); otherwise a fresh alert is created.

AlertStore.Ingest

internal/hub/alert_store.go:Ingest (L63). Begins a transaction, loads the bounded working set, calls Apply, upserts the result, commits, and only after commit calls routePush (L108) -> PushNotifier.RouteAlert. The push call is nil-safe: if no pusher was wired (e.g. in tests), delivery is silently skipped rather than erroring the ingest.

PushNotifier.RouteAlert

internal/hub/notification_router.go:RouteAlert (L51). Branches on a.Severity:

  • routeCritical (L75) -- bypasses quiet hours entirely, re-notifies at +15m and +2h, then becomes badge-only after a hard cap (criticalRenotifyCap); always pushes on resolve, even if never acked.
  • routeWarning (L105) -- a one-push-ever budget; if inside a configured quiet window, held server-side via MarkQuietHeld and flushed later by SweepQuietFlush.
  • routeInfo (L126) -- always delivered live, no suppression.

fanOut / Pusher.Push

internal/hub/push_notifier.go:fanOut (L284) -> the Pusher interface, implemented in production by internal/hub/apns.go (token-auth ES256 JWT over HTTP/2 to Apple APNs). A 410 Unregistered (ErrDeviceGone) deletes the device row outright; any other failure is logged and recorded as a fail row in the runs ledger -- there is no explicit retry loop here, only the next real-world signal or a scheduled sweep.

Human resolve (ack / dismiss)

internal/hub/alerts_api.go, wired at internal/hub/server.go:790-791. POST /alerts/{id}/ack marks acked (the app's convention: opening the detail view acks it); it returns ErrAlertResolved (conflict) if the alert already auto-resolved underneath the human. POST /alerts/{id}/dismiss marks resolved/dismissed. Read surface: GET /alerts.json (alerts_api.go:15).

A dual-purpose caveat worth keeping in mind

PushNotifier is genuinely dual-purpose: RouteAlert/ PushAlertNotification is the alert-specific path drawn above. PushUpdate, PushSkillResult, PushApproval, LiveActivityUpdate, and the chat-turn Tick fan-out are separate, non-alert notification kinds that share the same fanOut/APNs plumbing -- see notification/push fan-out for those.

Key tables

  • migrations/0126_alerts.sql -- alerts table, one open row per dedup key enforced by a partial unique index.
  • migrations/0127_alert_push.sql -- push_stage/push_cycle_started_at/ quiet_held columns, plus the parallel approval_pushes table.
  • migrations/0128_alerts_deep_link.sql -- deep_link column.
  • docs/explanation/REQUIREMENTS.md US-11 (taxonomy/lifecycle/quiet-hours) and US-36 (this native pipeline, explicitly superseding the Grafana relay as source while keeping US-11's lifecycle) corroborate this diagram.