Skip to content

Alert store wiring (ARGUS-1246)

What broke

cmd/hub/main.go constructed three separate *hub.AlertStore instances against the same Postgres pool:

  1. alertStore at the top of main() -- the instance every native detector (watchdog no-shows, watchdog-blind, landing queue CI-red/rebase-failed, deploy completion/failure, capture-heartbeat staleness, noise-rate and backlog trackers) calls .Ingest() on.
  2. A throwaway instance minted inline for PushNotifier.WireAlertPush's push-stage bookkeeping (the critical-renotify and quiet-flush sweeps).
  3. NewServer's own internal instance, backing /alerts.json, /alerts/*, and the ARGUS-740 detector test-fire door.

Only instance 3 ever received SetPusher/SetEvents. Instance 1 -- the one every detector actually feeds -- never did, so detector alerts persisted to the alerts table (push_stage=0) but never reached APNs and never published an SSE kind=alert event. Because the test-fire door exercises instance 3, testing the alert path reported success while the real detector path went nowhere. Production evidence (2026-08-05): 20 critical and 180 warning rows at push_stage=0, versus 3 critical and 9 warning that pushed -- all the latter Grafana-relayed, entering through instance 3's /runs wiring, not native detectors.

The fix

One AlertStore now serves the whole hub. main() constructs it, wires SetEvents unconditionally (the SSE broker always exists) and SetPusher whenever APNs is configured, and hands that exact instance to NewServer via the new hub.WithAlertStore option instead of letting NewServer mint its own. The push-sweep bookkeeping site (former instance 2) now takes the same variable instead of minting a third. AlertStore.VerifyWiring is a startup self-check main() calls right after wiring: it logs loudly (slog.Error) if the SSE broker or (when APNs is configured) the pusher somehow ends up unset. See TestLiveProofDetectorAlertReachesPushAndSSE (internal/hub/argus1246_alert_wiring_test.go) for the live proof: it raises a real detector signal (SignalWatchdogBlind, not the test-fire door) and asserts both a push attempt and an SSE event.

Backfill decision: dropped, not replayed

The ~200 historical push_stage=0 rows (production, as of the 2026-08-05 audit) are intentionally not replayed. Reasoning:

  • They are historical detection events, not current state. Replaying them now -- days to weeks after the fact for some -- would page Aaron about conditions that, for the resolved ones, no longer exist; a late push for an already-resolved alert is misleading, not useful.
  • Any condition that is STILL actually firing today (e.g. a heartbeat still stale) self-heals through this same fix: native detectors re-evaluate and re-Ingest on their normal sweep cadence, and the now-wired store pushes and emits SSE on the very next cycle without any special backfill code.
  • The build box this fix was authored on carries no production database credentials (established policy, ARGUS-1215), so a live backfill/replay pass is not something this change could execute even if it were the right call.

Net: nothing further will push for the 200 historical rows. They remain visible in /alerts.json//ui/inbox exactly as before (this fix does not touch read paths), just without a retroactive push. This is a deliberate, recorded decision, not a silent gap.