Skip to content

Tasks Module (M3 run-reporting + watchdog) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Every scheduled task run reports to the hub (enforced, not opt-in), and a task that was due but silent raises a visible no-show — closing spec F3/F4 and audit finding observability-ops-2.

Architecture: The hub gains a token-gated POST /runs reporting API and a schedule-aware watchdog that parses tasks/config.ini (the existing single source of when) to know what to expect. Enforcement lives in a tasks/run.py wrapper that Ofelia invokes instead of the task directly: it times the task, captures exit + output tail, and reports — and because a failed report leaves no run row, the watchdog converts "couldn't report" into a no-show alert automatically. No new services, no new config files, one tiny migration.

Tech Stack: Go (hub: chi, robfig/cron/v3 for schedule parsing), Python stdlib only (wrapper), existing Postgres runs table.

Explicitly out of scope (later milestones): Telegram push (F5), llm/agent task handlers + taskkit (F2/F12), per-task deliver channels, dashboard paging/UI rework. This plan makes the data trustworthy; the minimal UI then renders it.

Design decisions (review these first)

  1. Enforcement point = the wrapper in the Ofelia command line. command = python /tasks/run.py heartbeat -- python /tasks/heartbeat.py. The runner-wrapper is the only thing config.ini is allowed to invoke (reviewable convention; a stanza that bypasses it is visible in diff review). Baseline report: task name, start/end, exit status, last 2000 chars of combined output.
  2. A failed report becomes a no-show, not a silent gap. The wrapper exits non-zero if the hub POST fails (Ofelia logs it), and since no run row landed, the watchdog flags the task as due-but-absent. Reporting failure is indistinguishable from not running — which is exactly the honest signal.
  3. config.ini is the expectation source (spec section 5). The hub parses the same file Ofelia reads — mounted read-only into the hub container — so schedule and expectation can never drift apart. Supported schedule forms: @every <duration> and 5/6-field cron (what robfig/cron parses; Ofelia uses the same library family).
  4. Due rule (deterministic, testable): a task is a no-show when now > last_run_start + 2*interval + 5m (for @every), or when the most recent cron fire time strictly before now - 5m grace has no run row at-or-after it. A task with NO runs ever is a no-show once the hub has been up longer than one full interval (prevents boot-storm false alarms).
  5. Auth reuses ARGUS_CAPTURE_TOKEN. One shared bearer secret for the whole stack (single-operator; Tailscale-only). POST /runs is gated; GET endpoints stay open like the dashboard.
  6. Watchdog proves its own liveness by recording a watchdog run row each sweep — the dashboard shows the watcher being watched, and Gatus + heartbeat already cover the hub process itself.
  7. Status vocabulary grows by one: ok | fail + new noshow rows written by the watchdog itself (one row per detected gap, deduped per task per gap). Alerts are rows, so they appear on the dashboard and in psql with zero new machinery — Telegram forwarding later just reads them.
  8. Skipped ticks stay invisible for now. Ofelia no-overlap skips don't report (Ofelia doesn't exec the command at all); the no-show grace (2x interval) absorbs them. Revisit only if real noise appears.

File structure

  • migrations/0018_runs_noshow.sql — allow noshow status, add runs (task, status) partial index for the watchdog's dedup query
  • internal/hub/config.go (new) — parse config.ini → []Expectation{Name, Schedule}; pure, no I/O beyond the file read
  • internal/hub/config_test.go (new)
  • internal/hub/watchdog.go (new) — due computation + Sweep() (detect → record noshow rows) + the ticker loop
  • internal/hub/watchdog_test.go (new)
  • internal/hub/runs.go (modify) — accept noshow status; LastRunBefore/RanSince helpers; delete the M0 NoShows (superseded)
  • internal/hub/server.go (modify) — POST /runs (bearer-gated), GET /runs.json, GET /noshows.json; dashboard gains a no-show banner row
  • internal/hub/server_test.go (modify) — handler matrix for the new routes
  • cmd/hub/main.go (modify) — env (ARGUS_CAPTURE_TOKEN, ARGUS_TASKS_CONFIG), start watchdog goroutine
  • tasks/run.py (new) — the enforced wrapper (stdlib only: subprocess, urllib, json)
  • tasks/config.ini (modify) — both stanzas adopt the wrapper
  • compose.yaml (modify) — hub: token env + ./tasks/config.ini:/etc/argus/tasks.ini:ro; task-runner: ARGUS_HUB_URL=http://hub:8080

Task 1: Migration — noshow status + watchdog index

Files: - Create: migrations/0018_runs_noshow.sql - Test: internal/shared/db/db_test.go (extend TestMigrate)

  • [ ] Step 1: Write the failing test — extend TestMigrate to insert a noshow run row and expect success, plus assert the index exists:
// runs accepts the watchdog's noshow status (M3).
if _, err := pool.Exec(ctx, `
    INSERT INTO runs (task, status, started_at) VALUES ('wd-test', 'noshow', now())`); err != nil {
    t.Errorf("noshow run row rejected: %v", err)
}
  • [ ] Step 2: Run it — go test ./internal/shared/db/ — expect FAIL (CHECK constraint or current enum rejects, or passes if no constraint exists: then the test pins the contract and Step 3 only adds the index)
  • [ ] Step 3: Write the migration
-- M3 watchdog: a detected no-show is recorded as a runs row (status
-- 'noshow') so alerts are queryable/dashboard-visible with no new
-- machinery. (0001 created status as unconstrained TEXT; the Go layer
-- validates. This migration documents the vocabulary and adds the
-- watchdog's dedup index.)
CREATE INDEX runs_task_status_idx ON runs (task, status, started_at DESC);
  • [ ] Step 4: Run test → PASS; Step 5: Commit — db: runs noshow vocabulary + watchdog index (M3)

Task 2: config.ini parser

Files: - Create: internal/hub/config.go, internal/hub/config_test.go

  • [ ] Step 1: Failing test — parse a fixture string with two [job-exec "name"] stanzas (one @every 5m, one cron 0 3 * * *), assert []Expectation{{Name,"heartbeat",Schedule:"@every 5m"},...}; unknown sections ignored; a stanza missing schedule is an error (an unschedulable expectation is a config bug, not a skip)
  • [ ] Step 2: Implement ParseTasksConfig(r io.Reader) ([]Expectation, error) — line-based INI scan (section regex \[job-exec "(.+)"\], schedule = key; comments ;/#); no third-party INI dep
  • [ ] Step 3: PASS; commit — hub: parse tasks/config.ini into watchdog expectations (spec F4)

Task 3: due computation + sweep

Files: - Create: internal/hub/watchdog.go, internal/hub/watchdog_test.go - Modify: internal/hub/runs.go (helpers), delete NoShows

  • [ ] Step 1: Failing tests (pure function first):
// Due(now, schedule, lastRun, hubStart) (bool, error)
// @every 5m, last run 7m ago  -> not due (within 2x+grace)
// @every 5m, last run 16m ago -> DUE
// cron "0 3 * * *", last run yesterday 03:01, now 03:10+grace -> not due
// cron "0 3 * * *", no run since yesterday 03:00, now today 03:06 -> DUE
// never ran, hub up 2m, @every 5m  -> not due (boot grace)
// never ran, hub up 20m, @every 5m -> DUE
  • [ ] Step 2: Implement Due with robfig/cron/v3 (cron.ParseStandard for cron lines; time.ParseDuration for @every); cron previous-fire computed by stepping Next from a lookback window
  • [ ] Step 3: DB test for Sweep — seed runs, run Sweep(ctx, expectations, now): writes ONE noshow row per newly-missed task (dedup: no second row while the latest noshow for that task is newer than the last real run), records its own watchdog ok row, returns SweepReport{Checked, NoShows, Recorded} — never silent
  • [ ] Step 4: Implement Sweep + helpers; PASS; commit — hub: schedule-aware no-show watchdog (spec F4, audit observability-ops-2)

Task 4: hub HTTP — POST /runs + JSON reads

Files: - Modify: internal/hub/server.go, internal/hub/server_test.go, internal/hub/runs.go

  • [ ] Step 1: Failing handler-matrix test (same style as memory's TestServerRouteMatrix):
  • POST /runs without bearer → 401; bad JSON → 400; missing task/status/started_at → 400; valid → 201 {id}
  • GET /runs.json?limit=N → 200 list (newest first); bad limit → 400
  • GET /noshows.json → 200 (current noshow rows newer than each task's last real run)
  • dashboard / still 200 and shows a NO-SHOW banner row when one exists
  • [ ] Step 2: Implement — NewServer(runs *Runs, check func() error, opts ...Option) gains WithReportToken(string); reuse memory's requireBearer shape; Run JSON decode with DisallowUnknownFields
  • [ ] Step 3: PASS; commit — hub: run-reporting API + JSON endpoints (spec F3)

Task 5: wire main.go

Files: - Modify: cmd/hub/main.go

  • [ ] Step 1: read ARGUS_CAPTURE_TOKEN (warn-if-empty like memory), ARGUS_TASKS_CONFIG (default /etc/argus/tasks.ini; missing file = loud warn, watchdog disabled — the hub must still serve)
  • [ ] Step 2: start watchdog goroutine: parse config at boot, Sweep every 5m via time.Ticker, log every report; re-parse the file each sweep (config changes land on redeploy, but re-reading is free and removes a restart dependency)
  • [ ] Step 3: build + existing tests PASS; commit — hub: wire reporting token + watchdog loop

Task 6: tasks/run.py — the enforced wrapper

Files: - Create: tasks/run.py - Test: tasks/test_run.py (pure-python; runs in CI via the existing Go-test job? NO — add a python3 -m unittest line to ci.yml test steps)

  • [ ] Step 1: Failing test — fake hub via http.server in-process: wrapper runs -- python -c 'print("hi")', asserts POST body {task, status:"ok", started_at, finished_at, detail} with detail containing "hi"; failing child (exit 3) posts status:"fail" and exits 3 (Ofelia sees the real failure); unreachable hub → wrapper exits non-zero with a loud stderr line
  • [ ] Step 2: Implement (stdlib only):
#!/usr/bin/env python3
"""run.py NAME -- CMD... : the enforced baseline reporter (spec F3).
Wraps every Ofelia task: times it, captures exit + output tail, POSTs the
run to the hub. Fail-loud: if the hub is unreachable the wrapper exits
non-zero and no run row lands -- the watchdog then flags the task, so a
broken reporting path surfaces as a no-show instead of a silent gap."""
import json, os, subprocess, sys, time, urllib.request, datetime

def main() -> int:
    name = sys.argv[1]; assert sys.argv[2] == "--"
    cmd = sys.argv[3:]
    hub = os.environ["ARGUS_HUB_URL"].rstrip("/")
    token = os.environ.get("ARGUS_CAPTURE_TOKEN", "")
    started = datetime.datetime.now(datetime.timezone.utc)
    proc = subprocess.run(cmd, capture_output=True, text=True)
    finished = datetime.datetime.now(datetime.timezone.utc)
    tail = (proc.stdout + proc.stderr)[-2000:]
    body = json.dumps({
        "task": name,
        "status": "ok" if proc.returncode == 0 else "fail",
        "started_at": started.isoformat(),
        "finished_at": finished.isoformat(),
        "detail": f"exit={proc.returncode} {tail}".strip(),
    }).encode()
    req = urllib.request.Request(hub + "/runs", data=body, method="POST",
        headers={"Content-Type": "application/json",
                 **({"Authorization": "Bearer " + token} if token else {})})
    with urllib.request.urlopen(req, timeout=10) as resp:
        if resp.status != 201:
            raise RuntimeError(f"hub answered {resp.status}")
    sys.stdout.write(proc.stdout); sys.stderr.write(proc.stderr)
    return proc.returncode

if __name__ == "__main__":
    try:
        sys.exit(main())
    except Exception as e:  # report failure loudly; no row -> watchdog flags it
        print(f"run.py: REPORTING FAILED: {e}", file=sys.stderr)
        sys.exit(70)
  • [ ] Step 3: PASS; commit — tasks: run.py enforced baseline reporter (spec F3)

Task 7: adopt the wrapper + compose wiring

Files: - Modify: tasks/config.ini, compose.yaml, .env.example

  • [ ] Step 1: config.ini commands become python /tasks/run.py heartbeat -- python /tasks/heartbeat.py (same for distill)
  • [ ] Step 2: compose: hub gets ARGUS_CAPTURE_TOKEN + ./tasks/config.ini:/etc/argus/tasks.ini:ro volume; task-runner gets ARGUS_HUB_URL: http://hub:8080 (token already present)
  • [ ] Step 3: bash harness/init.sh smoke PASS; commit — tasks: every Ofelia job reports through run.py; hub reads expectations

Task 8: end-to-end verification + docs

  • [x] Step 1: deployed locally; wrapped Ofelia ticks report: distill ok 00:47:36 / 00:49:36, heartbeat ok 00:50:36, detail carries exit + output tail
  • [x] Step 2: kill test — happened ORGANICALLY, better than staged: the first deploy left Ofelia on the old unwrapped commands (compose does not recreate on bind-mounted file content changes, and Ofelia caches its INI at startup). Tasks ran but never reported, and the watchdog flagged BOTH as no-shows within one boot-grace interval ("last real run: never"), deduped the persisting gap across 3 sweeps (exactly 1 noshow row per task), showed the dashboard banner, and cleared everything ({"noshows":[]}, banner gone) once the force-recreated Ofelia started reporting. The watchdog's first catch was a real misdeployment — exactly the failure class it exists for. OPERATIONAL NOTE: config.ini changes need docker compose up -d --force-recreate ofelia (the hub re-reads per sweep; Ofelia does not).
  • [x] Step 3: roadmap updated in the closing commit

Self-review notes

  • Spec F3 coverage: baseline report enforced by wrapper (name/start/end/exit/output) — yes; cost + deliver are taskkit opt-ins, explicitly deferred.
  • F4: schedule-aware due rule from the same file Ofelia reads — yes; never-ran case covered; boot grace covered.
  • observability-ops-2 (Ofelia failure visibility): wrapper posts fail rows with exit + stderr tail — closed.
  • No placeholder steps; the wrapper code above is the real implementation modulo review feedback.
  • Risk: Ofelia job-exec exec environment must have the env vars (task-runner container env, not Ofelia's) — verified: job-exec runs inside argus-task-runner, which carries the env from compose.