Alert stack — end-to-end overview

Scope. How the MVP alert stack (#1307) composes CloudWatch, Sentry, and Better Stack into one paging surface. This is the map; the per-component reference (every alarm name, threshold, PromQL query, and the Better Stack provisioner) lives in ../runbooks/observability.md. Read this first to understand who gets paged on what and why; drop into the runbook for the exact knobs.

The MVP stack is deliberately three independent tools, not one platform. Each watches a different failure class from a different vantage point, and no tool depends on another being up — a Better Stack outage doesn't blind CloudWatch, and a Slack outage doesn't stop Sentry from recording. No PagerDuty, no Datadog: those are post-PMF optimizations and should not be wired before product-market fit (see ../runbooks/observability.md § MVP Cost Floor).

The three vantage points

System	Watches from	Catches	Source of truth
CloudWatch	inside AWS	ECS task health (CPU / memory / running count), ALB 5xx surge, OOMKills, structured-log spikes (specialist_mismatch, agent_backend_request_failed)	infra/terraform/modules/monitoring
Sentry	inside each process	unhandled exceptions + release-health regressions across api / agent / frontend	per-surface DSN — see ../runbooks/observability.md § Sentry
Better Stack	outside, from us / eu / as (frontend us / eu only)	synthetic uptime — is the public endpoint answering at all	scripts/provision-uptime.sh

The split matters: CloudWatch sees a healthy ECS task that is silently 500-ing only via the ALB 5xx alarm; Better Stack sees the same outage from the client's side regardless of what AWS thinks; Sentry tells you the stack trace. A real P0 typically lights up all three within a minute, from three directions.

End-to-end flow

                         ┌──────────────────────────────────────────┐
   in-cluster signals →  │ CloudWatch alarms + EventBridge (OOM)     │
   (CPU, mem, 5xx,       │  modules/monitoring/main.tf               │
    log-metric filters)  └───────────────┬──────────────────────────┘
                                         │ alarm_actions / ok_actions
                                         ▼
                          ┌─────────────────────────────────┐
                          │ SNS topic  humanwork-<env>-alerts│  ← one fan-in
                          └───────────────┬─────────────────┘
                                         │ lambda subscription
                                         ▼
                          ┌─────────────────────────────────┐
                          │ Slack subscriber Lambda          │  slack_subscriber.py
                          │  (webhook URL in Secrets Manager)│
                          └───────────────┬─────────────────┘
                                         ▼
   process exceptions →   Sentry ───────────────────────►  Slack  #ops-alerts
   (api/agent/frontend)        (own alert rules)             ▲
                                                             │
   public endpoint down → Better Stack ──────────────────────┘ + on-call SMS/push
   (us/eu/as probes)            (own integration + on-call rotation)

Three producers, one human destination (#ops-alerts), but three independent delivery paths. CloudWatch reaches Slack through SNS → Lambda; Sentry and Better Stack each carry their own Slack integration. Better Stack additionally drives the on-call rotation (SMS + push), which is the only leg that wakes a human who isn't watching the channel.

CloudWatch → SNS → Lambda → Slack (the AWS leg)

1. Every alarm in modules/monitoring publishes to a single SNS topic, humanwork-<env>-alerts, on both alarm_actions and ok_actions — so recovery (OK) posts to Slack too, not just the failure.
2. The OOMKill path is the exception that proves the rule: a SIGKILLed container writes nothing to stdout, so OOMKilled never reaches CloudWatch Logs. EventBridge catches the ECS Task State Change event (stopCode=OutOfMemory) instead and an input transformer reshapes it into a CloudWatch-alarm-shaped SNS payload, so the subscriber has a single code path.
3. The SNS topic fans out to one subscriber: an inline Python Lambda (slack_subscriber.py) that translates the alarm payload into Slack blocks (red for ALARM, green for OK, yellow for INSUFFICIENT_DATA) and POSTs to the incoming-webhook URL it reads from Secrets Manager. The handler never raises on a malformed payload — it logs and returns ok so SNS doesn't retry forever and spam the channel.

The SNS topic policy grants publish to exactly two principals — cloudwatch.amazonaws.com (scoped by SourceAccount) and events.amazonaws.com (scoped to the OOM rule's specific ARN) — so no other EventBridge rule can reuse the policy to inject messages into the alert topic.

Sentry (the in-process leg)

Sentry is wired into all three surfaces (humanwork-api, humanwork-agent, humanwork-frontend, #845) and routes to Slack through its own integration and alert rules — it does not pass through the SNS topic. api/agent events carry org_id + osa_id tags for triage. Release health gates the deploy pipeline (reusable-ecs-deploy.yml): a release is created before rollout and finalized only on success, so a rolled-back deploy leaves the release unfinalized and out of the regression baseline. Never paste customer PII from a Sentry event into Slack or GitHub — link to the event.

Better Stack (the synthetic / on-call leg)

Better Stack probes the public endpoints on a 30s interval — the api and agent /health checks from us / eu / as, and the frontend root from us / eu at 60s — provisioned idempotently by scripts/provision-uptime.sh. The api and agent /health endpoints return a static {status:"ok"} with no DB/Redis reach-through, so a probe failure means the process is gone, not a slow dependency. Better Stack carries its own Slack integration and the on-call rotation — it is the only producer that pages a human by SMS/push.

Who gets paged on what severity

Severity definitions and response times are owned by ../runbooks/incident-response.md § Severity Levels. The stack maps to them as follows:

Severity	Typical trigger	Which producer fires	Reaches a human via
P0 — all clients down / auth down	Better Stack: 2 consecutive probe failures from multiple regions; CloudWatch: api-running-below-desired, alb-5xx-rate-critical	Better Stack + CloudWatch	#ops-alerts and on-call SMS/push (Better Stack on-call rotation)
P1 — major feature broken for all orgs	CloudWatch: *-memory-high (OOM precursor), oom-killed, agent-backend-failed-spike; Sentry: exception spike	CloudWatch + Sentry	#ops-alerts; on-call only if Better Stack also trips
P2 — degraded / some orgs	CloudWatch: *-cpu-high, specialist-mismatch-spike; BullMQ depth warnings	CloudWatch + Sentry	#ops-alerts
P3 — minor / workaround exists	low-rate Sentry issues, single-region Better Stack blips	Sentry / Better Stack	#ops-alerts (no page)

The escalation boundary is Better Stack's on-call rotation, not Slack. Everything lands in #ops-alerts; only Better Stack's rotation (weekly, Mon 09:00 UTC) escalates to SMS/push for someone who isn't watching the channel. specialist_mismatch is the one CloudWatch signal that is explicitly not auto-triaged — an Expert may be intentionally testing an unbound runtime; page the Platform Lead before disabling the alarm.

Cooldowns & de-duplication

The stack relies on each tool's native de-dup rather than a shared throttle:

• CloudWatch notifies on state transition, not continuously. An alarm posts once when it crosses into ALARM and once when it returns to OK; it does not re-page every evaluation period while breaching. The "cooldown" is the alarm's evaluation_periods × period window before it transitions (e.g. memory-high = 1×5m, cpu-high = 2×5m, running-below-desired = 2×5m). A flapping metric is damped by requiring N consecutive breaching periods.
• Better Stack confirms an outage with a 60s confirmation_period (≈2 consecutive 30s probe failures) before alerting, then waits team_wait (180s) before escalating to the on-call rotation — so a 90-second blip pages the channel but not a person's phone.
• Sentry groups events into issues and applies its own per-rule rate limits; a fan-out (e.g. OpenAI rate-limit storm) collapses into one issue, not N alerts.

There is no cross-tool cooldown — a true P0 intentionally pages from all three paths at once. That redundancy is the design, not a bug.

Bypass / degradation path

The stack degrades safely and can be routed around without alarm churn:

• Slack delivery is optional and decoupled from alarm creation. The Slack subscriber Lambda is gated on alerts_slack_webhook_secret_arn being set (local.enable_slack). An empty ARN (e.g. a staging env with no channel) creates the SNS topic and every alarm but no Lambda — alarms still fire and record state, they just don't post to Slack. The env can wire a different protocol later (email, https, PagerDuty) by subscribing it to the existing topic, with zero alarm churn.
• Adding a new destination is a one-line SNS subscription against the sns_topic_arn output — the alarm side only knows the topic, never the subscriber, so swapping Slack for PagerDuty needs no alarm edits.
• Muting is per-tool: silence a CloudWatch alarm by detaching its alarm_actions, mute a Better Stack monitor in the UI, or snooze a Sentry issue. Because the paths are independent, muting one does not blind the others — which is exactly why you must page the Platform Lead before muting specialist-mismatch rather than silently disabling it.
• The webhook secret is operator-provisioned out-of-band (Secrets Manager), not in Terraform state. Rotating the Slack webhook is a secret update + Lambda cold start; no terraform apply required.

One-time operator setup (not in Terraform)

Three things live outside the provisioner and must be wired by hand once per env (detailed steps in ../runbooks/observability.md § AWS-Side Alerts and § Better Stack):

1. Slack webhook secret — paste the #ops-alerts incoming-webhook URL into the Secrets Manager secret referenced by alerts_slack_webhook_secret_arn.
2. Better Stack on-call rotation — On-call → New rotation, weekly Mon 09:00 UTC; this is the leg that drives SMS/push.
3. Better Stack ↔ Slack integration — Integrations → Slack, mapping the monitors to #ops-alerts.

Where to go next

You want…	Go to
Exact alarm names, thresholds, PromQL, env vars	../runbooks/observability.md
Severity definitions + first-response checklist	../runbooks/incident-response.md
The Terraform that ships all of this	infra/terraform/modules/monitoring
Better Stack monitor provisioning	scripts/provision-uptime.sh
In-app notification pipeline (separate surface)	../runbooks/notification-failures.md