Observability

End-to-end overview: for how CloudWatch + Sentry + Better Stack compose into one paging surface — who gets paged on what severity, cooldowns, and the bypass path — read ../architecture/alert-stack.md first. This runbook is the per-component reference (every alarm name, threshold, and PromQL query).

BullMQ Queue Metrics

The API exposes BullMQ queue metrics on GET /metrics in Prometheus text format. Metrics are collected only when REDIS_URL is set.

Metrics:

Metric	Labels	Meaning
bullmq_queue_jobs	queue, state	Job count by BullMQ queue and state. States are waiting, active, delayed, completed, failed, and paused.
bullmq_queue_oldest_job_age_seconds	queue	Age in seconds for the oldest waiting job. Empty queues report 0.
bullmq_queue_depth_total	queue	Queue backlog depth, calculated as waiting + active + delayed.

Label conventions:

• queue is the literal BullMQ queue name from the API module registration.
• state is one of the six BullMQ states listed above.
• Queue metrics are service-level operational signals and do not include org ids or payload data.

Alert thresholds:

Alert	Expression	Duration	Severity
HumanworkBullMQQueueDepthWarning	bullmq_queue_depth_total > 100	5m	warning
HumanworkBullMQQueueDepthCritical	bullmq_queue_depth_total > 1000	2m	critical

Runbook: docs/runbooks/bullmq-queue-backlog.md.

Slack Layer 1 Hard-Trigger Routing (#2208)

The API exposes Prometheus counters on GET /metrics for Slack's Layer 1 hard-trigger routing (DM / @mention / thread continuation — the unconditional routes shipped in #2179). This telemetry is observability only; it does not change routing semantics. Emitted from api/src/channels/slack/slack.metrics.ts, incremented at the routing decisions in SlackService.handleRuntimeEvent and the dedup gate in SlackController.slackEvents.

Metric	Labels	Meaning
slack_hard_trigger_total	subtype	A hard trigger was routed. subtype ∈ dm, mention, thread_continuation.
slack_dedup_drop_total	source	An inbound event was dropped by the dedup gate. source=retry = Slack redelivery (X-Slack-Retry-Num > 0); source=event = duplicate event identity arriving by another path (LB retry, double-enqueue).
slack_engagement_failure_total	op, fallback	A slack_engaged_threads lookup/upsert failed. op ∈ lookup, mark, refresh. fallback=mention_only means the failure degraded routing to the mention-gate (a continuation may have been missed); fallback=none means the message still routed this turn.

Label conventions: service-level operational signals only — no org ids, channel ids, or message content (kept low-cardinality and PII-free, matching the BullMQ metrics convention).

Operator queries (PromQL)

Confirm hard triggers are being routed (and not silently dropped) — each subtype should be non-zero on a workspace with live traffic:

sum by (subtype) (rate(slack_hard_trigger_total[5m]))

Dedup drop rate, split by source (a retry spike means we're 5xx-ing or timing out and Slack is redelivering; an event spike points at LB/queue double-delivery):

sum by (source) (rate(slack_dedup_drop_total[5m]))

Engagement-store failure rate — the mention_only series is the one that silently degrades routing, so alert on it specifically:

sum by (op, fallback) (rate(slack_engagement_failure_total[5m]))

Alert thresholds

Alert	Expression	Duration	Severity
HumanworkSlackHardTriggersStalled	sum(rate(slack_hard_trigger_total[15m])) == 0 (only meaningful on a workspace with known steady traffic)	15m	warning
HumanworkSlackEngagementDegraded	sum(rate(slack_engagement_failure_total{fallback="mention_only"}[5m])) > 0.1	10m	warning
HumanworkSlackDedupRetryStorm	sum(rate(slack_dedup_drop_total{source="retry"}[5m])) > 1	5m	warning

A sustained slack_engagement_failure_total{fallback="mention_only"} rate means clients' thread follow-ups are being missed unless they @mention — cross-reference Redis/Postgres health (docs/runbooks/railway-outage.md) before assuming a code regression.

Sentry

Sentry is wired into the NestJS API, the FastAPI agent, and the Next.js frontend (#845 closes #828). Each surface has its own DSN — see docs/ops/railway-env-vars.md § Sentry.

Projects:

• humanwork-api — NestJS unhandled exceptions, BullMQ worker failures, Postgres connection errors. Alert hook on Redis disconnect from ThrottlerModule (#839).
• humanwork-agent — FastAPI exceptions (agent dispatch errors, OpenAI rate-limit fan-out, tool-call exceptions).
• humanwork-frontend — browser exceptions; release tags are wired to the Vercel deployment SHA.

Triage convention: every Sentry alert in humanwork-api and humanwork-agent carries org_id and osa_id tags (set via Sentry scope in OrgRlsInterceptor and the agent's request middleware). When triaging, never paste customer PII from a Sentry event into Slack or GitHub — link to the Sentry event instead.

Leader-Status (Multi-Pod)

The API supports multi-pod deployment (#865, #1007). Per-pod leader election is Redis-backed (leader-status key) and gates scheduled jobs (cron sweeps, Haystack reconciliation, billing rollup). Only the elected leader runs scheduled work; non-leaders run HTTP + queue workers only.

• GET /ops/leader-status (internal-only, behind SuperAdmin) reports the current leader pod ID, last-heartbeat timestamp, and TTL.
• Sentry breadcrumbs include leader=true|false so cron-only failures can be filtered to the leader pod.
• If leader-status shows no leader for > 60s, scheduled jobs are not running — page on-call.

Throttler Shadow Mode

ThrottlerModule (#812, #839) runs in shadow mode in production (see docs/ops/railway-env-vars.md § Throttler). Triage signals:

• Sentry event throttler.shadow_block — counts requests that would have been blocked. A sustained spike from a single org_id or ip is an abuse signal — page Platform Lead before flipping THROTTLER_SHADOW_MODE=false.
• Sentry alert throttler.redis_disconnect — Redis lost during throttler evaluation; throttler degrades to allow-all but emits a breadcrumb. Cross-reference with docs/runbooks/railway-outage.md § Redis.

External Uptime Probes (Better Stack)

Sentry catches errors that reach the app. It does not catch the case where the container is wedged, the ALB is broken, or AWS itself is unreachable. That gap is closed by an external synthetic probe.

Provisioned by scripts/provision-uptime.sh — idempotent, looks up monitors by pronounceable_name so re-runs PATCH rather than duplicate.

Monitors:

Surface	Endpoint	Interval	Regions	Alerts on
humanwork-prod-api	https://api.h.work/health	30s	us, eu, as	2 consecutive failures
humanwork-prod-agent	https://agent.h.work/health	30s	us, eu, as	2 consecutive failures
humanwork-prod-frontend	https://app.h.work/	60s	us, eu	2 consecutive failures
humanwork-staging-*	*-staging.h.work	same	same	same

The agent and API /health endpoints intentionally return a static {status: "ok"} without DB / Redis reach-through so a probe failure means the process is gone, not a downstream dependency. Dependency-aware probes belong in /health/deep (future).

Delivery: Better Stack → Slack #ops-alerts + on-call SMS via the team rotation.

One-time operator setup (NOT in the provisioner):

# 1) Get a team API token: https://uptime.betterstack.com/team/<team>/api-tokens
export BETTERSTACK_API_TOKEN=...
export BETTERSTACK_ENV=prod    # or staging
bash scripts/provision-uptime.sh

# 2) UI: Integrations → Slack → paste #ops-alerts incoming-webhook URL.
# 3) UI: On-call → New rotation. Members: Paul, Emanuele, AlexD. Weekly Mon 09:00 UTC.

AWS-Side Alerts (CloudWatch + SNS → Slack)

Provisioned by infra/terraform/modules/monitoring, wired into the stack module so it ships with every env (prod, staging).

Alerts published to a single SNS topic (humanwork-<env>-alerts) which fans out to a Slack subscriber Lambda. The webhook URL lives in Secrets Manager; set the ARN via alerts_slack_webhook_secret_arn in the env's terraform.tfvars. Empty ARN = SNS topic exists but no Slack delivery (the env can wire a different protocol later without alarm churn).

Resource alarms

Alarm	Threshold	Period
humanwork-<env>-api-cpu-high	CPU > 85%	10m
humanwork-<env>-api-memory-high	Memory > 90%	5m
humanwork-<env>-agent-cpu-high	CPU > 85%	10m
humanwork-<env>-agent-memory-high	Memory > 90%	5m
humanwork-<env>-api-running-below-desired	RunningCount < api_desired_count	10m
humanwork-<env>-agent-running-below-desired	RunningCount < agent_desired_count	10m
humanwork-<env>-alb-5xx-rate-critical	5xx / requests > 1%	5m

Log-metric-filter alarms

These rely on specific log lines being emitted by the services. Renaming the log event name without a paired Terraform change breaks the alarm.

Alarm	Log pattern	Emit site	Threshold
humanwork-<env>-specialist-mismatch-spike	specialist_mismatch	api/src/runtime-control-plane/runner-control-plane.service.ts:478	> 5/min over 5m
humanwork-<env>-agent-backend-failed-spike	agent_backend_request_failed	agent/audit_client.py (every failure path)	> 10/min over 5m

EventBridge-routed alarms

Some ECS lifecycle signals never reach CloudWatch Logs because the kernel sends SIGKILL and the process has no opportunity to flush stdout/stderr. The awslogs driver only forwards what the container writes, so we hook the ECS control-plane events instead.

Rule	Event source	Filter	Notes
humanwork-<env>-oom-killed	aws.ecs / ECS Task State Change	stopCode=OutOfMemory, clusterArn=<this env's cluster>	Routes through the same SNS topic; Slack subscriber receives a CloudWatch-alarm-shaped payload via EventBridge input transformer.

The SNS topic policy grants events.amazonaws.com sns:Publish scoped to this specific rule's ARN (not the service globally), so a different EventBridge rule can't reuse the policy to fan messages into the alert topic.

The specialist_mismatch event is the hottest signal in the runtime — a sustained spike means a Specialist manifest or claim is being mis-routed and the rejection is happening at claim time. Triage is not automatic; an Expert may be intentionally testing an unbound runtime. Page Platform Lead before disabling the alarm.

The agent_backend_request_failed event is emitted by audit_client.py on every transport / timeout / non-2xx response from the platform API. The event name is pinned by a regression test (agent/evals/test_audit_client.py::test_audit_client_emits_backend_failed_on_5xx) so a future refactor surfaces in CI.

Deploy Gates (Sentry Release Health + ECS Health Probe)

The reusable deploy workflow (.github/workflows/reusable-ecs-deploy.yml) runs five gates in order:

1. Build + push image to ECR.
2. Sentry release — create (best-effort; non-blocking on Sentry outage). Tags the release with ${{ github.sha }} and the deploy environment.
3. ECS service update + stabilization with circuit-breaker rollback detection.
4. Health probe — health-probe.sh against the public health URL, 3-minute timeout, 18 attempts at 10s intervals.
5. Sentry release — finalize ONLY on success. Rollbacks leave the release unfinalized so they don't pollute the regression baseline.

The Sentry steps are gated on secrets.SENTRY_AUTH_TOKEN + inputs.sentry_project being supplied — env without Sentry secrets simply skips the steps. Wire the org-level secrets SENTRY_AUTH_TOKEN and SENTRY_ORG once in GitHub repo settings to enable the gate.

Zero-Downtime Deploy Posture

Service	Prod desired	Staging desired	ZD on prod?
api	2	1	✓ (rolling, minimum_healthy_percent=100, maximum_percent=200)
agent	2	1	✓
frontend	n/a (Vercel)	n/a	✓ (edge)

Terraform's aws_ecs_service is wired with deployment_minimum_healthy_percent = 100 and deployment_maximum_percent = 200, so a deploy provisions new tasks before draining old ones. The aws ecs wait services-stable step plus the circuit-breaker rollback detection in reusable-ecs-deploy.yml catches any deploy that fails to take hold.

Staging runs single-replica by design — there's a brief drain-cutover window on every deploy. That's acceptable for staging; do not lower prod below 2 without a paired ADR.

Health Probe Script

agent/scripts/health-probe.sh is a zero-dep curl-based probe invoked by the GitHub-Actions deploy gate (.github/workflows/reusable-ecs-deploy.yml) AND used by operators for manual verification after a Terraform / task-def change. Validates HTTP 200 plus the response body's status field matches ok (agent /health) or ready (api /ready). Exit codes: 0 ok, 2 HTTP/network error, 3 body status mismatch.

MVP Cost Floor

Surface	Tool	Plan
App errors	Sentry	Free (5k events/mo) or team plan if release health needed
Synthetic uptime	Better Stack	Free (3-min interval) → paid (~$25/mo) for 30s + on-call
Log alarms	CloudWatch	Pay-per-alarm (~$0.10 each), included for the AWS-native surfaces above
Resource alarms	CloudWatch	Same
SNS → Lambda → Slack	AWS native	~$0/mo at humanwork volume

All AWS alerts converge on the SNS topic and from there a single Slack channel. No PagerDuty, no Datadog — those are post-MVP optimizations and should not be wired before product-market fit.