Skip to main content

Staging NFR smoke runbook

Owner: backend infra Last updated: 2026-05-27 (#810) Script: scripts/test-nfr-smoke.sh Nightly CI: .github/workflows/nightly-nfr-smoke.yml

TL;DR​

scripts/test-nfr-smoke.sh runs 12 active-active NFR acceptance checks against the deployed environment. Some items are pure HTTP probes the script handles automatically; some need manual / infra steps that this runbook documents in detail.

Run mode:

STAGING_BASE_URL=https://api.h853.work \
STAGING_FRONTEND_URL=https://app.h853.work \
STAGING_TOKEN_ORG_A=eyJ...  STAGING_TOKEN_ORG_B=eyJ... \
bash scripts/test-nfr-smoke.sh

Exit code 0 = no failures. Skips are expected for items that need infra ops.

What each item is checking​

Item 1 — /health returns 200 on all services​

Sanity probe. If this fails, nothing else matters — fix this first.

SymptomCauseAction
502 / 504ECS task crash-loopingcheck CloudWatch task logs for startup-invariant failure
200 from /health but 503 from /readydownstream (PG / agent / Redis) degradedfollow per-component runbook

Item 2 — Sentry alert path → Slack #engineering​

The script hits /admin/sentry-test which deliberately throws. Verify in Slack that the event landed in #engineering within ~30 seconds. If it didn't:

  • check SENTRY_DSN env var on the running task
  • check Sentry project's Slack integration config
  • check sentry.config.ts beforeSend isn't accidentally dropping events

Item 3 — Cross-org IDOR​

Org A's token tries to read org B's conversation. Expected 403/404. A 200 here is an active-incident severity — file an immediate Sentry alert and pull staging from public DNS until patched.

Item 4 — Inbound webhook dedup​

Posts the same Slack event_id twice. Both calls should return 200 (Slack expects 200), but only ONE conversation/message row should land in the DB. Verify in DB:

SELECT count(*) FROM messages WHERE metadata->>'eventId' = '<event_id_from_log>';
-- expected: 1

Item 5 — Outbound failure → DLQ retry (manual)​

The script can't inject a vendor outage from outside. Manual procedure:

  1. In staging, temporarily set RESEND_API_KEY=invalid-key on the API task definition.
  2. Force a deploy of that task.
  3. Send an expert reply through the workspace UI.
  4. Verify in CloudWatch: dispatchEmail.conv=<id>] attempt 3/3 failed + DLQ push entry.
  5. Restore the real RESEND_API_KEY and verify the DLQ retry worker re-sends within 5 min.
  6. Confirm the client received exactly one email (no duplicates).

Prometheus check:

channel_failures_pending
channel_failures_total
channels_outbound_jobs_total
channels_outbound_dispatch_duration_ms_count

Expected during this smoke: DLQ queued/retry/dead/drained counters move as rows transition, pending gauges refresh on the 5 min retry tick, and channels-outbound job success/retry/dead counters are present in /metrics.

Item 6 — Socket.io 2-pod 50-socket 5-min soak (manual)​

Cross-pod fan-out test. Manual procedure:

  1. SSH to two staging pods.
  2. Run the soak helper (scripts/load/socketio-fanout-test.mjs — see also docs/runbooks/local-multi-pod.md): 
    node scripts/load/socketio-fanout-test.mjs \
      --url wss://api.h853.work/notifications \
      --tokens <50 staging tokens> \
      --duration 300
  3. From a third terminal, emit broadcast events via the API. Each event must be received by all 50 sockets per pod = 100 receives per event.
  4. Acceptance: 5 min, zero gaps in the receive log (use the helper's --strict mode).

Item 7 — RDS Multi-AZ failover (infra, quarterly)​

Owned by @ethumanity / @grazianognoll. Procedure:

  1. Open AWS Console → RDS → humanwork-staging → Actions → "Reboot with failover".
  2. Watch the ALB target group health — the API task health checks should fail briefly then recover within ~30s as Postgres comes back on the standby.
  3. Confirm /ready recovers and no client requests returned 5xx for > 60s.

This runs quarterly per ops calendar, not nightly.

Item 8 — Production startup invariant​

The script probes /ready and fails if the body contains demo_mode: true. The real invariant lives in api/src/main.ts and refuses to start the process if APP_ENV=production && DEMO_MODE=true. The smoke probe is a defense-in-depth check that the deployed env can't be tricked into advertising demo-mode.

Item 9 — Status page / uptime monitor target​

The /health endpoint is the target for whatever uptime monitor we wire (Better Stack / Statuspage / etc.). Item 9 just verifies the URL is reachable; the upstream monitor config is owned by ops.

Item 10 — 2-pod 24h soak — 5 crons fire exactly once each (manual)​

Cross-pod cron singleton via BullMQ jobId dedup. Manual procedure:

  1. After a clean 24h soak window on staging with 2 pods running: 
    SELECT name, count(*) as fires
    FROM bullmq_jobs_archive  -- or equivalent
    WHERE created_at > now() - interval '24 hours'
      AND job_name IN ('daily-summary', 'review-summary', 'inventory-alert', 'weekly-report', 'snooze-resurface')
    GROUP BY name;
  2. Expected fires-per-job depends on cron pattern (see api/src/scheduler/scheduler.constants.ts). Each name should match its expected count exactly — never 2× (would indicate both pods fired the same cron).

Item 11 — Pod kill mid-deploy → BullMQ resumes (manual)​

Drain test. Manual procedure:

  1. While a load generator sends 100 inbound webhooks/min to staging: 
    scripts/nfr/inbound-loadgen.sh --rate 100/min --duration 60s &
  2. From AWS Console, stop one of the two staging tasks.
  3. ECS reschedules. The killed task should drain via SIGTERM handler (#808) within stopTimeout=120s.
  4. After load gen completes, verify in DB: 
    SELECT count(*) FROM messages WHERE created_at > <load gen start>;
    -- expected: 100, not 99 / 98

Item 12 — Idempotent expert respond (10× double-click) (semi-manual)​

LB may route consecutive expert clicks to different pods. Each click should be idempotent at the API layer (P0-9j atomic UPDATE...WHERE...status != terminal).

Procedure (full automation tracked under #744):

  1. Open an expert workspace queue item.
  2. Click "Send reply" 10 times in rapid succession.
  3. Expected: only 1 message in DB, only 1 outbound channel call, only 1 audit log entry.

When nightly CI alerts​

The .github/workflows/nightly-nfr-smoke.yml job runs test-nfr-smoke.sh at 02:00 UTC daily.

  • Slack alert on fail: posts a single message to #engineering linking to the failed run.
  • Skip-only nights: not noisy; alerts only on FAIL.
  • Token rotation: STAGING_TOKEN_ORG_A/B GitHub secrets must be refreshed monthly (they're long-lived staging JWTs).

Adding new checks​

Each item is a bash function in scripts/test-nfr-smoke.sh:

test_NN_my_check() {
  header "[NN/12] description"
  # ... probe ...
  if [ ok ]; then pass "msg"; else fail "msg"; fi
}

Register it in main(). Keep each function ≤ 30 lines; if probing logic gets bigger, extract a helper to scripts/nfr/<name>.sh.

Reference​

  • MVP P0 NFR FINAL EN — §Verification Plan
  • ADR-018 — multi-pod safety contract
  • #810 — original issue
Last updated on by Degen