BullMQ Queue Backlog Runbook

1. Signals

Alerts:

Alert	Meaning	First PromQL check
`HumanworkBullMQQueueDepthWarning`	`bullmq_queue_depth_total > 100` for 5 minutes	`bullmq_queue_depth_total{service="humanwork-api"}`
`HumanworkBullMQQueueDepthCritical`	`bullmq_queue_depth_total > 1000` for 2 minutes	`bullmq_queue_depth_total{service="humanwork-api"}`

Grafana:

Open the API observability dashboard and filter service=humanwork-api.
Check queue depth by queue: bullmq_queue_depth_total
Check state distribution: sum by (queue, state) (bullmq_queue_jobs)
Check oldest waiting job age: bullmq_queue_oldest_job_age_seconds
Check whether backlog is growing: delta(bullmq_queue_depth_total[10m])
Check failed job volume: bullmq_queue_jobs{state="failed"}

2. Identify The Queue

Use the alert's queue label. Map it to the owning API module:

Queue	Owning module	Primary producer / consumer
`scheduler`	`api/src/scheduler`	`SchedulerService` repeatable jobs, `SchedulerProcessor`
`correction-refinement`	`api/src/haystack`	`HaystackModule` hourly tick, `CorrectionRefinementProcessor`
`background-tasks`	`api/src/common`	`BackgroundTasksService`, `BackgroundTasksProcessor`
`telegram-inbound`	`api/src/channels/telegram`	`TelegramService`, `TelegramInboundProcessor`
`channels-inbound`	`api/src/channels`	`ChannelsController`, `ChannelsInboundProcessor`
`channels-outbound`	`api/src/channels`	`ChannelDispatcherService`, `ChannelsOutboundProcessor`
`workspace-provisioning`	`api/src/workspace-lifecycle`	`WorkspaceProvisioningProcessor` — `create-gsuite-user`, `archive-gsuite-user` (ADR-0002 Phase 2)
`persona-cleanup`	`api/src/meetings`	`PersonaCleanupProcessor`

3. First 5 Minute Triage

Check Redis health:

redis-cli -u "$REDIS_URL" ping
redis-cli -u "$REDIS_URL" info memory
redis-cli -u "$REDIS_URL" info clients

Confirm PONG, available memory headroom, and sane connected_clients.

Check API and worker process counts. Prod is on AWS ECS Fargate (#1101); dev/staging may still be on Railway during the migration window.

ECS / AWS (primary prod path):
```
aws ecs describe-services --cluster humanwork-<env> --services humanwork-api
aws ecs list-tasks --cluster humanwork-<env> --service-name humanwork-api
aws logs tail /ecs/humanwork-api --follow --since 15m
```
Confirm desiredCount / runningCount match and that tasks are not in a restart loop.

Railway (legacy / staging fallback):
```
railway status
railway service
railway logs --service humanwork-api --tail 200
```
Confirm the expected API/worker replicas are running and not restarting.

Check recent deploys:

ECS / AWS:

aws ecs describe-services --cluster humanwork-<env> --services humanwork-api \
  --query 'services[0].deployments'
gh pr list -R humanity-org/humanwork --state merged --base dev --limit 10

Railway (legacy / staging fallback):

railway deployments --service humanwork-api

Look for a deploy that changed the queue owner module or downstream integration.

4. Common Root Causes

Root cause	How to confirm	What to do
Worker crash or boot failure	Dev/Railway logs or prod CloudWatch logs (`/ecs/humanwork-api`) show repeated NestJS startup errors, unhandled BullMQ errors, or missing env vars. Queue `active` stays low while `waiting` rises.	Dev/Railway: `railway rollback --service humanwork-api --deployment <previous-deploy-id>` or fix env/config. Prod/ECS: `aws ecs update-service --cluster humanwork-<env> --service humanwork-api --task-definition <prev-revision>` or fix env/config. Scale healthy worker/API replicas after the process is stable.
Slow downstream API	Logs show timeouts or retries for Slack, Telegram, WhatsApp/Twilio, Haystack, Tavus, or email. `active` jobs remain high and oldest age rises.	Check vendor status. Reduce retry pressure if needed. Pause the affected integration if it is causing cascading failures.
Poison message storm	`failed` count rises quickly for one queue and logs show the same payload or org repeatedly failing.	Identify the payload/job id. Pause the queue if it is actively causing harm. Clean failed jobs only after preserving enough data for root cause analysis.
Redis memory exhausted	`redis-cli info memory` shows `used_memory` near max, evictions, or OOM errors. BullMQ operations time out.	Increase Redis memory, remove old completed/failed jobs, or reduce retention. Do not flush Redis without incident lead approval.
Traffic spike	HTTP/channel ingress metrics spike before queue depth. Workers are healthy but total input exceeds throughput.	Scale API/worker replicas, rate-limit noisy sources, or temporarily pause low-priority queues.

5. Mitigation Playbook

Add Workers

Scale the API worker service after confirming the backlog is not caused by poison jobs.

ECS / AWS (primary prod path):

aws ecs update-service \
  --cluster humanwork-<env> \
  --service humanwork-api \
  --desired-count 4
aws logs tail /ecs/humanwork-api --follow --since 5m

If the deployment has a separate worker service:

aws ecs update-service \
  --cluster humanwork-<env> \
  --service humanwork-worker \
  --desired-count 4

Railway (legacy / staging fallback):

railway scale --service humanwork-api --replicas 4
railway logs --service humanwork-api --tail 200

If the deployment has a separate worker service, scale that service instead:

railway scale --service humanwork-worker --replicas 4

Watch:

bullmq_queue_depth_total{queue="<queue>"}
rate(bullmq_queue_jobs{queue="<queue>",state="completed"}[5m])
bullmq_queue_oldest_job_age_seconds{queue="<queue>"}

Drain Queue

Only drain when the incident lead has confirmed the jobs are safe to discard. Preserve job ids or sample payloads first.

Clean failed jobs:

CONFIRM_DRAIN_QUEUE="<queue>" REDIS_URL="$REDIS_URL" node <<'NODE'
const { Queue } = require("bullmq");

const queueName = process.env.CONFIRM_DRAIN_QUEUE;
if (!queueName) throw new Error("set CONFIRM_DRAIN_QUEUE to the exact queue name");
if (!process.env.REDIS_URL) throw new Error("REDIS_URL is required");

const queue = new Queue(queueName, {
  connection: { url: process.env.REDIS_URL, maxRetriesPerRequest: null },
});

(async () => {
  const removed = await queue.clean(0, 1000, "failed");
  console.log(`removed failed jobs: ${removed.length}`);
  await queue.close();
})();
NODE

Clean waiting jobs only with explicit confirmation:

CONFIRM_DRAIN_QUEUE="<queue>" CONFIRM_DELETE_WAITING="yes" REDIS_URL="$REDIS_URL" node <<'NODE'
const { Queue } = require("bullmq");

const queueName = process.env.CONFIRM_DRAIN_QUEUE;
if (!queueName) throw new Error("set CONFIRM_DRAIN_QUEUE to the exact queue name");
if (!process.env.REDIS_URL) throw new Error("REDIS_URL is required");
if (process.env.CONFIRM_DELETE_WAITING !== "yes") {
  throw new Error("set CONFIRM_DELETE_WAITING=yes to delete waiting jobs");
}

const queue = new Queue(queueName, {
  connection: { url: process.env.REDIS_URL, maxRetriesPerRequest: null },
});

(async () => {
  const removed = await queue.clean(0, 1000, "wait");
  console.log(`removed waiting jobs: ${removed.length}`);
  await queue.close();
})();
NODE

Pause Vs Flush

Pause when jobs are valid but a downstream dependency is unhealthy. Resume after the dependency recovers.
Pause when a specific queue is harming the rest of the system and you need time to inspect payloads.
Flush or clean only when jobs are invalid, duplicated, or safe to regenerate.
Do not flush scheduler blindly. Scheduler jobs are repeatable control-plane work and should usually be fixed by restoring workers or removing a bad repeatable job by id.
Do not flush Redis. Redis may contain BullMQ metadata, circuit breaker state, throttling state, and other runtime keys.

Pause example:

QUEUE_NAME="<queue>" REDIS_URL="$REDIS_URL" node <<'NODE'
const { Queue } = require("bullmq");
const queue = new Queue(process.env.QUEUE_NAME, {
  connection: { url: process.env.REDIS_URL, maxRetriesPerRequest: null },
});
(async () => {
  await queue.pause();
  console.log(`paused ${process.env.QUEUE_NAME}`);
  await queue.close();
})();
NODE

Resume example:

QUEUE_NAME="<queue>" REDIS_URL="$REDIS_URL" node <<'NODE'
const { Queue } = require("bullmq");
const queue = new Queue(process.env.QUEUE_NAME, {
  connection: { url: process.env.REDIS_URL, maxRetriesPerRequest: null },
});
(async () => {
  await queue.resume();
  console.log(`resumed ${process.env.QUEUE_NAME}`);
  await queue.close();
})();
NODE

6. Postmortem Follow-Ups

Use the incident template in docs/runbooks/incident-response.md.

Required follow-ups:

Record the alert name, queue label, peak depth, peak oldest job age, and time to drain.
Link the deploy, PR, vendor incident, or traffic source that triggered the backlog.
Add or update a regression test when a code path caused poison jobs or worker crashes.
Tune worker concurrency, retry/backoff, or alert thresholds if the alert was noisy.
Confirm completed/failed job retention is low enough to protect Redis memory.

1. Signals​

2. Identify The Queue​

3. First 5 Minute Triage​

4. Common Root Causes​

5. Mitigation Playbook​

Add Workers​

Drain Queue​

Pause Vs Flush​

6. Postmortem Follow-Ups​