Runbook: Workspace Provisioning Failure

P1. Each OSA (Org × Specialist Assignment) needs a Gsuite user + email_alias to send/receive mail. A stuck provisioning job means an AM clicked "Assign" but the workspace never came online — the Specialist has no mailbox and inbound email bounces. See OWNERS.md for on-call contact.

This runbook covers workspace-provisioning BullMQ jobs that fail repeatedly (after the 3 automatic retries) when assigning a Specialist to an Org. Implementation: api/src/workspace-lifecycle/workspace-provisioning.processor.ts. Architecture: post-#1722 single-SA DWD model — see channels.md §System 4 for the current Specialist-email shape.

First, rule out disabled-mode. If jobs are completing with a skipped: gsuite disabled log (not failing), Gsuite isn't configured in this environment — that's expected behavior, not a failure. See gsuite-disabled-mode.md. This runbook is only for jobs that genuinely fail after retries.

Symptoms

AM Setup Wizard step 2 shows the assignment but the OSA detail page shows email_alias populated, gsuite_user_id = NULL, and gsuite_provisioned_at = NULL — the alias was computed but the actual Workspace user was never created.
Sentry [workspace-provisioning-processor] onFailed events firing for create-gsuite-user jobs.
BullMQ failed-jobs count > 0 for the workspace-provisioning queue (visible at /health/queues).
AM notification "Workspace provisioning failed for {Specialist} @ {Org} — please retry." (sent automatically by the processor's onFailed handler).
Inbound email bounces for the new email_alias — Gsuite Admin SDK returns "user not found" because the user was never created.

Step 1 — Triage (within 15 min)

1a. Identify the failed job

SELECT id, name, data, failed_reason, attempts_made, finished_on
FROM bullmq_failed_jobs  -- or via BullMQ UI / Arena
WHERE queue_name = 'workspace-provisioning'
  AND finished_on > NOW() - INTERVAL '24 hours'
ORDER BY finished_on DESC;

Pull the OSA id from the job data ({ osaId: "...", orgId: "...", specialistId: "..." }).

1b. Classify the failure mode

failed_reason typically falls into one of:

Pattern	Root cause	Next step
`unauthorized_client` / `invalid_grant`	Service account creds missing / wrong / Domain-Wide Delegation not granted	Step 2a
`Quota exceeded` / `User license cap exceeded`	Gsuite tenant out of seats	gsuite-license-cap.md
`entity already exists` / 409	`email_alias` collides with an existing user (e.g., manual creation)	Step 2b
`network timeout` / `ECONNRESET`	Transient	Step 3 (just retry)
`Could not decrypt OAuth refresh token`	`MASTER_ENCRYPTION_KEY` rotation broke an old row	oauth-token-refresh.md
`Service account key invalid` / "Bad JWT signature"	`GSUITE_SERVICE_ACCOUNT_KEY_JSON` env var corrupted or expired	Step 2a

If the failure mode doesn't match any of the above, capture the full stack from Sentry and escalate (see Step 5).

Step 2 — Fix the root cause

2a. Service account / DWD broken

Verify GSUITE_SERVICE_ACCOUNT_KEY_JSON env is set and parses as valid JSON (no truncation in Secrets Manager). (Legacy alias: GOOGLE_SERVICE_ACCOUNT_JSON — deprecated but still supported; canonical name takes precedence.)
Verify Domain-Wide Delegation in Google Admin → Security → API Controls → Domain-wide Delegation. Client ID must match the service account's client_id. Scopes must include the full WORKSPACE_DWD_SCOPES set from api/src/workspace-lifecycle/workspace-lifecycle.constants.ts — minimum-necessary 9 scopes (broader subsumes narrower where Google docs the relationship; Meet's two are independent):
- https://www.googleapis.com/auth/admin.directory.user
- https://www.googleapis.com/auth/admin.directory.user.security
- https://www.googleapis.com/auth/gmail.modify
- https://www.googleapis.com/auth/gmail.settings.basic
- https://www.googleapis.com/auth/calendar
- https://www.googleapis.com/auth/drive
- https://www.googleapis.com/auth/contacts
- https://www.googleapis.com/auth/meetings.space.created
- https://www.googleapis.com/auth/meetings.space.readonly
DWD matches scopes by exact string. If the API returns unauthorized_client, the most common cause is a scope present in code but missing in the Admin Console grant (or vice versa). Diff the two lists before deeper debugging.
Re-confirm GSUITE_ADMIN_USER_EMAIL is the impersonated super-admin in the tenant. (Legacy alias: GOOGLE_ADMIN_EMAIL — deprecated but still supported; canonical name takes precedence.)

2b. Provisioning gated off for this org

On dev/staging, provisioning is disabled by default unless the org has an explicit override. Check:

SELECT id, slug, gsuite_provisioning_enabled
FROM organizations
WHERE id = '<org_id>';

`gsuite_provisioning_enabled`	APP_ENV	Result
`NULL`	`production`	Provisioned
`NULL`	`development` / `staging`	Skipped
`true`	any	Provisioned
`false`	any	Skipped

Remediate: If the org should provision on dev/staging:

curl -X PATCH -H "Authorization: Bearer <superadmin-jwt>" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}' \
  https://api.<env>.h.work/admin/orgs/<org_id>/gsuite-provisioning

Then retry the assignment or trigger the provisioning hook manually.

2c. email_alias collision

Query the conflict:

curl -X GET "https://admin.googleapis.com/admin/directory/v1/users/<email_alias>" -H "Authorization: Bearer <token>"

If the user truly exists from manual creation, decide:
- If it's a stale manual test user → delete it in Google Admin, then retry the job.
- If it's a real prior assignment → file a P0 isolation bug; the OSA→email_alias rule was violated.

Step 3 — Retry the job

Once the root cause is fixed:

# Option A: re-enqueue via API (preferred — preserves OSA linkage):
curl -X POST -H "Authorization: Bearer <admin-jwt>" \
  https://api.<env>.h.work/admin/workspace-provisioning/retry/<osa_id>

# Option B: manual BullMQ retry from the UI / Arena / scripts:
psql -c "SELECT bullmq_retry_job('<job_id>');"

Watch the next job's outcome in Sentry / /health/queues. Successful retry → gsuite_user_id + gsuite_provisioned_at populate on the org_specialist_assignments row.

Step 4 — Rollback (if retry still fails after Step 2)

If the AM needs to un-stick the Setup Wizard while you debug:

Mark the OSA inactive without deleting it:

UPDATE org_specialist_assignments
SET gsuite_archived_at = NOW(),
    metadata = jsonb_set(coalesce(metadata, '{}'::jsonb), '{provisioning_failed}', 'true'::jsonb)
WHERE id = '<osa_id>';

Notify the AM the assignment is on hold pending ops review.
Continue debugging the failure in Sentry / Step 2.

Once fixed, clear the gsuite_archived_at and re-trigger the BullMQ job per Step 3.

Step 5 — Escalate

If after Step 2 + Step 3 the same OSA still fails twice in a row:

Page Platform Lead (Eusden) + escalation channel (#ops-incidents).
Capture: osa_id, failed_reason, Sentry event link, full Sentry stack, BullMQ job payload.
Document timeline in the incident channel.

For tenant-wide Gsuite outages (multiple OSAs failing simultaneously), check Google Workspace Status Dashboard before assuming our bug.

oauth-token-refresh.md — when the failure is OAuth-decrypt
gsuite-license-cap.md — when the failure is seat cap
gsuite-tenant-suspended.md — when ALL provisioning fails simultaneously
channels.md §System 4 — current Specialist-email schema + lifecycle

Symptoms​

Step 1 — Triage (within 15 min)​

1a. Identify the failed job​

1b. Classify the failure mode​

Step 2 — Fix the root cause​

2a. Service account / DWD broken​

2b. Provisioning gated off for this org​

2c. email_alias collision​

Step 3 — Retry the job​

Step 4 — Rollback (if retry still fails after Step 2)​

Step 5 — Escalate​

Related​