Skip to main content

Runbook: Gsuite License Cap Hit

P2 in steady state, P1 during active onboarding. Each OSA consumes one Gsuite user license. When the tenant hits its purchased seat cap, new assignments fail until we either upgrade or reclaim seats. See OWNERS.md for on-call contact.

This runbook covers Gsuite tenant seat-cap exhaustion — provisioning fails for new OSAs because we've allocated all purchased Workspace licenses. Implementation: provisioning processor surfaces Quota exceeded / User license cap exceeded. Architecture: ADR-0002 Phase 2 — Specialist Email (channels.md §System 3).

Symptoms​

  1. workspace-provisioning jobs failing with Quota exceeded for the current request or User license cap exceeded for domain.
  2. AM cannot assign Specialist — Setup Wizard step 2 → assignment confirmed, but BullMQ job dies, OSA row left half-provisioned (no gsuite_user_id).
  3. Bulk AM onboarding incident — multiple AMs simultaneously hitting "Assign" → spike of provisioning failures.
  4. Google Admin → Billing → Subscriptions shows assigned licenses = total purchased (no headroom).

Step 1 — Confirm the cap​

1a. Check tenant state​

In Google Admin → Billing → Subscriptions → Google Workspace Business/Enterprise plan:

  • Assigned licenses vs Total licenses purchased.
  • If assigned = total, the cap is real.

1b. Cross-reference with our DB​

SELECT count(*) AS active_osa_count
FROM org_specialist_assignments
WHERE gsuite_user_id IS NOT NULL
  AND gsuite_archived_at IS NULL;

This count should approximately equal Google's "assigned licenses" minus the <env>-admin service accounts we maintain. If our count is materially lower than Google's: orphan users exist in Google (manually-created users, leftover from old assignments). Audit + reclaim per Step 3.

Step 2 — Quick fix: purchase more seats​

For an active customer onboarding incident:

  1. Go to Google Admin → Billing → Manage subscriptions → Increase user count.
  2. Add headroom = (expected new OSAs in next 30 days) × 1.3. Don't size to exactly fit — assignment bursts always happen.
  3. Wait 5-15 min for Google to provision the new seats.
  4. Retry the failed workspace-provisioning jobs per workspace-provisioning-failure.md Step 3.

Cost: each seat is typically $5-25/month depending on the plan tier. Log the increase in #ops-finance for next-month reconciliation.

Step 3 — Long-term: reclaim unused seats​

Investigate why we're at cap if onboarding isn't expected to explain it:

3a. Archived but not deleted OSAs​

SELECT id, org_id, specialist_id, email_alias, gsuite_archived_at
FROM org_specialist_assignments
WHERE gsuite_archived_at IS NOT NULL
  AND gsuite_user_id IS NOT NULL
ORDER BY gsuite_archived_at ASC
LIMIT 50;

These OSAs are logically deactivated (e.g., client churned, Specialist re-assigned) but the Google Workspace user is still alive. Each one is still consuming a seat.

Decision: delete vs suspend.

  • Delete (frees seat immediately): DELETE via Google Admin SDK. Lose the inbox. Only do this for OSAs archived > 90 days.
  • Suspend (keeps inbox readable, frees the seat in some Google plans): Google Admin → User → Suspend. Note: not every Google plan frees a license seat on suspend; check your current subscription tier.

3b. Orphan Google users (not tied to any OSA)​

# Pseudo — listing all Workspace users via Admin SDK, then cross-checking against our DB
all_google_users = admin.users().list(domain=HWORK_DOMAIN).execute()
db_active_users = set(
    row.gsuite_user_id for row in OSA.where(gsuite_user_id IS NOT NULL AND gsuite_archived_at IS NULL)
)
orphans = [u for u in all_google_users if u.id not in db_active_users]

For each orphan: investigate (manual creation in panic? leftover from migration?), then delete in Google Admin if confirmed unused.

Rollback​

Adding more seats is non-reversible without contacting Google billing (you can downgrade at next billing cycle but not mid-cycle).

If you wrongly deleted a Workspace user as part of reclaim: Google retains deleted users for ~20 days before purging — you can restore from Google Admin → Users → Recently deleted. Beyond 20 days, the inbox is gone.

Step 4 — Capacity forecasting (prevent next time)​

SELECT
  date_trunc('week', assigned_at) AS week,
  count(*) AS new_assignments
FROM org_specialist_assignments
WHERE assigned_at > NOW() - INTERVAL '12 weeks'
GROUP BY 1
ORDER BY 1;

If the trend shows steady growth + 4 weeks until cap → schedule a license increase pre-emptively. Set a CloudWatch alarm: assigned_licenses / total_licenses > 0.85 → Slack #ops-finance.

Step 5 — Escalate​

  • If Step 2 (buy more seats) fails because we can't reach Google billing during a customer launch → page Platform Lead + Finance. Workaround: use service account gmail.users.create via API which sometimes bypasses the seat check, but does NOT actually free us from the seat cap — it just lets the user exist while billing reconciles.
  • If we hit cap because of a runaway provisioning bug (creating duplicate Workspace users for the same OSA) → P0, that's a different incident; see workspace-provisioning-failure.md Step 5.
Last updated on by Degen