Skip to main content

ADR-011: Google Workspace lifecycle — tenant model, Admin SDK quota, billing

Renumbered from ADR-009 in #937 to resolve a duplicate-number collision with 009-ragflow-shared-instance-vs-per-tenant.md. No content changes.

Date: 2026-05-25 Status: Accepted — Phase 2 shipped 2026-05-29 via #1065 (OSA Gsuite lifecycle schema) + #1074 (WorkspaceLifecycleModule) + #1076 (WorkspaceProvisioningProcessor) + #1079 (OAuthTokenService) + #1089 (assignSpecialist hook) + #1093 (inbound email OSA routing) + #1095 (outbound Gmail Send) + #1097. Issue: #679 (this ADR); parent epic #678; consumed by #587 Authors: paulhumanity Revision history:

  • 2026-05-25 (initial draft) — skeleton committed before tenant provisioning. Quota numbers cite published Google limits; final headroom calc pending real-tenant measurement. Billing line-items and incident runbook stubbed.

Context​

#678 commits us to per-(org × specialist) Google Workspace mailboxes in a single HP-owned tenant. Before any provisioning code lands (#680 and below), we need:

  1. The tenant itself standing up across staging + prod, with billing attached.
  2. Hard numbers on Admin SDK rate limits, so the provisioning hook (#680) is designed against a known ceiling rather than discovered at burst time.
  3. An auth-model decision (service account vs. domain-wide delegation) the rest of the workstream can build against without re-litigation.
  4. A billing posture (SKU, growth alerts) so unexpected user-count growth is detected before the invoice arrives.
  5. A workspace-side incident runbook (lockout, MX failure, suspension propagation) so Ops doesn't author it under fire.

Tenancy facts are settled in #678 and not re-litigated here:

  • Per-(org × specialist) mailbox in a single HP-owned tenant.
  • Address scheme: {specialist-firstname}.{client-slug}@h.work (prod), …@h852.work (dev). h853.work reserved for staging.
  • Slug uniqueness is a platform-wide invariant.
  • Plus-addressing rejected.

Decision​

1. Tenant layout​

EnvironmentDomainTenantNotes
Prodh.workHP-owned, primaryProduction mailboxes only.
Stagingh853.workHP-owned, separateMirrors prod auth model; TODO: provision.
Devh852.workHP-owned, separateAlready in use as HWORK_DOMAIN for dev environments. TODO: confirm same auth-model as prod.

Rationale for separate tenants per environment: dev/staging provisioning bugs must not be capable of touching prod billing or prod mailboxes. Domains are already split; tenants will be too.

2. Auth model​

Decision: service account with domain-wide delegation (DWD).

OptionProsConsVerdict
Service account (no DWD)Simplest, no impersonation surfaceCannot act as a specific user — can't send mail as bob.acme@h.work. Disqualified by #587 requirements.❌
OAuth user tokens per mailboxMost granular consent surface, matches per-(Specialist × Client) scope cleanlyRequires interactive consent per mailbox at provisioning — incompatible with onboarding SLA.❌
Service account + DWDOne credential controls all provisioned mailboxes; Admin SDK and Gmail API both available; revocable per scopeSingle high-value credential (mitigated by KMS encryption + rotation runbook); DWD grants are tenant-wide (mitigated by scoping to specific OAuth scopes, not all scopes).✅

Scopes granted to the DWD client (full set is WORKSPACE_DWD_SCOPES in api/src/workspace-lifecycle/workspace-lifecycle.constants.ts — Specialists are treated as full personal Gsuite identities, minimum-necessary list because broader Google scopes subsume narrower ones, see architecture/workspace-lifecycle.md §Scopes for the subsumption notes):

  • admin.directory.user / admin.directory.user.security — user lifecycle (separate scope for suspend/revoke).
  • gmail.modify — read, send, label management (subsumes gmail.send / gmail.readonly / gmail.metadata).
  • gmail.settings.basic — required to set per-pair send-as display name; NOT in gmail.modify.
  • calendar — full personal calendar (subsumes calendar.events).
  • drive — full My Drive (subsumes drive.file; required because clients share existing docs in).
  • contacts — Specialist's address book.
  • meetings.space.created + meetings.space.readonly — create Meet spaces and read any Meet conference (Specialist needs both to handle meetings clients invite them to; meetings.space.created alone only covers spaces the SA itself created).

Excluded by design: https://mail.google.com/ (legacy super-scope, triggers restricted-scope OAuth verification), tasks (out of scope for v1), admin.directory.group (DLs-per-client not shipped), contacts.other.readonly (auto-collected contacts surface not needed).

Credential storage: service-account JSON key persisted encrypted via the existing integration_credentials envelope (KMS-wrapped, see #551). Env var name: GOOGLE_WORKSPACE_SA_CREDENTIALS_REF (a reference id; the actual JSON is looked up from integration_credentials at runtime, never injected as env). Rotation runbook: see §5.

3. Admin SDK quota headroom​

Published Admin SDK limits (Directory API, as of 2026-05; verify against Google's quota page before merging):

OperationPer-project per-secondPer-project per-dayPer-user per-second
users.insert51,500 (write quota)1
users.update5shared above1
users.delete / suspend5shared above1

Worst-case provisioning burst: 50 Specialist×Client pairs assigned in a single onboarding day (per #679 acceptance criterion).

Headroom calculation:

  • 50 users.insert calls at 5/s = ~10 seconds of API time, well within per-second budget.
  • 50 inserts vs. the published ~1,500/day write quota = ~3.3% utilization. We can provision ≥ ~1,500 / 50 ≈ 30 days' worth of worst-case onboarding bursts before hitting the daily ceiling — i.e. headroom is daily-quota-bound, not burst-bound.
  • At current scale, throttling is not a realistic concern. Reconsider if onboarding velocity exceeds ~300 new pairs/day.

Caveats:

  • Numbers above are project-level; per-user quotas (service account acting as admin) match.
  • Daily quotas reset on Pacific Time midnight.
  • TODO once tenant exists: measure actual quotas in Cloud Console — Google's documented numbers have lagged the dashboard before.

Mitigation if we ever approach the ceiling:

  • Token-bucket rate-limit in the provisioning hook (#680).
  • Exponential backoff on 403 quotaExceeded.
  • Spread inserts across the day if a single onboarding event would queue >300 pairs.

4. Billing​

SKUPer-user/monthNotes
Business Standard$12Confirmed in #679 scope. 2TB pooled storage / user, Gmail + Calendar + Drive. Sufficient for our Specialist mailbox use case.
Business Plus (alt)$18Adds Vault retention, advanced endpoint mgmt. Not justified for Specialist mailboxes; revisit if compliance requires Vault.

Growth alerts:

  • Workspace Admin console → Reports → set alert at user_count > 1.2 × <last week's count> weekly.
  • Mirror in monitoring: daily cron compares SELECT count(*) FROM org_specialist_assignments WHERE deleted_at IS NULL against current Workspace user count via directory.users.list — alert on drift > 5%.
  • Cost model line item: (active org_specialist_assignments) × $12/month — tracked in finance projection.

Cost projection (illustrative):

  • 10 clients × ~5 Specialists each = 50 mailboxes = $600/month at Business Standard.
  • 100 clients × 5 = 500 = $6,000/month. Still rounding noise vs. infra spend.

5. Incident runbook (workspace-side failures)​

IncidentDetectionFirst responseOwner
Specialist account locked (Google sign-in)Outbound send fails with 403 userLockedUnlock via Admin console; if repeated, investigate brute-force / IP block.Ops (Paul / Eusden)
MX/DNS failure on h.workInbound email bounces; Cloudflare→Gmail webhook fails.Verify MX records (mx.google.com.); re-add if drift. Run dig MX h.work.DevOps
Service account credential leakAudit log shows unexpected actorRotate SA key (Cloud Console → SA → keys → revoke + create new); update integration_credentials; redeploy.Paul
Admin SDK 403 quotaExceeded (sustained)Provisioning hook logsBack off provisioning; file ticket with Google for quota increase; manual user.insert via Admin console for blocked pair.Ops
Whole-tenant suspensionAll API calls 403 forbiddenPage Google Workspace support; check billing/payment status; comms to affected clients.Paul + Finance
User suspended unexpectedlydirectory.users.get returns suspended:true without matching org_specialist_assignments.deleted_atTrigger reconciliation job (#684); investigate cause; unsuspend if assignment is still active.Ops

Escalation path: Workspace incidents → Paul → Eusden → Google Workspace support (priority based on SKU; Business Standard = standard support hours).

Credential rotation cadence: SA key rotated every 90 days, or immediately on any suspected leak. Calendar reminder owned by Paul.

Consequences​

Positive:

  • Provisioning hook (#680) has a fixed quota target; no surprises at burst time.
  • Single auth model (SA + DWD) avoids per-mailbox OAuth dance — onboarding SLA achievable.
  • Per-environment tenants prevent dev provisioning bugs from touching prod billing/users.
  • Cost is linear, predictable, and small compared to infra spend.

Negative / risks:

  • DWD is a high-trust credential; compromise = tenant compromise. Mitigated by KMS, rotation, and audit logging.
  • Daily Admin SDK write quota is a (very distant) ceiling; if HP grows to ~300 new pairs/day onboarding velocity, design must revisit batching.
  • Three tenants × three domains = 3× the Workspace admin surface to maintain. Acceptable cost for isolation.

Acceptance checklist​

  • Workspace tenant provisioned for prod (h.work).
  • Workspace tenant provisioned for staging (h853.work).
  • Dev tenant (h852.work) confirmed on same auth model.
  • Service-account credentials generated and persisted (encrypted) in integration_credentials; GOOGLE_WORKSPACE_SA_CREDENTIALS_REF documented in .env.example.
  • DWD configured with the four scopes above for each tenant.
  • Billing: Business Standard SKU confirmed, growth alert configured in Admin console.
  • Quota headroom verified against Cloud Console dashboard (not just published docs).
  • Incident runbook reviewed by Ops.
  • This ADR moved from Draft to Accepted.

Open questions​

  • Do we need Workspace Vault for compliance / data retention? (Drives SKU decision: Business Standard vs. Business Plus.)
  • Should h852.work (dev) share a billing account with prod for unified cost reporting, or stay financially isolated?
  • Quota-increase request to Google: pre-emptive, or wait until we approach the ceiling?

References​

Last updated on by Degen