Workspace lifecycle — Google Workspace provisioning, scoped tokens, and tenancy

Status: Authoritative architecture for per-(Specialist × Client) Google Workspace mailboxes. Anchors: #678 (workstream), #679 (this spike), #680 (provisioning), #681 (OAuth scoping), #682 (suspend lifecycle), #683 (token refresh), #684 (reconciliation), #587 (email channel consumer). Companion docs: ISOLATION_SANDBOXING_BINARIES.md — defines the "no ambient secrets" + control-plane credential proxy principles this doc realizes for Workspace. Decision authority: Eusden (product / tenancy), Emanuele (Workspace admin / DNS / DKIM), Paul (billing sign-off), Alex Dankov (isolation alignment).

---

TL;DR

• One Google Workspace tenant (h.work), three domains:
  • h.work — production
  • h853.work — staging
  • h852.work — dev
• Each Specialist assigned to a Client (org_specialist_assignments row) gets its own Workspace user at {firstname}.{client-slug}@{env-domain}. Same Specialist on two Clients = two distinct users.
• Three service accounts in a single GCP project (hwork-workspace-lifecycle), one per env: humanwork-workspace-{prod,staging,dev}. Each SA impersonates a per-env super-admin delegate (provisioner@{env-domain}) via Domain-Wide Delegation.
• DWD scopes are tenant-wide (Google does not support per-domain DWD scoping). The hard boundary is enforced in application code: every users.insert / gmail.send asserts targetDomain === env.WORKSPACE_DOMAIN before issuing the call.
• Service account JSON keys are short-term; live in integration_credentials (encrypted via #551). Long-term plan: migrate to Workload Identity Federation once runtime OIDC issuance is wired (follow-up issue to file post-spike).
• Removing an org_specialist_assignments row suspends the Workspace user (not delete). Audit retention required.

---

Tenancy decision

Considered alternatives

Option	Decision	Why
a) Three separate tenants (h.work, h853.work, h852.work as standalone Workspace tenants)	❌ rejected	3× admin overhead (DNS, MX, DKIM, billing, support contracts), 3× per-tenant Admin SDK quota story, 3× the audit/auth surface. No isolation gain that we can't get from per-env service accounts.
b) One tenant, secondary domains	✅ chosen	Single Workspace billing/admin boundary, env encoded in the domain, single per-tenant Admin SDK quota envelope. Seats pool across domains.
c) Cloudflare Agentic Inbox (cloudflare/agentic-inbox) instead of Workspace	❌ rejected for this scope	Email-only — would still need GSuite for Calendar / Drive / Docs. Re-evaluate when the email-only scope is on the table (today it isn't — Calendar/Drive are in roadmap).
d) Plus-addressing (bob+acme@h.work)	❌ rejected	Fragile on sender clients, mis-handled by many MTAs, blocks per-pair OAuth. Decided 2026-05-25.

Tenancy facts (do not re-litigate)

• Per-(org × specialist) mailbox. Same Specialist on two Clients = two distinct Workspace users with distinct addresses.
• Address scheme: {specialist-firstname}.{client-slug}@{env-domain}.
  • env-domain ∈ {h.work (prod), h853.work (staging), h852.work (dev)}.
  • client-slug uniqueness is a platform-wide invariant (relied on; do not change).
  • specialist-firstname is lowercased, ASCII-folded, single token.
• Seats are pooled across the three domains under one billing account.
• Lower-env reaping policy (explicit operational requirement): Weekly automated reap of any Workspace user in @h852.work / @h853.work suspended for >7 days. Suspension happens when the corresponding org_specialist_assignments row is deleted or the Client org has status='deactivated'. This bounds dev/staging seat growth to active testing load.

---

Auth model — DWD with three service accounts

Why three SAs (not one) — hard cross-env credential boundary

A single SA spanning all three envs would mean a compromised dev key reaches prod. Three SAs establish a hard cross-env credential boundary, giving us:

1. Separate key material — dev compromise does not expose prod credentials. No credential overlap across environments.
2. Separate impersonation delegate — each SA impersonates a per-env super-admin user, so the Workspace audit log shows which env initiated each Admin SDK call.
3. Separate rotation lifecycle — rotate dev independent of prod.

Why DWD scopes are tenant-wide

Google's Domain-Wide Delegation grants scopes against the tenant, not against a specific primary or secondary domain. A DWD-authorized SA can in principle impersonate any user in the tenant within the granted scopes. We accept this and enforce per-env isolation in application code (see "Hard boundary" below).

Hard boundary — enforced in app code

WorkspaceTokenService (and any direct Admin SDK / Gmail caller) must assert:

if (!targetEmail.endsWith(`@${env.WORKSPACE_DOMAIN}`)) {
  throw new Error(`Cross-env Workspace call denied: ${targetEmail} vs ${env.WORKSPACE_DOMAIN}`);
}

Where env.WORKSPACE_DOMAIN is one of h.work / h853.work / h852.work, set per deployment. The same assertion lives inside the users.insert provisioning hook (#680), the per-pair OAuth flow (#681), and the Gmail send/read paths (#587).

This is the load-bearing isolation primitive for Workspace. The Google policy layer is permissive; our app layer is the boundary.

Scopes granted to all three SAs

The DWD service account holds a minimum critically-necessary Workspace persona scope set — each provisioned (Specialist × Client) account is a real personal Gsuite user (Gmail + Calendar + Drive + Contacts + Meet) so clients can interact with bob.acme@h.work the same way they'd interact with any colleague. Broader Google scopes subsume narrower ones, so we grant only the broad scope in each case. The canonical list lives in api/src/workspace-lifecycle/workspace-lifecycle.constants.ts as WORKSPACE_DWD_SCOPES; reproduced here:

https://www.googleapis.com/auth/admin.directory.user
https://www.googleapis.com/auth/admin.directory.user.security
https://www.googleapis.com/auth/gmail.modify
https://www.googleapis.com/auth/gmail.settings.basic
https://www.googleapis.com/auth/calendar
https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/contacts
https://www.googleapis.com/auth/meetings.space.created
https://www.googleapis.com/auth/meetings.space.readonly

Subsumption notes (verified against Google's OAuth scopes master index 2026-06-08):

• gmail.modify ("Read, compose, and send emails from your Gmail account") ⊇ gmail.send + gmail.readonly + gmail.metadata. gmail.settings.basic is granted separately because it covers email settings/filters/send-as identity — Google docs it as a distinct scope, not implied by gmail.modify. Required to set the per-pair display name (Bob — Acme) on outbound mail (#587).
• calendar ("See, edit, share, and permanently delete all the calendars") ⊇ calendar.events ("View and edit events").
• drive ("See, edit, create, and delete all of your Google Drive files") ⊇ drive.file ("only the specific files you use with this app"). drive is the right choice because clients share existing docs into the Specialist's account, not just files the Specialist created.
• Meet's two scopes are NOT in a subset relationship. meetings.space.created covers Meet spaces the SA itself created ("Create, edit, and see information about your Google Meet conferences created by the app"). meetings.space.readonly is required additionally to read information about any of the Specialist's Meet conferences (e.g. meetings clients invited them to). Both granted.
• admin.directory.user.security ("Manage data access permissions for users on your domain") is not a subset of admin.directory.user ("View and manage the provisioning of users on your domain") — separate scope, required to revoke per-user OAuth grants on assignment removal (#682).

Excluded by design:

• https://mail.google.com/ (legacy full-Gmail super scope, "Read, compose, send, and permanently delete all your email") — only adds permanent-deletion-bypassing-trash on top of gmail.modify, triggers Google's restricted-scope OAuth verification, and broadens DWD blast radius. Never granted.
• https://www.googleapis.com/auth/tasks — Specialist task list not in scope for v1.
• https://www.googleapis.com/auth/admin.directory.group — DLs-per-client pattern (ADR-007) not shipped yet; add in the same PR that ships the feature.
• https://www.googleapis.com/auth/contacts.other.readonly — "Other Contacts" auto-collected API; contacts covers the address book the Specialist actually manages.
• https://www.googleapis.com/auth/meetings.space.settings — Meet workspace-wide settings; not needed for v1.

Drift assertion. DWD authorizes scopes by exact-string match on the Admin Console side. The canonical list in WORKSPACE_DWD_SCOPES is the single source of truth — when code starts calling a new Google API, the scope is added there AND granted on the DWD client in the Workspace Admin Console in the same PR. A scope drift between the two surfaces fails closed at runtime with unauthorized_client (see runbook workspace-provisioning-failure.md).

Why service-account JSON keys today (and why this is temporary)

Per Google org policy iam.disableServiceAccountKeyCreation (Secure by Default), creating SA JSON keys is denied at the org level. We temporarily disabled enforcement on the hwork-workspace-lifecycle project, minted three keys, and re-enabled the policy before any other project could create keys.

The keys live encrypted in integration_credentials (see #551 — closed). The follow-up to eliminate keys entirely is Workload Identity Federation:

1. Configure a WIF pool + provider per env, bound to the runtime's OIDC issuer (Railway / ECS / whatever the env runs on).
2. Grant the principal roles/iam.serviceAccountTokenCreator on the matching SA.
3. Delete the SA JSON keys.

Follow-up issue to file: "#XXX [P2][infra] Migrate Workspace SAs from JSON keys to Workload Identity Federation." Acceptance: no JSON key material in integration_credentials, no key material at rest, app-side change is GoogleAuth({ credentials: undefined }) and runtime ADC.

---

Credential storage — integration_credentials rows

Each env's SA + delegate config is one platform-scoped row in integration_credentials. Schema is the existing integration_credentials entity (integrationType, encryptedConfig, metadata); we add integrationType='google_workspace_admin' per env. Org scoping: because this is platform infrastructure (not per-Client), the rows use a sentinel platform-org UUID (00000000-0000-0000-0000-000000000000) and metadata.env distinguishes them.

Row shape (one per env)

Because integration_credentials has a UNIQUE(org_id, integration_type) constraint, the three env rows differentiate via integration_type suffix rather than colliding on a shared key. All three share the platform sentinel org_id.

Encryption contract (encrypted_config)

integration_credentials.encrypted_config is a JSONB column whose string values are individually encrypted by encryptConfig() in api/src/common/crypto.ts — each string becomes enc:<iv>:<tag>:<ciphertext> via AES-256-GCM (see issue #551 for the rollout). Non-string values (null, numbers, nested objects) pass through unencrypted. decryptValue() passes anything not starting with enc: through untouched, so a JSONB document can hold a mix of plaintext and ciphertext keys safely.

What this means for the lifecycle here:

• Secrets (service_account_json, client_id) live in encrypted_config and are encrypted by encryptConfig() per-key.
• Public identifiers (delegate_email, client_email, scopes, GCP project, env, domain) live in metadata, which is unencrypted by design. Keeping them out of encrypted_config avoids any partial-update ambiguity (no risk of a future jsonb_set merging plaintext into ciphertext space).
• Activation is a full-object overwrite, not a partial JSONB merge: the vault-access agent constructs the complete secret object in memory, wraps it with encryptConfig(), then UPDATE integration_credentials SET encrypted_config = $1, metadata = jsonb_set(metadata, '{placeholder}', 'false'). There is no in-place merge into an encrypted blob.

Placeholder row (seeded by migration 1751500000001)

{
  "id": "<uuid>",
  "org_id": "00000000-0000-0000-0000-000000000000",
  "integration_type": "google_workspace_admin_{prod|staging|dev}",
  "encrypted_config": {},
  "metadata": {
    "env": "prod" | "staging" | "dev",
    "workspace_domain": "h.work" | "h853.work" | "h852.work",
    "gcp_project": "hwork-workspace-lifecycle",
    "scopes": ["https://www.googleapis.com/auth/admin.directory.user", "https://www.googleapis.com/auth/admin.directory.user.security", "https://www.googleapis.com/auth/gmail.modify", "https://www.googleapis.com/auth/gmail.settings.basic", "https://www.googleapis.com/auth/calendar", "https://www.googleapis.com/auth/drive", "https://www.googleapis.com/auth/contacts", "https://www.googleapis.com/auth/meetings.space.created", "https://www.googleapis.com/auth/meetings.space.readonly"],
    "delegate_email": "provisioner@{env-domain}",
    "client_email": "humanwork-workspace-{env}@hwork-workspace-lifecycle.iam.gserviceaccount.com",
    "wif_migration_target": true,
    "placeholder": true
  }
}

Activated row (post vault-access agent)

{
  "id": "<uuid>",
  "org_id": "00000000-0000-0000-0000-000000000000",
  "integration_type": "google_workspace_admin_{prod|staging|dev}",
  "encrypted_config": {
    "service_account_json": "enc:<iv>:<tag>:<ciphertext>",
    "client_id": "enc:<iv>:<tag>:<ciphertext>"
  },
  "metadata": {
    /* same as placeholder shape, with `placeholder: false` */
  }
}

metadata.placeholder=true until the encrypted JSON is dropped in by the agent with prod-vault access. The provisioning hook (#680) must refuse to run against a placeholder row.

A seed migration that inserts the three placeholder rows is in api/migrations/1751500000001-WorkspaceLifecycleCredsPlaceholder.ts. It is idempotent and safe to run before keys exist — the provisioner refuses to act on placeholder rows.

---

Admin SDK quota & rate-limit headroom

Per-project quotas (single GCP project hwork-workspace-lifecycle shared across all three SAs):

API	Per-minute (per project)	Per-day (per project)
admin.directory.users.insert	2,400 / min	86,400 / day
admin.directory.users.update	2,400 / min	86,400 / day
admin.directory.users.delete (we use suspend, not delete)	2,400 / min	86,400 / day
gmail.users.messages.send	250 quota units / user / sec	— (per user, not per project)

Headroom calculation (worst case):

• Worst-case onboarding burst: 50 Specialist × Client pairs in one day = 50 users.insert calls. Against the 86,400/day per-project cap, this is 0.06% utilization. Per-minute peak (assume all 50 in 1 min) is 50 / 2,400 = 2.1%.
• Steady state: assume 200 active pairs, ~5% turnover/mo = 10 inserts + 10 suspends/mo. Negligible.
• Gmail send: per-user, not per-project. Each (Specialist × Client) mailbox is its own user, so the 250 qu/sec ceiling applies per-pair, not aggregated. Even a bulk-send Specialist will not hit it under normal review workflows.

Verdict: We're orders of magnitude under quota. No need for batching, exponential backoff with high retry counts, or quota uplifts. Standard exponential backoff on 429/503 is enough.

---

Per-org provisioning flag

The organizations.gsuite_provisioning_enabled column controls whether GSuite user provisioning fires for a given org. This is a cost-control mechanism for non-production environments.

Column value	Semantics
NULL	Use env default: production = enabled, dev/staging = disabled
true	Force-enable for this org regardless of environment
false	Force-disable for this org regardless of environment

Resolution logic (see isGsuiteProvisioningEnabledForOrg() in api/src/workspace-lifecycle/gsuite-provisioning-policy.ts):

1. Explicit true or false on the org → use that value.
2. NULL → check APP_ENV: only "production" returns true.

SuperAdmin override: Use PATCH /admin/orgs/:orgId/gsuite-provisioning with body { "enabled": true | false | null } to flip the flag for a specific org. Useful for enabling provisioning on a test org in dev without touching the environment default.

Env default rationale: Eusden direction: "control costs on dev so make sure only certain orgs have an org-level flag to have provisioned email access (default false on dev and staging, true on prod), that superadmin can edit."

---

Lifecycle hooks (sub-issue mapping)

Hook	Trigger	Sub-issue	Description
Provision	org_specialist_assignments INSERT	#680	Admin SDK users.insert against {firstname}.{slug}@{env-domain}. Set initial password (rotated immediately by token service), force-no-password-change flag, set display name. Gated by per-org provisioning flag (see above).
OAuth scoping	post-provision	#681	Generate per-pair OAuth client, persist scoped refresh token in integration_credentials (per-org row, integration_type='gmail_per_specialist', encrypted).
Suspend	org_specialist_assignments DELETE	#682	Admin SDK users.update setting suspended=true. See "Suspension lifecycle" section for retention policy.
Token refresh	scheduled, per-row	#683	Cron-driven refresh of per-pair Gmail refresh tokens. Alert via Slack #agents-complaints if refresh fails or expires within 48h.
Reconciliation	scheduled, nightly	#684	Diff org_specialist_assignments (assignments) vs Workspace tenant users (per domain). Repair or alert on drift.
Lower-env reaping	scheduled, weekly	explicit requirement	Reap (delete) any Workspace user in @h852.work / @h853.work suspended for >7 days. Bounds dev/staging seat burn.

---

Suspension lifecycle

When an org_specialist_assignments row is deleted (Specialist removed from Client, or Client org deactivated), the corresponding Workspace user is suspended, not deleted. This section defines retention, reaping, and reactivation policy per #682.

Retention policy (environment-specific)

Environment	Retention	Rationale
Production (h.work)	2 years from suspension date	Audit retention + legal hold requirements. Manual purge only (see below).
Staging (h853.work)	7 days from suspension date, then automatic reap (delete)	Bounds seat burn; no audit retention requirement for non-prod.
Dev (h852.work)	7 days from suspension date, then automatic reap (delete)	Bounds seat burn; no audit retention requirement for non-prod.

Reaping mechanism (dev/staging only): A weekly cron job queries all Workspace users in @h852.work and @h853.work with suspended=true and suspendedAt < now() - 7 days, then calls Admin SDK users.delete for each. Seat count drops immediately on delete. Prod users are NEVER auto-reaped.

Reactivation (assignment recreated)

If a Specialist is re-assigned to the same Client after suspension (same {firstname}.{client-slug}@{env-domain} address):

1. Check if Workspace user still exists (not yet reaped):
   • If user exists and suspended=true: call Admin SDK users.update setting suspended=false. The same user, address, Gmail history, and OAuth grants are restored intact.
   • If user was reaped (deleted): provision as new (same address, fresh user, no history carryover).
2. Reactivation preserves conversation history intentionally. When Specialist Alice hands off Client Acme to Specialist Bob, Alice's user (alice.acme@h.work) suspends and Bob's user (bob.acme@h.work) provisions. When Alice returns months later, her alice.acme@h.work reactivates with her original Gmail thread history intact (if still within the 2-year prod retention window). This is by design — handoff does NOT merge histories; each Specialist sees only their own threads.

Manual purge (prod only, admin-tool-only)

Deleting a Workspace user in prod (permanent, irreversible, loses all Gmail/Drive/Calendar history) requires:

1. Admin Console manual action by a Workspace super-admin (Emanuele or delegated DevOps), OR
2. Purpose-built admin tool (future, file follow-up issue #XXX [P3][admin] Workspace user purge tool for expired prod suspensions). The tool must gate on: (a) user is suspended, (b) suspension date > 2 years ago, (c) no active legal hold tag.

The platform's normal suspension hook (#682) NEVER calls users.delete on prod users — only users.update setting suspended=true. This prevents accidental permanent deletion.

Cross-human handoff and conversation history

When Specialist A hands off Client X to Specialist B:

• Specialist A's user (a.x@h.work) suspends (not deleted).
• Specialist B's user (b.x@h.work) provisions as a new, distinct user.
• No Gmail history is shared between A and B. Each sees only their own sent/received threads.
• Intentional isolation: even though both are "the Specialist on Client X," they operate as distinct mailbox identities. The Client's external correspondents see two different From: addresses over time.
• If Specialist A returns to Client X later (re-assignment), A's original user reactivates (if within retention window), restoring A's own conversation history.

This model prioritizes per-Specialist audit trail and no ambient access to another Specialist's threads over continuity-of-mailbox-identity.

---

Billing

• SKU: Google Workspace Business Standard ($12/seat/mo). Confirmed 2026-06-02 by Eusden. Rationale: we need Gmail + Calendar exposed via DWD; Business Standard is the lowest SKU that ships both with the security envelope (2FA, mobile management, basic eDiscovery via Vault is not required). Business Plus / Enterprise add features (advanced eDiscovery, DLP, S/MIME, Vault retention) that our tenancy model does not consume — Specialist mailboxes forward through Gmail and conversation retention lives in our application DB.
• Pricing model: per-user-per-month. Seats pool across the three domains, so dev/staging seats count against the same subscription as prod.
• Projected steady-state seat burn (200 prod pairs, 50 staging pairs, 50 dev pairs): ~300 seats. At $12/seat/mo (Business Standard) = ~$3,600/mo.
• Cost framing: seat cost is a small-margin line item against the per-Specialist revenue model ($3,000+/Specialist) — this is a provisioning-correctness lever, not a margin lever. Don't optimize the seat count; do optimize for not leaking seats.
• Alert: Set a Workspace billing alert at 400 seats as an anomaly/provisioning-bug detector (≈33% over projected steady state). Threshold is approximate — the value of the alert is detecting a runaway provisioning loop or missed reaping job within a billing cycle, not enforcing a hard cost ceiling. Tune as actual seat usage stabilizes.
• Lower-env reaping policy (see "Suspension lifecycle" section below): weekly reap of suspended workspace users in dev/staging to keep seat count bounded by active testing, not historical pairs.

---

Support runbook — incident scenarios

Incident	Owner	Procedure
Workspace user locked / cannot send	DevOps (Emanuele)	Admin Console → Users → unlock. Investigate cause; if abuse-detected, file #agent-complaints.
MX failure on h852.work / h853.work	DevOps	Confirm DNS in Cloudflare → verify Admin Console domain status → contact Google support if tenant-side.
SA key compromise	DevOps + Paul	Revoke key in GCP Console (iam.serviceAccounts.keys.delete), rotate via #551 encrypted-storage flow, update integration_credentials row, invalidate all in-flight Workspace operations for that env.
DWD grant revoked accidentally	Eusden or DevOps	Re-add DWD entry in Admin Console (Client ID + scope list above). Workspace operations resume on next token refresh (~5 min).
Per-pair refresh token revoked by Workspace	App-layer (#683)	Token refresh service alerts; on next inbound Gmail attempt for that pair, re-auth flow kicks via per-pair OAuth client.
Billing alert fires (>400 seats)	Paul + Eusden	Audit org_specialist_assignments vs Workspace user count. Reap orphans. Investigate provisioning hook for double-creates.

---

Open follow-ups (not blocking spike close)

1. #XXX WIF migration. Eliminate SA JSON keys entirely; runtime authenticates to GCP via OIDC. P2, file post-spike.
2. #XXX Workspace audit log → SIEM. Forward Workspace audit log to whatever audit sink ends up canonical. Tracking-issue only; not on critical path.
3. #XXX OAuth client per (Specialist × Client) — see #681. Already filed.
4. #XXX [P3][admin] Workspace user purge tool for expired prod suspensions. Purpose-built admin tool for manual prod purge (see "Suspension lifecycle" section). Gated on 2-year retention + no legal hold. P3, file post-spike.

---

Spike acceptance — closing checklist

• Workspace tenant provisioned (h.work primary + h852.work dev + h853.work staging as secondary domains, DKIM + MX validated).
• Auth model decided (DWD + 3 SAs, per-env delegate, app-code domain enforcement).
• Admin SDK quota / rate-limit headroom documented (above).
• Billing SKU + alert plan confirmed (Business Standard, ~$3,600/mo at 300 seats, 400-seat anomaly alert). Eusden sign-off 2026-06-02.
• Support escalation runbook (above).
• Service-account env-var / credential-row schema documented (above).
• Placeholder integration_credentials rows — deferred to the consuming tickets (#680 provisioning, #681 OAuth scoping). Originally scaffolded as migration 1751500000001-WorkspaceLifecycleCredsPlaceholder but pulled before ship: schema has org_id NOT NULL and there is no clean org context at ADR-ship time. Per-org rows get created by the provisioning code in #680 when there's an actual org_specialist_assignments row driving them.
• Actual SA JSON keys dropped into integration_credentials (deferred to vault-access agent — flips metadata.placeholder to false).
• End-to-end smoke (provision + suspend a test user on h852.work) — deferred to #680.

Closing #679 on merge of this ADR. #680 / #681 / #683 unblock; #682 / #684 stay blocked until #680 / #681 land (correct sequencing).