Verification Recap: Gsuite Disabled-Mode Graceful Fallback (#1179)

Scope: ADR-0002 Phase 2 OSA Gsuite lifecycle — graceful degradation when Gsuite is not configured (dev/test without GSUITE_SERVICE_ACCOUNT_KEY_JSON, or any env where Workspace Admin DWD has not been provisioned yet).

Date: 2026-06-22 · Issue: #1179 · Operational entry: gsuite-disabled-mode.md

The contract

Never crash, never silently write a half-provisioned OSA row, always surface a clear "disabled" indicator to the operator.

Pre-req env state assumed by these checks

• GSUITE_SERVICE_ACCOUNT_KEY_JSON unset
• GSUITE_ADMIN_USER_EMAIL unset (issue text says GSUITE_ADMIN_EMAIL; the live canonical var is GSUITE_ADMIN_USER_EMAIL, alias GOOGLE_ADMIN_EMAIL)
• MASTER_ENCRYPTION_KEY set (encryption util is shared infra, not Gsuite-gated)

How "disabled" is determined

Two independent layers gate Gsuite work — both verified:

1. Per-org policy gate (isGsuiteProvisioningEnabledForOrg, gsuite-provisioning-policy.ts) decides whether to enqueue a provisioning job. NULL flag → enabled only in production; dev/staging default off. Applied by the producer in organizations.service.ts (_enqueueWorkspaceProvisioning).
2. Infrastructure gate (GoogleAdminClient.isEnabled()) reflects whether a service-account credential actually loaded. This is the layer the issue targets, and it is now enforced defensively in the processor so a job that slips past the policy gate (e.g. the explicit AM "Provision" action, which overrides the env default) still degrades cleanly.

---

Results

Legend: ✅ verified pass · 🛠 gap found → fixed in this PR · ➖ not applicable

Startup invariants

Bullet	Status	Evidence
API boots (/health 200) with all GSUITE_* unset	✅	GoogleAdminClient.onModuleInit returns early when creds absent; @googleapis/admin is lazy-required only when creds exist, so the module graph loads without it.
No Sentry/error log at boot mentioning Gsuite (warn OK, error not)	✅	All-unset path logs a single warn, never error or Sentry. Asserted by google-admin.client.spec.ts → "logs a warn (not error) once at boot when fully disabled".
Clear "Gsuite disabled — no service account configured" log once at boot	🛠	Message reworded to exactly: Gsuite disabled — no service account configured (… unset …). Workspace provisioning is skipped; lifecycle jobs complete as no-ops.

Provisioning path

Bullet	Status	Evidence
workspace-provisioning job for a fresh OSA completes (not failed) with a clear "skipped: gsuite disabled" terminal log	🛠	This was the core gap. The processor had no disabled-mode guard; the AM-triggered provisionSpecialistAssignment overrides the env-default gate and can enqueue a CREATE job in a creds-less env, where createUser → getDirectory() threw → job FAILED (spurious Sentry + AM "provisioning failed" alert). Added a guard at the top of WorkspaceProvisioningProcessor.process(). Only the intentional no-credentials disable is a no-op — it logs Job … skipped: gsuite disabled (no-credentials) — OSA … left unprovisioned and returns (job completes). A misconfiguration (invalid-credentials / config-incomplete) instead throws so the job fails → retries → onFailed alerts — see the credential-rot note below.
email_alias may be populated; gsuite_user_id / gsuite_provisioned_at stay NULL	✅	The short-circuit returns before any osaRepo.update, so no Gsuite columns are written. The producer may have set email_alias (a pure string compute) — allowed. Asserted: "CREATE completes as a no-op … and writes nothing".
oauth_refresh_token_encrypted stays NULL	✅	No token path is touched in disabled mode; nothing writes that column during provisioning.
Direct provisionUser(osaId) returns a typed "disabled" result, not a raw throw	✅ (adapted)	There is no GsuiteService.provisionUser in the codebase; the equivalent surface is WorkspaceProvisioningService.createUser. Rather than change its return type, the predicate isGsuiteEnabled() / getGsuiteDisabledReason() is the typed disabled signal callers consult; createUser still throws a clear, typed message if invoked directly while disabled.

Token read path

Bullet	Status	Evidence
OAuthTokenService for an OSA with oauth_refresh_token_encrypted = NULL returns a clean "no token" result / specific exception type, not a generic 500	🛠	getAccessToken previously threw a bare Error for the no-token case. Introduced OAuthTokenUnavailableError (carries osaId); thrown for both the never-granted and revoked cases. Callers can now catch (e instanceof OAuthTokenUnavailableError) and no-op. The sole HTTP caller (channels.controller.ts Gmail webhook) already wraps the call in try/catch → controlled 500 with a clear log (no pod crash).
Token rotation cron / job is a no-op for OSAs without a token; no error noise	➖	No OAuth-token rotation cron exists in the codebase (grep for rotat/@Cron/BullMQ repeat over the OAuth service found none — token refresh is lazy, on-demand inside getAccessToken). N/A; documented so a future rotation job is built to catch OAuthTokenUnavailableError.

UI rendering

Bullet	Status	Evidence
OSA detail panel shows email_alias, with gsuite_user_id / gsuite_provisioned_at as "Not provisioned" — not blank/undefined/JS error	✅	frontend/.../specialists/[assignmentId]/page.tsx: {osa.gsuiteUserId ?? "Not provisioned"} and formatIso(value) returns "—" for null. StatusBadge renders Pending when unprovisioned. No code change needed.
AM Setup Wizard step 2 shows a "Gsuite integration disabled in this environment" banner	⚠️ deferred	The wizard does not render a dedicated disabled-mode banner today; the per-org GSuite provisioning toggle on the client detail page is the operator control. Cosmetic-only and out of the backend contract scope — captured here as a follow-up, no crash today.
Specialist detail page hides/grays Gmail-Send / Drive-fetch when the OSA has no Gsuite creds	⚠️ deferred	Same: actions are not conditionally disabled in disabled mode. They route through the now-hardened backend, which fails clean rather than crashing. Follow-up, not a contract crash.

Negative scenarios (regression guard)

Bullet	Status	Evidence
Invalid GSUITE_SERVICE_ACCOUNT_KEY_JSON → boot still succeeds; typed "credentials invalid" error distinct from "disabled"	🛠	Was a crash: onModuleInit called parseServiceAccountKey unguarded, so malformed JSON threw during bootstrap (boot fail / crash-loop under active-active). Now caught → stays disabled with disabledReason = "invalid-credentials", logs a loud (error-level, non-fatal) line; getDirectory() throws … credentials are invalid …. The processor treats this distinctly from "disabled" — a misconfig fails the job loudly (Sentry + AM alert) instead of no-op'ing (see "Credential-rot safety" below). Asserted: "does NOT crash boot when the key JSON is not parseable" + processor "misconfiguration → fail loudly".
Only GSUITE_SERVICE_ACCOUNT_KEY_JSON set, admin email unset → boot succeeds; "config incomplete"	🛠	Already booted (early return), but returned the generic "disabled" reason. Now distinguished as disabledReason = "config-incomplete" with its own error message; getDirectory() throws … configuration is incomplete ….

---

Credential-rot safety (PR #3000 review)

The disabled guard intentionally splits two cases rather than treating all "disabled" states the same:

• no-credentials — an intentional absence (dev/test, or prod before DWD is provisioned). Jobs no-op and complete. Silent success is correct here.
• invalid-credentials / config-incomplete — a misconfiguration, e.g. a service-account secret that rotated incorrectly in production. Jobs throw → retry → onFailed fires Sentry + an AM notification. This is essential for the security-sensitive suspend and archive ops: silently no-op'ing them under credential rot would leave a Specialist with Google Workspace access they should have lost, with no per-job signal. Boot also logs an error for these states. (Also satisfies the #1179 negative-scenario contract, which asks for a distinct, surfaced "credentials invalid" error — not a silent skip.)

Code changes in this PR

File	Change
api/src/workspace-lifecycle/google-admin.client.ts	Boot never throws on missing/invalid/incomplete creds; GsuiteDisabledReason classification (no-credentials / config-incomplete / invalid-credentials / null); getDisabledReason(); reworded disabled log; reason-aware notConfiguredError().
api/src/workspace-lifecycle/workspace-provisioning.service.ts	isGsuiteEnabled() + getGsuiteDisabledReason() passthroughs.
api/src/workspace-lifecycle/workspace-provisioning.processor.ts	Disabled-mode guard in process() — no-credentials jobs complete as no-ops ("skipped: gsuite disabled"); invalid-credentials / config-incomplete jobs throw so the misconfig surfaces via onFailed (Sentry + AM alert).
api/src/workspace-lifecycle/oauth-token.service.ts	OAuthTokenUnavailableError typed exception for the no-token / revoked case.
specs	New disabled-mode coverage across the four units + e2e mock updated.

Alignment with "What this is NOT"

• Not a happy-path test — happy paths remain covered by the existing specs.
• Not a fix for the agent /audit/log 401 — untouched.
• Not setting up real Gsuite creds in dev — none added; this hardens the absence path.

Follow-ups (non-blocking, no crash today)

1. AM Setup Wizard step 2 disabled-mode banner (UI cosmetic).
2. Conditionally gray out Specialist Gmail-Send / Drive-fetch actions in disabled mode (UI cosmetic).

Neither is a silent crash or partial-write; both are surfaced cleanly by the hardened backend. Per the issue's acceptance, any silent crash/partial-write would be filed separately — none were found; the two boot/processor gaps above were in-scope contract violations and are fixed here.