Skip to main content

humanwork — MVP/P0 UAT Runbook (v2.5, dev-only)

Audience: human testers (no engineering background) + agents. Organized by role × P0 user story. Assumes the backend is functional — this is top-level E2E.

Source map: user-stories.md P0 entries, mvp-launch-checklist.md. Companion: the runbook is canonically authored in Notion (linked from the task hub) and this repo file is the markdown source of truth — Notion is a presentation copy.

Test environment: https://dev.h852.work (frontend), https://api-dev-XXXX.up.railway.app (API). DEMO_MODE is on — log in via the role buttons under the email field on /login; no password.

0. How to use this runbook​

  • Each story has two parts: Human walkthrough (step-by-step) and Agent recipe (technical specifics).
  • Walk a story top to bottom. Mark PASS / FAIL at the end. Capture a screenshot on FAIL.
  • Naive assumption: if the backend is broken, the screen will reflect that — file a bug, move to the next story. Don't debug.
  • Stop after the first FAIL inside a role; that section is GO/NO-GO for the role.

Logging in (dev): visit https://dev.h852.work/login. Below the email field, the DEBUG MODE panel exposes four buttons: Login as Client, Login as Expert, Login as Account manager, Login as Superadmin. No password.

The client onboarding wizard is a separate 3-step layout (Workspace setup → Your h.work team → Done) and renders identically regardless of token state. (Distinct from the AM "Set up new client" wizard described in SA-C1, which is 4 steps.) If the invite token is invalid or expired, the wizard shows a friendly error in red — NOT a stack trace.

Role: Client​

A person at the customer organization (Kaito's admin in the launch scenario) who logs into the portal to message their Specialist and manage their workspace.

C-O1 — Accept invite, set password, log in​

"As a client admin, I want to accept an email invite, set my password, and log into the portal so that I can start configuring my account."

Human walkthrough​

  1. Open the invite email you received from your Account Manager.
  2. Click the "Activate your workspace" link. The portal opens at the onboarding wizard (https://dev.h852.work/onboarding/start?token=…).
  3. Enter the 6-digit OTP code from the OTP email (sent automatically).
  4. Set a password (minimum 12 characters).
  5. Walk three steps: Workspace Setup → Your h.work Team → Done.
  6. Click "Continue to your Workspace" → you land at https://dev.h852.work/client/chat already logged in.

PASS criteria (DEMO_MODE smoke): you reach /client/chat without seeing a login screen. The top-left header shows your org's slug (not the literal text "h.work"). The Specialist's first name (e.g. "Alex") is visible.

PASS criteria (real invite path): see C-O0 — Client confirmation flow for the full 7-item checklist (org=active, slug immutable, trial started, /client/chat access, AM notification, link invalidated, welcome summary email). DEMO_MODE short-circuits all of these.

FAIL signals: raw JSON error on screen · stack trace text · invitation-expired page when token is fresh · landing back at /login.

Agent recipe (technical)
  • Demo path (dev only): POST /auth/demo-login {"role":"client"} → JWT cookie set; navigate to /client/chat.
  • Real invite path: POST /auth/send-invite-otp then POST /auth/verify-invite-otp with token from INVITE_URL. Assert response contains JWT with platformRole='org_admin' and orgMemberships[0].orgRole in {'owner','admin'}.
  • Anchors after success: snapshot contains "Your Threads", "Your Specialists". Must NOT contain "Direct Messages" or "Your Experts".
  • Expired-token negative: load an artificially expired INVITE_URL → snapshot must contain "This invitation has expired" (NOT raw "statusCode":410).

C-O2 — Connect Slack workspace​

"As a client admin, I want to connect my Slack workspace to my Specialist so that my team can interact with the Specialist directly in Slack."

Human walkthrough​

  1. Log in as Client. From the chat page, click the gear icon → Channels (or go directly to https://dev.h852.work/client/settings/channels).
  2. Find the Slack row in the channels table.
  3. Click "Connect Slack" → you're redirected to Slack's OAuth consent screen.
  4. Approve the install for your Slack workspace.
  5. You're returned to the channels page; Slack now shows status Connected.

PASS criteria: Slack row shows "Connected" with a green dot after the OAuth roundtrip. Send a test message in the connected Slack workspace to your Specialist — it should appear in your /workspace/queue (Expert view).

Slack is Available (unblocked 2026-05-26 via #816 Slack Connect OAuth flow; per-org signing_secret rotation landed 2026-06-01 via #1219). Earlier "Coming Soon / BLK-004 / #120" status is now closed — if a fresh env still shows "Coming Soon", check that the SLACK_* env vars are wired (see docs/ops/deploy.md).

Agent recipe (technical)
  • Inbound endpoint: POST /channels/slack/events. HMAC-SHA256 verify via SLACK_SIGNING_SECRET + 5-min replay window.
  • Outbound: ChannelDispatcherService (PR #567) posts to Slack on Expert release. heldForReview=true blocks pre-release outbound.
  • Negative test: stale-timestamp event → 401. Forged-signature event → 401.
  • Feature-flag check: GET /admin/config?key=slack-configured returns false if creds aren't set — section becomes BLOCKED.

C-O3 — Connect inbound email​

"As a client admin, I want to connect an inbound email address to my Specialist so that emails to that address are handled by the Specialist automatically."

Human walkthrough​

  1. Log in as Client → https://dev.h852.work/client/settings/channels.
  2. Note your Specialist's email address shown in the Email row (e.g. alex@[your-slug].h852.work).
  3. The Email row should already show "Connected" — email is wired automatically when the org is created.
  4. Send a test email from your corporate email to the Specialist's address.
  5. Watch the chat page (or your Expert's queue) — the message should appear as a new conversation within 30 seconds, with an "Email" channel badge.

PASS criteria: Email row reads "Connected"; sending an email creates a new conversation visible in /client/chat or /workspace/queue within 30 s.

FAIL: email is sent but no conversation appears within 60 s. Check the sender domain is on the org's corporate-email whitelist (under settings) — unknown senders get a one-shot "registered users only" reply.

Agent recipe (technical)
  • Inbound (primary, 2026-06-05+): Gmail API polling — cron job (*/30 * * * * *) calls gmail.users.history.list + messages.get against the Specialist's gsuite mailbox via DWD impersonation (PR #1722, commit d8333558). Messages are enqueued to BullMQ for processing.
  • Inbound (fallback during 2-week soak): Cloudflare email worker → POST /channels/email/inbound with X-Webhook-Signature HMAC-SHA256 of body using CLOUDFLARE_EMAIL_WEBHOOK_SECRET. Retired by #1441 after the soak.
  • Sender authorization: org_memberships membership is primary check; email_whitelists only consulted as fallback.
  • Subdomain normalization: bob@dev.h852.work → bob@h852.work for DB lookup.
  • Outbound (Specialist replies): Gmail API users.messages.send as the impersonated Specialist (DWD) via single Google Workspace service account. From = Specialist's gsuite address. Resend is not used for Specialist conversation channels.
  • Outbound (transactional): Resend remains for invites/onboarding/transactional via email/email.service.ts. DKIM signed.
  • Threading: In-Reply-To preserved; fallback to most-recent open conversation for (from, org) pair.

C-W1 — Send a message to your Specialist​

"As a client, I want to send a message to my Specialist in the portal chat so I can request work or ask questions."

Human walkthrough​

  1. Log in as Client → land at https://dev.h852.work/client/chat.
  2. Click "+ New thread" in the left sidebar to open the composer. (/client/chat lands on a "Today's briefing" overview; the composer is inside a thread, not on the landing page.)
  3. If you already have threads, you can alternatively click an existing thread to open it.
  4. Type a message in the composer at the bottom (e.g. "Hi Alex, can you help with our campaign analytics?") and press Enter or click Send.

PASS criteria (immediate): your message appears immediately in the thread. The thread enters status PENDING (a held-draft state) and a holding indicator like "[Specialist first name] will be with you shortly" appears (e.g. "Alex will be with you shortly" — exact text depends on the org's assigned Specialist persona). On dev's Acme Financial seed the assigned Specialist is "Eleanora Ogilvie", so you'll see "Eleanora will be with you shortly" — this is the Specialist persona, NOT the Expert seed user (which happens to share the first name on this seed).

PASS criteria (end-to-end, requires Expert release): an Expert logged into /workspace/queue picks up the held draft and releases it — a reply attributed to your Specialist (e.g. "Alex") then arrives in the thread within 30 s of release. The reply is NOT labelled "agent" or "AI" or "bot" — it's the Specialist persona.

Why two PASS criteria? Default org config is autoReplyMode='none' and autoRespondThreshold=101, which holds every draft for Expert review. A Specialist reply does NOT arrive autonomously — it requires an Expert in the loop. On cold dev with no Expert active, only the immediate criteria can be observed.

FAIL: greeting like "Hi Alex" is rejected with a prompt-injection error · message stuck in "sending" state · no holding indicator after 30 s · the held draft becomes visible to the client (issues #347, #350).

Agent recipe (technical)
  • Endpoint: POST /conversations/:id/messages with JWT.
  • Default org config: autoReplyMode='none' and autoRespondThreshold=101. In this state, the AI draft is heldForReview=true and an Expert must send.
  • Real-time path: Socket.io agent_message_sent event; 8-second polling fallback.
  • Held-draft invisibility check: as client, GET /conversations/:id + /conversations/search?q=… must NOT surface any heldForReview=true messages (issues #347, #350).
  • E2E loop validation: pair with Expert flow E-R3 → E-R6 (release) to observe the full client-visible reply.

C-W2 — Receive your Specialist's response in real time​

"...so the experience feels like messaging a colleague rather than submitting a support ticket."

Human walkthrough​

  1. Continue from C-W1, or open any thread with the Specialist.
  2. Send a message. Do NOT refresh the page.
  3. Once an Expert releases the held draft (see C-W1's end-to-end criteria), the reply should appear in the thread without you doing anything (typing indicator may show briefly).

PASS criteria: Specialist reply appears within 30 s of Expert release with no page refresh. Threads list (left sidebar) also updates the snippet/timestamp.

FAIL: you have to refresh to see the reply (real-time push broken) · reply takes > 60 s after Expert release.

Agent recipe (technical)
  • Verify: Network tab shows an open Socket.io connection. After Expert release, an agent_message_sent event fires within 30 s.
  • Polling fallback: GET /conversations/:id called every ~8 s; new message appears in next poll cycle.

C-W3 — Create a ticket for a specific work request​

"As a client, I want to create a ticket for a specific, trackable work request (e.g. "update all listings for SKU X") so I can follow progress and close the task when done."

Human walkthrough​

  1. From /client/chat, click "+ New thread" in the left sidebar.
  2. Type the request as a single message (e.g. "Please reconcile our October Amazon orders against Shopify and flag any mismatches"). Send.
  3. The thread appears in the sidebar with a snippet of your message + timestamp + green active dot.

PASS criteria: new thread appears in the sidebar with the correct Specialist name. You can leave the page, come back, and find the thread waiting for you.

Agent recipe (technical)
  • Endpoint: POST /conversations → 201; sidebar updates via conversation_created Socket.io event.
  • Persistence: last-viewed thread saved in localStorage['hw_last_thread_'+jwt.sub] → restored on next visit.

Client-portal AC-PC mapping (added 2026-05-25)​

Source of truth: docs/acceptance/CLIENT_PORTAL_CHAT.md. This section walks the 12 AC-PC entries with dev-observed status. Statuses on the source doc and this table are kept in sync.

ACSpec criterion (one line)Status (2026-05-25 on dev)Evidence / pointer
AC-PC-01Top-left sidebar avatar shows org identity, not hardcoded "H"OPEN BUG → FIXED in PR #719PortalChat.tsx:288 was literal H; patched to {orgName?.charAt(0)?.toUpperCase() ?? "H"}. Avatar text now derives from orgName (already shown in adjacent label). Dev verification: Acme Financial seed shows A after the fix.
AC-PC-02Specialist first name + avatar in sidebar / chat header / Workforce cardACTIVE — RIGHT-PANEL PROFILEPersona "Eleanora" surfaces in the /client/chat dashboard right panel. Full profile data comes from GET /client/specialists; multiple assigned Specialists use carousel dots/arrows, while a single Specialist renders without carousel controls.
AC-PC-03"Message" button on Specialist card creates a threadOBSOLETE — REPLACEDZero >Message< strings in frontend/src/components/client/. Specialist discovery is embedded in the /client/chat right panel; thread creation stays on sidebar + New thread (AC-PC-06). Do not re-add the standalone /client/specialist discovery page.
AC-PC-04Type+send → optimistic update; reply within 3s if availableACTIVE — PASS-IMMEDIATESend verified on dev. End-to-end reply gated by Expert release per autoRespondThreshold=101 default (covered in C-W1's two-tier PASS criteria).
AC-PC-05Labels: "Your Threads" / "Your Specialists" (not "Direct Messages" / "Your Experts")OBSOLETE — REPLACEDZero matches for any of the four label strings. Sidebar collapsed two-section layout into a flat thread list; discovery + approvals moved to top-level routes.
AC-PC-06Sidebar + New Thread creates a conversationACTIVE — PASSClick increased thread count, composer rendered. Canonical thread-creation path.
AC-PC-07Thread row metadata: name, snippet, timestamp, active dotACTIVE — PASSReprobe 2026-05-25 found 14 rows on dev. Each row's structure: [avatar initial] [thread name] [relative timestamp] [status badge] — example: `E
AC-PC-08Chat header: name+avatar, online status, status badgeACTIVE — PASS-PARTIALReprobe 2026-05-25 found the chat header via button[aria-label='Back to briefing'] parent div. Header content on dev: thread subject + status badge (Hi + NEEDS INPUT). The AC's specifically-named specialist name/avatar at header level is drift — on dev the specialist is shown per-message rather than in the chat header. Not a regression (the spec's "Online · Support Specialist" line was removed in the redesign).
AC-PC-12Inline-edit thread title (P2)ACTIVE — PASSReprobe 2026-05-25: 14 .hw-thread-rename-btn buttons exist (one per row, aria-label="Rename thread: <name>"), opacity: 0 until row :hover (touch-friendly hide). Force-click triggers the rename input field (aria-label="Rename thread"). Saves via localStorage[hw_threadNames] + UI commit. Working as specced.
AC-PC-09Empty state has clear CTA, no dead endACTIVE — PASS-VIA-DIFFERENT-CTA+ New thread reachable without existing threads. AC's specific "Message" button is obsolete (AC-PC-03), no-dead-end PASS criterion is met.
AC-PC-10Mobile responsive < 640px (sidebar collapses with thread open)ACTIVE — PASSAt 480px viewport with thread open, sidebar width=0/height=0. Back-button / swipe-return not exercised.
AC-PC-11Last-viewed thread restored from localStorageOBSOLETE — REPLACEDZero matches for hw_last_thread_*. Restoration is via /client/chat/[id] URL routing — URL-as-state is strictly better than localStorage.

PASS criteria for this section overall: all ACTIVE rows resolve to PASS or PASS-PARTIAL with documented evidence on dev. OBSOLETE — REPLACED rows do not count for/against the runbook PASS; they're historical reference. Re-test rhythm: walk this table whenever PortalChat.tsx changes materially.

Agent recipe (probe selectors for future walks)
  • AC-PC-01: await page.locator("aside > button > div").first.innerText() → first character should match orgName.charAt(0).toUpperCase().
  • AC-PC-02 / 07 / 08: probe aside [class*='ThreadListItem'] for row metadata; probe chat-pane top region (not header) for header info.
  • AC-PC-04: page.locator("textarea").first.fill(...) → press Enter; verify message appears in DOM within 1s.
  • AC-PC-06: count aside button matching + New thread; click and verify composer appears.
  • AC-PC-10: page.set_viewport_size({"width": 480, "height": 900}); verify sidebar getBoundingClientRect().width === 0 when a thread is open.
  • AC-PC-11: check URL pattern — /client/chat/<conv-id> after navigating to a thread.

C-O0 — Client confirmation flow (Phase 2, full)​

"As a client admin invited by an AM, I want to complete the onboarding wizard — verify my account, review the AM-entered company info, meet my Specialist, and activate the workspace — so the org goes live."

Test type: NEEDS-LIVE-TOKEN. DEMO_MODE bypasses this entire flow. To exercise it, an AM must first run SA-C1 end-to-end with paul@humanity.org as the admin email, then a human or this agent reads the OTP from that inbox.

Human walkthrough​

  1. Step 2.1 — Account Verification. Open the invite email, click the activation link. You land at /onboarding/start?token=…. Enter the 6-digit OTP from a second OTP email. Set a password (≥12 chars). Fill profile: name (required), title (optional), phone (optional). Continue.
  2. Step 2.2 — Review & Confirm Company Info. AM-entered data is pre-populated. You can edit: company legal name, slug (this is your one chance — slug becomes IMMUTABLE after confirm), industry, website, country, timezone, admin email. You cannot edit the primary corporate email domain (locked by AM at Phase 1). Confirm.
  3. Step 2.3 — Meet Your Specialist. Read-only card(s) for each assigned Specialist: name, avatar, email ({firstname}@{HWORK_DOMAIN}), skill tags, bio. The fact that the Specialist is an AI + Expert composition is NOT visible — you only see the persona.
  4. Step 2.4 — Onboarding Complete. Click "Complete". Org status flips pending_client_confirmation → active. Trial clock starts (default 30 days). You're redirected to /client/chat. The AM gets an email notification; you get a welcome-summary email. The original invite token is marked accepted and the link can no longer be used.

PASS criteria (7-item checklist, from mvp-launch-checklist.md):

  1. Org record status = active.
  2. Slug permanently reserved and immutable (subsequent edit attempts fail with a slug_locked 409).
  3. Trial period started (organizations.trialStartedAt = now(), default 30-day window).
  4. Client Admin can log in fresh and lands on /client/chat.
  5. AM receives notification email of onboarding completion.
  6. Onboarding link is invalidated (token.status = accepted); replaying the link shows the expired-invitation page.
  7. Welcome summary email is delivered to the Client Admin.

FAIL signals: slug field on Step 2.2 remains editable after confirm (immutability bug) · status stays pending_client_confirmation after Step 2.4 · /client/chat redirect lands you back on /login · no AM notification email within 60 s · token status stays pending (link still reusable).

Known dev-environment gap (2026-05-25): loading /onboarding/start?token=<expired-or-bogus> renders the 3-step wizard skeleton and the red error "Failed to load company details. Please refresh." with a "Retry" button — NOT the runbook-specified "This invitation has expired" friendly message. The error containment is OK (no stack trace, no raw JSON) but the wording will leave a real user retrying forever. Track as a follow-up bug.

Agent recipe (technical)
  • Endpoints in order: GET /onboarding/start?token=… → POST /auth/verify-invite-otp → POST /auth/set-password → PATCH /onboarding/company (Step 2.2 review-and-edit) → GET /onboarding/specialists → POST /onboarding/complete.
  • Slug immutability: enforced server-side via pessimistic write-lock on organizations.slug at completion time. Race condition: two confirmations on different orgs racing for the same slug; only one wins, the other gets a 409 slug_taken and is forced to pick a new one.
  • JWT after Step 2.1 contains platformRole='org_admin' and orgMemberships[0].orgRole='owner'.
  • Activation side-effects: org status transition pending_client_confirmation → active, trialStartedAt = now(), trialEndsAt = now() + 30d, two outbound emails (welcome summary to client, completion notification to AM), invite-token state pending → accepted.

C-S1 — Billing & Quota page​

"As a client admin, I want to see my plan, trial status, usage, and rate limits so I can monitor adoption and know when to upgrade."

Human walkthrough​

  1. Log in as Client → https://dev.h852.work/client/settings/billing (or click "Billing" in the /client/settings left nav).
  2. The page shows:
    • Subscription summary — per-Specialist subscriptions (since Phase 7 #1016 removed the fixed plan-tier model), each with monthly rate and current usage.
    • Saved payment method — last-4 / brand if a card is on file; otherwise an Add payment method CTA that opens Stripe Checkout (#1275).
    • Lago portal link — one-shot URL to the Lago self-service portal (#1155).
    • Message Volume (last 24h) — sparkline / chart placeholder.

PASS criteria: page renders without error; per-Specialist subscription rows match the OSAs in /ops/clients/[id]; clicking Add payment method opens a Stripe Checkout session and the saved card surfaces after success; clicking the Lago portal link opens the customer portal in a new tab.

FAIL signals: raw "404" or "Failed to load" error · subscription rows empty when OSAs exist · Stripe Checkout returns an error · Lago portal URL is null or 500.

Agent recipe (technical)
  • Endpoint: GET /client/billing/summary returns the Lago-derived per-OSA subscription summary (#1155). The pre-#1016 /billing/summary route and the legacy fixed-plan-tier shape are gone.
  • Add-payment-method CTA target: GET /client/billing/checkout-url (#1275) returns a one-shot Stripe Checkout session URL. Stripe is Lago's PSP; there is no parallel Stripe billing rail.
  • Portal CTA target: GET /client/billing/portal-url (#1155) returns a one-shot Lago portal URL.

C-T1 — Invite a teammate​

"As a client admin, I want to invite a colleague by email so that they can access the portal and interact with the Specialist."

Human walkthrough​

  1. Log in as Client → https://dev.h852.work/client/settings/team.
  2. Click "Invite Teammate" (top right).
  3. Enter the colleague's email + select role (Member / Admin).
  4. Click "Send Invite".
  5. The colleague's email row appears in the table with status Invited.

PASS criteria: row added with "Invited" badge; colleague receives an OTP email within 60 s; clicking the link drops them on the same onboarding wizard.

Agent recipe (technical)
  • Endpoint: POST /admin/members/invite — accessible to platform role org_admin with org-role admin or owner.
  • Invite token: 32-byte, 7-day expiry; bcrypt-hashed 6-digit OTP. Resending extends window.

Client-role edge cases (added 2026-05-25)​

These are the recurring edge cases worth probing whenever the Client onboarding or settings surface changes.

#Edge caseWhere it surfacesExpected behaviorKnown gap
EDGE-1Invitation expiry (7-day token TTL)/onboarding/start?token=<expired>Friendly page: "This invitation has expired. Ask your account manager for a new link." Optionally a "Request a new invite" CTA.BUG (2026-05-25): dev shows "Failed to load company details. Please refresh." instead of the friendly expired message. Tracked as follow-up.
EDGE-2Stale orgs (90+ days in pending_client_confirmation)Backend cron + /ops/clients SA viewOrgs auto-archived (status → deactivated); a stale-marker badge appears in the SA list. AM can manually un-archive.Cron job behavior not directly tested — requires time travel or seeded data.
EDGE-3Slug race conditionTwo clients confirming Step 2.2 simultaneously with the same chosen slugPessimistic write-lock at POST /onboarding/complete. First commit wins; second gets HTTP 409 slug_taken and is forced to pick a new slug.Hard to test deterministically without parallel sessions; covered by API integration tests.
EDGE-4Re-entry CTA for pending orgs/ops/clients SA/AM viewOrgs with status pending_client_confirmation show a "Resend invite" or "Complete Setup" CTA in the row or detail page.Not visible at row-level on dev (2026-05-25 probe). May live on the SA-C3 drill-down page.

For non-engineering testers: if you encounter any of these states on dev, screenshot the page and file the case as EDGE-N in the run summary. These edge cases do NOT need to be exercised on every UAT pass — they're a checklist for product-quality review.

Role: Expert​

The Humanity Protocol team member sitting behind the Specialist persona. Reviews AI drafts, edits, sends as the Specialist, resolves tickets, captures corrections.

E-O1 — Accept invite, create Expert account​

"As a new expert, I want to accept an email invite and create my account so I can access the Expert workspace."

Human walkthrough (dev short-path)​

  1. On dev, the Expert seed user (Eleanora) is preconfigured. Visit https://dev.h852.work/login and click Login as Expert.
  2. You land at https://dev.h852.work/workspace/queue.

In production: an Account Manager invites you via the AM wizard → you receive an email → click → set password → log in.

PASS criteria: you land at /workspace/queue (NOT /login or /client/chat). Role-based redirect works.

Agent recipe (technical)
  • Demo path: POST /auth/demo-login {"role":"expert"} → JWT with platformRole='expert'.
  • Auth redirect: getRoleRedirect() in AuthContext.tsx routes expert → /workspace/queue.

E-R1 — View queue sorted by risk​

"As an expert, I want to see all pending escalations in my queue, sorted by risk level (critical → high → medium → low), so I always work on the most urgent item first."

Human walkthrough​

  1. Log in as Expert → /workspace/queue.
  2. Look at the queue table. Items should be ordered by risk descending.

PASS criteria: CRITICAL rows appear at the top; LOW at the bottom. Each row has a coloured risk badge.

Prerequisite for E-R1..E-R7: the Expert queue is empty on a cold dev environment. To populate it, run client-side C-W1 first — sending a message creates a held draft that lands in the Expert queue within 10 s. If C-W1 has not been run for the current dev seed, E-R1..E-R7 cannot be exercised. (No dedicated seed script exists yet; a npm run seed:expert-queue helper is a logical follow-up if testers find themselves doing this often.)

Agent recipe (technical)
  • Endpoint: GET /expert-queue — server-side sort by risk_level DESC, created_at ASC.
  • Real-time: Socket.io expert_queue_item_added event; 8-second polling fallback.
  • Scoping: ADR-007 expert_access table (replaced org_experts 2026-05-22). Items filtered to orgs the Expert has access to.
  • Seed-data note: no dedicated seed script exists today — to populate the Expert queue on a cold dev seed, walk client-side C-W1 first (this creates a held draft that lands in the Expert queue within 10 s of send). A npm run seed:expert-queue script is a follow-up; track it in a separate ticket if you find yourself doing the C-W1 detour repeatedly.

E-R2 — Filter queue by status​

"As an expert, I want to filter the queue by status so I can focus on items that need my attention."

Human walkthrough​

  1. On /workspace/queue, find the status tabs at the top: "All" / "Pending" / "Awaiting Reply" / "Resolved" / "Snoozed" (five tabs on dev — earlier runbook versions listed only three; that was stale).
  2. Click each tab in turn. The list filters accordingly.

PASS criteria: clicking each of the five tabs filters the list immediately (client-side or single API call). Counters next to each tab match the filtered counts. The "All" tab is the broadest — it removes the status equality filter entirely so items in any state are reachable.

Agent recipe (technical)
  • Query param: GET /expert-queue?status=all|pending|awaiting_client|resolved|snoozed — values exactly match the UI tabs (Awaiting Reply tab maps to awaiting_client). Default when omitted is pending (per expert-queue.service.ts:79).
  • The legacy unread value from earlier runbook versions is NOT accepted by the controller — do not use it.
  • The all sentinel removes the status equality predicate; safeLimit still caps the result set so an unfiltered query can't dump the whole table.
  • Counters: separate /expert-queue/counts endpoint or returned in headers.

E-R3 — View the full customer conversation thread​

"...so I have all the context I need before responding."

Human walkthrough​

  1. From /workspace/queue, click any queue row.
  2. You land on the queue-item detail page. The customer's full thread is visible in chronological order.

PASS criteria: every prior message in the conversation is visible. Channel of origin (Email / WhatsApp / Slack / Web) is shown as a badge.

Agent recipe (technical)
  • Endpoint: GET /conversations/:id?include=messages.
  • Internal-note filter: is_internal=true messages visible to Expert, NOT to client (DB-level filter, conversations.service.ts:386).

E-R4 — See AI suggested response + confidence score​

"...so I can quickly accept, edit, or discard the draft rather than write from scratch."

Human walkthrough​

  1. On the queue-item detail page, look at the right pane.
  2. You should see: the AI's draft reply, a confidence score (0–100), a risk badge, and a one-word escalation reason.

PASS criteria: draft text is non-empty; confidence is a number 0-100; risk badge is one of LOW/MEDIUM/HIGH/CRITICAL; escalation reason is human-readable (e.g. "agent_flagged" or "risk_high").

Agent recipe (technical)
  • Fields in the queue-item response: message.metadata.confidence, message.risk_level, queueItem.escalation_reason.
  • Risk classification: keyword heuristic today (will become LLM-based per features/agentic-tasks.md).

E-R5 — Edit the AI draft before sending​

"...so I can correct factual errors, adjust tone, or add information the AI missed."

Human walkthrough​

  1. On the queue-item detail page, click "Edit" on the draft pane.
  2. The draft becomes an editable textarea.
  3. Make your edits.
  4. Click "Send Edited Response".

PASS criteria: edited text replaces the draft AND a correction row is recorded (used for learning). The client sees the edited version, never the original draft.

Agent recipe (technical)
  • Endpoint: POST /expert-queue/:id/respond with body { text: <edited>, mode: 'edit' }.
  • Correction capture: a type='edit' row is written to corrections table with originalText + editedText delta.

E-R6 — Send the response as the Specialist​

"...so the customer's experience is consistent regardless of which Expert handled the item."

Human walkthrough​

  1. From the queue-item detail page, click "Approve & Send" (verbatim) OR send your edited version.
  2. Switch to the corresponding channel (or the client's /client/chat in another browser/incognito) and verify the message arrives.

PASS criteria: the message arrives to the client. Sender displayed = the Specialist's name ("Alex" / "Amy" / etc.), NEVER the Expert's real name. The client cannot tell whether the response was the verbatim AI draft or an edited human reply (it appears as the Specialist either way).

Agent recipe (technical)
  • Outbound dispatch goes through ChannelDispatcherService (PR #567): WS for portal, Twilio for WhatsApp, Resend for email, Bolt for Slack.
  • Email From-header: always Specialist routing address, never the Expert's mailbox.
  • Persona check: the new message has role='expert' in DB but is rendered with the Specialist's display name on every client-side surface.

E-R7 — Resolve a ticket​

"...so it's removed from the active queue and logged as closed."

Human walkthrough​

  1. After sending the response, click "Resolve" on the queue item.
  2. Optionally rate satisfaction 1-5; submit.
  3. The item disappears from the Pending tab and appears under Resolved.

PASS criteria: queue counter for Pending decrements; item visible under Resolved; the originating conversation is marked complete on the client side as well.

Agent recipe (technical)
  • Endpoint: POST /expert-queue/:id/resolve { satisfaction?: 1-5 }. Sets resolvedAt + resolvedBy.

E-L1 — Submit a correction (the learning loop)​

"...so the learning pipeline captures my feedback for model improvement."

Human walkthrough​

  1. On any queue item, after editing the draft (E-R5) or discarding it and writing your own, the system automatically captures the delta as a correction.
  2. Optionally, on the resolve dialog, fill in a "What went wrong?" note.

PASS criteria: editing-and-sending produces a correction row in the database. Visible in the SuperAdmin learning page (out of scope here — just confirm the response was sent without error).

Agent recipe (technical)
  • Auto-capture: any non-zero text delta between AI draft and Expert-sent text writes a correction type='edit' row.
  • Manual rejection: POST /expert-queue/:id/respond { mode: 'write', text: ... } writes type='rewrite'.
  • Verification: GET /learning/corrections?conv={id} returns the row with non-empty originalText + editedText.

Role: Account Manager + SuperAdmin​

Internal Humanity Protocol staff. AM provisions and onboards a single client. SuperAdmin sees all clients and runs the platform.

SA-C1 / AM-001 — Create a new client​

"As a superadmin/AM, I want to create a new client with an initial admin user so they can log in and configure their account."

Human walkthrough​

  1. Log in as Account Manager → https://dev.h852.work/ops/clients.
  2. Click "Set up new client" (top right). (This button was labelled "+ New Client" in earlier runbook versions — on dev it is "Set up new client".)

Phase 1 — AM-driven setup (4 sub-steps, mirrors Konstantin's onboarding UAT granularity; the "Your Profile" step was retired via PR #1562 — wizard is now Company Info → Assign Specialists → Assign Experts → Send Invite, per frontend/src/components/shared/StepProgressBar.tsx WIZARD_LABELS):

1.1 Company Information. Fill in:

  • Company Legal Name (required, placeholder e.g. Acme Financial).
  • Subdomain (slug) (required, placeholder acme-corp). Regex ^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$. Reserved list blocks api, admin, app, www, etc. Real-time validation: "Continue →" stays disabled with inline hint ("Enter a company name." / "Subdomain unavailable.") until the field is valid.
  • Industry (required, dropdown).
  • Approved Corporate Email Domains (required, Press Enter to add, accepts multiple). Personal domains (gmail.com, yahoo.com, outlook.com) MUST be rejected — flag if accepted.
  • Client Admin Email (required, must match an approved domain — auto-fills the domain list if entered first).
  • Timezone (required, defaults to America/New_York).
  • Optional: Billing email, additional domains (collapsible).
  • PASS gate: companyInfoComplete=true after Continue.

1.2 Assign Specialists. Step 2 paginated catalog (~31 pages on dev with per-specialist monthly rate). Select the Specialist (e.g. "Alex"). Set Confidence Threshold to 101 (always escalate to Expert) for safety. Mark Primary.

  • PASS gate: specialistAssigned=true.

1.3 Assign Experts. Step 3 (ADR-007 expert_access). Pick Scope (Whole org / Specific Specialist) and Role (Reviewer). The wizard treats this as mandatory (Send Invitation stays disabled until at least one expert-access row exists); however, this gate is enforced by the frontend wizard step rather than the GET /am/orgs/:orgId/phase0-status endpoint, which returns only the three gates listed below.

  • PASS gate: wizard step "Continue" enables only after an Expert grant is created (POST /am/orgs/:orgId/expert-access returns 201).

1.4 Send Invitation. Step 4 (final). The wizard exposes a trial-end-date field (no fixed default — AM enters a date directly). A common convention is 30 days from invitation send, but this is operator policy, not enforced by the API. Option to "Copy Onboarding Link" (manual share) or "Send Invitation" (welcome email to client admin: subject contains org name + AM name; body contains tokenised onboarding URL with 7-day expiry).

  • PASS gate: trialEndDateSet=true AND companyInfoComplete, specialistAssigned true AND wizard-enforced Expert grant exists → Send Invitation enables.

PASS criteria (Phase 1 overall):

  • All 3 phase-0 API gates (companyInfoComplete, specialistAssigned, trialEndDateSet) plus the wizard-enforced Expert grant resolve true before Send Invitation enables. (The amProfileComplete gate was removed when the "Your Profile" wizard step was retired via #1562; if your local API still returns the field for back-compat, treat it as deprecated.)
  • After clicking Send Invitation, org status flips to pending_client_confirmation.
  • Org now appears in /ops/clients with a "Pending" badge.
  • NEEDS-LIVE-INBOX (skippable on cold dev — see C-O1, C-T1 for the same caveat). Welcome email lands at the admin inbox containing the onboarding URL. DEMO_MODE bypasses outbound email so this PASS gate is non-blocking on cold dev; it is blocking on a NEEDS-LIVE-TOKEN run where the OTP relay is being exercised. Pair this check with the SA-C1 run that feeds C-O0 / C-T1.
  • Step 4 → completed-state CTA shows "Complete Setup" for re-entry to the wizard.

Known dev gate (navigation): Step 4 cannot be reached via deep-link ('Failed to load org status'); the wizard requires forward-only sequential navigation.

Agent recipe (technical)
  • Endpoints in order: GET /orgs/slug-check → POST /orgs → POST /am/orgs/:orgId/specialist → POST /am/orgs/:orgId/expert-access → GET /am/orgs/:orgId/phase0-status → POST /am/orgs/:orgId/invite.
  • Phase-0 API gates (GET /am/orgs/:orgId/phase0-status response shape): companyInfoComplete, specialistAssigned, trialEndDateSet — all must be true for the AM wizard to consider Phase 1 complete. The legacy amProfileComplete field was retired when the "Your Profile" wizard step was removed (PR #1562); some back-compat shapes may still emit it but it is no longer a wizard gate. The Expert grant is enforced at the wizard layer (no expertAssigned field in the API response — verified against organizations.service.ts:1878+); the runbook gate is the presence of at least one expert_access row.
  • Slug rules: regex ^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$; reserved list blocks api admin app www etc.
  • Invite token: 32-byte random, 7-day expiry, bcrypt-hashed 6-digit OTP.
  • Trial: organizations.trialEndDate is set by the AM (raw date input via the org-update path); trialEndDateSet = !!org.trialEndDate. The 30-day convention is operator policy, NOT an API default — there is no trialDurationDays field in the invite payload. org.trialStartedAt is set by the onboarding service when Phase 2 completes (status flips to active); the trial countdown displayed in the client portal uses trialEndDate - now().

SA-C2 — List all clients​

"As a superadmin, I want to view a list of all clients so I can monitor onboarding status and platform adoption."

Human walkthrough​

  1. Log in as SuperAdmin → https://dev.h852.work/ops/clients.
  2. The page shows a table of every client org with: name, slug, status badge (Active/Pending/Trial), trial-end date, assigned Specialist, member count.

PASS criteria: every org you've created (including the throwaway from SA-C1) appears. Status badges match the org's current lifecycle stage. Counts are correct.

Agent recipe (technical)
  • Endpoint: GET /ops/orgs. Returns all orgs for SA; AM-scoped view returns only orgs the AM owns.

SA-C3 — Drill into a client's detail​

"...to see their members, ticket activity, assigned experts, and plan."

Human walkthrough​

  1. From /ops/clients, click any client row.
  2. You see the client detail page with tabs: Overview, Tools, Team, Specialists, Billing, Channels.

PASS criteria: every tab loads without error. Overview shows the org's basic facts; Team tab shows current members; Specialists tab shows the assigned Specialist + threshold.

Agent recipe (technical)
  • Endpoint: GET /ops/orgs/:id — returns nested members, specialists, channels, billing summary.
  • Impersonate: SuperAdmin-only POST /admin/impersonate { orgId, userId } → issues a scoped JWT for support purposes.

SA-C4 — Assign an Expert directly to a client​

"As a superadmin, I want to assign Experts directly to a client org so escalations route to the right people."

Human walkthrough​

  1. On the client detail page, open the Team tab (or Specialists / Access tab depending on dev build).
  2. Click "Add Expert" (or similar).
  3. Pick an Expert from the dropdown of HP staff.
  4. Confirm. The Expert now appears in the client's access list.

PASS criteria: Expert added without error. Logging in as that Expert, you now see queue items for this client's org (validated by ADR-007 expert_access).

Agent recipe (technical)
  • Endpoint (canonical, ADR-007): POST /admin/expert-access with body { orgId, expertId, scope: 'org' | 'specialist', specialistId? }.
  • Legacy org_experts table deprecated 2026-05-22 (#537); removed in #540 after soak. Always read via ExpertAccessService.

Coverage matrix — P0 user stories vs this runbook​

This runbook covers every P0 row in user-stories.md Priority Index, under the Kaito-lens revision (2026-05-25). Each runbook case ID maps 1:1 to a user-story ID.

Run summary​

Fill this in at the end of the run. Each role section is GO/NO-GO; the whole runbook is GO only if all three are GO.

RoleGO/NO-GONotes
Client
Expert
AM + SuperAdmin

Final GO/NO-GO: ____ (Paul)

Changelog​

v2.5 (2026-06-04)​

  • AM wizard re-aligned to 4 steps (Company Info → Assign Specialists → Assign Experts → Send Invite), matching frontend/src/components/shared/StepProgressBar.tsx WIZARD_LABELS. The "Your Profile" Step 1.4 documented in v2.2-v2.4 was retired via PR #1562; the sibling docs/test-cases/On-boarding test.md was synced on 2026-06-03 (commit 04898f09/5ada6858) but this runbook had not been until now.
  • SA-C1 Phase 1 walkthrough rewritten: dropped 1.4 Your Profile + the amProfileComplete phase-0 gate; renumbered Send Invitation from 1.5 → 1.4.
  • Phase-0 API gates documented as 3 (companyInfoComplete, specialistAssigned, trialEndDateSet) instead of 4; amProfileComplete flagged as deprecated for back-compat callers.
  • Line 26 disambiguated: the 3-step layout described there is the client onboarding wizard, distinct from the 4-step AM "Set up new client" wizard.
  • v2.0 history note corrected: the "dev has 5 steps vs staging's 4" claim was historical and is now superseded — both environments are 4 steps as of #1562.

v2.4 (2026-05-25, AC-PC reprobe + EDGE-1 fix)​

After the v2.3 mapping flagged AC-PC-07, AC-PC-08, AC-PC-12 as INCONCLUSIVE due to walker-selector gaps and EDGE-1 as an OPEN BUG, this update reprobes with tighter selectors and ships the EDGE-1 fix.

  • AC-PC-07 flips INCONCLUSIVE → PASS. 14 rows on dev, structure [avatar initial] [thread name] [relative timestamp] [status badge]. Walker: aside button[aria-label^='Open thread:'].
  • AC-PC-08 flips INCONCLUSIVE → PASS-PARTIAL with spec drift. Header located via button[aria-label='Back to briefing'] parent. The "Online · Support Specialist" line from the spec was removed in the redesign; identity is per-message now. AC needs spec update or product decision to re-add.
  • AC-PC-12 flips INCONCLUSIVE → PASS. 14 .hw-thread-rename-btn buttons (per-row, opacity: 0 until hover). Force-click reveals the rename input. Caveat: renames are local-only (localStorage[hw_threadNames]) — PATCH /conversations/:id not wired. Cross-device persistence is a follow-up if wanted.
  • EDGE-1 expired-invitation wording bug fixed in PR #721. step2.tsx bare-catch was swallowing 410-GoneException from the API; replaced with ApiError-status branching that renders a friendly "This invitation has expired" page for 401/403/404/410 and keeps the existing Retry path for 5xx/network failures.

v2.3 (2026-05-25, this PR follow-up)​

Folded in docs/acceptance/CLIENT_PORTAL_CHAT.md (Tyler/E spec, 2026-05-01) as a first-class testable section after a dev walkthrough validated each of the 12 AC-PC entries.

  • New: Client-portal AC-PC mapping section (after C-W3, before C-O0). Walks AC-PC-01..12 with dev-observed status, per-row evidence, and an agent recipe for repeatable probes.
  • Source doc updated in parallel — each AC in CLIENT_PORTAL_CHAT.md carries a Status (2026-05-25) line: ACTIVE / OBSOLETE — REPLACED / OPEN BUG. The two docs are kept in sync.
  • Real product bug surfaced and fixed: AC-PC-01's hardcoded H avatar in the sidebar (PortalChat.tsx:288) → fixed in PR #719 (1-line, derives initial from orgName).
  • Four ACs marked OBSOLETE — REPLACED (UI evolved past spec): AC-PC-03 ("Message" button), AC-PC-05 ("Your Threads/Specialists" labels), AC-PC-11 (localStorage last-thread → URL-as-state via /client/chat/[id]), and partially AC-PC-09 (CTA pattern). These remain in the source doc as historical context but do not count for/against the PASS criteria.

v2.2 (2026-05-25, this PR follow-up)​

Folded in deltas from Konstantin's onboarding UAT (positive-path AM↔Client flow, Phase 1 + Phase 2, edge cases). Konstantin's spec is narrower (onboarding only) but deeper than v2.1's compressed SA-C1 + C-O1; this update preserves the wider runbook surface and inherits his Phase-2 granularity.

  • New: C-O0 Client confirmation flow (Phase 2, full). Real-invite path covering Steps 2.1-2.4 (account verification w/ OTP + password + profile; review-and-edit company info w/ slug becoming IMMUTABLE; meet-your-Specialist read-only card; activation w/ status flip, trial clock, AM notification, welcome summary email, token invalidation). 7-item PASS checklist replaces v2.1's 2-gate criteria. Marked NEEDS-LIVE-TOKEN since DEMO_MODE bypasses this entire flow.
  • New: C-S1 Billing & Quota page. /client/settings/billing surface — Plan & Usage, Rate Limits, Message Volume, Plan Features, Upgrade Plan CTA. PASS-UI verified against dev seed.
  • New: Client-role edge cases table. EDGE-1 invite expiry (with dev wording bug flagged), EDGE-2 90-day auto-archive, EDGE-3 slug race + write-lock, EDGE-4 re-entry CTA for pending orgs.
  • C-O1 PASS criteria split into DEMO_MODE smoke (existing 2-gate) and real-invite path (new 7-gate via C-O0 cross-reference).
  • SA-C1 Phase-1 walkthrough expanded from 7 flat steps to 4 named sub-steps (1.1 Company Info w/ real-time validation + personal-domain rejection callout; 1.2 Assign Specialists w/ catalog pagination note; 1.3 Assign Experts mandatory gate; 1.4 Send Invitation w/ welcome-email + 7-day-token + 30-day-trial spec). (Originally documented as 5 sub-steps with a "1.4 Your Profile" gate; that step was retired via PR #1562 and the runbook was re-aligned to the 4-step StepProgressBar.tsx WIZARD_LABELS in v2.5 — 2026-06-04.)

v2.1 (2026-05-25, this PR)​

  • C-W1 PASS criteria split into immediate (thread enters PENDING with holding indicator) and end-to-end (Expert release → reply within 30 s of release). Resolves the inconsistency between the human walkthrough ("reply within 30s") and the agent recipe (default autoRespondThreshold=101 holds the draft).
  • C-W1 Step 2 rewritten: clarify that /client/chat lands on the "Today's briefing" overview, and the composer is reached by clicking "+ New thread" in the left sidebar.
  • SA-C1 button label updated from "+ New Client" to "Set up new client" to match the dev UI.
  • E-R1..E-R7 prerequisite note added — Expert queue is empty on cold dev and requires walking C-W1 first to populate.
  • E-R2 rewritten to match the actual dev UI: 5 status tabs ("All" / "Pending" / "Awaiting Reply" / "Resolved" / "Snoozed"), not the stale 3 ("Pending" / "In Review" / "Resolved") from earlier runbook versions. Agent recipe API values updated to all|pending|awaiting_client|resolved|snoozed (verified against expert-queue.service.ts:79 default pending); the legacy unread value is explicitly called out as not-accepted.
  • SA-C1 agent recipe extended with the Step 3 "Assign Experts" endpoint (POST /am/orgs/:orgId/expert-access) and the expertAssigned phase-0 gate.

v2.0 (2026-05-25)​

Dev-only re-shoot. Per Paul's directive, humanwork test environment is now exclusively dev.h852.work (never staging/prod).

  • Replaced all 15 image blocks with fresh dev captures.
  • Patched all URLs and environment callouts from staging.h852.work to dev.h852.work.
  • AM new-client wizard: 4 steps (Company Info → Assign Specialists → Assign Experts → Send Invite) on both dev and staging — per frontend/src/components/shared/StepProgressBar.tsx WIZARD_LABELS. (v2.0 originally claimed dev had a 5-step wizard with a "Your Profile" step; that step was retired via PR #1562 and the historical claim is corrected in v2.5 — 2026-06-04.) "Assign Experts" Step 3 documented in SA-C1 (Scope: Whole org / Specific Specialist; Role: Reviewer; aligns with ADR-007 expert_access).
  • Specialist catalog (Step 2) is paginated on dev (31 pages with per-specialist monthly rates) vs staging's 5 hardcoded entries.
  • Expert workspace surface on dev includes the full feature set absent from staging: correction-tag chips (Factual / Tone / Missing context / Wrong action / My preference), Show-AI-reasoning toggle, private Ask-Eleanora composer, escalation-reason callout, Agent Healthcheck indicator.
  • Slack: Available (unblocked 2026-05-26 via #816 Connect OAuth flow; per-org signing_secret landed 2026-06-01 via #1219). Earlier "Coming Soon / BLK-004 / #120" status is closed. UAT-SL- cases now active* — run them as part of P0 sign-off.
  • Step 4 of the AM wizard cannot be reached via deep-link ('Failed to load org status'); the wizard requires forward-only sequential navigation — documented as a UX runbook callout.

Companion documents​

Last updated on by Degen