LLM Cost Statistics — Platform / Client / Specialist Levels
Status: Approved plan (spec-of-record) · Owner: Franky · Date: 2026-07-23 Chinese version: llm-cost-statistics.zh.md Related: #1231 (Langfuse integration — promised spend-by-org / spend-by-specialist dashboards) · #3196 (Expert-labour cost — the other half of SA-B5) · #4662 (measurement answer key, metric #7 cost-per-task) · billing-and-cost-guardrails.md (design-of-record for the billing model) · ADR-041 Langfuse self-host (open PR #4735)
1. Summary & goals
With the Langfuse integration complete (#4749 env contract, #4750 five-hunk plugin patch — verified on dev 2026-07-22), every Hermes chat turn now produces a trace with correct cost (≈$0.046/turn on anthropic/claude-sonnet-4) and tenant attribution (user_id = org_id, session_id = conversation UUID, metadata.specialist_id). This spec plans the next step: cost statistics at three levels — platform (cross-org), client (org), and Specialist — combining Langfuse-native analytics with authoritative in-product statistics.
Goals
- G1 — Ops can answer "what did the platform spend yesterday / this month" without manual SQL.
- G2 — Ops can rank clients (orgs) by LLM spend and see each org's trend.
- G3 — Ops can attribute spend to a Specialist, both within an org and across orgs.
- G4 — Numbers used for anything billing-adjacent come from our own DB, with Langfuse as a cross-check (
api/src/llm/is the implementation authority).
Non-goals (adjacent, tracked elsewhere — see §6 Phase 3)
- Gross-margin computation (needs Expert-labour cost — #3196).
- Client-facing / AM-facing cost views (product decision not yet made).
- Budget-enforcement unification (API
CostBudgetServicevs agentcost_cap.py). - Infrastructure-cost attribution.
2. Current state (verified 2026-07-23)
2.1 Two disjoint DB ledgers
| Ledger | Written by | Captures | Read by |
|---|---|---|---|
messages.cost_cents (+ token_input/output, model_name, osa_id) | ConversationsService on every agent-message persist, from the agent /chat response | Main Hermes chat turns | LlmCostRollupService → the only billing read path |
llm_runs (org_id, osa_id, conversation_id, purpose, cost_cents, cost_micro_usd) | CostBudgetService.recordSpend, called only from LlmService | In-process auxiliary calls (summary, subject, classify, embed, …) | Nobody (write-only audit today) |
Known weaknesses of the messages path:
- Cost originates from the agent regex-scraping
cost=lines out of Hermes-voutput (agent/hermes_parser.py_HERMES_COST_RE) — fragile per provider/format; the #4750 saga proved OpenRouter cost surfacing is unreliable. - Integer cents rounding: sub-half-cent turns record 0 — there is no
cost_micro_usdonmessages. - Expert-consultation turns hard-code
cost_usd = None(agent/main.py). - No
org_idcolumn — every rollup joins throughconversations.
Known weaknesses of the llm_runs path: osa_id is in the type (api/src/llm/llm.types.ts) but no caller passes it; stream() is unmetered by design.
2.2 Existing product surface
GET /ops/billing/orgs/:orgId/llm-cost(SuperAdmin-only, api/src/billing/interface/ops-billing.controller.ts) + the "LLM Cost (this month — token cost only, not billed)" card on frontend/src/app/ops/billing/page.tsx. Single org, single month, byosa_id.- No platform-level (cross-org) aggregation exists anywhere. The platform billing overview (#1979) is revenue-only.
- No cross-org per-Specialist view exists.
- Margin is "not computed anywhere" (billing-and-cost-guardrails.md §4).
2.3 Langfuse dimensions today
| Surface | user_id | session_id | tags | metadata |
|---|---|---|---|---|
| Hermes agent traces | org_id (UUID) | conversation UUID | constant ["hermes","langfuse"] | org_id, conversation_id, specialist_id, customer_id, run_id, agent_role, hermes_session |
API LlmService telemetry | — (unset) | — (unset) | — (unset) | orgId, conversationId (camelCase; osaId never passed, specialistId absent) |
Consequences: per-org grouping works natively only for agent traces (Langfuse Users view); Specialist is metadata-only (not natively groupable in the UI); the two surfaces cannot be unified in any built-in Langfuse view; an org's Langfuse "user cost" undercounts by all API-side auxiliary spend.
3. Architecture decisions
D1 — Langfuse = exploration + cross-check; own DB = billing-grade numbers
Keep the settled decision ("billing pivots on llm_runs/messages, not Langfuse"). Langfuse Metrics API / dashboards are the right tool for fast internal ops exploration (Phase 0) but not the statistics source of record, because: ClickHouse retention is configurable (a TTL change would silently truncate "billing" history); the self-hosted instance has no HA story and ingestion is fire-and-forget (droppable — not a ledger); trace cost depends on the build-time plugin patch (a pin bump can regress it); traces contain message content, so widening dashboard access widens PII exposure; and two cost derivations already exist and disagree at the margins — D4 turns that into a guard instead of a liability.
D2 — Make Specialist a first-class Langfuse dimension via value-bearing tags
Keep user_id = org_id and session_id = conversation_id (do not overload user_id with a composite scheme — it would break the existing org view and all historical traces). Add value-bearing tags on every trace: specialist:<specialist_id>, source:agent / source:api, and purpose:<purpose> (API side). Tags are the first-class filterable dimension in the Langfuse UI; today's constant tags carry zero grouping power. Keep metadata.specialist_id as the canonical structured key and align API-side metadata key names to the agent's snake_case (org_id, conversation_id, specialist_id). Whether the self-hosted Metrics API supports group-by on tag values (vs filter-only) is a Phase 0 spike output, not an assumption — if filter-only, Langfuse specialist views degrade to saved per-specialist filters, which is acceptable because Phase 2 makes the DB the grouping engine.
D3 — Unify the ledgers: dual-write Hermes turn cost into llm_runs (purpose='chat-hermes')
Building three-level statistics on a UNION of two schemas would permanently double every query and bake the cents-rounding loss into platform numbers. Instead, at the point where the API persists a Hermes turn, also call CostBudgetService.recordSpend with purpose='chat-hermes', orgId, osaId, conversationId, tokens, and cost_micro_usd. Cost is computed server-side from agent-reported tokens × model-pricing.ts (micro-USD); the regex-scraped agent cost becomes a cross-check, not the source. messages.cost_cents keeps being written during the transition (expand-contract); the messages-based read path is retired only at Phase 2 cutover. Backfill: last 90 days of messages history into llm_runs, marked as backfilled (cents precision) so precision-sensitive analysis can exclude those rows. (Backfill depth decided by product owner 2026-07-23.)
D4 — Nightly Langfuse ↔ DB reconciliation (alert-only)
A nightly job compares D-1 per-org and platform-total cost: llm_runs (post-unification) vs the Langfuse Metrics API grouped by user. Structured warning + ops-page badge when |drift| > 5% or > $1/day. This directly guards the two known fragilities (the regex scrape and the build-time patch). Deliberately not a data-repair job.
4. Phased plan
Phase 0 — Langfuse-native analytics (this week, zero code deploys)
| # | Task | Acceptance criteria | Effort |
|---|---|---|---|
| 0.1 | Build Langfuse dashboards: platform daily cost trend; cost by user (= org) top-N; session drill-down. Save as shared dashboards. | Ops answers "platform spend yesterday" and "top 5 orgs this week" from Langfuse without engineering. | S |
| 0.2 | Metrics API spike: verify group-by/filter capabilities on userId / tags / metadata for our self-hosted version. | Findings appended to this doc; go/no-go for tag-based specialist grouping (feeds D2, D4). | S |
| 0.3 | Runbook section in billing-and-cost-guardrails.md: how to read the dashboards + caveats. | Caveats explicitly documented: per-org Langfuse numbers undercount (API-side calls carry no user_id yet); stream() unmetered; expert-consult turns cost-NULL in DB. | S |
Risk: pre-Phase-1 numbers undercount — without task 0.3's caveat, ops will treat them as complete.
Phase 1 — Trace-dimension completion
| # | Task | Acceptance criteria | Effort | Deps |
|---|---|---|---|---|
| 1.1 | Agent traces: add value-bearing tags specialist:<id>, source:agent — extend agent/main.py _hermes_env + apply_langfuse_tenant_patch.py (new env var → tag list, same mechanism as #4749/#4750). | Fresh-image-built trace carries the tags; Langfuse filter specialist:<id> returns it. Patch suite extended. | S–M | 0.2 |
| 1.2 | API telemetry unification: applyTelemetry (api/src/llm/llm.service.ts) sets userId=orgId, sessionId=conversationId, tags [source:api, purpose:<p>, specialist:<id>]; metadata keys aligned to agent snake_case; add specialistId to the costContext type. Verify exact @langfuse/otel attribute mapping as part of the task — do not assume key names. | An API-side summary call and an agent turn for the same conversation appear under the same Langfuse user and session. | M | 0.2 |
| 1.3 | Thread osaId/specialistId through LlmService callers that have the context (conversation summary/subject, learning, composer, transcripts) into both telemetry and recordSpend (today no caller passes osaId). Callers with genuinely no specialist context pass null deliberately. | Chat-adjacent purposes carry osa_id in llm_runs and specialist tags in Langfuse. | M | 1.2 |
| 1.4 | (Open decision) Meter LlmService.stream() — or at minimum emit telemetry for streamed calls. | Streamed calls visible in Langfuse; ideally recorded via recordSpend. | M | 1.2 |
| 1.5 | Update Phase-0 dashboards to group/filter by specialist tag. | Ops answers "which Specialist burns the most across all orgs". | S | 1.1, 1.2 |
Risks: the agent change is build-time — verify on a fresh image build, not a running container; tag cardinality is fine at current Specialist counts (re-check at thousands); historical traces lack the tags — dashboards are forward-looking.
Phase 2 — Authoritative in-product statistics (deliverable of record)
| # | Task | Acceptance criteria | Effort | Deps |
|---|---|---|---|---|
| 2.1 | Ledger unification (D3): dual-write Hermes turn cost into llm_runs (purpose='chat-hermes', org/osa/conversation, tokens, server-side cost_micro_usd). messages.cost_cents continues to be written. | Every new chat turn produces exactly one llm_runs row; micro-USD matches the pricing table; expert-consult turns costed (or the exclusion documented). | M | — |
| 2.2 | Backfill script: last 90 days of messages → llm_runs, marked backfilled, idempotent. | Row counts reconcile; re-run is a no-op. | S–M | 2.1 |
| 2.3 | New read model PlatformCostStatsService (new service in api/src/billing/, reads llm_runs only — do not stretch LlmCostRollupService, whose contract is single-org-from-messages): platform daily trend, monthly top-N orgs, per-specialist within an org, cross-org per-specialist, split by purpose (chat vs background). Includes an llm_runs(created_at) index migration (existing indexes are org-prefixed; platform scans need it) — expand-contract, CONCURRENTLY. | Unit-tested aggregations match manual SQL on staging. | M | 2.1 |
| 2.4 | Ops endpoints (SuperAdmin, same guard stack as the existing per-org endpoint): GET /ops/billing/llm-cost/platform, …/llm-cost/orgs?period=, …/llm-cost/specialists?period=; extend the per-org endpoint with purpose breakdown. | Correct aggregates; cross-org endpoints 403/404 for non-SuperAdmin (tested). | M | 2.3 |
| 2.5 | Ops UI on frontend/src/app/ops/billing/page.tsx: platform trend chart, top-N orgs table, specialist rollup with org drill-down. | All three levels answerable in-product without Langfuse. | M | 2.4 |
| 2.6 | Reconciliation job (D4): nightly BullMQ repeat job, D-1 totals llm_runs vs Langfuse Metrics API; warn + ops badge on >5% or >$1/day drift. | Seeded-drift test fires the alert; clean day silent. | M | 2.1, 0.2, Phase 1 |
| 2.7 | Cutover: switch LlmCostRollupService consumers and the CostBudgetService daily gate to llm_runs; retire the messages-based read path (column stays for history); update billing code + guardrails doc. DONE 2026-07-23 (#4799) — the "stable ≥ 2 weeks" precondition was waived because the messages ledger had no accepted consumer. #4798 reconciliation remains the ongoing guard. | One ledger of record; existing endpoint behavior preserved or versioned. | M |
Risks: dual-write divergence during transition (2.6 is the guard — run it before cutover); backfilled rows are cents-precision (flagged); the budget gate's semantics shift slightly at cutover (micro-USD accumulation vs cents) — snapshot-compare a week of gate decisions first.
Phase 3 — Explicitly out of scope (adjacent)
- Margin view = this plan's LLM cost + Expert-labour cost (#3196). Phase 2 endpoints are shaped so #3196 can join onto them.
- Budget unification: point agent
cost_cap.pyand APICostBudgetServiceat the single ledger; per-Specialist budgets become possible only after 2.7. - Client-facing cost transparency (user-stories.md line 23) — product decision pending.
5. Suggested issue breakdown (planning only — issues to be created after approval)
| Suggested issue | P | Phase | Links |
|---|---|---|---|
| Umbrella: LLM cost statistics — platform / org / specialist (SA-B5 cost half) | P1 | all | #1231, #3196, #4662 |
| Langfuse ops cost dashboards + runbook | P2 | 0 | umbrella |
| Spike: Langfuse Metrics API group-by capabilities (self-hosted) | P2 | 0 | umbrella |
| Agent traces: value-bearing tags (specialist/source) via tenant patch | P1 | 1 | #4749, #4750 |
| Unify API-side Langfuse telemetry (userId/sessionId/tags/specialistId) | P1 | 1 | umbrella |
| Thread osaId/specialistId through LlmService callers + recordSpend | P2 | 1 | above |
| Hermes chat-turn cost into llm_runs (purpose='chat-hermes', micro-USD) | P1 | 2 | umbrella, #1231 |
| Backfill llm_runs from messages (90 days, flagged, idempotent) | P2 | 2 | above |
| Platform/org/specialist cost stats: read model + ops endpoints + ops UI | P1 | 2 | #1231, #1979 |
| Nightly Langfuse↔DB cost reconciliation with drift alert | P2 | 2 | umbrella |
| Cutover billing reads to llm_runs; retire messages-based rollup | P2 | 2 | all Phase 2 |
Also: close stale #2750 with a pointer to the umbrella (its plan/seat/overage framing predates the flat-fee model).
6. Open decisions
| # | Decision | Recommendation | Status |
|---|---|---|---|
| OD-1 | Turn-cost primary source: server-side tokens × pricing vs agent-scraped cost | Server-side (D3); confirm agent token counts are reliable — the reconciliation job will show which source drifts | Open |
| OD-2 | stream() metering (task 1.4): Phase 1 or deferred | Include in Phase 1 if 1.2 touches the same code paths anyway | Open |
| OD-3 | Backfill depth | — | Decided: 90 days (2026-07-23) |
7. Spike findings — Langfuse Metrics API (task 0.2, #4791)
Probed live against the self-hosted instance (project humanwork-dev, Langfuse v3.174.1) on 2026-07-23 with real trace data. Transport quirks first: the Metrics API is GET /api/public/metrics?query=<url-encoded JSON> — POST returns 405 — and the Cloudflare in front of langfuse.hptestingsite.com returns 403 (error 1010) to non-browser user agents, so any client (including the Phase-2 reconciliation job, #4798) must send a browser-like User-Agent.
Capability matrix
| Capability | Result | Evidence |
|---|---|---|
Group by userId (= org) | ✅ | Per-org sum_totalCost rows returned; unattributed API-side spend appears as a userId: null row |
Group by sessionId (= conversation) | ✅ | Per-conversation cost rows |
| Group by tags | ✅ but by the whole tag array, not per tag value | {"tags": ["hermes","langfuse"], …} comes back as one group — with exactly one specialist:<id> tag per trace, arrays become distinct per specialist, so this works as de-facto specialist grouping |
| Filter by single tag value | ✅ | {"column":"tags","operator":"any of","value":["hermes"],"type":"arrayOptions"} |
| Filter by metadata key = value | ✅ | {"column":"metadata","operator":"=","key":"specialist_id","value":"<uuid>","type":"stringObject"} returned the correct single-org row |
| Group by a metadata key | ❌ | Invalid dimension metadata. Must be one of id,name,tags,userId,sessionId,release,version,environment,timestampMonth |
| Time-series (day granularity) | ✅ | "timeDimension":{"granularity":"day"} → platform daily trend |
| Observations view (by model) | ✅ | providedModelName dimension with totalCost/totalTokens sums |
| Legacy daily rollup | ✅ | GET /api/public/metrics/daily → per-day totals + per-model breakdown |
| Dashboards public API | ❌ (404) | No dashboards API in 3.174.1 — dashboards are UI-managed only (manual recipe in the guardrails runbook §6.1) |
Verdict
GO for value-bearing tags (D2 / #4792). Tags are the only tenant-shaped dimension that is both filterable (any of) and de-facto groupable (via whole-array grouping); metadata is filter-only, so cross-specialist grouping in one query is impossible until the tags land. metadata.specialist_id stays the canonical structured key for point filters. The reconciliation job (#4798) can rely on group-by-userId + the daily endpoint — both proven.