LLM Cost Statistics — Platform / Client / Specialist Three-Tier
Status: Approved plan (spec-of-record) · Owner: Franky · Date: 2026-07-23 English version: llm-cost-statistics.md Related: #1231 (Langfuse integration — promised spend-by-org / spend-by-specialist dashboards) · #3196 (Expert labor cost — the other half of SA-B5) · #4662 (metrics answer table, metric #7 cost-per-task) · billing-and-cost-guardrails.md (billing model design-of-record) · ADR-041 Langfuse self-hosting (open PR #4735)
1. Summary and Goals
Now that the Langfuse integration is complete (#4749 env contract, #4750 five-hunk plugin patch — verified in dev 2026-07-22), every Hermes conversation-turn trace carries correct cost (anthropic/claude-sonnet-4 ≈ $0.046/turn) and tenant attribution (user_id = org_id, session_id = conversation UUID, metadata.specialist_id). This spec plans the next step: three-tier cost statistics at platform level (cross-org) / client level (org) / Specialist level — combining Langfuse's native analytics with an authoritative in-product statistics layer.
Goals
- G1 — Ops can answer "how much did the platform spend yesterday/this month" without hand-writing SQL.
- G2 — Ops can rank clients (orgs) by LLM spend and view per-org trends.
- G3 — Ops can attribute spend to Specialists — both within an org and across orgs.
- G4 — Any billing-relevant number comes from our own DB, with Langfuse as cross-validation (
api/src/llm/is the implementation authority).
Non-goals (adjacent items, tracked separately — see §6 Phase 3)
- Gross margin calculation (needs Expert labor cost — #3196).
- Client/AM-facing cost views (product decision pending).
- Unifying budget enforcement mechanisms (API
CostBudgetServicevs agentcost_cap.py). - Infrastructure cost attribution.
2. Current State (verified 2026-07-23)
2.1 Two disconnected DB ledgers
| Ledger | Writer | Coverage | Reader |
|---|---|---|---|
messages.cost_cents (+ token_input/output, model_name, osa_id) | ConversationsService, on every agent-message persist, sourced from the agent /chat response | Hermes main-chat turns | LlmCostRollupService → the only billing read path |
llm_runs (org_id, osa_id, conversation_id, purpose, cost_cents, cost_micro_usd) | CostBudgetService.recordSpend, called only by LlmService | In-process auxiliary calls (summarization, topics, classification, embeddings, etc.) | Nobody (currently a write-only audit table) |
Known weaknesses of the messages path:
- The cost source is a regex scrape by the agent of
cost=lines in Hermes-voutput (agent/hermes_parser.py_HERMES_COST_RE) — sensitive to provider/format; the #4750 investigation proved OpenRouter's cost surfacing is unreliable. - Rounded to whole cents: turns under half a cent are recorded as 0 —
messageshas nocost_micro_usd. - Expert-consultation turns hardcode
cost_usd = None(agent/main.py). - No
org_idcolumn — every rollup has to joinconversations.
Known weaknesses of the llm_runs path: osa_id exists in the type (api/src/llm/llm.types.ts) but no caller passes it; stream() is not metered by design.
2.2 Existing product surface
GET /ops/billing/orgs/:orgId/llm-cost(SuperAdmin only, api/src/billing/interface/ops-billing.controller.ts) + the "LLM Cost (this month — token cost only, not billing)" card on frontend/src/app/ops/billing/page.tsx. Single org, single month, grouped byosa_id.- No platform-level (cross-org) aggregation exists anywhere. The platform billing overview (#1979) only has revenue.
- No cross-org per-Specialist view exists.
- Margin "has nowhere to be computed" (billing-and-cost-guardrails.md §4).
2.3 Current state of Langfuse dimensions
| Surface | user_id | session_id | tags | metadata |
|---|---|---|---|---|
| Hermes agent traces | org_id (UUID) | conversation UUID | constant ["hermes","langfuse"] | org_id, conversation_id, specialist_id, customer_id, run_id, agent_role, hermes_session |
API LlmService telemetry | — (unset) | — (unset) | — (unset) | orgId, conversationId (camelCase; osaId is never passed, specialistId is missing) |
Consequence: native grouping by org only works for agent traces (the Langfuse Users view); Specialist only lives in metadata (the UI can't natively group by it); the two surfaces can't be unified in any built-in Langfuse view; an org's Langfuse "user cost" will undercount the entirety of API-side auxiliary spend.
3. Architecture Decisions
D1 — Langfuse = exploration + cross-validation; our own DB = billing-grade numbers
Maintain the standing decision ("billing pivots on llm_runs/messages, not Langfuse"). The Langfuse Metrics API / dashboards are suited to fast internal ops exploration (Phase 0), but cannot serve as the statistics source of record because: ClickHouse retention is configurable (a single TTL change silently truncates "billing" history); the self-hosted instance has no HA, and ingestion is fire-and-forget (droppable — not a ledger); trace cost depends on a build-time plugin patch (a pin bump could regress it); traces contain message content, so opening access for a "cost dashboard" would expand PII exposure; and two cost derivations already exist and diverge at the edges — D4 turns that from a liability into a guard.
D2 — Make Specialist a first-class Langfuse dimension via value-bearing tags
Keep user_id = org_id, session_id = conversation_id (do not override user_id with a composite scheme — it would break existing org views and all historical traces). Add value-bearing tags to every trace: specialist:<specialist_id>, source:agent / source:api, purpose:<purpose> (API side). Tags are a first-class filterable dimension in the Langfuse UI; the current constant tags have no grouping ability. metadata.specialist_id remains the canonical structured key, and API-side metadata key names are aligned to the agent's snake_case (org_id, conversation_id, specialist_id). Whether the self-hosted Metrics API supports group-by on tag values (vs. filter-only) is the output of the Phase 0 spike — no assumption is made; if it's filter-only, the Langfuse specialist view degrades to a saved filter per specialist, which is acceptable since Phase 2 makes the DB the real grouping engine.
D3 — Unified ledger: dual-write Hermes turn cost into llm_runs (purpose='chat-hermes')
Building three-tier statistics on a UNION of the two schemas would permanently double every query and bake the cent-rounding precision loss into platform numbers forever. Instead: at the point where the API persists a Hermes turn, also call CostBudgetService.recordSpend, writing purpose='chat-hermes', org/osa/conversation, tokens, and cost_micro_usd. Cost switches to being computed server-side from agent-reported tokens × model-pricing.ts (in micro-USD); the regex-scraped agent cost is demoted to cross-validation, no longer the data source. During the transition messages.cost_cents keeps being written (expand-contract); the messages read path is retired only at Phase 2 cutover. Backfill: the last 90 days of messages history into llm_runs, tagged as backfilled (cent-precision), which precision-sensitive analyses can exclude. (Backfill depth was decided by the product owner on 2026-07-23.)
D4 — Nightly Langfuse ↔ DB reconciliation (alert-only)
A nightly job compares D-1 per-org and platform totals: llm_runs (post-unification) vs. the Langfuse Metrics API grouped by user. When |drift| > 5% or > $1/day, emit a structured alert + an ops-page badge. This directly guards the two known fragile points (regex scraping and the build-time patch). Deliberately does not auto-remediate data.
4. Phased Plan
Phase 0 — Native Langfuse analytics (this week, zero-code deploy)
| # | Task | Acceptance criteria | Effort |
|---|---|---|---|
| 0.1 | Build Langfuse dashboards: platform daily cost trend; top-N cost by user (= org); session drill-down. Save as a shared dashboard. | Ops can answer "how much did the platform spend yesterday" and "top 5 orgs this week" without engineering support. | S |
| 0.2 | Metrics API spike: verify the self-hosted version's group-by/filter support for userId / tags / metadata. | Findings appended to this document; go/no-go on tag-based specialist grouping (feeds D2, D4). | S |
| 0.3 | Add a runbook section to billing-and-cost-guardrails.md: how to read these dashboards + caveats. | Explicitly documents caveats: Langfuse per-org numbers undercount (API-side calls don't yet have user_id); stream() isn't metered; expert-consultation turns have NULL cost in the DB. | S |
Risk: numbers before Phase 1 are undercounted — without the 0.3 caveats, ops will treat them as complete numbers.
Phase 1 — Complete trace dimensions
| # | Task | Acceptance criteria | Effort | Dependency |
|---|---|---|---|---|
| 1.1 | Agent trace: add value-bearing tags specialist:<id>, source:agent — extend agent/main.py _hermes_env + apply_langfuse_tenant_patch.py (new env var → tag list, same mechanism as #4749/#4750). | Traces from a freshly built image carry these tags; filtering by specialist:<id> in Langfuse hits. Patch test suite extended accordingly. | S–M | 0.2 |
| 1.2 | Unify API telemetry: applyTelemetry (api/src/llm/llm.service.ts) sets userId=orgId, sessionId=conversationId, tags [source:api, purpose:<p>, specialist:<id>]; metadata keys aligned to the agent's snake_case; costContext type gets specialistId. The exact key names for @langfuse/otel attribute mapping are verified as part of the task — no assumption made. | An API-side summarization call and an agent turn from the same conversation appear under the same user and same session in Langfuse. | M | 0.2 |
| 1.3 | Pass osaId/specialistId through to LlmService callers that have context (session summary/topic, learning, composer, transcripts), into both telemetry and recordSpend (currently no caller passes osaId). Callers genuinely lacking specialist context pass null explicitly. | Chat-related purposes carry osa_id in llm_runs and a specialist tag in Langfuse. | M | 1.2 |
| 1.4 | (Open decision) Metering LlmService.stream() — at minimum emit telemetry for streaming calls. | Streaming calls are visible in Langfuse; ideally also recorded via recordSpend. | M | 1.2 |
| 1.5 | Update the Phase 0 dashboards to group/filter by specialist tag. | Ops can answer "which Specialist has spent the most across all orgs." | S | 1.1, 1.2 |
Risk: the agent change is build-time — it must be verified against a fresh image build, not just a running container; tag cardinality is fine at the current Specialist count (revisit at thousands); historical traces have no tags — dashboards only work going forward.
Phase 2 — Authoritative in-product statistics (the formal deliverable of this plan)
| # | Task | Acceptance criteria | Effort | Dependency |
|---|---|---|---|---|
| 2.1 | Ledger unification (D3): dual-write Hermes turn cost into llm_runs (purpose='chat-hermes', org/osa/conversation, tokens, server-side cost_micro_usd). messages.cost_cents keeps being written. | Every new chat turn produces exactly one llm_runs row; micro-USD matches the pricing table; expert-consultation turns have cost (or the exclusion reason is explicitly recorded). | M | — |
| 2.2 | Backfill script: last 90 days of messages → llm_runs, tagged as backfilled, idempotent. | Row counts reconcile; re-running is a no-op. | S–M | 2.1 |
| 2.3 | New read model PlatformCostStatsService (a new service under api/src/billing/, reads llm_runs only — don't bloat LlmCostRollupService, whose contract is single-org-from-messages): platform daily trend, monthly top-N org, per-specialist within an org, per-specialist across orgs, broken down by purpose (chat vs. background). Includes a llm_runs(created_at) index migration (existing indexes are all org-prefixed; platform-level scans need this) — expand-contract, CONCURRENTLY. | Aggregations have unit tests and match hand-written SQL on staging. | M | 2.1 |
| 2.4 | Ops endpoints (SuperAdmin, same guard stack as the existing single-org endpoint): GET /ops/billing/llm-cost/platform, …/llm-cost/orgs?period=, …/llm-cost/specialists?period=; the single-org endpoint gets a purpose breakdown. | Aggregation is correct; non-SuperAdmin access to cross-org endpoints returns 403/404 (with tests). | M | 2.3 |
| 2.5 | Ops UI (frontend/src/app/ops/billing/page.tsx): platform trend chart, top-N org table, specialist summary table (drillable to org). | All three tiers can be answered in-product without depending on Langfuse. | M | 2.4 |
| 2.6 | Reconciliation task (D4): nightly BullMQ repeat job, comparing D-1 llm_runs totals vs. the Langfuse Metrics API; alert + ops badge when drift is >5% or >$1/day. | An injected-drift test triggers the alert; a normal day stays silent. | M | 2.1, 0.2, Phase 1 |
| 2.7 | Cutover: point LlmCostRollupService consumers and the CostBudgetService daily budget gate at llm_runs; retire the messages read path (columns kept for history); update billing code + the guardrails doc. Completed 2026-07-23 (#4799) — the "stable ≥ 2 weeks" precondition was waived because the messages ledger had no accepted consumer. #4798 reconciliation remains a continuous guard. | Single ledger; existing endpoint behavior stays the same or is versioned. | M |
Risk: dual-write divergence during the transition (2.6 is the guard — run it before cutover); backfilled rows are cent-precision (already flagged); budget-gate semantics shift subtly at cutover (micro-USD accumulation vs. cent accumulation) — snapshot-compare a week of gate decisions first.
Phase 3 — Explicitly out of scope for this plan (adjacent)
- Gross margin view = this plan's LLM cost + Expert labor cost (#3196). The Phase 2 endpoint shapes are designed so #3196 can join directly.
- Budget unification: pointing the agent's
cost_cap.pyand the API'sCostBudgetServiceat a single ledger; per-Specialist budgets are only possible after 2.7. - Client-facing cost transparency (user-stories.md line 23) — product decision pending.
5. Suggested Issue Breakdown (planning only — create after approval)
| Suggested issue | P | Phase | Related |
|---|---|---|---|
| Umbrella: LLM cost statistics — platform / org / specialist (the cost half of SA-B5) | P1 | All | #1231, #3196, #4662 |
| Langfuse ops cost dashboards + runbook | P2 | 0 | umbrella |
| Spike: Langfuse Metrics API group-by capability (self-hosted) | P2 | 0 | umbrella |
| Agent trace: value-bearing tags (specialist/source), via tenant patch | P1 | 1 | #4749, #4750 |
| Unify API-side Langfuse telemetry (userId/sessionId/tags/specialistId) | P1 | 1 | umbrella |
| Pass osaId/specialistId through LlmService callers + recordSpend | P2 | 1 | previous item |
| Write Hermes chat-turn cost into llm_runs (purpose='chat-hermes', micro-USD) | P1 | 2 | umbrella, #1231 |
| Backfill llm_runs from messages (90 days, tagged, idempotent) | P2 | 2 | previous item |
| Platform/org/specialist cost statistics: read model + ops endpoints + ops UI | P1 | 2 | #1231, #1979 |
| Nightly Langfuse↔DB cost reconciliation and drift alerting | P2 | 2 | umbrella |
| Switch billing reads to llm_runs; retire messages rollup | P2 | 2 | all of Phase 2 |
Also: close the stale #2750 with a note pointing to the umbrella (its plan/seat/overage framework predates the current flat-fee model).
6. Open Decisions
| # | Decision | Recommendation | Status |
|---|---|---|---|
| OD-1 | Primary source of turn cost: server-side tokens × pricing table vs. agent-scraped cost | Server-side (D3); confirm agent token reporting is reliable — the reconciliation task will show which source is drifting | Open |
| OD-2 | stream() metering (task 1.4): Phase 1 or deferred | If 1.2 is touching the same code path anyway, fold into Phase 1 | Open |
| OD-3 | Backfill depth | — | Decided: 90 days (2026-07-23) |
7. Spike Findings — Langfuse Metrics API (task 0.2, #4791)
Tested 2026-07-23 against real trace data on the self-hosted instance (project humanwork-dev, Langfuse v3.174.1). Transport-layer gotchas first: the Metrics API is GET /api/public/metrics?query=<url-encoded JSON> — POST returns 405 — and the Cloudflare front for langfuse.hptestingsite.com returns 403 (error 1010) for non-browser UAs, so any client (including the Phase 2 reconciliation task #4798) must send a browser-style User-Agent.
Capability matrix
| Capability | Result | Evidence |
|---|---|---|
Group by userId (= org) | ✅ | Returns a sum_totalCost row per org; unattributed API-side spend appears as a userId: null row |
Group by sessionId (= conversation) | ✅ | Per-session cost rows |
| Group by tags | ✅ but groups by the entire tag array, not individual tag values | {"tags": ["hermes","langfuse"], …} is returned as a single group — since each trace has exactly one specialist:<id> tag, the array naturally differentiates by specialist and can effectively be used as specialist grouping |
| Filter by an individual tag value | ✅ | {"column":"tags","operator":"any of","value":["hermes"],"type":"arrayOptions"} |
| Filter by metadata key = value | ✅ | {"column":"metadata","operator":"=","key":"specialist_id","value":"<uuid>","type":"stringObject"} returned the correct single-org rows |
| Group by metadata key | ❌ | Invalid dimension metadata. Must be one of id,name,tags,userId,sessionId,release,version,environment,timestampMonth |
| Time series (day granularity) | ✅ | "timeDimension":{"granularity":"day"} → platform daily trend |
| Observations view (by model) | ✅ | providedModelName dimension + totalCost/totalTokens sums |
| Legacy daily-summary endpoint | ✅ | GET /api/public/metrics/daily → daily totals + per-model breakdown |
| Public dashboards API | ❌ (404) | 3.174.1 has no dashboards API — dashboards can only be managed in the UI (manual build steps in guardrails runbook §6.1) |
Conclusion
Value-bearing tags (D2 / #4792): GO. Tags are the only tenant-shaped dimension that is both filterable (any of) and effectively groupable (by whole tag array); metadata is filter-only, so cross-specialist grouping within a single query is not possible until tags land. metadata.specialist_id remains the canonical structured key for point-query filtering. The reconciliation task (#4798) can rely on grouping by userId + the daily endpoint — both are verified.