Skip to main content

ADR-031 — SigNoz / OpenTelemetry APM (api + agent)

Status: Accepted (foundation shipped via #1230; collector decision below resolves the issue gate) Date: 2026-06-19 Authors: Claude Code agent (foundation), @andrei (review) Issue: #1230 — [P1][feat][observability] SigNoz APM integration — api + agent Related: #1231 (Langfuse LLM tracing — builds on this bootstrap)

Context​

The platform had error tracking (Sentry, #828) and structured logs (the agent's canonical chat_completed line, US-016) but no distributed tracing. There was no way to see a single request as it crosses the api → agent /chat hop, to get p50/p95/p99 latency per route, or to build an APM service map. Issue #1230 required choosing a backend, wiring OpenTelemetry into both services, and propagating trace context across the api→agent boundary.

humoongate already runs a production SigNoz/OTel integration across its TS services; the conventions here are lifted from that reference implementation and adapted to humanwork's APP_ENV + active-active model.

Decision​

1. Backend / collector — SigNoz Cloud (US region)​

We export OTLP/HTTP directly to SigNoz Cloud, US region (https://ingest.us.signoz.cloud:443). We do not self-host an OTel collector. Rationale:

  • A self-hosted collector is stateful infrastructure someone must run, which contradicts the active-active "stateless pods" rule (repo CLAUDE.md). All pods export to the same managed endpoint; pods differentiate via the auto-detected host.name resource attribute. No per-pod state.
  • HP already runs a paid SigNoz Cloud tenant for humoongate (same dashboards / alerting team / on-call muscle memory).
  • Authentication is a shared-secret signoz-ingestion-key header, stored as a platform secret (Railway / k8s Secret), never committed.

Retention / cost ceiling follow the SigNoz Cloud plan default; revisit if the chat-heavy span volume approaches the plan's GB ceiling (the batch tunables + future tail sampling are the lever).

2. One shared OTel SDK per process; #1231 extends it​

Each process runs exactly one OTel SDK / TracerProvider. #1230 stands up that single bootstrap (api/src/instrumentation/tracing.ts, agent/instrumentation/tracing.py) with the SigNoz exporter. #1231 (Langfuse) must add a second SpanProcessor/exporter to the same provider — it must not stand up a parallel SDK (two global providers double-patch and conflict). The Node bootstrap exports registerSpanProcessor() and marks the extension point in code for exactly this.

3. Off by default, fail-open​

Telemetry is OFF unless OTEL_ENABLED=true and SIGNOZ_INGESTION_KEY is present. Dev / CI / test stay dark (no network export). Init is wrapped so a bad endpoint / missing key / SDK throw is logged and swallowed — a telemetry error never crashes the service, mirroring the existing Sentry fail-open posture. OTEL_SDK_DISABLED=true is a hard kill switch.

4. Load order​

OTel must initialize before Sentry and before any app module so its instrumentations patch http/express/pg/ioredis/undici (api) and fastapi/httpx/logging (agent) before those modules load. The api imports ./instrumentation/tracing as the first line of main.ts; the agent calls init_tracing() before FastAPI() is constructed.

5. Cross-service propagation​

W3C TraceContext (traceparent) is the default propagator. The api injects it on the outbound /chat call via @opentelemetry/instrumentation-undici (load-bearing — AgentClient uses native fetch/undici, not axios). The agent extracts it via opentelemetry-instrumentation-fastapi, making the agent's /chat span a child of the api span.

6. PII posture​

ADR-020 isolation classification — Telemetry spans/logs are observability data, not a tenant resource. They carry only pseudonymized identifiers (org_id / specialist_id / hashed customer_id) and framework/IO metadata. No raw customer PII, message bodies, or tool args are ever placed on SigNoz spans. Raw LLM content belongs to Langfuse (#1231) under SuperAdmin-only access, not here. The (deferred) wide-events interceptor is responsible for stamping the pseudonymized identifiers; the auto- instrumentation in this PR captures HTTP/DB/Redis metadata only.

Service naming​

humanwork-api-<APP_ENV> and humanwork-agent-<APP_ENV> (e.g. humanwork-api-staging), derived from APP_ENV when OTEL_SERVICE_NAME is unset.

Scope of the foundational PR (#1230)​

Shipped: the two SDK bootstraps, load-order wiring, graceful-shutdown flush, api→agent propagation via undici→fastapi, trace-id stamping on the agent's canonical log lines, env vars, this ADR, and OPENTELEMETRY.md.

Deferred (tracked as #1230 follow-ups): NestJS wide-events interceptor, BullMQ scheduled-task emitter, SigNoz dashboards, the "api 5xx > 1% over 5min → Slack" alert, the Haystack circuit-breaker gauge re-export, and frontend RUM.

Consequences​

  • Single managed backend, no collector ops, active-active safe.
  • #1231 has a clean extension point and will not fork the SDK.
  • Telemetry is invisible until explicitly enabled per env, so this PR is a no-op in CI/dev and safe to merge dark.
Last updated on by Degen