Skip to main content

ADR-029 — Tool Execution Runtime Architecture

2026-06-05 status: E2E spec coverage shipped via #1608 (commit 5e7c82e4).

Status: Accepted (shipped 2026-06-04 via #1359 Sub-PRs 1-5)
Date: 2026-06-04
Authors: @ethumanity (implementation), @eng-agent (documentation)
Epic: #1359 (Tool execution end-to-end)

Context​

Prior to this ADR, the HumanWork platform had stubbed tool execution endpoints. The Specialist AI could only paraphrase client messages and draw from KB context — any factual lookup ("what's the status of my order?", "what's my account balance?") required manual escalation to a human Expert.

This architectural gap prevented the platform from delivering autonomous, data-driven Specialist responses. This ADR documents the tool execution runtime architecture that enables LLMs to invoke external vendor APIs (Shopify, QuickBooks, Jumio, etc.) via a controlled, audited gateway.

Decision​

Architecture Overview​

Tool execution follows a 4-layer architecture:

┌─────────────────────────────────────────────────────────────┐
│  Layer 1: LLM (Hermes / Claude)                             │
│  - Emits tool_use blocks based on HERMES_TOOLSETS env       │
│  - Receives tool results and incorporates into reply        │
└─────────────────────────────────────────────────────────────┘
                         ↓ tool_use block
                         ↓
┌─────────────────────────────────────────────────────────────┐
│  Layer 2: Agent Runtime (Python FastAPI)                    │
│  - Intercepts tool_use from LLM stream                      │
│  - Calls ToolGatewayClient → NestJS gateway                 │
│  - Feeds result back to LLM session (resume)                │
└─────────────────────────────────────────────────────────────┘
                         ↓ HTTP POST
                         ↓
┌─────────────────────────────────────────────────────────────┐
│  Layer 3: Tool Gateway (NestJS)                             │
│  - Validates tool in AgentRun.toolsManifest                 │
│  - Evaluates policy (allow / block / require_approval)      │
│  - Records ToolCall ledger row (input/output redacted)      │
│  - Delegates execution to RuntimeToolExecutor               │
│  - Appends audit events to Runtime Ledger                   │
└─────────────────────────────────────────────────────────────┘
                         ↓ dispatch
                         ↓
┌─────────────────────────────────────────────────────────────┐
│  Layer 4: Runtime Tool Executor (NestJS)                    │
│  - Fetches vendor credentials (AES-256-GCM encrypted)       │
│  - Dispatches to vendor-specific handler:                   │
│    • Nango proxy (OAuth integrations)                       │
│    • Shopify SDK (e-commerce)                               │
│    • Amazon SP-API (marketplace)                            │
│    • Jumio (KYC/identity verification)                      │
│  - Normalizes vendor response → ToolResult                  │
└─────────────────────────────────────────────────────────────┘

Data Flow (Happy Path)​

  1. User message arrives → "What's the status of my order #12345?"

  2. AgentRun created with toolsManifest populated from tool_permissions table (filtered to enabled=true for this Org/Specialist)

  3. Hermes spawned with HERMES_TOOLSETS env containing tool definitions from manifest

  4. LLM emits tool_use block:

    {
      "type": "tool_use",
      "id": "toolu_abc123",
      "name": "order_lookup",
      "input": {
        "action": "lookup",
        "orderId": "12345"
      }
    }

  5. Agent runtime intercepts tool_use, calls ToolGatewayClient.execute():

    result = await tool_gateway_client.execute_tool_via_gateway(
        run_id=run.id,
        tool_name="order_lookup",
        action="lookup",
        params={"orderId": "12345"},
        org_id=org.id,
        correlation_id=generate_correlation_id(),
    )

  6. Tool Gateway validates manifest:

    // Check if 'order_lookup.lookup' is in run.metadata.runtimeManifest.allowedTools
    if (!run.metadata.runtimeManifest.allowedTools.includes(`${toolName}.${action}`)) {
      throw new ForbiddenException('Tool not in manifest');
    }

  7. Policy evaluated:

    • Risk level assessed (low/medium/high)
    • Decision: allow | observe | require_expert_approval | require_client_approval | block
    • If require_*_approval: create ApprovalRequest, set ToolCall.status = 'waiting_approval', pause run
    • If block: set ToolCall.status = 'blocked', return error to LLM
    • If allow or observe: proceed to executor

  8. ToolCall ledger row created (status: 'running'):

    const toolCall = toolCallRepo.create({
      orgId,
      runId,
      osaId: run.osaId,
      toolName,
      action,
      correlationId,
      status: 'running',
      inputRedacted: redactSecrets(params), // apiKey → '[REDACTED]'
      policyDecisionId: decision.id,
      riskLevel: decision.riskLevel,
    });

  9. RuntimeToolExecutor dispatches to vendor handler:

    // Fetch encrypted credentials
    const cred = await credRepo.findOne({ where: { orgId, integrationType: 'shopify' } });
    const decrypted = decrypt(cred.encryptedConfig, MASTER_ENCRYPTION_KEY);

    // Call Shopify SDK
    const order = await shopifyClient.getOrder(params.orderId, decrypted.access_token);

    // Normalize result
    return {
      orderId: order.id,
      status: order.fulfillment_status,
      trackingNumber: order.tracking_company,
      items: order.line_items.map(i => ({ name: i.name, quantity: i.quantity })),
    };

  10. Gateway updates ToolCall (status: 'succeeded'):

    toolCall.status = 'succeeded';
    toolCall.outputRedacted = redactPii(executorResult);
    toolCall.durationMs = Date.now() - startTime;
    await toolCallRepo.save(toolCall);

  11. Audit event appended to Runtime Ledger:

    await runtimeLedger.append({
      type: 'tool_call.succeeded',
      orgId,
      runId,
      resourceId: toolCall.id,
      metadata: { toolName, action, durationMs: toolCall.durationMs },
    });

  12. Result returned to agent runtime:

    {
      "status": "succeeded",
      "toolCallId": "tool-uuid",
      "result": {
        "orderId": "12345",
        "status": "shipped",
        "trackingNumber": "TRACK-001",
        "items": [...]
      }
    }

  13. Agent runtime feeds result to LLM (Hermes resume):

    {
      "type": "tool_result",
      "tool_use_id": "toolu_abc123",
      "content": "{\"orderId\": \"12345\", \"status\": \"shipped\", ...}"
    }

  14. LLM generates final reply incorporating tool data:

    "I've looked up order #12345. It's currently shipped with tracking number TRACK-001. You can expect delivery by [estimated date]."

  15. Reply sent to customer, conversation continues.

Failure Modes​

1. Feature Flag Disabled​

Scenario: tool_calling_enabled FeatureFlag is false for the Org.

Behavior:

  • Gateway checks feature flag before policy evaluation
  • If disabled: ToolCall.status = 'blocked', errorCode = 'FEATURE_DISABLED'
  • Returns error message to LLM: "Tool execution is not enabled for this organization"
  • LLM replies to user: "I'm unable to look up that information right now. Let me connect you with an Expert."

Audit Trail:

  • ToolCall row created with status='blocked'
  • Runtime Ledger event: tool_call.blocked with reason='feature_disabled'

2. Tool Not in Manifest​

Scenario: LLM requests a tool not in AgentRun.toolsManifest.allowedTools.

Behavior:

  • Gateway validation fails before policy evaluation
  • Throws ForbiddenException (403)
  • ToolCall.status = 'blocked', errorCode = 'NOT_IN_MANIFEST'
  • Error propagated to LLM: "The requested tool is not available in this conversation context"

Why it happens:

  • Tool not enabled in tool_permissions for this Org
  • Specialist manifest doesn't include the tool
  • AgentRun.toolsManifest not properly populated from tool_permissions (Sub-PR 4 regression)

Audit Trail:

  • ToolCall row with status='blocked', errorCode='NOT_IN_MANIFEST'
  • Runtime Ledger event: tool_call.blocked with reason='not_in_manifest'

3. Policy Requires Approval​

Scenario: PolicyEngine returns decision: 'require_expert_approval' (high-risk tool like wire_transfer).

Behavior:

  • ToolCall.status = 'waiting_approval'
  • ApprovalRequest created, linked to ToolCall
  • AgentRun.status = 'waiting_approval', pendingApprovalCount += 1
  • Message sent to Expert queue
  • LLM receives "waiting_approval" status, replies: "I've prepared this action for expert review. You'll receive an update once it's approved."

Resolution:

  • Expert approves → ToolCall.status = 'approved' → executor runs → result to LLM
  • Expert rejects → ToolCall.status = 'blocked', errorCode = 'EXPERT_REJECTED'

Audit Trail:

  • ToolCall row with status='waiting_approval' → 'approved' or 'blocked'
  • ApprovalRequest row with status='approved' or 'rejected'
  • Runtime Ledger events: tool_call.approval_requested, tool_call.approved or tool_call.rejected

4. Vendor API Error (500)​

Scenario: External vendor API returns HTTP 500 or times out.

Behavior (current — no retry):

  • RuntimeToolExecutor catches error from vendor SDK
  • ToolCall.status = 'failed', errorText = sanitized error message
  • Error returned to LLM (not raw 500 text, sanitized)
  • LLM receives: "Tool execution failed due to a temporary service issue"
  • LLM replies: "I'm having trouble accessing the order system right now. Please try again in a moment, or I can connect you with an Expert."

Conversation continuity: AgentRun.status remains 'running' (tool failure does NOT crash the run).

Future retry behavior (not implemented in Sub-PRs 1-5): If retry logic is added later:

  • Retry up to 3 times with exponential backoff (1s, 2s, 4s)
  • Only retry on 5xx errors (not 4xx)
  • After 3 failures: mark ToolCall.status = 'failed', record retryCount: 3
  • Return final error to LLM after exhaustion

Audit Trail:

  • ToolCall row with status='failed', errorText='Vendor API error: ...'
  • Runtime Ledger event: tool_call.failed with metadata={'vendorStatusCode': 500}

5. Missing Credentials​

Scenario: No integration_credentials row exists for the required vendor (e.g., Org has order_lookup enabled but no Shopify credentials).

Behavior:

  • RuntimeToolExecutor throws NotFoundException
  • ToolCall.status = 'failed', errorCode = 'CREDENTIALS_NOT_FOUND'
  • Error to LLM: "Tool execution failed: integration not configured"
  • LLM replies: "I don't have access to the order system yet. Let me connect you with an Expert who can help."

Audit Trail:

  • ToolCall row with status='failed', errorCode='CREDENTIALS_NOT_FOUND'
  • Runtime Ledger event: tool_call.failed with reason='credentials_missing'

Security & Data Governance​

Credential Storage​

  • All vendor credentials stored in integration_credentials.encrypted_config (JSONB)
  • Encrypted with AES-256-GCM using MASTER_ENCRYPTION_KEY env var
  • Key rotation NOT currently supported (future ADR)
  • Credentials scoped to orgId — no cross-Org leakage

Input Redaction​

Before writing to ToolCall.input_redacted:

  • API keys, tokens, passwords → '[REDACTED]'
  • Credit card numbers → '[REDACTED]'
  • SSN, Tax ID → '[REDACTED]'
  • Non-sensitive params (orderId, productId) → preserved

Implementation: RedactionService.redactSecrets(params) uses regex patterns + field name heuristics.

Output Redaction​

Before writing to ToolCall.output_redacted:

  • Customer email, phone → '[REDACTED]'
  • Payment method details → '[REDACTED]'
  • Full addresses → partial redaction (zip code preserved)
  • Non-PII data (order status, tracking number) → preserved

Why redact in ledger but not in LLM response?

  • LLM needs full data to generate helpful replies
  • Ledger is long-term storage, subject to compliance audits
  • Conversation data is ephemeral and already scoped to Org × Specialist (ADR-020)

Audit Trail​

Every tool execution generates 3+ audit artifacts:

  1. ToolCall row (persistent ledger):

    • Input/output (redacted)
    • Duration, status, error details
    • Policy decision reference
    • Correlation ID (traces back to LLM session)

  2. Runtime Ledger events:

    • tool_call.requested
    • tool_call.succeeded / tool_call.failed / tool_call.blocked
    • Event metadata: orgId, runId, resourceId (toolCallId)

  3. Policy Decision row (if policy evaluated):

    • Decision outcome (allow/block/require_approval)
    • Risk level (low/medium/high)
    • Timestamp, user context

Manifest & Authorization​

AgentRun.toolsManifest Population​

When AgentRun is created (Sub-PR 4):

const enabledTools = await toolPermissionsRepo.find({
  where: { orgId, enabled: true },
});

const allowedTools = enabledTools.map(tp => `${tp.toolName}.${tp.action}`);

run.metadata = {
  runtimeManifest: {
    allowedTools, // e.g., ['order_lookup.lookup', 'shopify_query.get_product']
  },
};

Specialist-Specific Tools (Future)​

Currently tools are Org-scoped. ADR-020 row 3 specifies tools will be Specialist-scoped via tool_specialist_bindings table. That binding is NOT yet enforced (deferred to post-Sub-PR 5 work).

When implemented:

  • tool_permissions filtered by BOTH orgId AND specialistId
  • Cross-Specialist tool reuse requires explicit binding rows

Cost Tracking​

Tool execution cost is NOT currently tracked in ToolCall rows (no costCents column exists). Future work:

  • Add ToolCall.costCents (Nango charges per API call, Jumio per lookup)
  • Rollup to AgentRun.totalCostCents (currently only LLM token cost)
  • Monthly invoice includes tool execution costs in messages.cost_cents (post ADR-020 message billing merge)

Retry Policy​

Current implementation (Sub-PRs 1-5): NO retry logic. Tool failures are immediate and terminal for that invocation.

Rationale:

  • Vendor APIs have their own retry/circuit breaker logic
  • Retrying blindly can amplify outages (thundering herd)
  • Idempotency is hard (some tools are not idempotent, e.g., send_invoice)

Future considerations:

  • Add retry for specific error codes (503 Service Unavailable, 429 Rate Limit)
  • Require tools to declare idempotent: true in catalog to enable retry
  • Track retryCount in ToolCall ledger

Consequences​

What This Enables​

  • ✅ Specialists can answer factual queries autonomously (order status, account balance, KYC results)
  • ✅ Full audit trail for compliance (who invoked what tool, when, with what data)
  • ✅ Policy-based controls (block high-risk tools, require approval for financial actions)
  • ✅ Vendor credential isolation (Org A cannot access Org B's Shopify store)

What Changed​

Database schema:

  • tool_calls table (new): ledger of all tool invocations
  • agent_runs.metadata.runtimeManifest (populated): allowedTools array
  • integration_credentials.encrypted_config (existing, now used)

Runtime:

  • HumanWorkRuntimeToolExecutor (new): replaces NoopRuntimeToolExecutor
  • ToolGatewayService (modified): now actually executes tools (was ledger-only stub)
  • agent/tool_gateway_client.py (new): HTTP client to call NestJS gateway
  • agent/tool_exec.py (modified): removed stubs, now calls gateway client
  • Hermes HERMES_TOOLSETS env (now populated per-OSA from manifest)

Deployment:

  • MASTER_ENCRYPTION_KEY env var required in production (for credential decryption)
  • tool_calling_enabled FeatureFlag controls rollout (default: false, enable per-Org)

Limitations​

  1. No retry logic — vendor 500 errors are terminal (addressed in future ADR)
  2. No cost tracking — tool execution cost not rolled into billing (future work)
  3. No cross-Specialist tool isolation — ADR-020 row 3 binding not enforced (future)
  4. No credential rotation — credentials are static until manually updated (future ADR)
  5. Limited tool catalog — only 3 tools shipped (order_lookup, shopify_query, amazon_query); more coming

Migration Path​

Feature flag rollout:

  1. Deploy Sub-PRs 1-5 to staging with tool_calling_enabled=false (default)
  2. Run manual smoke test (see docs/staging/smoke-test-tool-execution.md)
  3. Enable for internal Orgs first (tool_calling_enabled=true for HumanWork's own support org)
  4. Monitor ToolCall ledger for errors, tune redaction rules
  5. Gradual rollout to beta customers (1 Org per day, monitor for 24h)
  6. General availability after 2 weeks of stable operation

Rollback plan:

  • Set tool_calling_enabled=false globally → LLMs stop invoking tools
  • Existing ToolCall ledger rows preserved (read-only)
  • Revert NestJS deployment → NoopRuntimeToolExecutor stub

References​

  • Epic #1359 (Tool execution end-to-end)
  • ADR-020 (Isolation classification — tools are Specialist-scoped, row 3)
  • api/src/runtime-control-plane/tool-gateway.service.ts — Gateway implementation
  • api/src/runtime-control-plane/runtime-tool-executor.service.ts — Executor implementation
  • agent/tool_gateway_client.py — Python client for agent → gateway HTTP
  • docs/staging/smoke-test-tool-execution.md — Manual test procedure
Last updated on by Degen