Expert Computer

Expert Computer gives an authorized Expert or Superadmin an interactive macOS desktop inside /workspace/computer. The current implementation is a managed remote-desktop system: the platform assigns one machine to an organization, authorizes each connection, selects the best available transport, and records the session. It is not the Specialist/Hermes execution runtime and it does not move AgentFS or SessionDB into the browser session.

This folder is the canonical technical documentation for the feature on dev. Component READMEs should contain only local build instructions and link back here. Historical documents describing the original VNC-only PoC or the hardware-video stopping decision are not implementation specifications.

Document map

Document	Audience	Contents
Architecture and lifecycle	Engineers, reviewers	Components, data model, authorization, connection selection, fallback behavior, security boundaries
API and protocols	API, frontend, connector engineers	REST and WebSocket contracts, signaling, media and DataChannel protocols, session states
Deployment and operations	Device and platform operators	Machine provisioning, connector enrollment, TCC, TURN, Tailscale, direct WSS, verification and recovery
Testing and observability	Engineers, QA, on-call	Automated tests, regional latency harness, metrics, Sentry diagnostics and release checks

Shipped system at a glance

Expert browser (/workspace/computer)
        |
        | platform JWT + orgId
        v
NestJS API
  - authorization + Computer entitlement
  - org_machines pool and assignment
  - connection plan and session audit
  - WebRTC signaling through Redis pub/sub
  - short-lived ICE/TURN credentials
        |
        | outbound signaling WebSocket
        v
Guest macOS hwork-desktop-connector
  - ScreenCaptureKit -> VideoToolbox H.264 -> WebRTC video
  - validated input DataChannel -> CGEvent
  - RFB DataChannel -> guest 127.0.0.1:5900 fallback

WebRTC machine fallback:
  native WebRTC video -> P2P RFB -> server-provided RFB fallback (currently API relay)

Direct-WSS machine fallback:
  direct Cloudflare WSS -> API VNC relay

The connector runs inside the macOS guest's logged-in Aqua session. That is a correctness and security boundary: it must see the guest WindowServer, own the guest user's Screen Recording and Accessibility grants, and reach the guest's loopback VNC service. Installing it on the Mac Mini host or as a system LaunchDaemon captures and controls the wrong session.

Eligibility invariant

A machine is visible and connectable only when all of the following are true:

1. The caller is an Expert with ExpertAccessService access to the organization, or is a Superadmin.
2. The organization has metadata.features.computerAccess = true.
3. Exactly one org_machines row is assigned to the organization.
4. The machine has status = 'active' and pool_status = 'assigned'.

Client roles are rejected. Pooled, maintenance, and retired machines have no organization binding, are not Expert-visible, and are never dialed. The partial unique index on assigned org_id enforces at most one assigned machine per organization.

Connection policy

The machine row owns connect_mode; the browser's default auto choice follows the server plan. relay and p2p are explicit operator/development overrides.

Machine mode	Primary path	Automatic recovery
webrtc	Native H.264 WebRTC video plus the input DataChannel	P2P RFB, then the relay descriptor returned by the API
direct_wss	noVNC through a per-machine Cloudflare Tunnel	Retry direct once with a new token, then API relay
proxy	noVNC through the API's raw WebSocket-to-VNC relay	No lower transport

For webrtc, a peer-reflexive, server-reflexive, or host ICE pair is direct; a relay candidate means TURN is carrying the WebRTC session. “P2P” therefore describes the protocol path, not a guarantee that TURN is absent.

The native video path uses a 12-second connection timeout and can attempt up to three ICE restarts after a transient failure. The direct WSS handshake has an eight-second watchdog. Failures are intentionally recoverable: audit and telemetry writes are best-effort and must not prevent the Expert from reaching a fallback path.

Component ownership

Area	Source of truth
Expert UI, noVNC and WebRTC receiver	frontend/src/app/workspace/computer/
Browser API types/helpers	frontend/src/lib/api.ts
Authorization, connection planning, relay and audit	api/src/remote-desktop/
WebRTC signaling, ICE endpoint and telemetry	api/src/remote-desktop-webrtc/
Machine/session entities	api/src/common/entities.ts
macOS connector	remote-desktop-agent/
ScreenCaptureKit/VideoToolbox encoder	remote-desktop-agent/native-video/
Guest enrollment and launchd files	devices/mac-mini/
Direct WSS connect guard and Cloudflare templates	devices/mac-mini/connect_guard.py, devices/mac-mini/cloudflared/
Regional latency harness	e2e/latency/

Product behavior

The Computer page lists only entitled, assigned machines the caller may access. It displays live reachability, machine display metadata, placement, the active transport, RTT, adaptive quality and recent session records. The organization selector is locked while a session is connecting or connected. The Expert can disconnect, enter fullscreen and inspect the latest close reason.

For native video, the guest cursor should be excluded from capture with HWDC_VIDEO_SHOW_CURSOR=false; the browser renders a local cursor immediately to avoid round-trip cursor lag. The fallback noVNC paths render the remote VNC cursor normally.

Terminology

Term	Meaning
Expert Computer	The complete platform feature documented here
connector	hwork-desktop-connector, the device-side WebRTC endpoint; it is not an AI agent
guest	The macOS VM whose desktop the Expert sees and controls
native video	ScreenCaptureKit and VideoToolbox H.264 over WebRTC
P2P RFB	Raw VNC/RFB carried in a WebRTC rfb DataChannel
direct WSS	noVNC through Cloudflare Tunnel and the device connect guard
API relay	Raw browser WebSocket bridged by the API to the machine's raw VNC port
entitlement	Organization feature flag metadata.features.computerAccess

Non-goals and boundaries

• Expert Computer is not the Hermes agent runtime, the Specialist chat runtime, a workspace file-sync contract, or an AgentFS transport.
• The API relay carries raw RFB. org_machines.vnc_port must identify a raw VNC endpoint, not websockify.
• The WebRTC connector opens an outbound signaling socket and requires no public inbound device port.
• public_hostname is used by direct_wss; it is not required for native WebRTC signaling.
• Current source code and migrations are authoritative if a historical issue, test record, or archived planning document disagrees with this set.