Architecture and lifecycle

End-to-end request flow

1. The Computer page calls GET /v1/remote-desktop/machines/mine with the platform JWT. The API filters by role, Expert access, Computer entitlement and assigned machine state.
2. The Expert selects an organization and calls POST /v1/remote-desktop/connect with { "orgId": "..." }.
3. The API resolves the current assigned machine and returns a server-owned connection plan. Device hostnames and ports remain server-side.
4. In webrtc mode the browser connects to the signaling broker as offerer. The connector is the answerer and maintains its own authenticated outbound signaling socket.
5. The browser receives H.264 and sends input over the input DataChannel. If native media cannot establish or later fails, the UI tries P2P RFB and then the relay descriptor.
6. The signaling or relay layer opens and closes a remote_desktop_sessions audit row. Client telemetry is reported asynchronously.

Transport architecture

Native WebRTC video

The browser adds a recv-only H.264 transceiver and creates the reliable, ordered input DataChannel. The connector uses Pion as a send-only WebRTC peer. Its Swift child process captures the guest display with ScreenCaptureKit, encodes low-delay H.264 with VideoToolbox and writes Annex-B access units to the Go connector. Pion packetizes those access units into RTP.

SPS and PPS precede every IDR. Browser PLI/FIR feedback forces a keyframe, and the connector resends the last keyframe after ten seconds of encoder inactivity so a static desktop does not let a VPN, TURN allocation, or middlebox reap the media flow.

The browser samples live RTT and recent packet loss every four seconds for automatic quality selection. A new tier must be selected by two consecutive samples before it is applied.

Tier	Resolution	Bitrate	Auto threshold
High	Configured base, default 1920x1080	8 Mbit/s	RTT under 120 ms and loss under 2%
Medium	1280x800	3 Mbit/s	RTT under 300 ms and loss under 5%
Low	960x600	1.5 Mbit/s	Anything worse

The browser sends {"type":"quality","tier":"high|medium|low"} on the input channel. The connector reconfigures the live encoder and forces a keyframe; it does not tear down the peer connection.

P2P RFB fallback

The browser and connector establish another WebRTC session using a reliable rfb DataChannel. The connector bridges the DataChannel byte stream to the guest's raw VNC service at 127.0.0.1:5900 by default, and noVNC handles the RFB protocol in the browser. This path avoids the API media relay but does not use native H.264 capture.

Direct WSS fallback

When connect_mode='direct_wss', public_hostname is set, tunnel_healthy=true, and Redis is available, the API creates a pending audit session and stores a 32-byte, base64url connect token in Redis. The browser opens wss://<public_hostname>/vnc?token=<token> through Cloudflare Tunnel.

The local connect guard consumes the token through Redis GETDEL, validates the runner organization and machine binding, activates the audit session, then bridges WebSocket frames to raw VNC. Missing, expired, replayed or mismatched tokens all fail without disclosing which check failed. If Redis is unavailable, the API does not issue an unverifiable token and returns the proxy plan.

API relay

The browser opens the exact raw WebSocket path /v1/remote-desktop. The API validates the platform token during upgrade, resolves the assigned machine and opens a raw TCP connection to tailscale_hostname:vnc_port. Binary bytes are copied in both directions; the API does not speak RFB and does not require websockify on this path.

When the address is in the Tailscale CGNAT range, the shared tailnet-aware connector uses TS_SOCKS_PROXY. Non-tailnet targets use a normal TCP socket.

Machine data model

org_machines combines inventory, placement, assignment and connection configuration:

Field	Contract
status	Static provisioning state: active, offline, or disabled
reachable_status	Live signal: online, unreachable, or unknown
pool_status	Lifecycle: provisioning, pooled, assigned, maintenance, or retired
org_id	Null unless the machine is assigned
tailscale_hostname, vnc_port	Server-side raw VNC relay target
connect_mode	webrtc, direct_wss, or proxy
public_hostname, tunnel_healthy	Direct WSS eligibility
jurisdiction, region, city, labels	Placement and assignment suggestions
metadata	Browser-safe display fields plus protected VNC credentials

The metadata display fields are model, osName, osVersion, cpu, and memoryGb. vncUsername and vncPassword are sensitive. They are omitted from /admin/machines and returned only by the entitlement-gated Expert machine/connect responses because Apple Remote Desktop authentication needs both values on the noVNC fallback paths.

machine_assignment_changes is append-only and records create, update, assign, unassign, maintenance, retire and restore transitions with actor, reason, and before/after snapshots. Credential metadata is deliberately excluded.

remote_desktop_sessions records organization, machine, user, start/end, close reason, transport and pending|active|ended state. WebRTC and relay audit persistence is best-effort so an audit outage does not strand a live desktop; operations should still alert on failed audit writes.

Machine lifecycle

create -> pooled <-> assigned
            |
            +-> maintenance -> pooled
            |        |
            +--------+-> retired -> pooled

An assigned machine must be unassigned before maintenance or retirement.

Only Superadmins create, edit, assign, unassign, maintain, retire, restore, or mint connector tokens. Account Managers may read the fleet overview with runtime endpoint details redacted. Assignment suggestions rank pooled machines using the organization's legal country and the jurisdiction mapping, but a Superadmin owns the final assignment.

Enabling Computer access is a separate action and is rejected unless an active assigned machine already exists. Revoking the entitlement immediately removes the machine from Expert listing and prevents new HTTP and WebSocket connects.

Reachability

For WebRTC machines, the connector's live signaling socket is authoritative: the broker marks the machine online on connect and every 60 seconds, then unreachable on disconnect. A one-minute leader-only TCP health job probes entitled assigned machines for relay reachability, but skips machines updated by connector presence in the previous 90 seconds so an API pod without tailnet reachability cannot overwrite an accurate connector signal.

Direct WSS health is separate. connect_guard.py posts a heartbeat every 30 seconds by default; the API offers the direct plan only while tunnel_healthy=true.

Authorization and isolation

The same four gates apply to list, connect, signaling and relay paths: caller role, Expert organization access, Computer entitlement, and active assigned machine. Superadmin bypasses Expert access but not the entitlement or assigned machine checks for a requested organization. Client roles cannot connect.

The browser never chooses a machine identifier for signaling. It supplies an organization; the API resolves the assigned machine and stamps a new server session ID onto all browser signaling frames. A connector token is a runner-audience JWT bound to one machine. The broker accepts it only while that machine remains active and assigned.

Security invariants

• Never expose tailscale_hostname, vnc_port, runner tokens, connector tokens, TURN API secrets, Cloudflare tunnel credentials or raw VNC credentials in logs.
• Never bake a connector token into a golden image. Enroll every clone after cloning and store the token only in the login Keychain.
• Never place HWDC_MACHINE_TOKEN in a production plist, environment file or process arguments. It is a local-development override only.
• Keep the connector in the guest Aqua session. Native capture and CGEvent injection fail closed without their exact TCC grants.
• Redis is required for active-active signaling and single-use direct tokens. In-process signaling is valid only for a single API pod.
• Keep no public inbound port for the WebRTC connector; it dials signaling outbound.
• Treat REMOTE_DESKTOP_TARGET_HOST and REMOTE_DESKTOP_TARGET_PORT as local test overrides, never as a production machine registry.