Skip to main content
Every chat client talks to an HQ over one WebSocket at /ws/chat. The socket carries JSON text frames in both directions. The client sends commands (start a turn, answer a question, cancel, queue a follow-up, stream microphone audio, drive a browser), and the HQ streams back the turn’s lifecycle, agent events, voice state, and browser frames. The same protocol is served on three routes to the same HQ. The HQ also serves a separate, server-to-client projects WebSocket for Desktop. The relay forwards /ws/chat upgrades to the HQ’s channel server on 9200, so a relayed socket behaves exactly like a loopback one. The mobile listener exists only when the HQ was started with both a management token and a chat token. The phone’s “Mobile bearer” and Desktop’s “chat token” are the same secret. The HQ accepts one chat token (--chat-token) on every /ws/chat listener and under /mobile/v1 over HTTP. The administrative management token is never accepted on /ws/chat.

Authentication

The HQ checks credentials during the upgrade. The rules are the same on every /ws/chat listener:
  1. If an Authorization header is present and non-empty, it must equal Bearer <chat-token>. Query parameters are ignored, and a ticket is not redeemed.
  2. With no Authorization header, the socket is accepted when ?token=<chat-token> matches or ?ticket=<ticket> redeems a valid ticket.
  3. Otherwise the upgrade completes and the HQ immediately closes the socket with code 4001 and reason Unauthorized.
Native clients send the header and keep the token out of URLs:
Desktop on loopback may use the query fallback:

WebSocket tickets

Browsers cannot set headers on a WebSocket upgrade, and the web client should not put the chat token in a URL. It mints a single-use ticket over HTTP with the Mobile bearer instead:
Then it connects with wss://<host>/ws/chat?ticket=<ticket>. A ticket:
  • Expires 30 seconds after it is minted.
  • Is consumed by its first redemption attempt, whether or not that attempt was in time.
  • Is redeemable on any /ws/chat listener of the HQ that minted it. One ticket store serves the whole HQ.
  • Is ignored, and left unredeemed, when the upgrade also carries an Authorization header.
See HQ Mobile API for the rest of the /mobile/v1 surface.

Versioning and capabilities

The socket has no in-band handshake or version frame. A client learns what the HQ supports from GET /mobile/v1/health (or GET /health on loopback) before it opens the socket: No capability advertises the browser live view. A client sends browser_view_start and handles browser_view_error with code: "unavailable". Compatibility rules:
  • Unknown server frame types. Clients reject a server frame whose type they do not know. iOS closes the socket and shows Update required. Desktop fails the running turn with Update Dash: the HQ sent an unsupported chat frame. A new server frame type is therefore a breaking change: update the clients before the HQ that sends it. browser_session is such a frame, and it can arrive mid-turn on any subscribed socket.
  • Unknown event variants. Clients must preserve or safely ignore agent events they do not recognize, so newer HQs remain compatible.
  • Unknown enum values. Clients decode voice_state.state, voice_stopped.reason, and the browser_view_* enums leniently, so a new value does not break the socket.
  • Unknown client frames. The HQ answers any frame it cannot parse with an error frame, code validation_failed, and keeps the socket open. See Errors.

Connection lifecycle

A socket is multiplexed. One socket can carry several conversations, one voice session, and one browser viewer at the same time. Every frame is correlated by its id, and most also carry a conversationId.

Resumable turns

Resumable turns are the protocol every current client uses. The turn outlives the socket.
  1. The client sends message with a client-generated turn id and resumable: true.
  2. The HQ persists the user and assistant messages and sends accepted. accepted always precedes the turn’s first event.
  3. The HQ streams event frames: text deltas, tool calls, questions, and a final response.
  4. The turn ends with exactly one terminal frame: done with outcome: "completed" or "cancelled", or a sequenced error.
Every accepted, event, done, and sequenced error carries a durable seq. The HQ writes the frame to the conversation’s event log before it sends it. Sequences are per conversation, start at 1, and keep increasing across turns. Transient events, currently only subagent_progress, are sent without a seq and are never replayed. The socket that started a turn is also subscribed to the conversation for the rest of its life.

Reconnecting

Closing a socket only detaches it. Provider work continues on the HQ. To catch up after a gap:
  1. Reconnect and authenticate.
  2. Optionally read the gap over HTTP with GET /mobile/v1/agents/:agentId/conversations/:conversationId/events?sinceSeq=<last-seq> and apply entries in ascending seq order.
  3. Send resume with the same turn id and the last seq you applied. The HQ replays every logged frame with a greater seq for that conversation, then attaches the socket to the turn if it is still running.
Never resend the message with a new turn id. Resending the same message with the same id is safe: the HQ does not run it twice. It sends that turn’s accepted again, replays the log from that point, and attaches the socket if the turn is still live.

Cancellation

Send cancel with the turn id. A cancelled turn ends with done and outcome: "cancelled". cancel for a turn that is no longer running does nothing.

Legacy turns

A message without resumable: true runs a legacy, connection-owned turn. Its frames carry id and, when the HQ has an event log, a seq, but no conversationId. cancel on a legacy turn stops it and answers immediately with {"type":"done","id":"<turn-id>"}. Closing the socket cancels every legacy turn it started. While a legacy turn runs, a second message from the same socket on the same conversation with streamingBehavior: "steer" or "followUp" is delivered to the running turn instead of starting a new one. New clients should use resumable turns and the pending-work commands.

Pending work

With conversation-control-v2, a client can queue work behind a running turn:
  • watch replays the conversation from a cursor and subscribes the socket to its queue. The HQ answers with watched, which includes the current queue.
  • follow_up queues a message to run after the current turn. interrupt_and_send queues a priority message and cancels the named active turn so it runs next.
  • edit_pending and remove_pending change a queued item, guarded by its version.
  • stop_conversation cancels the active turn and pauses the queue. resume_pending restarts a paused queue.
Each command is answered on the issuing socket by a command_receipt. Every watcher of the conversation receives queue_changed whenever the queue changes. Queued items start their own turns with a fresh turn id. Their accepted carries requestId (the follow_up command’s id) and pendingItemId, and is delivered to watchers. Watch the conversation to see turns started from the queue. Command IDs are idempotent. Retrying a command with the same id and the same payload returns the original receipt with status: "already_applied". Reusing an id with a different payload is rejected with an error frame, code validation_failed. A conversation holds at most 100 pending items.

Subscriptions

A socket receives a turn’s frames if it started or resumed that turn. It also receives turns it did not start in two cases:
  • subscribe, and the implicit subscription from message or resume, deliver turns the client could not have started itself: server-initiated notification turns (origin: "notification") and sub-agent child turns (origin: "parent"). Ordinary user turns started from another device are not delivered this way.
  • watch delivers every turn on the conversation, plus queue_changed.
subscribe and unsubscribe send no acknowledgement. browser_session frames go to both subscribers and watchers.

Heartbeats and timeouts

The HQ sends no application-level heartbeat on /ws/chat and enforces no idle timeout on it. A client that needs liveness detection must provide its own, such as WebSocket pings. Two timers apply inside features: the voice drain timer (8 seconds) and the browser-view safety acknowledgement (5 seconds).

Client frames

Every client frame is a JSON object with a string type and a string id. The HQ validates the fields listed as required; a frame that fails validation is answered with an error frame and does not change any state. Common field rules:
  • id is client-generated. The contract specifies a UUID. For turn frames it is the turn ID; for commands it is the command ID; for voice and browser frames it is the session or viewer ID.
  • conversationId must be 1 to 128 characters and must not contain /, \, or .., or start with .. The contract narrows it to a UUID, or sub_ plus a 26-character ULID for a sub-agent conversation.
  • images is an array of { "mediaType", "data" } objects, where data is standard base64. On resumable message, follow_up, interrupt_and_send, and edit_pending, the HQ accepts at most four images, only image/jpeg, image/png, image/gif, or image/webp, at most 5 MiB each and 12 MiB combined after decoding.

message

Starts a turn. With resumable: true the turn is durable and survives the socket. Only the HQ’s own voice session sends modality: "voice". A dictated message is ordinary typed text once transcribed, so a client must omit modality or send "text". location fields: JavaScript’s getTimezoneOffset() is west-positive and must be negated.
A resumable message fails with an error frame, code conversation_busy, while the conversation’s previous turn is still settling. The frame’s activeTurnId names that turn when the HQ knows it.

resume

Replays missed frames and reattaches the socket to a running turn.
The replay covers every turn in the conversation after sinceSeq, not only the named turn. The socket is also subscribed to the conversation. A conversation that does not exist or belongs to another agent returns error with code not_found.

answer

Answers a question event from the running turn.
If the turn is not running, the HQ returns error with code not_found.

cancel

Stops a running turn.
A resumable turn ends with done and outcome: "cancelled", and the cancelling socket receives it even if it did not start the turn. A legacy turn is answered immediately with a bare done.

subscribe

Receives notification turns and sub-agent child turns for a conversation this socket did not start. See Subscriptions.
No frame is sent on success. An unknown conversation returns error with code not_found.

unsubscribe

Ends a subscribe. Always succeeds, even for a deleted conversation, and sends no frame. It does not end a watch.

watch

Replays a conversation from a cursor, then subscribes the socket to all of its turns and its queue. Requires conversation-control-v2.
The HQ sends the replayed frames first, then watched. A watch lasts until the socket closes.

follow_up

Queues a message to run as its own turn after the current one. Requires conversation-control-v2.
If the conversation is idle and its queue is running, the item starts immediately.

interrupt_and_send

Queues a priority message and cancels the active turn so the message runs next. Requires conversation-control-v2.
If expectedActiveTurnId is not the running turn, the receipt is rejected with reason stale_execution and nothing is cancelled. Interrupting also sets the queue back to running.

stop_conversation

Cancels the active turn, if any, and pauses the queue. Requires conversation-control-v2.

resume_pending

Sets a paused queue back to running and starts the next item. Requires conversation-control-v2.
An empty queue is rejected with reason queue_empty.

edit_pending

Replaces the text and images of a queued item that has not started. Requires conversation-control-v2.
The receipt is rejected with not_found, already_claimed (the item has started), or version_conflict (the version changed).

remove_pending

Removes a queued item that has not started. Requires conversation-control-v2.
Rejections match edit_pending.

Voice and browser client frames

The voice_* client frames are documented under Voice mode, and the browser_view_* client frames under Browser live view.

Server frames

Server frames are JSON objects with a string type. Turn frames carry the turn’s id.

accepted

The HQ admitted a resumable turn and persisted its user and assistant messages. Always the turn’s first frame. origin and kind are omitted for an ordinary user turn in a user conversation, except on frames delivered to a watch, which always include them. Frames replayed from the event log never carry origin, kind, or requestId.
A sub-agent child turn:

event

One agent event from the running turn.
Agent event types: Persisted transcripts from older HQs may also contain the retired worker_spawned, worker_status, and worker_done events; clients drop them. New variants can be added at any time, so ignore types you do not know.

done

The turn ended.

error

Either a turn’s durable terminal failure, or the rejection of a client frame. The two are told apart by seq. An error without seq is not journaled and is never replayed. It does not end any running turn.

watched

Acknowledges a watch. Sent after the replayed frames.

command_receipt

The result of a pending-work command. Sent only to the socket that issued the command.
A command that the HQ cannot admit at all, such as one aimed at a conversation with a running legacy turn, is answered with an error frame instead of a receipt.

queue_changed

The conversation’s queue changed. Sent to every socket watching the conversation.

Queue snapshot

watched, command_receipt, and queue_changed carry the queue in this shape: Each item:

Voice and browser server frames

The voice_* server frames are documented under Voice mode, and the browser_view_* and browser_session frames under Browser live view.

Voice mode

Hands-free voice mode runs a spoken conversation over the same socket. The client streams microphone audio; the HQ detects utterances, transcribes them, runs each one as a resumable turn, and streams the spoken reply back. It requires the speech-v1 capability. Every voice_* frame in both directions carries the session id, which the client chooses in voice_start. A socket holds at most one voice session.

Session lifecycle

  1. The client sends voice_start. The HQ checks that speech is configured, that the conversation belongs to the agent, and that a speech provider is available. If all pass, it sends voice_state with state: "listening".
  2. The client streams voice_audio. The client may start streaming immediately; audio that arrives before listening is silently dropped.
  3. When the user stops speaking, the HQ sends voice_state transcribing, then a voice_transcript with final: true and a new turnId.
  4. The HQ starts a resumable turn with that turnId, channelId: "ios", and modality: "voice", and sends voice_state thinking. The socket receives the turn’s ordinary accepted, event, and done frames; accepted usually arrives just before thinking. The voice_transcript carrying the turnId always arrives before the turn’s accepted, so the client can render its user row first.
  5. As the reply is synthesized, the HQ sends voice_state speaking, then voice_speech chunks.
  6. The client sends voice_played as its playback drains. Once the turn is done and the client has acknowledged the last chunk, the HQ returns to listening.
  7. The session ends with voice_stopped.
Speaking while the HQ is speaking interrupts it (barge-in), once the reply has played for at least 300 ms. The HQ cancels the running turn, discards queued speech, returns to listening, and treats the interrupting utterance as the next turn. An utterance spoken during a turn without interrupting it is queued: its voice_transcript is sent without a turnId when it is heard, and again with a turnId when its turn starts. When the agent asks a question, the HQ speaks it, and the next utterance is sent as the answer. That transcript is sent without a turnId.

voice_start

Client frame. Starts a session. A second voice_start on the same socket first stops the running session with voice_stopped and reason: "replaced".
Failures are reported with voice_error: unavailable when speech is not configured or no provider is available, and invalid when the conversation does not exist or belongs to another agent.

voice_audio

Client frame. One chunk of microphone capture.
A pcm over the limit is answered with an ordinary error frame and code validation_failed, not voice_error. Audio is dropped while the session is muted.

voice_mute

Client frame. Mutes or unmutes the microphone. Playback continues.
Muting sends voice_state muted, and no other voice_state is sent while muted. Unmuting sends the state the session reached while muted.

voice_played

Client frame. Every voice_speech up to and including seq has finished playing.
After a turn’s last voice_speech, the HQ stays in speaking until it receives a voice_played with a seq at least that high, or until an 8-second safety timer fires. Clients that flush playback when the session leaves speaking must send it, or the last sentence of every reply is cut off. A stale or out-of-order seq is ignored. Do not acknowledge audio that a barge-in discarded.

voice_stop

Client frame. Ends the session. Idempotent: stopping when no session is running is not an error.
The HQ cancels any running voice turn and sends voice_stopped with reason: "client". A session stopped before its first listening ends without a voice_stopped. voice_audio, voice_mute, and voice_played sent with no session on the socket are answered with voice_error, code invalid.

voice_state

Server frame. The session’s phase.
The HQ never sends state: "stopped", although it is a valid value. Every session ends with a voice_stopped frame instead.

voice_transcript

Server frame. Text recognized from the user’s speech.
An utterance that transcribes to nothing produces no transcript and no turn.

voice_speech

Server frame. A chunk of the spoken reply.
voice_state speaking always precedes a turn’s first voice_speech.

voice_error

Server frame. A speech or session failure.
Most errors leave the session running:
  • A transcription that fails, or takes longer than 20 seconds (network), drops that utterance and returns to listening.
  • A synthesis failure with unavailable or network skips that sentence.
  • A turn that fails is reported with provider once its partial speech has played.
These end the session with voice_stopped and reason: "provider":
  • Three consecutive synthesis failures, or one with any other code.
  • A turn rejected because the conversation is busy (unavailable), missing, or unauthorized (invalid).

voice_stopped

Server frame. The session ended. No more voice_* frames follow for this id.

Browser live view

When an agent opens a browser in a conversation, a client can watch it live and take over input. The view is a JPEG screencast of the session’s active tab. Frames and viewer input belong to the person watching: the agent never sees them. Every browser_view_* frame carries the viewer id the client chose in browser_view_start. There is no capability flag. If the HQ has no live view, every browser_view_* frame is answered with browser_view_error and code: "unavailable".

Viewer rules

  • One viewer per socket. A second browser_view_start on the same socket stops the first with reason: "replaced".
  • One viewer per browser session, HQ-wide. A viewer on another socket, or another listener, replaces this one the same way.
  • Flow control. At most one browser_view_frame is unacknowledged at a time. While it is in flight, newer frames replace a single held frame; intermediate frames are dropped, not queued. Acknowledge each frame with browser_view_ack. If the client does not acknowledge a frame within 5 seconds, the HQ acknowledges it itself and sends the next one.
  • Control. control is "agent" while a turn is running in the conversation and "user" otherwise. The HQ re-evaluates it every second and before each input, and announces changes with browser_view_state. Input sent while control is "agent" is silently dropped.
  • The view stops with reason: "session_closed" when the browser closes and with "socket_closed" when the socket closes.

browser_view_start

Client frame. Starts viewing the conversation’s browser.
The HQ answers with browser_view_started, or with browser_view_error: invalid when the conversation does not belong to the agent, no_session when no browser is open in it, and unavailable when the view fails to start.

browser_view_ack

Client frame. Acknowledges a browser_view_frame.
An ack for any frame other than the one in flight is ignored.

browser_view_input

Client frame. Sends pointer, keyboard, text, or scroll input to the page. Effective only while control is "user". The HQ scales frame coordinates to the page. modifiers uses the Chrome DevTools Protocol bitmask: Alt 1, Ctrl 2, Meta 4, Shift 8.

browser_view_stop

Client frame. Stops the view. Idempotent.
The HQ answers with browser_view_stopped and reason: "client" when a view was running. browser_view_ack and browser_view_input for a viewer id that is not running are answered with browser_view_error, code invalid.

browser_view_started

Server frame. The view is running.
No other frame for this viewer precedes it.

browser_view_frame

Server frame. One screencast frame.

browser_view_state

Server frame. The page or control changed.

browser_view_stopped

Server frame. The view ended.

browser_view_error

Server frame. A live-view request failed.

browser_session

Server frame. The agent opened or closed a browser in a conversation. Sent to every socket that is subscribed to or watching the conversation, and often mid-turn. It is not sequenced and not replayed.
A sub-agent’s browser is not announced separately.

Projects WebSocket

Desktop’s Projects views receive live updates from a separate, broadcast-only socket on the management server. It exists only when the HQ has a projects database.
  • Listener: the loopback management server (--management-port, default 9300). It is not served on the mobile listener, and the relay’s device-facing surface accepts only /ws/chat upgrades.
  • Auth: the management token in the token query parameter. The chat token is not accepted. A missing or wrong token closes the socket with 4001 Unauthorized. When the HQ runs without a management token, no token is required.
  • Direction: server to client only. The HQ ignores anything the client sends.
  • Subscription: every client receives every event. There is no filtering, replay, or acknowledgement.
Each frame is a JSON object with a topic and a payload:
For the issue_id topics, refetch the issue’s detail over the management API. See Projects for the feature itself.

Loopback channel server

The channel server on 127.0.0.1:9200 serves the same /ws/chat protocol documented on this page, with the same frames, the same chat token, and the same HQ-wide ticket store. It differs from the mobile listener only in transport: plain ws:// on loopback instead of pinned TLS on all interfaces. It is also the listener the relay forwards device sockets to, so a relayed socket and a loopback socket behave identically. Desktop connects with the query-token fallback:
Change the port with --channel-port.

Errors and close codes

Error frames

The HQ never closes a healthy socket because of a bad frame. It answers with an error frame and keeps the socket open. Codes the socket can return on error: The contract’s MobileApiErrorCode list defines further codes shared with the HTTP API, so treat any unknown code as a generic failure. Legacy turns return error frames with only id, error, and sometimes seq.

Close codes

The HQ sends no other application close code. Through the relay, a device can also see the relay’s own 4401 and 4429 closes. See Relay API. When a socket closes, the HQ stops that socket’s voice session (reason: "socket"), stops its browser viewer (reason: "socket_closed"), drops its subscriptions and watches, and cancels its legacy turns. Resumable turns keep running.