/ws/chat. The socket carries JSON text
frames in both directions. The client sends commands (start a turn, answer a question, cancel,
queue a follow-up, stream microphone audio, drive a browser), and the HQ streams back the turn’s
lifecycle, agent events, voice state, and browser frames.
The same protocol is served on three routes to the same HQ. The HQ also serves a separate,
server-to-client projects WebSocket for Desktop.
The relay forwards
/ws/chat upgrades to the HQ’s channel server on 9200, so a relayed socket
behaves exactly like a loopback one. The mobile listener exists only when the HQ was started with
both a management token and a chat token.
The phone’s “Mobile bearer” and Desktop’s “chat token” are the same secret. The HQ accepts one
chat token (--chat-token) on every /ws/chat listener and under /mobile/v1 over HTTP. The
administrative management token is never accepted on /ws/chat.
Authentication
The HQ checks credentials during the upgrade. The rules are the same on every/ws/chat
listener:
- If an
Authorizationheader is present and non-empty, it must equalBearer <chat-token>. Query parameters are ignored, and aticketis not redeemed. - With no
Authorizationheader, the socket is accepted when?token=<chat-token>matches or?ticket=<ticket>redeems a valid ticket. - Otherwise the upgrade completes and the HQ immediately closes the socket with code
4001and reasonUnauthorized.
WebSocket tickets
Browsers cannot set headers on a WebSocket upgrade, and the web client should not put the chat token in a URL. It mints a single-use ticket over HTTP with the Mobile bearer instead:wss://<host>/ws/chat?ticket=<ticket>. A ticket:
- Expires 30 seconds after it is minted.
- Is consumed by its first redemption attempt, whether or not that attempt was in time.
- Is redeemable on any
/ws/chatlistener of the HQ that minted it. One ticket store serves the whole HQ. - Is ignored, and left unredeemed, when the upgrade also carries an
Authorizationheader.
/mobile/v1 surface.
Versioning and capabilities
The socket has no in-band handshake or version frame. A client learns what the HQ supports fromGET /mobile/v1/health (or GET /health on loopback) before it opens the socket:
No capability advertises the browser live view. A client sends
browser_view_start and handles browser_view_error with code: "unavailable".
Compatibility rules:
- Unknown server frame types. Clients reject a server frame whose
typethey do not know. iOS closes the socket and shows Update required. Desktop fails the running turn with Update Dash: the HQ sent an unsupported chat frame. A new server frame type is therefore a breaking change: update the clients before the HQ that sends it.browser_sessionis such a frame, and it can arrive mid-turn on any subscribed socket. - Unknown
eventvariants. Clients must preserve or safely ignore agent events they do not recognize, so newer HQs remain compatible. - Unknown enum values. Clients decode
voice_state.state,voice_stopped.reason, and thebrowser_view_*enums leniently, so a new value does not break the socket. - Unknown client frames. The HQ answers any frame it cannot parse with an
errorframe, codevalidation_failed, and keeps the socket open. See Errors.
Connection lifecycle
A socket is multiplexed. One socket can carry several conversations, one voice session, and one browser viewer at the same time. Every frame is correlated by itsid, and most also carry a
conversationId.
Resumable turns
Resumable turns are the protocol every current client uses. The turn outlives the socket.- The client sends
messagewith a client-generated turnidandresumable: true. - The HQ persists the user and assistant messages and sends
accepted.acceptedalways precedes the turn’s firstevent. - The HQ streams
eventframes: text deltas, tool calls, questions, and a finalresponse. - The turn ends with exactly one terminal frame:
donewithoutcome: "completed"or"cancelled", or a sequencederror.
accepted, event, done, and sequenced error carries a durable seq. The HQ writes
the frame to the conversation’s event log before it sends it. Sequences are per conversation,
start at 1, and keep increasing across turns. Transient events, currently only
subagent_progress, are sent without a seq and are never replayed.
The socket that started a turn is also subscribed to the conversation for the rest of its life.
Reconnecting
Closing a socket only detaches it. Provider work continues on the HQ. To catch up after a gap:- Reconnect and authenticate.
- Optionally read the gap over HTTP with
GET /mobile/v1/agents/:agentId/conversations/:conversationId/events?sinceSeq=<last-seq>and apply entries in ascendingseqorder. - Send
resumewith the same turnidand the lastseqyou applied. The HQ replays every logged frame with a greaterseqfor that conversation, then attaches the socket to the turn if it is still running.
message with a new turn id. Resending the same message with the same id
is safe: the HQ does not run it twice. It sends that turn’s accepted again, replays the log
from that point, and attaches the socket if the turn is still live.
Cancellation
Sendcancel with the turn id. A cancelled turn ends with done and
outcome: "cancelled". cancel for a turn that is no longer running does nothing.
Legacy turns
Amessage without resumable: true runs a legacy, connection-owned turn. Its frames carry
id and, when the HQ has an event log, a seq, but no conversationId. cancel on a legacy
turn stops it and answers immediately with {"type":"done","id":"<turn-id>"}. Closing the
socket cancels every legacy turn it started. While a legacy turn runs, a second message from the
same socket on the same conversation with streamingBehavior: "steer" or "followUp" is delivered to the running turn
instead of starting a new one. New clients should use resumable turns and the
pending-work commands.
Pending work
Withconversation-control-v2, a client can queue work behind a running turn:
watchreplays the conversation from a cursor and subscribes the socket to its queue. The HQ answers withwatched, which includes the current queue.follow_upqueues a message to run after the current turn.interrupt_and_sendqueues a priority message and cancels the named active turn so it runs next.edit_pendingandremove_pendingchange a queued item, guarded by itsversion.stop_conversationcancels the active turn and pauses the queue.resume_pendingrestarts a paused queue.
command_receipt. Every watcher of the
conversation receives queue_changed whenever the queue changes. Queued items start their own
turns with a fresh turn id. Their accepted carries requestId (the follow_up command’s
id) and pendingItemId, and is delivered to watchers. Watch the conversation to see turns
started from the queue.
Command IDs are idempotent. Retrying a command with the same id and the same payload returns
the original receipt with status: "already_applied". Reusing an id with a different payload
is rejected with an error frame, code validation_failed. A conversation holds at most 100
pending items.
Subscriptions
A socket receives a turn’s frames if it started or resumed that turn. It also receives turns it did not start in two cases:subscribe, and the implicit subscription frommessageorresume, deliver turns the client could not have started itself: server-initiated notification turns (origin: "notification") and sub-agent child turns (origin: "parent"). Ordinary user turns started from another device are not delivered this way.watchdelivers every turn on the conversation, plusqueue_changed.
subscribe and unsubscribe send no acknowledgement. browser_session frames go to both
subscribers and watchers.
Heartbeats and timeouts
The HQ sends no application-level heartbeat on/ws/chat and enforces no idle timeout on it. A
client that needs liveness detection must provide its own, such as WebSocket pings. Two timers
apply inside features: the voice drain timer (8 seconds) and the browser-view safety
acknowledgement (5 seconds).
Client frames
Every client frame is a JSON object with a stringtype and a string id. The HQ validates
the fields listed as required; a frame that fails validation is answered with an error frame
and does not change any state.
Common field rules:
idis client-generated. The contract specifies a UUID. For turn frames it is the turn ID; for commands it is the command ID; for voice and browser frames it is the session or viewer ID.conversationIdmust be 1 to 128 characters and must not contain/,\, or.., or start with.. The contract narrows it to a UUID, orsub_plus a 26-character ULID for a sub-agent conversation.imagesis an array of{ "mediaType", "data" }objects, wheredatais standard base64. On resumablemessage,follow_up,interrupt_and_send, andedit_pending, the HQ accepts at most four images, onlyimage/jpeg,image/png,image/gif, orimage/webp, at most 5 MiB each and 12 MiB combined after decoding.
message
Starts a turn. Withresumable: true the turn is durable and survives the socket.
Only the HQ’s own voice session sends
modality: "voice". A dictated message is
ordinary typed text once transcribed, so a client must omit modality or send "text".
location fields:
JavaScript’s
getTimezoneOffset() is west-positive and must be negated.
message fails with an error frame, code conversation_busy, while the
conversation’s previous turn is still settling. The frame’s activeTurnId names that turn when
the HQ knows it.
resume
Replays missed frames and reattaches the socket to a running turn.sinceSeq, not only the named turn. The
socket is also subscribed to the conversation. A conversation that does not exist or belongs to
another agent returns error with code not_found.
answer
Answers aquestion event from the running turn.
error with code not_found.
cancel
Stops a running turn.done and outcome: "cancelled", and the cancelling socket receives
it even if it did not start the turn. A legacy turn is answered immediately with a bare
done.
subscribe
Receives notification turns and sub-agent child turns for a conversation this socket did not start. See Subscriptions.error with code not_found.
unsubscribe
Ends asubscribe. Always succeeds, even for a deleted conversation, and sends no frame. It does
not end a watch.
watch
Replays a conversation from a cursor, then subscribes the socket to all of its turns and its queue. Requiresconversation-control-v2.
watched. A watch lasts until the socket closes.
follow_up
Queues a message to run as its own turn after the current one. Requiresconversation-control-v2.
interrupt_and_send
Queues a priority message and cancels the active turn so the message runs next. Requiresconversation-control-v2.
expectedActiveTurnId is not the running turn, the receipt is rejected with reason
stale_execution and nothing is cancelled. Interrupting also sets the queue back to running.
stop_conversation
Cancels the active turn, if any, and pauses the queue. Requiresconversation-control-v2.
resume_pending
Sets a paused queue back to running and starts the next item. Requiresconversation-control-v2.
rejected with reason queue_empty.
edit_pending
Replaces the text and images of a queued item that has not started. Requiresconversation-control-v2.
rejected with not_found, already_claimed (the item has started), or
version_conflict (the version changed).
remove_pending
Removes a queued item that has not started. Requiresconversation-control-v2.
edit_pending.
Voice and browser client frames
Thevoice_* client frames are documented under Voice mode, and the
browser_view_* client frames under Browser live view.
Server frames
Server frames are JSON objects with a stringtype. Turn frames carry the turn’s id.
accepted
The HQ admitted a resumable turn and persisted its user and assistant messages. Always the turn’s first frame.origin and kind are omitted for an ordinary user turn in a user conversation, except on
frames delivered to a watch, which always include them. Frames replayed from the event log
never carry origin, kind, or requestId.
event
One agent event from the running turn.
Persisted transcripts from older HQs may also contain the retired
worker_spawned,
worker_status, and worker_done events; clients drop them. New variants can be added at any
time, so ignore types you do not know.
done
The turn ended.error
Either a turn’s durable terminal failure, or the rejection of a client frame. The two are told apart byseq.
An
error without seq is not journaled and is never replayed. It does not end any running
turn.
watched
Acknowledges awatch. Sent after the replayed frames.
command_receipt
The result of a pending-work command. Sent only to the socket that issued the command.error frame instead of a receipt.
queue_changed
The conversation’s queue changed. Sent to every socket watching the conversation.Queue snapshot
watched, command_receipt, and queue_changed carry the queue in this shape:
Each item:
Voice and browser server frames
Thevoice_* server frames are documented under Voice mode, and the
browser_view_* and browser_session frames under Browser live view.
Voice mode
Hands-free voice mode runs a spoken conversation over the same socket. The client streams microphone audio; the HQ detects utterances, transcribes them, runs each one as a resumable turn, and streams the spoken reply back. It requires thespeech-v1 capability.
Every voice_* frame in both directions carries the session id, which the client chooses in
voice_start. A socket holds at most one voice session.
Session lifecycle
- The client sends
voice_start. The HQ checks that speech is configured, that the conversation belongs to the agent, and that a speech provider is available. If all pass, it sendsvoice_statewithstate: "listening". - The client streams
voice_audio. The client may start streaming immediately; audio that arrives beforelisteningis silently dropped. - When the user stops speaking, the HQ sends
voice_statetranscribing, then avoice_transcriptwithfinal: trueand a newturnId. - The HQ starts a resumable turn with that
turnId,channelId: "ios", andmodality: "voice", and sendsvoice_statethinking. The socket receives the turn’s ordinaryaccepted,event, anddoneframes;acceptedusually arrives just beforethinking. Thevoice_transcriptcarrying theturnIdalways arrives before the turn’saccepted, so the client can render its user row first. - As the reply is synthesized, the HQ sends
voice_statespeaking, thenvoice_speechchunks. - The client sends
voice_playedas its playback drains. Once the turn is done and the client has acknowledged the last chunk, the HQ returns tolistening. - The session ends with
voice_stopped.
listening, and
treats the interrupting utterance as the next turn. An utterance spoken during a turn without
interrupting it is queued: its voice_transcript is sent without a turnId when it is heard, and
again with a turnId when its turn starts.
When the agent asks a question, the HQ speaks it, and the next utterance is sent as the
answer. That transcript is sent without a turnId.
voice_start
Client frame. Starts a session. A secondvoice_start on the same socket first stops the running
session with voice_stopped and reason: "replaced".
voice_error: unavailable when speech is not configured or no
provider is available, and invalid when the conversation does not exist or belongs to another
agent.
voice_audio
Client frame. One chunk of microphone capture.pcm over the limit is answered with an ordinary error frame and code validation_failed,
not voice_error. Audio is dropped while the session is muted.
voice_mute
Client frame. Mutes or unmutes the microphone. Playback continues.voice_state muted, and no other voice_state is sent while muted. Unmuting sends the state the session reached while muted.
voice_played
Client frame. Everyvoice_speech up to and including seq has finished playing.
voice_speech, the HQ stays in speaking until it receives a voice_played
with a seq at least that high, or until an 8-second safety timer fires. Clients that flush
playback when the session leaves speaking must send it, or the last sentence of every reply is
cut off. A stale or out-of-order seq is ignored. Do not acknowledge audio that a barge-in
discarded.
voice_stop
Client frame. Ends the session. Idempotent: stopping when no session is running is not an error.voice_stopped with reason: "client". A
session stopped before its first listening ends without a voice_stopped.
voice_audio, voice_mute, and voice_played sent with no session on the socket are answered
with voice_error, code invalid.
voice_state
Server frame. The session’s phase.state: "stopped", although it is a valid value. Every session ends with a
voice_stopped frame instead.
voice_transcript
Server frame. Text recognized from the user’s speech.voice_speech
Server frame. A chunk of the spoken reply.voice_state speaking always precedes a turn’s first voice_speech.
voice_error
Server frame. A speech or session failure.- A transcription that fails, or takes longer than 20 seconds (
network), drops that utterance and returns tolistening. - A synthesis failure with
unavailableornetworkskips that sentence. - A turn that fails is reported with
provideronce its partial speech has played.
voice_stopped and reason: "provider":
- Three consecutive synthesis failures, or one with any other code.
- A turn rejected because the conversation is busy (
unavailable), missing, or unauthorized (invalid).
voice_stopped
Server frame. The session ended. No morevoice_* frames follow for this id.
Browser live view
When an agent opens a browser in a conversation, a client can watch it live and take over input. The view is a JPEG screencast of the session’s active tab. Frames and viewer input belong to the person watching: the agent never sees them. Everybrowser_view_* frame carries the viewer id the client chose in browser_view_start.
There is no capability flag. If the HQ has no live view, every browser_view_* frame is answered
with browser_view_error and code: "unavailable".
Viewer rules
- One viewer per socket. A second
browser_view_starton the same socket stops the first withreason: "replaced". - One viewer per browser session, HQ-wide. A viewer on another socket, or another listener, replaces this one the same way.
- Flow control. At most one
browser_view_frameis unacknowledged at a time. While it is in flight, newer frames replace a single held frame; intermediate frames are dropped, not queued. Acknowledge each frame withbrowser_view_ack. If the client does not acknowledge a frame within 5 seconds, the HQ acknowledges it itself and sends the next one. - Control.
controlis"agent"while a turn is running in the conversation and"user"otherwise. The HQ re-evaluates it every second and before each input, and announces changes withbrowser_view_state. Input sent whilecontrolis"agent"is silently dropped. - The view stops with
reason: "session_closed"when the browser closes and with"socket_closed"when the socket closes.
browser_view_start
Client frame. Starts viewing the conversation’s browser.browser_view_started, or with browser_view_error: invalid when the
conversation does not belong to the agent, no_session when no browser is open in it, and
unavailable when the view fails to start.
browser_view_ack
Client frame. Acknowledges abrowser_view_frame.
browser_view_input
Client frame. Sends pointer, keyboard, text, or scroll input to the page. Effective only whilecontrol is "user".
The HQ scales frame coordinates to the page.
modifiers uses the Chrome DevTools Protocol
bitmask: Alt 1, Ctrl 2, Meta 4, Shift 8.
browser_view_stop
Client frame. Stops the view. Idempotent.browser_view_stopped and reason: "client" when a view was running.
browser_view_ack and browser_view_input for a viewer id that is not running are answered
with browser_view_error, code invalid.
browser_view_started
Server frame. The view is running.browser_view_frame
Server frame. One screencast frame.browser_view_state
Server frame. The page or control changed.browser_view_stopped
Server frame. The view ended.browser_view_error
Server frame. A live-view request failed.browser_session
Server frame. The agent opened or closed a browser in a conversation. Sent to every socket that is subscribed to or watching the conversation, and often mid-turn. It is not sequenced and not replayed.Projects WebSocket
Desktop’s Projects views receive live updates from a separate, broadcast-only socket on the management server. It exists only when the HQ has a projects database.- Listener: the loopback management server (
--management-port, default9300). It is not served on the mobile listener, and the relay’s device-facing surface accepts only/ws/chatupgrades. - Auth: the management token in the
tokenquery parameter. The chat token is not accepted. A missing or wrong token closes the socket with4001Unauthorized. When the HQ runs without a management token, no token is required. - Direction: server to client only. The HQ ignores anything the client sends.
- Subscription: every client receives every event. There is no filtering, replay, or acknowledgement.
topic and a payload:
For the
issue_id topics, refetch the issue’s detail over the management API. See
Projects for the feature itself.
Loopback channel server
The channel server on127.0.0.1:9200 serves the same /ws/chat protocol documented on this
page, with the same frames, the same chat token, and the same HQ-wide ticket store. It differs
from the mobile listener only in transport: plain ws:// on loopback instead of pinned TLS on all
interfaces. It is also the listener the relay forwards device sockets to, so a relayed socket and
a loopback socket behave identically. Desktop connects with the query-token fallback:
--channel-port.
Errors and close codes
Error frames
The HQ never closes a healthy socket because of a bad frame. It answers with anerror frame and
keeps the socket open.
Codes the socket can return on
error:
The contract’s
MobileApiErrorCode list defines further codes shared with the HTTP API, so
treat any unknown code as a generic failure. Legacy turns return error frames with only id,
error, and sometimes seq.
Close codes
The HQ sends no other application close code. Through the relay, a device can also see the
relay’s own
4401 and 4429 closes. See Relay API.
When a socket closes, the HQ stops that socket’s voice session (reason: "socket"), stops its
browser viewer (reason: "socket_closed"), drops its subscriptions and watches, and cancels its
legacy turns. Resumable turns keep running.