Wire Formats

Every socket transport carries the same object model: elements sent by the client and output elements sent by the pipeline. Only the encoding differs. It is selected per connection and per direction with the inputFormat and outputFormat parameters (see Pipeline Configuration).

Format Frame / payload Encoding

TEXT

Text frame, or the Socket.IO text event

Human-readable strings. Control messages use the #name:params scheme; anything else is a plain text token.

JSON

Text frame, or the Socket.IO text event

JSON object with a type discriminator property naming the element type.

MSGPACK

Binary frame, or the Socket.IO msgpack event

MessagePack map with exactly one key — the element type name — whose value is the element object.

TEXT is the default in both directions.

TEXT Encoding

Client to Pipeline

The engine inspects the string and produces an element accordingly:

Message Resulting element

#commit

Commit — flush the accumulated input of the current turn.

#url:<url>

URL.

#binary:<base64> or <contentType>,<base64>

Binary. Without an explicit content type the payload is treated as raw audio.

#transcript:text=<text>&isInterim=<bool>

Transcript with provider set to Client. A sentence separator is appended to text when missing.

#element:<json>

Any element, given as the JSON encoding described below. Use this to send an element type that has no # shorthand over a TEXT connection.

anything else

Token carrying the string verbatim.

A token whose text starts with # is an action trigger: the pipeline resolves it against the agent’s and project’s action mappers, so #intro, #end, #ready, #action:rating and friends are all sent as ordinary tokens. Parameters use the #name:key=value&key=value form.

Raw audio does not need the #binary: wrapper on the plain WebSocket transport — send a binary frame instead, which avoids base64 overhead.

Pipeline to Client

Element Rendering

Any textual element (command, token, speech text, error)

Its text form. Commands render as #name or #name:key=value&key=value.

Speech Item

#speech-item:<json of item>

Visual State

#visual-state:<json of state>

Navigation Specification

#navigation-specification:<json of specification>

Evaluation Status

#evaluation-status:<json of status>

Multimodal Interaction

#multimodal-interaction:<json of interaction>

An element with no TEXT representation is replaced by an #error:message=… line, so a client that expects a type the engine cannot render in TEXT should switch to JSON or MSGPACK.

JSON Encoding

The element is serialized as a JSON object whose type property names the type:

{"type": "token", "text": "What time is it?"}

MSGPACK Encoding

The same objects are encoded with MessagePack, but the type name becomes the single outer key instead of a type property:

{ "token": { "text": "What time is it?" } }
{ "error": { "sessionId": "…", "source": "Socket", "message": "…", "isRecoverable": false } }

Binary payloads (audio, images) are carried natively, which is why this format is the most compact of the three and is used by the mobile clients.

Input Element Reference

Type names are the type values in JSON and the wrapper keys in MSGPACK.

Type Fields Description

token

text

A text token. Either user text, or an action trigger when it starts with #.

transcript

text, isInterim, isSynthetic, isAsynchronous, provider

A speech recognition result produced by the client, when the client runs its own recognition.

binary

bytes (base64), contentType

Raw audio (audio/raw, audio/basic) or an image (image/*).

url

url

A resource URL for the agent to consider. Only images are supported, and only on realtime models that accept image input.

commit

—

Flush the input accumulated so far, ending the input phase of the turn (push-to-talk mode).

sensory

voiceDirection, temperature, illuminance, ambientNoise, isWhiteCaneDetected, isWheelChairDetected, persons

Environment and presence data from a physical device, made available to the agent as context.

action

name, text, parameters

A resolved action. Normally produced by the engine from a #name token rather than sent by a client; text is the wording handed to the language model.

Output Element Reference

Every output element carries sessionId, turnId (may be null for session-scoped elements) and time (server timestamp in milliseconds) in addition to its own fields.

Type Fields Description

command

name, parameters

Pipeline control command — ready, transcript, sessionShouldEnd, exit, error and others. See Command.

speech-item

item

One audio chunk of the agent’s synthesized speech. item carries provider, sampleRate, frameRate, isLast, bytes, isRawAudio, created and the animation data (visemes, blendShapeWeights).

speech

hash, text

The textual form of an utterance the agent is about to speak, with the hash identifying its cached audio.

token

text

A streamed language-model output token, when the pipeline streams text as it is generated.

transcript

text, isInterim, isSynthetic, isAsynchronous, provider

A recognition result produced by the engine. Interim results also arrive as a transcript command in protocol version 1 clients.

visual-state

state

Visual state for the client to render: emotions, reflectiveStates and signals.

multimodal-interaction

interaction

A UI element the agent wants displayed — image, web page, input prompt, choices or rating.

navigation-specification

specification

A request to navigate the user to a specific place in the host application.

evaluation-status

status

Evaluation results delivered over the open connection, in the same shape as /api/v2/evaluation/status.

error

source, type, message, description, isRecoverable

An error. source names the component that failed and isRecoverable says whether the session could continue. Note that the element’s own type property shares its name with the JSON type discriminator, so treat source and message as the primary fields when parsing JSON.

Protocol Versions

The protocolVersion parameter (default 1) affects output type naming: with version 1 the multimodal interaction element is emitted under the legacy type name MultimodalInteractionElement instead of multimodal-interaction. Both names are accepted when parsing, so a client can be written against either.