Pipeline Socket

The engine exposes real-time pipeline communication over three socket transports:

Transport Engine Port Endpoint Notes

Socket.IO

9011

/socket/io/pipeline?key={key} namespace

Two events carry all traffic: text (a string payload, interpreted as TEXT or JSON) and msgpack (a binary payload). Default format TEXT.

Plain WebSocket

9010

/socket/pipeline/{key}

Text frames carry TEXT/JSON, binary frames carry raw audio or MSGPACK. Default format TEXT.

Twilio WebSocket

9010

/socket/twilio/{key}

Twilio Media Streams JSON protocol (events: connected, start, media, dtmf, mark, stop). See Twilio Telephony Integration.

The {key} path segment identifies the agent/pipeline configuration — see The Pipeline Key. On the Socket.IO transport it is passed as the key handshake query parameter instead.

In a deployed environment both listeners are reachable on the deployment host over TLS: the reverse proxy routes /socket/io/ and /socket.io/ to the Socket.IO listener and everything else, including /socket/pipeline/ and /socket/twilio/, to the HTTP listener. The port numbers above therefore only apply to a locally running engine.

A fourth, non-socket transport runs a conversation over a plain HTTP request — see HTTP Pipeline Channel.

Pipeline Configuration

All parameters can be supplied either as socket URL query parameters or as HTTP request headers. When both are present the query parameter takes precedence.

Query Parameter HTTP Header Default Description

sessionId

—

(auto-generated UUID)

Reuse an existing session by its UUID. If omitted a new session ID is generated.

locale

Accept-Language

en

BCP 47 language tag(s) for the session (e.g. en-US, cs-CZ). Multiple values are accepted in standard Accept-Language priority order.

timeZone

X-Time-Zone

Europe/Prague

IANA time-zone identifier used for time-aware responses (e.g. America/New_York).

deviceRef / deviceId

X-Device-Ref / X-Device-Id

(session ID)

Stable device identifier. Allows the pipeline to associate multiple sessions with the same physical device.

deviceType

X-Device-Type

—

Client device category. One of: iOS, Android, Desktop, Virtual, External, Link. Used by test agents only.

inputFormat

X-Input-Format

TEXT

Wire format for messages sent by the client. TEXT, JSON or MSGPACK — see Wire Formats.

outputFormat

X-Output-Format

TEXT

Wire format for messages sent by the pipeline. TEXT, JSON or MSGPACK.

protocolVersion

X-Protocol-Version

1

Output protocol version. Affects the JSON type name of some output elements; keep 1 unless instructed otherwise.

inputMode

X-Input-Mode

Continuous

Audio input mode. Continuous — microphone is always open; PTT — push-to-talk.

inputSampleRate

X-Input-Sample-Rate

24000

Sample rate (Hz) of the raw audio sent by the client.

outputSampleRate

X-Output-Sample-Rate

24000

Sample rate (Hz) requested for synthesized speech chunks.

outputChunkSize

X-Output-Chunk-Size

—

Maximum size in bytes of each #speech-item audio chunk. If omitted the synthesizer’s natural chunk size is used.

outputMedia

X-Output-Media

—

Output audio container/codec type (e.g. mp3, ogg). If omitted raw PCM is delivered. Only taken into account on InternalRealtimeModel

inputEncoding

X-Input-Encoding

LINEAR16

Encoding of the raw audio sent by the client, also used when handing the audio to speech recognition. One of: LINEAR16, ALAW, MULAW.

speechProcessing

X-Speech-Processing

true

Enable server-side output speech processing using Audio2Face. Set to false to disabled it (e.g. when no humanoid avatar is rendered on the client side).

textOnly

X-Text-Only

false

When true the pipeline skips TTS and returns only text tokens.

noSystemPrompt

X-No-System-Prompt

false

When true the pipeline omits the built-in system prompt, relying solely on agent-defined instructions. Used primarily by test/benchmarking agents.

anonymousMode

X-Anonymous-Mode

false

When true the session is associated with an anonymous user derived from the session ID rather than the authenticated identity.

testMode

X-Test-Mode

false

Enables test mode marking all data like turns and LLM logs as testing (should be set to true by both manual and automated testers).

useAudioFromStart

X-Use-Audio-From-Start

false

When true the pipeline begins input audio processing from the moment the connection opens, otherwise ignores all audio until the first turn finishes.

useCaching

X-Use-Caching

false

Enable TTS audio caching, so repeated utterances are not synthesized again. Set to true to opt in.

ignoreAfterTurnStartAudio

X-Ignore-After-Turn-Start-Audio

—

Milliseconds of client audio to discard right after a turn ends. Prevents the agent from hearing the tail of its own speech through an open microphone.

ignoreShortAudioBursts

X-Ignore-Short-Audio-Bursts

—

Discard audio bursts shorter than this many milliseconds, suppressing clicks and other noise.

realtimeConfigurationId

X-Realtime-Configuration-ID

—

UUID of a stored realtime (speech-to-speech) configuration to run the session with, overriding the agent’s own setting.

trackingRef

X-Tracking-Ref

—

Caller-defined reference stored on the session, for correlating it with a record in your system.

testedAgentRef

X-Tested-Agent-Ref

—

Used by automated testing only: ref of the agent under test when this session drives another agent.

avatarRef

—

—

Reference identifier of the visual avatar to use for this session (can be used to enforce specific server-side avatar rendering - e.g. using LiveAvatar.com).

authorization

Authorization

—

JWT bearer token. Alternatively passed as a standard Authorization: Bearer <token> header.

Message Formats

All socket transports support three wire formats, selected per connection and independently for each direction via inputFormat and outputFormat.

Format Description

TEXT (default)

Human-readable command strings. Commands use the #name:params scheme; audio is base64-encoded with a #binary: prefix.

JSON

Full JSON serialization of the Element (input) or OutputElement (output) type hierarchy, discriminated by a "type" field.

MSGPACK

The same object model encoded with MessagePack and exchanged as binary payloads. The most compact option, used by the mobile clients.

The examples throughout this page show the TEXT and JSON forms. The complete encoding rules of all three formats, together with the element type reference, are in Wire Formats.

Typical Communication Flow

The table below shows the complete lifecycle of a session, from connection open through session end.

Phase Client Engine / Interaction Pipeline

Session Start (Preconnect)

Client emits #ready once the socket is open to indicate that it is ready to start interaction.

TEXT

#ready

JSON

{
  "type": "token",
  "text": "#ready"
}

Pipeline emits #ready once the session is initialized in pipeline (pre-session operations have been performed). Client should wait for it by display loading screen before interaction can start. Client must not emit any inputElements (including #intro) until #ready is received.

TEXT

#ready

JSON

{
  "type": "command",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": null,
  "name": "ready",
  "parameters": {}
}

Agent Introduction (optional)

Client triggers the intro greeting by sending action #intro.

TEXT

#intro

JSON

{
  "type": "token",
  "text": "#intro"
}

Note: client can directly proceed with audio or text input if user is starting interaction (e.g. by text input What’s the weather?)

Pipeline streams the agent’s spoken introduction as #speech-item chunks (see Speech Item). isLast: true marks the final audio chunk.

Audio Input

Client sends raw audio as binary WebSocket frames (or #binary:<base64> over Socket.IO).

TEXT (Socket.IO / text-only transport)

#binary:UklGRiQAAABXQVZFZm10IBAAAA...

JSON

{"type": "binary", "bytes": "UklGRiQAAABXQVZFZm10IBAAAA...", "contentType": "audio/raw"}

Pipeline sends interim #transcript commands (with isInterrim=true) while speech recognition is running, then emits the final (isInterrim=false) #transcript.

Depending on the upstream model, #speech-item’s containing the agent’s response and #transcript s of the user’s speech may be interleaved. The final #transcript may be emitted even after the final (isFinal=true) #speech-item

Interim #trancripts indicate barge-in Client should stop speech playback and flush its SpeechItem buffer whenever it receiver an interim #transcript.

The pipeline may also indicate "barge-in", the detection of user starting to speak, with an empty transcript containing only a dot. #transcript:text=.&isInterim=true

TEXT — interim transcript

#transcript:text=hel&isInterim=true

JSON — interim transcript

{
  "type": "command",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "name": "transcript",
  "parameters": {
    "text": "hel",
    "isInterim": "true"
  }
}

Pipeline emits the final #transcript followed by the agent’s spoken reply as #speech-item chunks.

TEXT — final transcript

#transcript:text=Hello, how can I help you today?&isInterim=false

JSON

{
  "type": "command",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "name": "transcript",
  "parameters": {
    "text": "Hello, how can I help you today?",
    "isInterim": "false"
  }
}

TEXT — speech reply chunk

#speech-item:{"provider":"google","sampleRate":24000,"frameRate":0,\
"isLast":false,"blendShapeWeights":[],"visemes":[],\
"bytes":"<base64-encoded-PCM>","isRawAudio":true,\
"created":"2024-03-25T12:00:01Z"}

JSON

{
  "type": "speech-item",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "item": {
    "provider": "google",
    "sampleRate": 24000,
    "frameRate": 0,
    "isLast": false,
    "blendShapeWeights": [],
    "visemes": [],
    "bytes": "<base64-encoded-PCM>",
    "isRawAudio": true,
    "created": "2024-03-25T12:00:01Z"
  }
}

Text Input

Client sends one or more token messages. The final token must end with a sentence separator (., ?, or !). A sentence separator is appended automatically if missing. Because the text is already known, this always yields a final transcript directly — no interim phase.

TEXT — single token

What time is it?

TEXT — multi-token (last token closes the sentence)

What time
is it?

JSON

{"type": "token", "text": "What time is it?"}

Pipeline emits a final #transcript immediately (no interim events), then the agent’s spoken reply as #speech-item chunks.

TEXT

#transcript:text=What time is it?&isInterim=false

JSON

{
  "type": "command",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000002",
  "name": "transcript",
  "parameters": {
    "text": "What time is it?",
    "isInterim": "false"
  }
}

Turn End

After the last speech chunk the pipeline signals the end of the turn with #exit. Following elements will have a new turnId. TEXT

#exit

JSON

{
  "type": "command",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "name": "exit",
  "parameters": {}
}

Session End by Agent

Before #exit on the final turn, the pipeline emits #sessionShouldEnd to signal that the session should be closed by client. #exit follows as usual to complete the turn.

TEXT

#sessionShouldEnd
#exit

JSON — session-should-end

{
  "type": "command",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000003",
  "name": "sessionShouldEnd",
  "parameters": {}
}

Client acknowledges the session end by sending #end.

TEXT

#end

JSON

{
  "type": "token",
  "text": "#end"
}

The connection stays open while the pipeline runs post-session operations (evaluation and webhooks); it closes once they finish. Depending on what is configured for the project and agent, an evaluation-status or navigation-specification output element may still arrive on the connection during this window — see Output Element Reference. There is no fixed order between these two and the ordinary end-of-turn output; treat the connection as open until the server actually closes it, not merely after #exit/#end. After-session feedback is not delivered over this connection — fetch it separately with GET /api/feedback/specification.

Session End by Client

Client initiates session end at any time by sending #end.

IT IS STRONGLY RECOMMENDED TO USE SEND ACTION #end IN ORDER TO CLOSE SOCKET BY ENGINE RATHER THEN CLOSE IT DIRECTLY AS THIS WILL PUT SESSION TO DISCONNECTED STATE AND WILL TAKE ENGINE ADDITIONAL 30+ SECONDS TO ABANDON SESSION AND EXECUTE REQUIRED POST SESSION OPERATIONS.

TEXT

#end

JSON

{
  "type": "token",
  "text": "#end"
}

Input Elements

Input elements are messages sent from the client to the pipeline.

Text Token (plain text or command trigger)

TEXT

Hello, what's the weather today?

TEXT (pipeline-defined action such as #intro)

#intro

JSON

{"type": "token", "text": "Hello, what's the weather today?"}

JSON

{"type": "action", "name": "intro"}

Transcript (client-side STT result)

Used when the client performs its own speech recognition and sends the result to the pipeline.

TEXT — interim

#transcript:text=what's the weath&isInterim=true

TEXT — final (a sentence separator is appended automatically if missing)

#transcript:text=What's the weather today?&isInterim=false

JSON — final

{
  "type": "transcript",
  "text": "What's the weather today?",
  "isInterim": false,
  "isSynthetic": false,
  "isAsynchronous": false,
  "provider": "Client"
}

Binary (raw audio)

Send audio to let the pipeline perform server-side speech recognition.

WebSocket — send a binary frame containing raw audio bytes (no text prefix needed).

TEXT (Socket.IO or text-only transports) — base64-encoded audio:

#binary:UklGRiQAAABXQVZFZm10IBAAAA...

JSON

{"type": "binary", "bytes": "UklGRiQAAABXQVZFZm10IBAAAA...", "contentType": "audio/raw"}

URL

Pass a URL resource into the pipeline (e.g. for multimodal processing). The only supported content type is an image and is only supported on OpenAiRealtime.

TEXT

#url:https://example.com/image.png

JSON

{"type": "url", "url": "https://example.com/image.png"}

Commit

Explicitly commit the current turn input (flush accumulated input in pipeline PTT mode).

TEXT

#commit

JSON

{"type": "commit"}

Arbitrary Element (JSON-in-TEXT wrapper)

Send any JSON-serialized Element over a TEXT connection:

TEXT

#element:{"type":"transcript","text":"Hello.","isInterim":false,...}

End Session

Signal that the client is closing the session (Socket.IO only; WebSocket clients simply close the connection).

TEXT

#end

Output Elements

Output elements are messages sent from the pipeline to the client.

Command

Generic pipeline command carrying a name and optional key=value parameters.

TEXT — no parameters

#ready

TEXT — with parameters

#transcript:text=Hello there.&isInterim=false

JSON

{
  "type": "command",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "name": "transcript",
  "parameters": {
    "text": "Hello there.",
    "isInterim": "false"
  }
}

Common command names:

Name Description

ready

Session initialized; client may now send input.

transcript

STT result (interim or final). Parameters: text, isInterim.

sessionShouldEnd

The session is ending after this turn. The client should send #end once ready to close the connection.

exit

Current turn completed; pipeline is ready for the next user input.

error

An error occurred. See Wire Formats for the error fields. The connection is closed immediately after.

Speech Item

An audio chunk of the agent’s synthesized reply. Multiple chunks are emitted per turn; the last one has isLast: true.

TEXT

#speech-item:{"provider":"elevenlabs","sampleRate":44100,"frameRate":0,\
"isLast":true,"blendShapeWeights":[],"visemes":[],\
"bytes":"<base64-encoded-PCM>","isRawAudio":true,\
"created":"2024-03-25T12:00:02Z"}

JSON

{
  "type": "speech-item",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "item": {
    "provider": "elevenlabs",
    "sampleRate": 44100,
    "frameRate": 0,
    "isLast": true,
    "blendShapeWeights": [],
    "visemes": [],
    "bytes": "<base64-encoded-PCM>",
    "isRawAudio": true,
    "created": "2024-03-25T12:00:02Z"
  }
}

visemes contains lip-sync timing data when produced by a supporting TTS provider (e.g. Azure). blendShapeWeights carries per-frame facial blend shape data for 3D avatar animation.

Visual State

Updates the client’s UI state (e.g. background, overlay, avatar pose).

TEXT

#visual-state:{emotions={boredom=0.357666015625, contemplation=0.2357177734375, calmness=0.2235107421875}, reflectiveStates={}, signals=[userNotSpeaking])}

JSON

{
  "type": "visual-state",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "state": {
    "emotions": {
        "boredom": 0.357666015625,
        "contemplation": 0.2357177734375,
        "calmness": 0.2235107421875
    },
    "reflectiveStates": {},
    "signals": [ "userNotSpeaking" ]
  }
}

Multimodal Interaction

Triggers a multimodal display element (image, video, form, etc.) on the client.

TEXT

#multimodal-interaction:{"ref":"product-card","data":{"title":"Widget X","price":49.99}}

JSON

{
  "type": "MultimodalInteractionElement",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "interaction": {
    "ref": "product-card",
    "data": {"title": "Widget X", "price": 49.99}
  }
}

Navigation Specification

Announces that a navigation entry has just been unlocked for the user. Emitted when an evaluation satisfies the entry’s condition, and carries the same object the project payload uses — see Navigation API. Match it to the client’s stored navigation by ref: replace the entry when the ref is known, insert it when it is not.

turnId is always null, and the delivered entry always arrives unlocked — its disabledMessage is null.

TEXT

#navigation-specification:{"type":"route","ref":"onboardingAssistant",\
"parentRef":"assistants","image":null,"name":"Onboarding assistant",\
"description":"Starts a conversation with the onboarding assistant.","order":2,\
"disabledMessage":null,"route":"/a/onboardingAssistant"}

JSON

{
  "type": "navigation-specification",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": null,
  "time": 1774000000000,
  "specification": {
    "type": "route",
    "ref": "onboardingAssistant",
    "parentRef": "assistants",
    "image": null,
    "name": "Onboarding assistant",
    "description": "Starts a conversation with the onboarding assistant.",
    "order": 2,
    "disabledMessage": null,
    "route": "/a/onboardingAssistant"
  }
}

Output Token

A streamed text token from the language model (used when streaming LLM output directly to the client).

TEXT

Hello

JSON

{
  "type": "token",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": "00000000-0000-0000-0000-000000000001",
  "text": "Hello"
}

Error

TEXT

#error:session.id=00000000-0000-0000-0000-000000000001&source=Socket&type=CommunicationException$Socket&message=Emitting input to pipeline failed&description=null&isRecoverable=false

JSON

{
  "type": "error",
  "sessionId": "00000000-0000-0000-0000-000000000001",
  "turnId": null,
  "source": "Socket",
  "message": "Emitting input to pipeline failed",
  "description": null,
  "isRecoverable": false
}

The connection is closed by the server immediately after an error is emitted.

Authentication

Pass a JWT bearer token either as an HTTP header or as a query parameter:

Authorization: Bearer <token>

or

/socket/pipeline/agent:some-agent?authorization=<token>

Unauthenticated connections are permitted when the pipeline configuration allows anonymous access.