Twilio Telephony Integration

The engine can conduct a conversation over a phone call. Audio is exchanged with Twilio using the standard Media Streams protocol over a WebSocket; the engine translates it into the same pipeline session that a web or mobile client would run.

Connection

wss://{host}/socket/twilio/{key}

{key} is the pipeline key of the agent that should answer. Because a colon is awkward in a stream URL, a hyphen in the last path segment is translated back to a colon — agent-acme-support addresses the key agent:acme-support.

Unlike the other transports, no pipeline configuration parameters are read from the URL. The session is configured from the call itself: μ-law audio at the sample rate Twilio announces, continuous input mode, server-side speech processing and TTS caching disabled.

Inbound Calls

Point the phone number’s voice webhook at a TwiML document that connects the call to the engine:

<Response>
  <Connect>
    <Stream url="wss://{host}/socket/twilio/agent-acme-support">
      <Parameter name="from" value="+420111222333"/>
      <Parameter name="input" value="#intro"/>
      <Parameter name="acceptedLanguages" value="en-US"/>
      <Parameter name="zoneId" value="Europe/Prague"/>
      <Parameter name="trackingRef" value="…"/>
    </Stream>
  </Connect>
</Response>

The custom parameters arrive on the start event and configure the session:

Parameter Required Description

from

Yes

Caller’s number. It identifies the user — the engine derives the session’s device ref and username from it — so a returning caller is recognized across calls.

input

No

First input handed to the pipeline once the stream starts. Defaults to #intro, which makes the agent greet the caller.

acceptedLanguages

No

Locale of the conversation. Defaults to the deployment default.

zoneId

No

IANA time zone used for time-aware responses.

trackingRef

No

Reference stored on the session for correlation with your own records.

Outbound Calls

To have the platform place the call, use POST /api/call/initiate. The engine builds the TwiML shown above from the request and dials out; the resulting stream connects to this endpoint.

Protocol

The implementation follows Twilio’s Media Streams message format. Messages are JSON text frames discriminated by their event property.

Messages from Twilio

event Handling

connected

Acknowledged; carries protocol and version.

start

Starts the session. Carries streamSid and a start object with accountSid, callSid, tracks, customParameters and mediaFormat (encoding, sampleRate, channels).

media

Inbound audio. media.payload is base64 μ-law; it is decoded and fed to the pipeline as audio input. Carries track, chunk and timestamp.

dtmf

A key press. dtmf.digit is passed to the agent as a text token, so an agent can ask the caller to enter a number. The # key is passed as . to avoid being read as an action trigger.

stop

Ends the input stream and thus the conversation.

mark

Playback acknowledgement. When the engine’s final SessionEnd mark is echoed back, the socket is closed — this guarantees the caller hears the agent’s closing words before the call ends.

Messages to Twilio

event Meaning

media

A chunk of the agent’s synthesized speech. The engine converts 16-bit PCM to μ-law and base64-encodes it into media.payload.

clear

Discard audio Twilio has buffered but not yet played. Sent when the engine emits a transcript, i.e. when the caller starts speaking — this is how barge-in is implemented on the phone.

mark

Turn boundary. Listening is sent after each completed turn; SessionEnd is sent as the last message of the conversation.

Limitations

  • Telephony audio is μ-law 8 kHz mono, so speech quality is lower than on the other transports.

  • Server-side speech processing (facial animation data) is disabled — there is no visual channel.

  • The wire formats described in Wire Formats do not apply here; this transport always speaks Twilio’s own protocol.