The wire protocol, v2

A WebSocket carrying JSON, one message per frame. The reference host is Unreal’s PromethistLinkPixelStreaming plugin, but nothing here is Unreal-specific. Implement this, and any tablet built on @promethist/kiosk-tablet will drive your system. This document is the contract: a host on another engine needs nothing from the reference implementation beyond what is written here.

The whole protocol is two ideas: the host publishes state, the tablet sends commands. Everything below details those two.

"Tablet" here means any client on the control channel, not specifically a touchscreen. We use the term because touchscreens are the overwhelmingly common case. Physical buttons on a microcontroller, a phone, or a building-management system speak exactly the same protocol. Where this document says a tablet renders something, substitute a lamp or a relay and nothing else changes.

Three kinds of button

Every control on a tablet is one of three things. The difference is entirely about what the host publishes for it:

Sends Host publishes Renders

Fire-and-forget

the command, nothing back

nothing

always the same; a tap animates and that is all

Flip

the desired value (!current)

a bool key

pressed or idle, from that key

Fire-and-forget with progress

the command, once

a bool key while it works

busy — a spinner — until the key clears

A flip button never toggles itself; it draws what the host says. That is why there is no acknowledgement anywhere in this protocol. The state echo is the receipt. A tablet that flipped optimistically would lie whenever a command was dropped and disagree with the second tablet in the room.

The third kind applies the same idea to time rather than value. It is how a host reports progress with no ack to report it on. The command goes out once. The host publishes <something>Pending while the work is outstanding and clears it when done. The tablet shows a spinner and refuses a second press meanwhile. Two things follow from this being state rather than a per-client reply: every tablet in the room shows the same wait, and a tablet reloading mid-wait comes back showing it, because hello carries the key like any other.

A host owning one of these owes it a clear on start-up, not only on completion. If the work was interrupted — a level torn down, a process restarted — nothing is left to clear the key. A tablet is then holding a spinner over an action nobody is doing.

The same obligation applies to a flip button, for a different reason. A tablet cannot know which way an unpublished toggle is about to move, so a flip button whose key has never been published stays disabled. This is deliberate — !undefined would make it claim "on" is next — but a host forgetting to publish the initial value ships a permanently greyed control, not a control that guesses.

Adding a button is therefore a tablet-side change plus, for the last two kinds, one published key. There is nothing to register.

Framing

  • Client → host: text frames. ws.send(JSON.stringify(…​)).

  • Host → client: may be binary. The Unreal host can only emit binary frames (a limitation of the engine’s WebSocket plugin), so a client must set binaryType = 'arraybuffer' and decode UTF-8 itself. A host that can send text should; clients must accept either.

  • Every message carries v (protocol major) and t (type). A client announcing an unsupported major is answered with versionMismatch and disconnected.

Connecting

  1. Client opens the socket.

  2. If the host has no auth token configured, it sends hello immediately.

  3. The client sends its own hello (with token when one is configured). The host replies with hello.

-> { "v":2, "t":"hello", "token":"...", "client":"tablet-1" }

<- { "v":2, "t":"hello",
    "server": { "app":"StreamingDemo", "boot":"5f3c...", "rev":42, "proto":[2] },
    "state":  { "paused":false, "volume":100, "sessionActive":true } }

hello always carries a complete state snapshot, never a patch, so a freshly connected or reconnected tablet never renders from stale assumptions.

A client may receive hello more than once on the same connection, and must treat every one as a wholesale replacement of its state mirror. Three things cause this: a trusted local client is authorised the moment the socket opens and gets one unprompted, so its own hello draws a second; sys.resync produces one; and the reference host re-sends hello to every authorised client when its command vocabulary changes. server.rev is the same counter the state patches carry, so seed your gap detection from it rather than from the first patch. server.boot is a per-process id for diagnostics. A client does not have to compare it, because replacing state wholesale on every hello already handles a restarted host. The reference client stores it nowhere and compares nothing.

The client’s client field is a free-form name for logs, and optional in both directions. The reference client always sends one and the reference host never reads it, preferring the peer address because that identifies a tablet in a room with three of them. Send it or don’t. A host that wants to name connections has it, and a host that doesn’t misses nothing.

Until authorised, any message other than hello is answered with unauthorized. The host closes the connection after a short grace period or a few bad frames.

Start listening as early as you can. A client has no way to learn that a host exists other than a successful connect attempt. The socket opening is the entire discovery mechanism, and every second before it is a second of spinner. The reference host binds its port from the game instance subsystem’s Initialize, before any level has loaded. The tablet is then live, if briefly stateless, while the world is still coming up. Clients should expect this: a hello may arrive carrying almost no state, and keys appear as the host learns them.

State

The host holds an authoritative map of flat, dotted keyspaused, volume, custom.blackout. Keys are never nested, because a patch like {"custom": {"blackout": true}} is ambiguous about whether it replaces the whole custom object or merges into it. Flat keys have exactly one meaning and cannot lose a sibling.

<- { "v":2, "t":"state", "rev":43, "patch":{ "paused":true, "muted":true } }

rev increments by one per patch. On a gap, the client sends sys.resync and takes the resulting full snapshot. A null value means the key was removed.

This is the only channel a control renders from. A tablet that flipped a button optimistically would lie every time a command was dropped, disagreeing with the second tablet in the room. State is also how a host locks a control while it is busy — see below.

The reference host publishes:

Key

connection

Disconnected, Connecting, Ready, Listening, Speaking, … or Unavailable when no avatar is behind the socket

sessionActive

a conversation exists

sessionBusy

a session change is in flight — a start, a restart, a language switch

agentUnready

there is no live conversation to act on (the inverse of "ready")

paused, muted

operator toggles the host owns. A pause outranks the mute: the host mutes on pause and refuses a micMuted:false until the pause lifts, so a tablet should treat paused as a lock on that button rather than expecting its press to land

blackout

the installation is out of service: the host has ended any conversation and is refusing to connect until this is lifted

blackoutManual

the same, but only the half a person latched — see below

health.failed

the names of the host’s failing self-diagnostic checks; empty means it believes it works

volume

0–100

locale, locales

the current language, and the list on offer

notice is read by the reference tablet — it renders any string published under that key as a banner — but nothing publishes one today. It is the cheap way for a host to say something without the protocol growing a message type for it.

One thing to know before you publish a notice: that banner has three possible sources. The reference tablet ranks them err > health.failed > notice. A project’s own message is replaced while a self-diagnostic check is failing — deliberately, since a queue length or a "closing in ten minutes" line is less urgent than the kiosk not working, and the operator needs to know which. That ordering is a tablet decision, not a protocol rule. A tablet is free to choose differently, or to draw all three. This is noted only so a host author is not surprised by a message silently vanishing.

subtitles and assistancePending are published by the project, not by the reference host. They are two worked examples of a button getting its state without the host learning what the button means: subtitles a flip for a setting in the project’s own UI, and assistancePending a progress key for an action ending in a human being fetched — the one place where how long it takes is a deployment’s business and nobody else’s.

Note what blackout implies for a tablet: a control taking the host out of service must not be gated on the host being healthy. Ours deliberately carries no busy key, because the moment you reach for it is the moment everything else is already broken.

Two latches behind one blackout

blackout is published as the effective state. The reference host reaches it two ways that must not overrule each other: a person latched it, or the host’s own self-diagnostics decided this box cannot work. The installation is out of service while either holds it.

That is why blackoutManual exists beside it. The split is a rule about honesty rather than a convenience:

A tablet’s out-of-service toggle renders blackoutManual it owns exactly one of the two latches, and must not light up for a fault nobody touched and pressing it cannot clear

A tablet’s fog, or anything saying "unavailable"

renders blackout

the visitor-facing truth is the OR

A toggle bound to the effective key gets both halves wrong. It lights up on a fault the operator did not cause. When pressed to "turn it off", the key does not move — the control reads as broken exactly when somebody is trying to work out what is broken. Bound to blackoutManual, it follows every tap, and the fog keeps standing over it for as long as the host has a reason of its own.

health.failed turns that fog into information. It carries names, not sentences, because a sentence has a language and the host does not know which one the tablet is showing. The reference tablet keeps the wording beside its other strings, falling back to the raw name for an unknown check. A host with no self-diagnostics publishes neither key, and nothing here applies.

Asking the visitor something

A host with a question does not need a message type for it either. It publishes a name — prompt is the key the reference tablet offers for this — and the tablet decides what that name looks like, how long it stays and what an answer is. The reply comes back as an ordinary command named after the prompt. A five-star rating is {"t":"cmd","cmd":"feedback","args":{"value":"4"}} and nothing else is involved.

This is the same choice as <x>Pending above, and for the same reasons: a tablet reloading mid-question comes back showing it because hello carries the key, and every tablet in the room agrees. It is also why v2 has no event frame — a one-shot notification cannot survive either.

Two things a host owes such a key, and the second is easy to miss:

  • Clear it when the question is over, as with any published key. One left set is a question a tablet will find on its next connect and ask about again, hours later.

  • Clear it before asking twice. A store that compares before it stores — the reference host’s does, so a host may mirror its whole state every tick — emits no patch for a value that has not changed. Re-publishing "feedback" after the first visitor produces nothing at all.

Both obligations come from treating the key as an event, and a tablet can decline to. Everything a host publishes is visible to it. A question whose moment the tablet can see for itself only needs the key to say whether to ask — making it a setting, and a setting needs no clearing. The reference tablet’s rating is this: it opens when sessionActive goes false and prompt reads feedback. The host publishes that once and never undoes it. Every visitor is asked, and a freshly connected tablet cannot be fooled by the key it finds — because a snapshot contains no edge to trigger on.

Worth stating as the general rule, since it applies to more than prompts: a key that gates something the client can already detect is safer than a key that commands it. The first cannot go stale.

Commands

-> { "v":2, "t":"cmd", "cmd":"pause", "args":{ "value":true } }

That is the entire client-to-host vocabulary. cmd is a string the host and the tablet have agreed on; args.value may be a string, a number or a bool.

args.value is optional in the envelope — a command like restart that names its whole action carries none. But a host can and should require one per command. Once coerced, an absent value is indistinguishable from a false one. A host treating "missing" as "false" lets a single malformed frame lift a lockout or mute a kiosk. The reference host requires a value for pause, volume, micMuted, blackout and locale, and a numeric one for volume. Commands arriving without required values are logged and ignored, never guessed at.

Nothing is acknowledged. There is no correlation id and no ack, because the confirmation a control actually needs is the returned state: press Pause, and the button changes when paused does. A host reports progress or refusal by publishing state, which every connected tablet sees.

sys.resync needs no registration and makes the host re-send hello. Like any message, it is subject to authorisation and the rate budget.

Finding out what a host accepts

State is self-describing: hello carries every key a host publishes, letting a client learn them from the wire. Commands are not, but a host may close that gap by declaring its vocabulary:

<- { "v":2, "t":"hello",
    "server": { "app":"StreamingDemo", "boot":"5f3c...", "rev":42, "proto":[2],
                "commands":[
                  {"cmd":"pause",    "value":"bool",   "reflects":"paused"},
                  {"cmd":"blackout", "value":"bool",   "reflects":"blackoutManual"},
                  {"cmd":"volume",   "value":"number", "reflects":"volume"},
                  {"cmd":"restart"},
                  {"cmd":"say",      "value":"string"}
                ] },
    "state": { ... } }
  • valuebool, number or string. Absent means the command takes no value at all, which is different from "any value will do".

  • reflects — the state key confirming the command landed. Absent when nothing does.

Three rules exist because the obvious assumptions are wrong:

Absent commands is not an empty one. It means the host does not say. A client reading it as "accepts nothing" would refuse to send anything.

A listed vocabulary is not exhaustive. A host may accept commands it never declared — declaring buys discoverability, never permission. An unlisted command merits a warning during development, but nothing at runtime.

Never assume the command is named after the key. Read reflects. On the reference host, five of ten commands do not share a name with the key they affect, and one actively misleads: blackout latches only the operator’s half of the lockout, while the blackout key represents that half or the host’s own diagnostics. A client assuming matching names would send blackout:false, watch the key stay true, and conclude its command failed.

This is not a manifest to render from. A tablet declares its own buttons. That is the whole difference between v2 and the v1 this replaced, letting a project add a button without the host learning anything. The list exists for the person writing a client, and for development-time checks ensuring every drawn button sends something the host lists.

How a control locks

A host wanting a button unavailable publishes a bool state key, and the tablet names it as that button’s busy key. Two consequences are worth stating:

  • It is positive-polarity: there is no way to spell "locked unless X". A host needing that publishes the inverse as its own key — the reference host publishes agentUnready alongside sessionActive for exactly this reason. A negation micro-syntax was considered and rejected: every host and client would have to implement it, and !missingKey has no good answer.

  • It is shared. v1 locked a control by remembering which command that one client had outstanding, so tablet A pressing Stop left tablet B’s button live. A second press then raised an error for an already satisfied intent. A state key locks every tablet at once.

A toggle needs no lock against double-tapping. If re-tapped before its state echo arrives, it reads the old state, sends the same value again, and the host no-ops. Only a stateless action can genuinely fire twice, and handling that is the client’s business — the reference tablet debounces those for 500 ms.

Errors

err is connection-level only — it never refers to a command:

<- { "v":2, "t":"err", "code":"versionMismatch", "message":"This host speaks protocol v2; reload the page." }

versionMismatch, unauthorized, badRequest, rateLimited, tooManyClients.

It exists because the alternative for these five is closing the socket in silence, causing an endless reconnect on the tablet with nothing to diagnose. A page cached before a protocol change is the most common cause and remains invisible without this frame.

Four of the five need a person and should reach the screen. rateLimited is the exception in both directions, worth detailing for either side of the wire:

  • A client should treat it as transient — log it, but do not render it as a condition. It disappears within the second, and nothing an operator can see or do applies to it.

  • A host should size its per-client budget above the rate its own tablet’s continuous controls generate, not how fast a finger can tap. A throttled slider sends steadily for as long as a drag lasts. The reference client’s slider sends at 8.3/s, and the reference host’s original budget was 4/s, meaning a five-second drag spent its last two seconds being refused. A budget refusing a slider used as intended measures the wrong thing. The reference host now allows 20/s sustained with a burst of 40 — comfortably above the slider, yet low enough to catch a control bound straight to pointermove (which emits 60–120/s).

  • A host dropping a command must log the drop. This is the one refusal a client cannot reasonably show, so the host’s log is the only place the press exists.

Liveness

-> { "v":2, "t":"ping" }
<- { "v":2, "t":"pong" }

Worth implementing on both sides: a host with a wedged main thread leaves the socket open while serving nothing. Silence — not disconnection — becomes the only symptom. The reference client marks the host stale after 8 s without a frame.

Batching

A host whose transport can only deliver one frame per tick may coalesce a frame’s worth of messages. Clients must unwrap this and process the entries in order.

<- { "v":2, "t":"batch", "msgs":[ {"v":2,"t":"pong"}, {"v":2,"t":"state","rev":44,"patch":{"paused":false}} ] }

Changes from v1

v1 carried a capability manifest in hello: the host declared every control with a 15-field descriptor (kind, group, order, localised labels, icons, state key, busy key, range bounds, enum options). The tablet rendered whatever it was told, and each command was correlated and acknowledged with ack / err and a client-side timeout.

It was an RPC framework serving eight buttons that all completed synchronously. v2 drops the manifest, the registration API behind it, correlation ids, and ack. A tablet declares its own buttons; the host publishes state and answers to strings.

Removed: capabilities in hello, ack, the id correlation field, and the command-level error codes (unknownCommand, unsupported, disabled, badArgs, busy, internal).

Added, later in v2: server.commands in hello. This is not the manifest returning — the difference is what the declaration is for. v1’s manifest drove rendering (fifteen fields per control), and registration was mandatory, meaning an unregistered command could not work. The v2 list simply names commands, value types, and the key each command reflects. It drives nothing, and an unregistered command behaves exactly as before. It exists because a README stops being an adequate answer the moment a second host implementation exists. It is optional and additive, so a client written before it still works.

Also removed, later in v2: the event frame — a one-shot notification channel ({"t":"event","name":…,"data":…}) that no host ever sent and no tablet ever read. Everything a tablet needs to render is persistent, making it state. A frame type existing only in this document was just one more thing for a host author to implement for nothing. Nothing else in the wire format has changed, so a client written against earlier v2 still works.