Audio & video calls

Multi-party calling through a Selective Forwarding Unit. Each participant uploads one stream no matter how many people are in the room, and no client ever learns another's IP address.

The model #

• Calls happen in rooms. There is no separate call object to create — you open the RTC socket for a room you are a member of, and you are in the call.
• Same token as chat. The bundle you already minted carries rtc_websocket_url and ice_servers. It must grant rtc:subscribe to receive and rtc:publish to send media.
• Signalling on the socket, media on UDP. Only SDP and ICE candidates ride the WebSocket; RTP goes directly between the client and the SFU.
• No client-side register frame. The server registers your connection from the verified token. A register frame from a client is rejected by design — identity is never taken from client input.

Connecting #

const pc = new RTCPeerConnection({ iceServers: bundle.ice_servers });
const ws = new WebSocket(bundle.rtc_websocket_url,
                         ["chatbox", "bearer." + bundle.access_token]);
// Optional third subprotocol, listed first: every forwarded track's msid then
// names its publisher's principal — see "Who published what" below.
//   ["chatbox.msid-principal", "chatbox", "bearer." + bundle.access_token]

Use the ice_servers array exactly as handed to you. The TURN credentials inside are per-session and time-limited — hard-coding them will work in the office and fail in the field.

Publishing #

const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true });
stream.getTracks().forEach((t) => pc.addTrack(t, stream));

const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
ws.send(JSON.stringify({ cmd: "offer", sdp: pc.localDescription }));

pc.onicecandidate = (e) => {
  if (e.candidate) ws.send(JSON.stringify({ cmd: "candidate", candidate: e.candidate }));
};

Receiving, and the re-offer loop #

This is the part integrations most often get wrong. When another participant publishes, the SFU sends you a fresh offer; you must answer it and echo its request_id, or the answer will not be matched to the negotiation.

ws.onmessage = async (ev) => {
  const f = JSON.parse(ev.data);

  if (f.event) return handleRoster(f);    // joined | error | peers | peer.joined | peer.left

  if (f.type === "answer") {              // answer to the offer we sent
    await pc.setRemoteDescription(f);
    return;
  }

  if (f.type === "offer") {               // SFU re-offer: someone else published
    await pc.setRemoteDescription(f);
    const answer = await pc.createAnswer();
    await pc.setLocalDescription(answer);
    ws.send(JSON.stringify({ cmd: "answer", sdp: answer, request_id: f.request_id }));
  }
};

pc.ontrack = (e) => attachToUi(e.streams[0]);   // remote media arrives here

Who published what #

A remote track arrives as a transport track, not a person. Two things connect them. First, the socket sends roster frames, discriminated by event:

{ "event": "peers",       "peers": [ { "client_id": "13622361611987684004", "principal_id": "u_42" } ] }
{ "event": "peer.joined", "client_id": "13622361611987684004", "principal_id": "u_42" }
{ "event": "peer.left",   "client_id": "13622361611987684004" }

• peers arrives once, before any SDP, and lists everyone already in the call except you ([] if you are first). peer.joined is an upsert — a client reconnecting on a fresh socket is announced again under the same client_id.
• client_id is a decimal string, not a number: it is a 64-bit id that routinely exceeds 2⁵³, where JSON.parse rounds. Compare it as a string. It identifies one connection — a second tab or device gets its own — so group tracks by client_id and attribute them to a person by principal_id, which is stable across all of that person's connections.
• peer.left is a roster hint, not a media event: the SFU stops forwarding that peer's tracks on its own, so do not tear anything down on it. Delivery is best-effort; the peers snapshot on your next connect is the resync. Ignore event values you do not recognise.

Second, every forwarded track's msid names its publisher. By default the SFU rewrites the stream id to peer-<client_id>-<original>, and you join it to the roster. Or list the subprotocol chatbox.msid-principal ahead of chatbox when you open the socket: when the server echoes it as the selected subprotocol, every forwarded track carries a=msid:<publisher principal_id> <publisher's own track id> instead, so event.streams[0].id is the verified principal in every browser — no regex, no roster lookup for attribution. Per socket and optional; a socket that offers only chatbox is unchanged.

// Default form: the msid is peer-<client_id>-<original>; the roster maps client_id → principal.
const names = new Map();
ws.addEventListener("message", (e) => {
  const msg = JSON.parse(e.data);
  if (msg.event === "peers")            for (const p of msg.peers) names.set(p.client_id, p.principal_id);
  else if (msg.event === "peer.joined") names.set(msg.client_id, msg.principal_id);
  else if (msg.event === "peer.left")   names.delete(msg.client_id);
});
pc.ontrack = (event) => {
  const publisher = /^peer-(\d+)-/.exec(event.streams[0]?.id ?? "")?.[1];
  render(event.track, names.get(publisher));      // undefined until the roster catches up
};

// With chatbox.msid-principal selected (check ws.protocol once open):
pc.ontrack = (event) => render(event.track, event.streams[0]?.id);   // the verified principal_id

Simulcast and layer selection #

A publisher may send up to three encodings (simulcast). The SFU forwards one layer per subscriber on a stable stream — you always see exactly one inbound-rtp per remote track, and there is no layer-request frame on the client. Which layer you get is decided in two ways:

• The adaptive policy, by default: each subscriber starts on the cheapest layer, climbs to the best one its link carries when the link has been clean, and backs off on loss. The measurement that drives it (bandwidth estimate, loss) is per subscriber, so one bad link never degrades anyone else's picture. “Clean” is measured, and it is measured from your client’s own reports: we fold each subscriber’s transport-cc feedback into one-second windows, call a window clean under 2 % loss, move up a layer after three clean windows, keep that move once it survives five windows, abandon it on two consecutive windows at 5 % or more, wait eight clean windows before trying again (doubling after a failure), and drop a layer after two windows at 10 % or more.
• Keep the transport-cc header extension enabled on every transceiver you receive on. A client that negotiates it away sends us no reports, so no window ever closes and that viewer stays on the cheapest layer for the whole call — no error, no warning, nothing in a log. If quality never improves on a good connection, check that extension first. A pinned layer (below) is unaffected either way.
• A preference your server pins, per member and per publisher track — the "show this speaker in high quality" control:

PUT /v1/server/rooms/{room_id}/members/{principal_id}/preferences     Authorization: Bearer ak_…  (members:write)
{ "tracks": [ { "publisher": "u_42", "track_id": "<u_42's track id>", "layer": "high" } ] }
# → { "principal_id": "…", "in_call": true, "tracks": [ …what is pinned, sorted… ] }

GET /v1/server/rooms/{room_id}/members/{principal_id}/preferences     (rooms:read)

• Keyed by what the member's client already sees: the publisher's principal_id and the publisher's own track id — exactly the stream id and track id a chatbox.msid-principal socket reads off the msid. No client change.
• low, medium and high name the publisher's layers by measured bitrate (with two layers, medium is the dearer). A pinned layer that stops is replaced by the dearest live layer below it and taken back when it returns. auto hands the track back to the policy.
• The whole list is replaced on each PUT; [] clears everything. Applied on the media loop's next layer report, about a second, to every connection the member has. Held for the member's session and across a PeerConnection rebuild (a 30 s grace after the last connection closes); a preference set before the member joins waits the same grace. in_call in the answer says which state they are in.
• Refusals: 403 without members:write, 404 for a non-member, 422 for a layer outside the four words or a stray field, 400 for more than 128 entries or a track listed twice.

Pausing and resuming a track

Muting a camera or stopping a screen share by setting the transceiver's direction to inactive and renegotiating, then resuming it later on the same m-section — direction sendonly again, the same track or a replaceTrack, same RIDs or not — is supported for simulcast and single-encoding tracks alike. The SFU forwards the track again within about a second of its first resumed packet; subscribers see the forwarding leave and return as a re-offer, with a new ontrack each time. A new m-section per restart works too; leave the old one inactive. Before 2026-10-10 a simulcast section resumed on the same mid stayed silent for every subscriber, and a second pause was answered with a direction the browser refused; single-encoding tracks and audio were never affected.

Codecs

Audio is Opus; video is negotiated from VP8, VP9, H.264 and AV1. H.265/HEVC is not registered: Chrome and Edge offer it only on machines with hardware HEVC support, which would give two browsers of the same version different negotiated codec sets. Offer it if you like; it is left out of the answer. A forwarded track is offered to you with the codecs the publisher and you both negotiated — a publisher codec you did not negotiate is left out rather than blocking the track, and only a track with nothing in common is withheld. If you author SDP by hand, do not reorder or renumber payload types in your answer.

Screen share #

A second video track, not a mode — so a participant can send camera and screen simultaneously and subscribers receive them as independent streams.

const display = await navigator.mediaDevices.getDisplayMedia({ video: true });
pc.addTrack(display.getVideoTracks()[0], display);
// renegotiate: createOffer → setLocalDescription → cmd:"offer"

Multi-device #

Pass a distinct device_id when minting each token. Call legs are keyed by (tenant, room, principal, device), so without it a second device replaces the first instead of joining alongside it.

Knowing a call started #

A room emits call.started when its first participant joins and call.ended when the last one leaves. Both carry the same call_id.

{ "v": 1, "event_id": 0, "type": "call.started", "room_id": "0f5c…",
  "payload": { "call_id": "5a3e…", "initiator": "alice" },
  "occurred_at": "2026-09-10T11:04:22Z" }

These reach you three ways, which together cover every device state:

• Chat socket — for a device that is online but not in the call.
• Webhook — for your app-server, e.g. to trigger your own call UI or logging.
• Push — for a backgrounded or killed device. Both push: call.started rings, and call.ended follows as a silent retraction to the members who were rung — to every device each has registered at that moment — so your app can cancel the notification it posted. A device registered after the ring can receive a cancel for a call it never rang; ignore a call_id you have no notification for.

What is bounded #

• Media threads are fixed; a room is pinned to one media loop, so a room's call is one core's work.
• Per-room call participant cap is configured at deploy time (64 by default). This is separate from the 5,000-member cap on a room's membership, which applies on every plane.
• Keyframes are requested and relayed, so a late subscriber gets video quickly.
• Call quality is measured per subscriber (bandwidth estimate, loss fraction) and exported as aggregate metrics; the same measurement drives the adaptive layer policy.
• A momentarily full send queue on the signalling socket drops that one frame, counted in the server's metrics; it does not end the call.
• A deploy closes the socket with 1001. The SFU's state does not survive the restart, so reconnect with backoff and rebuild the PeerConnection: re-offer your publications, let the server offer the others' tracks again, and expect a fresh peers snapshot. A release does this once (Deploys).