Recordings and replay
How Synento's layout-agnostic recording model works, and how to replay sessions with configurable layouts.
Every session creates a recording the moment it starts. Replay and MP4 export compose those stored tracks into a layout at playback time.
The stream-native recording model
Capture (per participant) → S2 append log (separated tracks + event timeline)
↓
┌─────────────────────┴─────────────────────┐
↓ ↓
Composable replay (client) Export (server, MP4)
layout descriptor + director track same layout vocabularyThere is no encoding delay. As soon as the last participant leaves a session, the recording is replayable at full fidelity. Presentation is never baked in: the stored data is a timeline of media tracks and typed events, and the player (or exporter) composes the view at playback time.
Replay is included up to an allowance: 1,500 replayed participant-minutes
per billing period on both plans (then $0.003 each on pay-as-you-go). One
participant's track played for one minute is one replayed participant-minute,
so a viewer watching a 4-person call for 10 minutes uses 40. Every playback
path counts, public sessions and replay links included, since each viewer reads
the recording afresh. On Free, past the allowance, playback is refused with
403 PLAN_LIMIT (a WebSocket replay closes with code 4003); on pay-as-you-go
it is billed as overage. GET /v1/billing/limits reports replayMinutesUsed and
includedReplayMinutes.
What gets recorded
Each session under a/{projectId}/rooms/{sessionId}/ persists several stream types:
1. Separated media tracks (media/{userId}/{kind})
Each participant writes up to three independent track streams:
| Stream | Kind | Contents |
|---|---|---|
media/{userId}/cam | Camera video | WebCodecs EVC1 (H.264/VP9/VP8) or MJPEG image bytes |
media/{userId}/mic | Microphone audio | Opus packets where the browser can encode Opus, otherwise raw PCM (s16le, 16 kHz mono); the codec is recorded per record |
media/{userId}/screen | Screen share | Same video formats as cam |
Track separation lets playback select only what a layout needs (e.g. active-speaker uses audio independently of video; screen-share focus reads the screen track alone).
Video payloads are self-describing:
- WebCodecs chunk (preferred):
EVC1magic + 18-byte header (codec id, keyframe flag, dimensions, timestamp µs) + elementary stream (annexb H.264, or VP9/VP8). - MJPEG image (legacy): raw JPEG/WebP bytes, from recordings made before WebCodecs.
The server relay stores and forwards payloads opaquely. It never re-encodes.
2. Event timeline (events stream)
A versioned, append-only JSON timeline. Every event shares one envelope:
{ "v": 1, "type": "participant.joined", "ts": 1699876000123, "actor": "alice", "data": {} }Event types written today:
| Type | Purpose |
|---|---|
participant.joined / participant.left | Lifecycle |
track.added / track.removed | Track lifecycle (kind: cam, mic, screen) |
screenshare.started / screenshare.stopped | Screen-share lifecycle |
speaking.started / speaking.stopped | Voice activity, recorded when your client reports it (see below) |
director.cut | Computed layout decision (focus participant, reason) in the director stream |
participant.redacted | GDPR/removal annotation |
caption.cue events are written by transcripts, into the
captions/{lang} streams rather than events. branding.applied and
highlight.marked are reserved: the player understands them, but nothing in
Synento writes them yet, and there is no API for adding them.
Speaking events come from your app. @synento/client does not detect
voice activity itself. Send each transition from your own detector with
room.sendControl({ type: "speaking", speaking: true }) (and false when
they stop); the server records it on the timeline. Without them there is no
director track, and active-speaker layouts show the first participant with a
camera instead of whoever is talking.
The meta stream still receives legacy join/leave/control records for compatibility, but events is the canonical timeline for reconstructing session state.
3. Director track (director stream)
When a session ends, the server computes active-speaker cuts from the speaking timeline and persists them as director.cut events (only when there are speaking events to compute from). Clients then read pre-computed focus switches instead of re-deriving them from raw audio on every replay.
Director cuts honour a minimum hold time (default 1.5 s) to avoid rapid switching.
4. Chat (chat stream)
Text messages between participants, one JSON record each (base64-encoded in playback windows):
{"text": "Hello everyone!", "user_id": "bob", "timestamp": 1699876012345}5. Captions (captions/{lang} streams)
A session's transcript, one stream per language, as
caption.cue events: actor is the speaker, data holds text, end and
lang. Replay windows include them, so the player draws them as captions
(captions: false in the layout turns that off, captionLang picks one
language). Once a transcript holds cues the manifest lists captions in
capabilities, and streams.captions_by_lang maps each language to its
stream. After a participant is redacted, a language's stream may carry a
generation suffix (captions/en.1): S2 cannot reuse a deleted stream name
straight away, so the remaining cues move to a new stream.
Session manifest
GET /v1/sessions/:sessionId/manifest returns enough to bootstrap playback
without reading every stream:
{
"data": {
"version": 1,
"session_id": "9b1c4e7a2d3f4a8b9c0d1e2f3a4b5c6d",
"epoch": 1699876000000,
"start_ts": 1699876000000,
"end_ts": 1699876300000,
"participants": [
{
"user_id": "alice",
"tracks": [
{ "kind": "cam", "stream_id": "a/.../media/alice/cam", "start_ts": 1699876001000, "end_ts": 1699876299000 },
{ "kind": "mic", "stream_id": "a/.../media/alice/mic", "start_ts": 1699876001100, "end_ts": 1699876298800 }
]
}
],
"streams": {
"events": "a/.../events",
"chat": "a/.../chat",
"director": "a/.../director"
},
"capabilities": [
"separated-tracks",
"event-timeline",
"configurable-layouts",
"director-track"
]
},
"error": null
}All timestamps are epoch milliseconds. A track is listed only once it holds
media, and a participant only once they have a track, so the manifest is the
evidence of what was actually recorded. director-track appears only when the
session has a director track. Use the manifest to see which tracks exist and
where the recording starts and ends.
Public sessions: GET /v1/public/sessions/:sessionId/manifest.
Session lifecycle and recording states
| State | Description | Timestamp field |
|---|---|---|
| Created | Session created, no participants yet | created_at set |
| Started | First participant joined | started_at set |
| Streaming | Active session with participants connected | - |
| Ended / Recording Ready | Last participant left, recording complete; director track computed shortly after | ended_at set |
A session's recording is the persisted streams between start and end. To check if a recording is ready, look for ended_at !== null. If people rejoin an ended session, ended_at returns to null until they leave again.
Accessing recordings (replay)
Step 1: Get the manifest (recommended)
curl https://api.synento.com/v1/sessions/{sessionId}/manifest \
-H "Authorization: Bearer sk_live_xxxxx"Returns participants, separated track index, time bounds, and capabilities.
Alternatively, GET /v1/sessions/{sessionId}/playback/timeline returns a
lighter summary: { participantCount, streamIds, totalEventsEstimated, minTs, maxTs }.
Step 2: Read playback windows (iterative)
The playback API reads the session's streams (events, director, chat, and
every participant's tracks) in time-based windows. from_ts is an epoch-millisecond
time, so start at the manifest's start_ts; window_size is in milliseconds
(default 30000):
curl "https://api.synento.com/v1/sessions/{sessionId}/playback?from_ts=1699876000000&window_size=30000" \
-H "Authorization: Bearer sk_live_xxxxx"{ "data": {
"start_ts": 1699876000000,
"end_ts": 1699876030000,
"window_size_ms": 30000,
"streams": [{ "stream_id": "a/.../media/alice/cam", "event_count": 900 }],
"events": [{ "stream_id": "a/.../media/alice/cam", "timestamp": 1699876001000, "body_base64": "..." }]
}, "error": null }Request the next window with from_ts set to the previous end_ts, and stop
once you pass the manifest's end_ts. A window that starts before the
recording (for example from_ts=0) comes back empty.
Each event includes stream_id (full S2 path), timestamp (epoch ms), and body_base64. Media track events carry raw payload bytes (no outer wire header). Microphone events also carry audio_codec ("opus" or "pcm"), because the payload alone does not say which.
Step 3: Compose the view (client side)
The composition layer decodes the tracks a layout needs and draws them into a single canvas:
- Reduce session state. Process
eventsanddirectorevents up to the playhead (participants, active speaker, screen sharers). - Resolve layout. Map a layout descriptor + session state to normalized regions (0..1 coordinates).
- Decode tracks. WebCodecs
VideoDecoderforEVC1chunks,createImageBitmapfor MJPEG; one offscreen surface per visible track. - Draw. Composite regions onto the output canvas; optional caption and branding overlays.
Available layout types (in the browser player):
| Layout | Behaviour |
|---|---|
active-speaker | Focus the current speaker (from director/speaking events); thumbnails for others. With no speaking events, focuses the first participant with a camera |
presenter | Focus a designated participant (presenter, default the first); their screen share if they are sharing, otherwise their camera |
grid | Equal tiles for active cameras (up to 8 by default, options.maxTiles) |
screen-share | Full-frame screen share with camera thumbnails; falls back to active-speaker |
selected | Grid of chosen participants only |
custom | Explicit normalized regions |
The Synento dashboard's replay player exposes a layout selector (Speaker / Grid / Presenter / Screen), built on the same @synento/player compositor.
Replay controls (dashboard player)
- Seek. Jump to any point; the player re-reads from that
from_ts. - Speed control. Play at 0.5x, 1x (default), or 2x.
- Layout switching. Change composition instantly without re-fetching (client-side).
- Pause / resume. Stop the playhead without losing position.
Player SDK (@synento/player)
Before you choose this path: the playback endpoints need your API key, and a
sk_live_key must never reach a browser. For a customer-facing page, proxy the two endpoints below through your own backend (which holds the key and checks your user may see the session) exactly as the Synento dashboard does, or make the session public and read the unauthenticated/v1/public/sessions/:id/...endpoints. If you would rather serve a plain video file, export to MP4.
The player SDK has two parts that share one clock: Compositor turns playback
windows into a composed frame on a canvas, and ReplayAudio decodes and mixes
every participant's microphone. You fetch the windows and drive the playhead;
this is the same loop the dashboard runs:
import {
Compositor,
type LayoutDescriptor,
ReplayAudio,
type StreamEvent,
} from "@synento/player";
// Your backend forwards these to https://api.synento.com/v1/..., adding the API key.
const api = "/api/synento";
const manifest = (await (await fetch(`${api}/sessions/${sessionId}/manifest`)).json()).data;
const start: number = manifest.start_ts;
const end: number = manifest.end_ts;
const canvas = document.querySelector("canvas")!;
const playButton = document.querySelector("button")!;
const compositor = new Compositor({ canvas, width: 1280, height: 720 });
const audio = new ReplayAudio();
const layout: LayoutDescriptor = { type: "active-speaker" };
// Keep windows back to back and fetch the next one before it is needed, so
// the decoders see one continuous run of frames.
let events: StreamEvent[] = [];
let bufferedTo = start;
let fetching = false;
async function extend() {
if (fetching || bufferedTo >= end) return;
fetching = true;
const res = await fetch(`${api}/sessions/${sessionId}/playback?from_ts=${bufferedTo}&window_size=10000`);
const { data } = await res.json();
events = [...events.filter((e) => e.timestamp >= bufferedTo - 15000), ...data.events];
bufferedTo = data.end_ts;
fetching = false;
}
await extend();
playButton.onclick = () => {
audio.resume(); // browsers only start sound from a user gesture
let playhead = start;
let last = performance.now();
function frame(now: number) {
// Advance by real time, but never past what has been downloaded.
playhead = Math.min(playhead + (now - last), bufferedTo); // 1x speed
last = now;
if (playhead >= end) {
audio.update(events, playhead, false);
return;
}
if (bufferedTo - playhead < 6000) void extend();
compositor.render(layout, events, playhead);
audio.update(events, playhead, true);
requestAnimationFrame(frame);
}
requestAnimationFrame(frame);
};audio.update(events, playhead, playing, speed) takes the same events and
timestamp as compositor.render. Pass playing: false to pause; a seek is just
a jump in the timestamp. Sound waits for a user gesture: call audio.resume()
from your play button, and audio.running tells you whether the browser has
allowed it yet. setMuted and setVolume control the output. At speeds other
than 1x the audio plays faster or slower and its pitch changes with it.
Microphone records in a playback window carry audio_codec: "opus" for Opus
packets, or "pcm" for raw s16le 16 kHz mono. ReplayAudio handles both; read
it if you decode audio yourself.
Lower-level helpers (resolveLayout, reduceSessionState, parseSessionEvents, availableTracks, resolveActiveCaption, resolveBranding) are pure and reusable.
The package also exports a PlaybackEngine class. In 0.2.0 it does not fit
this flow: it hands back decoded video events rather than the raw window events
Compositor needs, and its fetchTimeline() expects a response shape the API
does not send. Use the loop above.
See @synento/player on npm for the package.
Exporting to MP4
Export is async and parameterized by layout. The web dashboard can enqueue and download exports while logged in. For scripts and integrations, pass a project API key on each request:
# Enqueue export (returns job id)
curl "https://api.synento.com/v1/sessions/{sessionId}/export?layout=presenter&orientation=vertical&focus=alice" \
-H "Authorization: Bearer sk_live_xxxxx"Response (202):
{ "data": { "job_id": "..." }, "error": null }Query parameters:
| Parameter | Values | Default |
|---|---|---|
layout | grid, presenter | grid |
orientation | landscape (1280×720), vertical (720×1280) | landscape |
focus | Participant user id (for presenter layout) | first participant, by user id |
Unrecognised values are ignored and the default is used. Export offers only
grid and presenter: presenter fills the frame with the focus
participant's camera and shows up to four others as thumbnails.
Poll status:
curl https://api.synento.com/v1/exports/{jobId} \
-H "Authorization: Bearer sk_live_xxxxx"status is queued, processing, completed or failed (with an error
message), and progress_pct gives a rough percentage while it runs. When
status is completed, download via download_url:
curl https://api.synento.com/v1/exports/{jobId}/download \
-H "Authorization: Bearer sk_live_xxxxx" \
--output session.mp4Export behaviour:
- Reads each participant's
camandmictracks and mixes everyone's audio. Screen shares are not included in the MP4 - Output is H.264 video and 128 kbps AAC audio in an MP4, at the size the orientation names. Each participant is fitted in their camera's own shape and letterboxed: a portrait phone in a landscape export is not stretched or cropped
- Codec-aware: H.264 camera tracks (what
@synento/clientuses whenever the browser can encode H.264) are passed to FFmpeg as-is; legacy image tracks are decoded frame by frame, repeating frames across gaps; Opus and PCM audio are both handled - Each camera frame keeps the time it was captured at, so a camera sending fewer than 30 frames a second plays at real speed; while a camera is off, the last picture holds
- Current limits: a camera track recorded in VP8 or VP9 (the SDK's fallback when a browser cannot encode H.264) cannot be exported, and every participant in the recording needs camera video. An export with a participant who published none fails with a message naming them. Each participant's video is placed from the start of the file rather than at the moment they joined
- An export runs for at most 15 minutes before it is marked
failed - Export artifacts are stored by Synento and served from the download endpoint until you delete the export, delete the session, or retention removes it
- Enqueuing is refused with
503when the organisation's usage allowance is used up, the same check as creating a session - Asking again for the same session with the same options returns the existing
job (
"reused": true) instead of encoding it again, and does not count against your allowance - Exports are counted in exported participant-minutes: each participant's
camera and microphone, from their first to their last moment in the
recording. A 30-minute call with 4 people is 120. Both plans include 300 per
billing period. Past that, a new Free export is refused with
403 PLAN_LIMIT, and pay-as-you-go is billed $0.004 per exported participant-minute, so that call would cost $0.48. The playback timeline'sparticipantMinutesgives the figure before you export. Minutes are counted once the export has rendered; a failed export costs nothing.GET /v1/billing/limitsreportsexportMinutesUsed,includedExportMinutesandbillableExportMinutes - An export made before a participant was redacted is never reused: the next request renders a fresh file without them (see Participant redaction)
To cut an export into a short subtitled clip, see Edited clips with JSON2Video.
Deleting an export
Delete one export's file without touching the session:
curl -X DELETE https://api.synento.com/v1/exports/{jobId} \
-H "Authorization: Bearer sk_live_xxxxx"Response: { "data": { "deleted": true, "job_id": "…" }, "error": null }.
Afterwards the export answers 404 (so does deleting it again), and asking for
an export of the session with the same options renders a new file. A deleted
export's minutes still count against the period's allowance, and stay billed
if they were past it. An export that is still queued
or rendering is refused with 409 CONFLICT; delete it once it has finished.
From @synento/node: await synento.deleteExport(jobId).
Participant redaction
Remove a participant's media while keeping the rest of the session:
curl -X DELETE https://api.synento.com/v1/sessions/{sessionId}/participants/{userId} \
-H "Authorization: Bearer sk_live_xxxxx"This deletes all of the participant's track streams (cam, mic, screen) and appends a participant.redacted event to the timeline. The participant then no longer appears in the manifest or in playback; no placeholder is drawn in their place. Their chat messages and timeline events (joins, speaking) are kept.
MP4s exported before the redaction are left as they are, so they still contain the participant. They are never handed back as a reused export: the next export request renders a fresh file without them. The response lists the earlier exports that may still show the participant, so you can delete each one with DELETE /v1/exports/{jobId}:
{
"data": {
"redacted": "alice",
"deletedStreams": 3,
"earlierExports": 1,
"earlierExportIds": ["8d0c2f0e-…"]
},
"error": null
}An export still rendering at the time of the redaction is listed too, since it may have read the participant's media before it was deleted.
Retention
Each project has a retention policy, and recordings older than it are deleted, counted from when the session ended. Free accounts can keep recordings for at most 7 days. Pay-as-you-go can keep them for at most 90, and stores each recording free for its first 30. New projects start at 7 days on Free and 30 on pay-as-you-go. The shortest setting is 1 day.
curl -X PUT https://api.synento.com/v1/projects/{projectId}/retention \
-H "Authorization: Bearer sk_live_xxxxx" \
-H "Content-Type: application/json" \
-d '{"retention_days": 30}'Architecture diagram
flowchart TB
subgraph rec [Recording layer — layout agnostic]
cam[media/userId/cam]
mic[media/userId/mic]
scr[media/userId/screen]
ev[events timeline]
dir[director track]
end
rec --> comp[Composition layer]
comp --> lay[Layout descriptor]
comp --> cli[Client compositor — @synento/player]
comp --> srv[Server exporter — FFmpeg]
lay --> cli
lay --> srv
cli --> view[Interactive playback]
srv --> art[MP4, landscape or vertical]